This document discusses scalable data analytics and visualization using cloud optimized services. It provides examples of leveraging cloud technologies like AWS and Azure to build scalable implementations for processing and serving large geospatial and earth observation datasets. This includes architectures for hosting high resolution raster data and services, global hydrologic modeling, and near real-time flood forecasting using NOAA's National Water Model.
TeamStation AI System Report LATAM IT Salaries 2024
Scalable Data Analytics and Visualization with Cloud Optimized Services
1. Scalable Data Analytics and Visualization
with Cloud Optimized Services
Sudhir Raj Shrestha
sshrestha@esri.com
Steve Kopp
skopp@esri.com
AGU Fall Meeting, December 12, 2018 | Washington, DC
2. Living in Interoperable Open World
ETL
Google Earth
ENVI
Imagine
MapInfo
Ionic
GeoMedia
ArcGIS Server
ArcGIS Desktop
ArcGIS Online
ArcExplorer
Gaia
gvSig
OpenLayers
uDig
QGIS
AutoCAD
Services
Web Services
OGC
ArcGIS Pro
4. mosaic dataset
geoprocessing tools
optimization
consumption
multidimensional filter
raster functions
Interactive/dynamic
scalable/extensible
dynamic web services
web maps & apps
web based analytics
Analyze ShareManage VisualizeIngest
raster types
crawl disk
link to pixels
animate over slices
sophisticated renderers
vector & scalar fields
GISworkflows that scale
Designed for the characteristics of Earth Observations scientific data
Data Service
7. Auto Scaling group
ArcGIS Portal
Elastic Load
Balancer
EC2
AGS
EC2
Configure Store
Federated with
Portal
S3 Storage User
Raster Store,
zip, fgdb
Elastic IP for Portal
Professional Imagery /
Geospatial Analysts
Client consuming
Imagery Items
Client consuming
Hosted Image
Services
VPC
VPC
Imagery S3 Storage
ArcGIS ProImage Services
Image Server
Image Services
Image Server
EC2 EC2
Image Services
Image Server
EC2
Dynamic Image
Services
Raster Analytics
Elastic Load
Balancer
Client consuming
Dynamic Image
Services
Enterprise + ArcGIS Server + Image server + RDS
• Used when we need to do raster analytics
• Mosaic sit in Postgres ( RDS)
• All server machines are in cluster
• Config store is on a different ec2 machine
More info on how to do this Imageserver Deployment Image server Cloud Formation Template
More info on how to do ArcGIS enterprise Deployment ArcGIS Enterprise Cloud Formation Template
RDS
Postgres RDS
Portal : m5.2xlarge 1 machine
Image server : m5.2xlarge 3 machine
File Store : m5.xlarge 1 machine
Postgres : RDS
Load Balance for Image server stack
Elastic IP for the portal machine
All machine in same VPC
Scalable Cloud Implementation
8. Global Streamflow Services – A Paradigm Shift
Global
Data
PAST – Individual Hydrologic Forecasting
Global
Services
NOW – Global Hydrologic Forecasting
9. Current NWS River Fcst
points ~3,600
The National Water Model expands forecasting capabilities
NWM streamflow and velocity for ~
2.7 million river reaches
Current NWS AHPS points overlaid with
NWM Stream Reaches (State level scale)
Courtesy: NOAA OWP
11. Real time Flood Inundation Forecast
National Water Model + Height Above Nearest Drainage (HAND)
Compute national Height Above Nearest Drainage (HAND)
4 days of computation on 32 processors
COTS software on standard hardware
NHD Streams
10 meter NED
Elevation
Hourly download
Convert flow to depth
through rating curve
Real time Flood Forecasting
HAND Raster Map
National Water Model