STG205_#EarthOnAWS How NASA is Using AWS

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS re:INVENT
#EarthOnAWS: How NASA is Using
AWS
S T G 2 0 5
N o v e m b e r 2 8 , 2 0 1 7

Open data on AWS
J o e F l a s h e r , A W S O p e n G e o s p a t i a l D a t a L e a d

Why does AWS care about open data?
 Many of our commercial sector customers rely on quality open data as much as they
rely on our cloud infrastructure services
 Many of our public sector customers use AWS to make their data available to a
global community of researchers, entrepreneurs, students, and fellow government
agencies
Sharing data on AWS makes it accessible to a large and growing
community of researchers, entrepreneurs, and enterprises who use
the AWS cloud

“…data must be organized, well-
documented, consistently formatted, and
error free. Cleaning the data is often the
most taxing part of data science, and is
frequently 80% of the work.”
— Data Driven by DJ Patil and Hilary Mason
Undifferentiated heavy lifting

Landsat on AWS
Graph by Drew Bollinger (@drewbo19) at Development Seed

When data is shared in the cloud, anyone
can analyze any volume of data without
needing to download or store it.
Opening data is the beginning, not the end.
Users need to be educated and have access
to tools to analyze and process the data.

Elevation
models
Aerial
imagery
Climate
models
Satellite
imagery
High-resolution
radar
aws.amazon.com/earth

AWS Cloud Credits for Research
provides promotional AWS cloud
credits for anyone to conduct research
using Earth observation data.
aws.amazon.com/earth/research-credits

Preparing for a
big-data future
NASA Earth science data
K e v i n M u r p h y , N A S A H Q / D a n P i l o n e , E l e m e n t 8 4 , I n c .

NASA’s Earth Science Data Systems
Program
Actively manages NASA’s Earth science data as a
national asset (satellite, airborne, and field)
Develops capabilities optimized to support
rigorous science investigations
Processes (and reprocesses) instrument data to
create high quality long-term earth science data
records.
http://go.nasa.gov/2mMd5g1
Single largest repository of Earth Science
Data, integrating
multivariate/heterogeneous data from
diverse observational platforms.

Earth science open data policy
NASA’s earth observation data is collected continuously. For
over half a century these invaluable records of earth
processes have provided a critical resource for scientists and
researchers.
Since 1994 NASA earth science data have been free and open
to all users for any purpose as quickly as practical after
instrument checkout and calibration.

Earth Observing System Data and
Information System (EOSDIS)
EOSDIS
Applications
Capture
and clean
Education
Process
Archive
Transform*
Distribute Research
*Subset, reformat, reproject
Commercial

SIPS DAAC
Distributed Active Archive Centers (DAACs), collocated with centers of
science discipline expertise, archive and distribute standard data
products produced by Science Investigator-led Processing Systems
(SIPS)ASF DAAC
SAR products, sea ice,
polar processes
PO.DAAC
Ocean circulation
Air-sea interactions
NSIDC DAAC
Cryosphere, polar
processes
LPDAAC
Land processes
and features
GHRC
Hydrological cycle and
severe weather
ORNL
Biogeochemical
dynamics, EOS
land validation
ASDC
Radiation budget,
clouds, aerosols, tropo
composition
LAADS/MODAPS
Atmosphere
OB.DAAC
Ocean biology and
biogeochemistry
SEDAC
Human interactions in
global change
CDDIS
Crustal dynamics
Solid earth
GES DISC
Atmos composition
and dynamics, global
modeling, hydrology,
radiance
NCAR, U. of Co.
MOPITT
JPL
MLS, TES,
SNPP Sounder
U. of Wisc.
SNPP
Atmosphere
GHRC
AMSR-U,
LIS
GSFC
SNPP,
MODIS, OMI,
OBPG

EOSDIS core services
Open data APIs and
Free data download
Open service APIs
Open source clients

Lightning fast, always available
- 95% queries complete in <1s
- 99.98% uptime (last 365d)
Big-data ready
- 34K collections
- 367 million files indexed
- Prepared to scale 1B+ records
Standards-focused
Community-focused
Internationally recognized
Common metadata repository

Starting AWS migration
Since September 2016, EOSDIS has migrated
two of its core systems, Common Metadata
Repository (CMR) and Earthdata Search, into
the Amazon cloud to immense success
• One year migration effort
• Over 500K queries per day
• Open source
• Open access API

Data-centric users
https://search.earthdata.nasa.gov
Imagery-centric users
https://worldview.earthdata.nasa.gov

Preparing for the future

New instruments and
missions.
2017 NRC Decadal
Survey - Earth
Science and
Applications from
Space: National
Imperatives for the
Next Decade and
Beyond
User expectations
continue to evolve.
5 Years from
Today Landsat 9
(2020)
PACE (2022)
NI-SAR (2022)
SWOT (2021)
TEMPO (2018)
JPSS-2(NOAA)
RBI, OMPS-Limb (2018)
GRACE-FO (2) (2018)
ICESat-2 (2018)
CYGNSS (2016)
ISS
SORCE, (2017)
TCTE (NOAA)
NISTAR, EPIC (2019)
(NOAA’S DSCOVR)
QuikSCAT (2017)
EO-1
(2017)Landsat 7
(USGS)
(~2022)
Terra
(>2021)
Aqua(>2022)
CloudSat (~2018)
CALIPSO (>2022)
Aura
(>2022)
SMAP (>2022)
Suomi NPP
(NOAA) (>2022)
Landsat 8
(USGS) (>2022)
GPM (>2022)
OCO-2
(>2022)
GRACE (2)
(2018)
OSTM/Jason 2 (>2022)
(NOAA)
(Pre)Formulation
Implementation
Primary Ops
Extended Ops
Earth Science Instruments on ISS:
RapidScat, (2017)
CATS, (2020)
LIS, (2016)
SAGE III, (2016)
TSIS-1, (2018)
ECOSTRESS, (2019)
GEDI, (2018)
OCO-3, (2018)
CLARREO-PF, (2020)
TSIS-2 (2020)
Sentinel-6A/B (2020, 2025)
MAIA(~2021)
TROPICS (~2021)
EVM-2 (~2021)
Implementation
Formulation
Primary Ops
Extended Ops
InVEST – In-Space
Validation CubeSats:
RAVAN (2016)
HARP (2016)
IceCube (2016)
MiRaTA (2017)
RainCube (2017)
TEMPEST-D (2018)
CIRiS (2018)
CubeRRT (2018)
CIRAS (2018)
LMPC (TBD)
NASA Earth Science
Missions: Present through 2023

EOSDIS Data System Evolution
EOSDIS is the premier Earth science
archive, but we are always looking
for ways to improve
The current architecture will not be
cost effective as the annual ingest
rate increases from 4 to 50PB/year
It will become increasingly difficult
and expensive to maintain and
improve our current system as data
volumes and research demands
continue to increase exponentially
EOSDIS is developing open source
cloud native software for reuse
across the agency and throughout
the government
Petabytes
Cloud offers benefits like the ability to analyze data at scale, analyze multiple data sets
together easily and avoid lengthy expensive moves of large data sets allowing scientists
to work on data “in place”

NISAR quick facts
“The NASA-ISRO Synthetic Aperture
Radar (NISAR) mission is a joint project
between NASA and ISRO to co-develop and
launch a dual frequency synthetic aperture
radar satellite. The satellite will be the first
radar imaging satellite to use dual frequency
and it is planned to be used for remote
sensing to observe and understand natural
processes of the Earth.”
https://en.wikipedia.org/wiki/NISAR_(satellite)
Key scientific objectives:
• Understand the response of ice sheets to
climate change and the interaction of sea ice and
climate
• Understand the dynamics of carbon storage and
uptake in wooded, agricultural, wetland, and
permafrost systems
• Determine the likelihood of earthquakes,
volcanic eruptions, and landslides
Payload:
L-band (24-centimeter wavelength)
polarimetric SAR (NASA)
S-band (12-centimeter wavelength)
polarimetric SAR (ISRO)
Launch: 2021-ish from India

The Surface Water & Ocean Topography (SWOT) mission
brings together two communities focused on a better
understanding of the world's oceans and its terrestrial surface
waters. U.S. and French oceanographers and hydrologists and
international partners have joined forces to develop this new
space mission to make the first global survey of Earth's surface
water, observe the fine details of the ocean's surface
topography, and measure how water bodies change over time.
NISAR is expected to generate a tremendous volume of data over its
scheduled three-year mission—as much as 19.9 TB/day or 7.2 PB/year.
Key Scientific Objectives:
• Provide sea surface heights (SSH) and terrestrial water
heights over a 120 km wide swath with a +/-10 km gap at
the nadir track.
• Over the deep oceans, provide SSH within each swath
with a posting every 2 km x 2 km, and a precision not to
exceed 0.8 cm when averaged over the area.
• Over land, produce a water mask able to resolve 100
meter wide rivers and lakes of 250 meter2 in size,
wetlands, or reservoirs. Cover at least 90 percent of the
globe. Gaps are not to exceed 10 percent of Earth's
surface.
Payload:
Ka-band Radar Interferometer (JPL)
Nadir Altimeter (CNES)
Cross-Track Advanced Microwave radiometer (JPL)
Launch: ~2021

What could a future data system architecture look like?
EOSDIS works well, but can we do better?
• Can we evolve NASA archives to better support interdisciplinary
Earth science researchers?
• What system architecture(s) will allow our holdings to become
interactive and easier to use for research and commercial users?
• Can we afford additional functionality?
• How will data from multiple agencies, international partners, and
the private sector be combined to study the earth as a system?
• GOES-R, CubeSats, Copernicus…

Conceptual ‘data close to compute’
Large volume data storage: Centralized mission
observation and model datasets stored in auto
graduated AWS object storage (Amazon S3,
Amazon S3 IA, Amazon Glacier)
Scalable compute: Provision, access, and
terminate dynamically based on need. Cost by use
Cloud Native Compute: Cloud vendor service
software stacks and microservices easing
deployment of user based applications
EOSDIS applications and services: Application
and service layer using AWS compute, storage
(Amazon S3, Amazon S3 IA, Amazon Glacier), and
cloud native technologies
Non-EOSDIS/public applications and services:
Science community brings algorithms to the data.
Support for NASA and non-NASA
Bring customers to the data
The operational model of consolidating data—allowing
users to compute on the data in place with a platform
of common tools—is natural to cloud; it is a cost-
effective way to leverage cloud and could be
applicable to many businesses and missions

Looking to the cloud for scalability

EOS is many interconnected systems…
Worldview/Earthdata Search web
applications
Global Imagery Browse Services (GIBS)
Common Metadata Repository (CMR)
Data ingest, archive, and distribution
(Cumulus)
Metrics, authentication,
monitoring,
distribution services, etc.

But that’s not the worst of it

80 TBs/day
generation
400 TBs/day
reprocessing
300 GB
Granules
150 PBs @ 50 Gbps
processing speed for months

We have to change the paradigm

Global Imagery Browse Service (GIBS)
in the cloud service swap
Handlers
Generation
Ops
console
MRFGen
Product
Config
Product
configuration
Inventory
ZooKeepe
r
Subscriptio
n service
CM
Manager
Authenticatio
n
Sig Event
Service
Install
Amazon
S3
Amazon
DynamoDB /
SQS
Amazon
SNS/SQS
AWS
CloudFormation
Scheduler/
dispatcher
IAM
Amazon
CloudWatch
AWS
CloudFormation
Custom service
External NASA/GIBS Library
Cloud services
Data

GIBS-in-the-cloud ingest & processing
Handlers
Generation MRFGen
Product
config
Dispatcher
(106 LoC)
Scheduler
(66 LoC)AWS

On-premises responses with time in milliseconds AWS responses with time in milliseconds
On-premises implementation showed consistent performance
during load testing vs. more sporadic latencies in AWS
Cloud performance affected architecture

Discover Sync Process
Provider
Discover
HTTP tiles
Sync HTTP
URLS
Generate
thumbnails and
tiles
Imagery
storage
Source image
storage
Execution flow
Data store
Data fetch
Scheduler
Imagery
locks
Ingest: Earth science Imagery Processing
Product config

OCTOBER 17-20, 2017 AWS announcements !

Discover Sync Process
Provider
Discover
HTTP Tiles
Sync HTTP
URLS
Generate
thumbnails and
tiles
Imagery
storage
Source image
storage
Execution flow
Data store
Data fetch
Scheduler
Imagery
locks
Ingest: Earth science imagery processing…
Product config

AWS Step
Functions
Collections
table
Workflow Execution
Discover granules
Sync
Processing step 1
Processing step 2
Submit to CMR
Scheduler
Workflow data/
service interaction
Provider
Granule
storage
Temporary
storage
CMR
Dashboard
Distribution
Ingest & Archive with AWS Step Functions

AWS Step
Functions
Workflow Execution
Discover Granules
Sync
Processing Step 1
Processing Step 2
Submit to CMR
Scheduler
Dashboard

How do users use this data?

Different kinds of egress
EOSDIS data
distribution from Amazon S3
Application interactions
(e.g. CMR results,
Earthdata Search, GIBS, etc.)
EOSDIS data from
services
EOSDIS data to
AWS compute

Basic Amazon S3 egress
InternetAmazon S3

Amazon S3 with CloudFront
InternetAmazon S3 CloudFront

Amazon S3 through AWS Direct Connect
to on-premises distribution pipe
Internet
Amazon S3 Direct Connect

Request limiting using Lambda and API
Gateway
InternetAmazon S3
Lambda
DynamoDB
API Gateway

Egress costs range more than
13x across those models

Egress costs are a big deal…
…but they weren’t our only issue…

Hard cost controls are essential
The Anti-Deficiency Act (ADA)
disallows unbounded costs
We needed a means of
absolutely limiting egress costs

Enter the circuit breaker

Conceptual design
Lambda 1: Calculate Amazon S3 egress
• Watch each bucket’s "Bytes
Downloaded" via CloudWatch
• Post totals
Lambda 2: Break the circuit (if needed)
• If total from first billing period to now exceeds our threshold…
• …lock down Amazon S3 bucket policy

Cloud scale science data
• Data can be generated at scale in AWS and
placed in accessible buckets, avoiding massive
data moves
• Ingest, archival, validation, processing, etc. can
scale dynamically based on incoming data
streams, reprocessing needs, etc.
• Entire petabyte scale archive is directly
accessible, with no transfer time or costs, to
science users in the same region for longtime
series or multiproduct use
• Data processing, transformation, and analysis
services can be spun up, NASA funded or
completely independently, leveraging the data
with scalable compute and cost and access-
managed output targets.

Looking forward

What we’re working on now…
• Efficient data services access and distribution
• Cost effective large archive storage
• Data disaster recovery and preservation approaches
• Third party cloud native data use at scale
• Expanding the paradigm of an established community

Efficient data services access and
distribution

Different kinds of egress
EOSDIS data
distribution from Amazon S3
Application interactions
(e.g. CMR results, EDSC,
Earthdata pages, etc.)
EOSDIS data from
services
EOSDIS data to
AWS compute

“S3 is a distribution mechanism"
- M a r k K o r v e r @ A W S

Data services as transformations between
buckets

Cost Effective Large Archive Storage

Using predictive analytics and
machine Learning at the ASF DAAC
Courtesy: Chris Stoner cstoner5@alaska.edu
Scott Arko saarko@alaska.edu
D e t e r m i n i n g s t o r a g e l o c a t i o n a n d t e m p e r a t u r e o n t h e f l y

Predictive analytics for the data storage
Based on past user behavior, we can predict
future behavior
• > 4 times within 30 days?
• Within 30 days?
• Not ever?
Behaviors can map to storage locations
• > 4 times—archive in hot storage at an
sdge (Amazon S3)
• Within 30 days—at least warm storage
(Amazon S3 IA), maybe qualifies for hot
• Not ever—go directly to glacier/cold
Storage (cheapest storage)

Machine learning: Tie behavior to prediction
Machine learning service within AWS
• Extrapolates away the complexities of ML algorithms and models
• Trained on history
• Creates predictions based on new, incoming data
• For every product ingested in January 2017
• Get download history
• Set Y column based on downloads >= 4
• Upload to S3

Using those predictions
Once you have something good:
Real-time prediction to determine storage
temperature
>= N times where N is
minimum cost for hot storage
Hot Storage (S3)
> 0 tines? Warm Storage
(S3IA)
0 times? Cold Storage
(Glacier)

So where does this leave us?

Final thoughts
• NASA is experiencing the same data explosions felt across the
industry.
• The cloud provides an opportunity to change existing and
introduce new paradigms for Big Data
• Effectively exploiting cloud hosted data is still an open problem
(Jupyter notebook based workflows, scientific processing with
peer reviewed algorithms and code, old habits, etc.)
• Need to find ways to balance authentication and metrics with
open access and cloud native services
• Majority of code discussed today is Open Sourced:
https://github.com/nasa

Thank you!
J o e F l a s h e r < j f l a s h e r @ a m a z o n . c o m >
K e v i n M u r p h y
D a n P i l o n e < d a n i e l . j . p i l o n e @ n a s a . g o v >

STG205_#EarthOnAWS How NASA is Using AWS

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à STG205_#EarthOnAWS How NASA is Using AWS

Similaire à STG205_#EarthOnAWS How NASA is Using AWS (20)

Plus de Amazon Web Services

Plus de Amazon Web Services (20)

STG205_#EarthOnAWS How NASA is Using AWS