Making earth observation data available by using Amazon S3 is accelerating scientific discovery and enabling the creation of new products. Attend and learn how the scale and performance of Amazon S3 lets earth scientists, researchers, startups, and GIS professionals gather and analyze planetary-scale data without worrying about limitations of bandwidth, storage, memory, or processing power. Learn how AWS is being used to combine satellite imagery, social data, and telemetry data to produce new products and services. Learn also how Amazon S3 provides much more than storage, and how an open geospatial data lake on Amazon S3 can be used as the basis for planetary-scale applications built with Amazon EMR, Amazon API Gateway, and AWS Lambda. As part of this talk, AWS customer Digital Globe demonstrates how they use open data stored in S3 to distribute high-resolution satellite imagery to their customers around the world.

1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Jed Sundwall, Open Data Global Lead
November 29, 2016
Earth on AWS
Working with Planetary-scale Data in the Cloud

2. What to Expect from the Session
• About open data on AWS
• Advantages of Amazon S3 for sharing data
• How E&J Gallo and DigitalGlobe use AWS to work with
geospatial data

3. Why Does AWS Care About Open Data?
Sharing data on AWS makes it accessible
to a large and growing community of
researchers, entrepreneurs, and
enterprises who use the AWS cloud.

4. “…data must be organized, well-
documented, consistently formatted, and
error free. Cleaning the data is often the
most taxing part of data science, and is
frequently 80% of the work.”
- Data Driven by DJ Patil and Hilary Mason
Undifferentiated Heavy Lifting

5. Traditional Data Acquisition
Data Acquisition in the Cloud

6. When data is shared in the cloud, anyone
can analyze any volume of data without
needing to download or store it themselves.

7. Landsat on AWS

8. Landsat
Landsat 8 satellite Raster data

9. RGB
Visible light
Infrared
Vegetation
Shortwave infrared
Urban areas
Wellington, New Zealand

10. Landsat on AWS
Amazon
EC2
s3://landsat-pds
.tarUSGS
.tiff

11. Landsat on AWS
Graph by Drew Bollinger (@drewbo19) at Development Seed

13. landsatonaws.com
Serverless browser interface to Landsat on AWS

14. New open source tools
• GDAL – http://www.gdal.org/
• Rasterio – https://mapbox.github.io/rasterio/
• sat-utils suite – https://github.com/sat-utils

15. Elevation
Models
Aerial
Imagery
Climate
Models
Satellite
Imagery
High-resolution
Radar
aws.amazon.com/earth

16. AWS Cloud Credits for Research
provide promotional AWS cloud credits
for anyone to conduct research using
Earth Observation data.
aws.amazon.com/earth/research-credits

17. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
John Webb, Manager, Information Technology , E & J Gallo Winery
Wine and Grape Supply Data Lake
E & J Gallo

18. E&J GALLO WINERY
Background

19. E&J Gallo Winery
Largest Winery in the world
• Established in 1933 and headquartered in Modesto,
California, E. & J. Gallo Winery remains a privately-held
and ever-growing company that employs 6,000 people
worldwide.
• Largest land owner in the State of California
Products and Distribution
• With products available in more than 90 countries, E. &
J. Gallo Winery is the largest exporter of California wine,
and imports wines from Argentina, Italy, New Zealand
and Spain
• WE hold 90 brands and include table and sparkling
wines, beverage products, dessert wines and distilled
spirits

20. Gallo Portfolio
Popular Premium Ultra
Premium

21. WINEGROWING
Perennial v. Row Crop

22. Wine and Grape Supply
Grower Relations
• Manage external grape purchases
• Inform growers on best management practices
Gallo Vineyards Inc.
• Gallo owned vineyards
• Irrigation management
• Yield and Quality estimation
• Best management practices for vineyard management
• Operations planning and scheduling
Viticulture Chemistry and Enology
• Internal research division
• Quality and yield enhancement
• Variable rate irrigation
Winemaking

23. Complexities of Wine Growing
Not a Biomass Product
• Manage more than just nitrogen and water
• A number of activities go against a given vine
Perennial Crop
• Average vineyard lifecycle of 24 years
• Vine management is an on-going activity
• Canopy Management
• Shoot Thinning
• Vine Balancing
• Trellis Management
• Nutrients
• Pesticides
High Value Crop Management
• Direct correlation between ranch management plan and
quality
• Yield impact associated with canopy management
activities

24. DATA DRIVEN INSIGHTS
Data Lake Modeling

25. Why Data Matters
We operate in an ‘Activity’ based agricultural environment
from which we can drive insights
• Events or Transactions help us manage our business
• These events are both structured and unstructured
Structured Events such as…
• Tons of grapes delivered to a specific winery
• Distribution of soil across a vineyard
• Ranch management transactions applied (i.e. shoot
thinning, leafing, dropping fruit etc.)
Unstructured Events such as…
• Imagery
• Soil Moisture Nodes
• Yield Monitors
• Mechanized Asset Control

26. Data Driven Value
Having a robust data and analytics platform allows us to
drive insights
• Grape to Bottle data management
• Improve Quality
• Increase Yield
• Detect and prevent early on-set disease
• Maintain a competitive supply position through predictive
yield estimation

27. Data Lake Model
EC2
Elastic Load Balancing
S3
Redshift
SQL on DB
Instance
EMR
Customer Facing
Product Layer
EDW
Reports
Data Processing,
Analysis, R&D Layer
Data Collection
Layer
Dashboard
Insights
D
a
t
a
L
a
y
e
r

28. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Shay Har-Noy, VP & General Manager, Platform, DigitalGlobe
Getting Data Out of Jail
DigitalGlobe’s Geospatial Big Data Platform

29. @iheartcrowds

30. 3,500,000 km2 collected EVERY DAY
13,200,000,000,000 pixels

31. Last 3 Days

32. Last 14 Days

33. Last 3 Months

34. With frequent coverage of high interest areas

35. Goal: Large scale information extraction

36. Three trends working in our favor
1.

37. Three trends working in our favor
2. hidden layer 1 hidden layer 2 hidden layer 3

38. Three trends working in our favor
3.

39. Open up Geo to world of innovators working on Deep Learning
Solve a new class of problems
to impact business, government, and people

40. strategic approach: Leverage the ecosystem
Crowdsourcing

41. – Data out of jail!
Storage Compute
GBD Partner
Tools Tools
MapsAPI
Catalog API
Workflow API

42. Scale humans until machines get smarter
Use humans to create training data
Crowdsourcing

43. Training and testing set to allow benchmarking and iteration

44. We have common frameworks to accelerate prototyping
We need a common data set to evaluate performance

45. - 50cm WV-2 image w/8 spectral bands
- Each image 200m2
- Covering Rio de Janeiro, Brazil
- 7000 Geotiff images
Available for your computing pleasure on

46. Rio Olympics – Use Case
• Governments around the world were preparing for the
upcoming Olympics in Rio De Janeiro
• Thorough understanding of the security and safety of the
athletes is front of mind
• Knowledge of high risk areas, traffic patterns, line-of-sight
and overall structure distribution is critical to ensure a
secure games

55. What you can expect
1) Satellite imagery analytics will be way more common in your industry
2) Computer vision tasks are going to change the way we see the world
3) We are going to learn a lot

56. Try it yourself
aws.amazon.com/public-data-sets/spacenet/
developer.digitalglobe.com

57. Thank you!

58. Remember to complete
your evaluations!

59. Related Sessions