AWS Summit Paris closing keynote by Werner Vogels

1. Beyond the Fridge
The world of Connected Data !
Dr. Werner Vogels!
CTO, Amazon.com!

4. The amount of information generated during the first day of
a baby’s life today is equivalent to 70 times the information
contained in the Library of Congress"

5. I. Science"

7. Observations – Theory – Models – Facts"

9. Human Genome Project"
Collaborative project to sequence every single letter!
of the human genetic code.!
13 years and $billions to complete.!
Gigabyte scale datasets (transferred between sites on!
iPods!)!

10. Beyond the Human Genome"
45+ species sequenced: mouse, rat, gorilla, rabbit, !
platypus, nematode, zebra fish...!
Compare genomes between species to identify!
biologically interesting areas of the genome.!
100Gb scale datasets. Increased computational
requirements.!

11. The Next Generation"
New sequencing instruments lead to a dramatic!
drop in cost and time required to sequence a genome.!
Sequence and compare genetic code of individuals to!
find areas of variation. Much more interesting.!
Terabyte scale datasets. Significant computational
requirements.!

12. The 1000 Genomes Projects"
Public/private consortium to build world’s largest!
collection of human genetic variation.!
Hugely important dataset to drive new insight into!
known genetic traits, and the identification of new ones.!
Vast, complex data and computational resources required,
beyond reach of most research groups and hospitals.!

13. 1000 Genomes in the Cloud"
The 1000 Genomes data made available to all on AWS.!
Stored for free as part of the Public Datasets program.!
Updated regularly.!
200Tb. 1700 individual genomes. As much compute and
storage as required available to all.!

22. II. Consumer"

24. Dropcam
is
the
biggest
inbound
video
service
on
the
Web
• More
data
uploaded
per
minute
than
YouTube
• Petabytes
of
data
processed
every
month
• Billions
of
mo=on
events
detected

34. III. Retail"

35. UNCERTAINTY"

36. UNDERSTAND"
YOUR CUSTOMER"

37. Who
is
my
customer
really?
What
do
people
really
like?
What
is
happening
socially
with
my
products?
Where
do
people
consume
my
product?
How
do
people
really
use
your
product?

38. PERSONALIZE"

39. 75% of users select"
movies based on"
recommendations"

40. More than 27 million users!
~ 30 million plays per day!
More than 40 billion events per day !
~ 4 million ratings per day!
~ 3 million searches per day!
Geo-location data!
Device information!
Time of day and week (it now can verify that users watch more TV shows during
the week and more movies during the weekend)!
Metadata from third parties such as Nielsen!
Social media data from Facebook and Twitter!

41. BIGGER IS BETTER"

43. IV. Industrial"

49. V. Sports"

58. VI. Location"

68. VII. The Pipeline"

69. COLLECT
|
STORE
|
ORGANIZE
|
ANALYZE
|
SHARE

70. COLLECT
|
STORE
|
ORGANIZE
|
ANALYZE
|
SHARE

71. COLLECT
|
STORE
|
ORGANIZE
|
ANALYZE
|
SHARE

72. COLLECT
|
STORE
|
ORGANIZE
|
ANALYZE
|
SHARE

73. COLLECT
|
STORE
|
ORGANIZE
|
ANALYZE
|
SHARE

74. COLLECT
|
STORE
|
ORGANIZE
|
ANALYZE
|
SHARE

75. VIII. Real-time"

76. What was happening
yesterday?!

77. What ! right now?!
trades are executing!
is the exception rate!
is the ad click-through!
topics are trending"
inventory remains!
queries are slow!
are the high scores!
!
!

78. Kinesis!

80. Kinesis
architecture
Amazon Web Services
AZ AZ AZ
Durable, highly consistent storage replicates data
across three data centers (availability zones)
Aggregate and
archive to S3
Millions of
sources producing
100s of terabytes
per hour
Front
End
Authentication
Authorization
Ordered stream
of events supports
multiple readers
Real-time
dashboards
and alarms
Machine learning
algorithms or
sliding window
analytics
Aggregate analysis
in Hadoop or a
data warehouse
Inexpensive: $0.028 per million puts

81. AWS
Internal
Metering
Service
Capture
Submissions
Process in
Realtime
Store in
Redshift
Clients
Submitting
Data
Workload
• Tens of millions records/sec
• Multiple TB per hour
• 100,000s of sources
New features
• Scale with the business
• Provide real-time alerting
• Inexpensive
• Improved auditing

82. Workload
• Daily
load
of
billions
records
from
millions
of
files
from
hundreds
of
sources
• 3
hour
SLA
to
load
and
audit
data
• Hundreds
of
customers
• Hundreds
of
queries
per
hour
New
features
• Our
data
is
fresh,
we
ingest
every
6
hours
• Now
processing
triple
the
volume
in
less
than
25%
of
the
=me
• “Hammerstone”
ETL
solu=on
– Built
on
AWS
Data
Pipeline
– Build
business
specific
marts
– Build
workload
specific
clusters
• Supports
a
variety
of
analy=cs
tools:
Tableau,
R,
Toad,
SQL
Developer,
etc.
Internal
AWS
Data
Warehouse
Over 200 internal
data sources
Data staged in
Amazon S3
"Hammerstone:"
Custom ETL
using AWS
Data Pipeline
Data processing
Redshift cluster
Batch reporting
Redshift cluster
Ad hoc query
Redshift cluster

83. IX. Beyond the Display"

88. CONNECTED DATA
REQUIRES
NO LIMITS"

89. Cloud enables
connected data
collection!

90. Cloud enables
connected data
processing!

91. Cloud enables
connected data
collaboration!

92. werner@amazon.com