Data is eating the world. Don't let it consume your organization! In this talk, we take a deep dive into the various methods of ingesting the ever-increasing amount of data that your organization is generating, and we discuss how to effectively leverage that data as part of your data analytics pipeline. We cover topics such as batch and real-time ingestion with AWS services and open source tools. We also dive into some reference architecture implementations of common ingestion patterns. Finally, we discuss how you can make this data useful through catalogs and governance.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Keeping the Pace with Data
Ingestion
Paul Sears
Partner Solutions Architect
Amazon Web Services
G P S C T 4 0 2
Rohit Pujari
Partner Solutions Architect
Amazon Web Services

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Batch ingestion & analytics

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Real-time ingestion & batch analytics

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
value = data

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
value =
data
time

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Hourly server logs  Real-time metrics
Weekly or monthly bills  Real-time spending alerts
Daily website clickstream  Real-time clickstream analysis
Daily fraud reports  Real-time detection
Decision windows are shrinking

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Real-time ingestion & analytics

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Kinesis
Amazon Kinesis
Data Streams
Amazon Kinesis
Data Analytics
Amazon Kinesis
Data Firehose
Build custom applications
that process and analyze
streaming data
Easily process and
analyze streaming data
with standard SQL
Easily load streaming
data into AWS

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Simple pattern for streaming data
Continuously creates
data
Continuously writes
data to a stream
Can be almost anything
Data producer
Durably stores data
Provides temporary
buffer that preps data
Supports very high-
throughput
Streaming service
Continuously processes
data
Cleans, prepares, &
aggregates
Transforms data to
information
Data consumer

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Kinesis Data Streams
• Easy administration and low cost
• Build real-time applications with framework of choice
• Secure, durable storage

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Kinesis Data Analytics
• Powerful real-time applications
• Easy to use, fully managed
• Automatic elasticity

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Kinesis Data Firehose
• Zero administration and seamless elasticity
• Direct-to-data store integration
• Serverless, continuous data transformations
Amazon S3
Amazon Redshift

14. Speed (Real-time)
Ingest ServingData
sources
Scale (Batch)
Modern data architecture (AWS)
Insights to enhance business applications, new digital services
Data Warehouse
Amazon Redshift
Legacy Apps
Amazon RDS
Data analysts
Data scientists
Business users
Engagement platforms
Schemaless
Amazon ES
Direct query
Amazon Athena
Near-zero latency
Amazon DynamoDB
Automation / events
Amazon S3
Staged data
(Data lake)
Semi/Unstructured
Amazon EMR
Transactions
Web logs /
cookies
ERP
AWS Database
Migration Service
AWS Direct
Connect
Internet
interfaces
Amazon
Kinesis
Connected
devices
Social media
Amazon S3
Raw Data
Amazon EMR
ETL
Advanced
analytics
MLlib
Event Capture
Amazon Kinesis
Stream Analysis
Amazon EMR
AWS
CloudTrail
AWS
IAM
Amazon
CloudWatch
AWS
KMS

15. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.