In this session, we show you how to understand what data you have, how to drive insights, and how to make predictions using purpose-built AWS services. Learn about the common pitfalls of building data lakes and discover how to successfully drive analytics and insights from your data. Also learn how services such as Amazon S3, AWS Glue, Amazon Redshift, Amazon Athena, Amazon EMR, Amazon Kinesis, and Amazon ML services work together to build a successful data lake for various roles, including data scientists and business users.

1. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Jeet Shangari, Senior Technical Account Manager
Amazon Web Services
BDA305
Build Data Lakes and Analytics on AWS:
Patterns & Best Practices

2. VisualizationVariability
Big data: Different forms of challenges
Volume Velocity Variety Veracity Value

3. https://www.promptcloud.com
https://john-popelaars.blogspot.com
https://ww.signiant.com
https://www.linkedin.com/pulse/world-today-data-rich-information-poor-
guru-p-mohapatra-pmp/
Challenges are often driven by:
Data growth
faster than ever
Data variety is
increasing

4. AWS Data Lake helps address this
Quickly ingest and store any
type of data
Single source of truth
Run the right tool for the right
job without manually copying
data around

5. Data lakes from AWS
Analytics
Machine
learning
Real-time dataOn Premises
Data lake
on AWS
movementdata movement
Ingestion
Intelligence
Storage
Catalog
Variety of
ingestion tools
Decoupled
analytics from
storage/catalog

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What data do I have?

7. Hot Warm Cold
Volume MB–GB GB–TB PB–EB
Item size B–KB KB–MB KB–TB
Latency ms ms, sec min, hrs
Durability Low–high High Very high
Request rate Very high High Low
Cost/GB $$-$ $-¢¢ ¢
Hot data Warm data Cold data
Data characteristics: Hot, warm, cold

8. COLLECT
Devices
Sensors
IoT platforms
AWS IoT STREAMS
IoT
EventsData streams
Migration
Snowball
Logging
Amazon
CloudWatch
AWS
CloudTrail
FILES
DataTransport&Logging
Import/export
Files
Log files
Media files
Mobile apps
Web apps
Data centers AWS Direct
Connect
RECORDS
Applications
Transactions
Data structures
Database records
Type of data

9. Events
Files
Transactions
COLLECT
Devices
Sensors
IoT platforms
AWS IoT STREAMS
IoT
Data streams
Migration
Snowball
Logging
Amazon
CloudWatch
AWS
CloudTrail
FILES
DataTransport&Logging
Import/export
Log files
Media files
Mobile apps
Web apps
Data centers AWS Direct
Connect
RECORDS
Applications
Data structures
Database records
Type of data STORE
NoSQL
In-memory
SQL
File/object
store
Stream
storage

10. Which data store should I use?
Data structure → Fixed schema, JSON, key-value
Access patterns → Store data in the format you will access it
Data characteristics → Hot, warm, cold
Cost → Right cost

11. Data structure and access patterns
Access patterns What to use?
Put/Get (key, value) In-memory, NoSQL
Simple relationships → 1:N, M:N NoSQL
Multi-table joins, transaction, SQL SQL
Faceting, Search Search
Graph traversal GraphDB
Data structure What to use?
Fixed schema SQL, NoSQL
Schema-free (JSON) NoSQL, Search
Key-value In-memory, NoSQL
Graph GraphDB

12. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Gartner:
“Through 2018, 80% of data lakes will not include effective
metadata management capabilities, making them inefficient.”
”Metadata Is the Fish Finder in Data Lake.”
What data do I have?
Data lake
on AWS
Storage | Archival Storage | Data Catalog

13. Job AuthoringData Catalog Job Execution
Apache Hive Metastore compatible
Integrated with AWS services
Automatic crawl and discover data
Discover
Auto-generates ETL code
Python and Apache Spark
Edit, debug, and share
Develop
Serverless execution
Flexible scheduling
Monitoring and alerting
Deploy
AWS Glue components

14. IAM role
AWS Glue crawler Databases
Amazon
Redshift
Amazon S3
JDBC connection
Object connection
Built-in classifiers
MySQL
MariaDB
PostgreSQL
Amazon Aurora
Oracle
Amazon Redshift
Avro
Parquet
ORC
XML
JSON & JSONPaths
AWS CloudTrail
BSON
Logs
Apache (Grok), Linux (Grok), MS (Grok), Ruby, Redis,
and many others
Delimited
(comma, pipe, tab, semicolon)
< ALWAYS GROWING…>
What can crawlers discover?
Create additional custom
classifiers
Amazon
DynamoDB
NoSQL connection

15. But I have my own data formats …?
− There is a custom classifier for that …
Row-based
GROK Classifier
A grok pattern is a
named set of regular
expressions (regex)
that are used to match
data one line at a time.
XML
XML Classifier
XML tag that defines a
table row in the XML
document.
JSON
JSON Classifier
JSON path to the
object, array, or value
that defines a row of
the table being
created. Type the
name in either dot or
bracket JSON syntax
using AWS Glue-
supported operators

16. Other ways of populating the catalog
Call the AWS Glue CreateTable API
Create table manually DDL statement (in Amazon Athena or Amazon EMR)
Apache Hive
Metastore
AWS GLUE ETL AWS GLUE
DATA CATALOG
Import from Apache Hive Metastore

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How do I hydrate my data lake?

18. How do I drive value?
Amazon SageMaker
AWS Deep Learning AMIs
Amazon Rekognition
Amazon Lex
AWS DeepLens
Amazon Comprehend
Amazon Translate
Amazon Transcribe
Amazon Polly
Amazon Athena
Amazon EMR
Amazon Redshift
Amazon Elasticsearch Service
Amazon Kinesis
Amazon QuickSight
AWS Direct Connect
AWS Snowball
AWS Snowmobile
AWS Database Migration Service
AWS IoT Core
Amazon Kinesis Data Firehose
Amazon Kinesis Data Streams
Amazon Kinesis Video Streams
Data lake
on AWS
Storage | Archival Storage | Data Catalog
AnalyticsMachine learning
Real-time data movementOn Premises data movement

19. Ingest data based on the type of data
Open and comprehensive
• Data movement from on-premises data centers
• Dedicated network connection
• Secure appliances
• Ruggedized shipping container
• Database migration
• Gateway that lets applications write to the cloud
• Data movement from real-time sources
• Connect devices to AWS
• Real-time data streams
• Real-time video streams
AWS Direct Connect
AWS Snowball
AWS Snowmobile
AWS Database Migration Service
AWS Storage Gateway
AWS IoT Core
Amazon Kinesis Data Firehose
Amazon Kinesis Data Streams
Amazon Kinesis Video Streams
Data movement from
real-time sources
Data movement from On
Premises
Amazon S3
Amazon Glacier
AWS Glue

20. Amazon
Kinesis Data
Firehose
Real-time data movement and data lakes on AWS
AWS Glue
Data Catalog
Amazon
S3 data
Data lake
on AWS
Amazon
Kinesis Data
Streams
Data definitionKinesis Agent
Apache Kafka
AWS SDK
LOG4J
Flume
Fluentd
AWS Mobile SDK
Kinesis Producer Library

21. Amazon S3
Amazon Glacier
AWS Glue
IMPORTANT: Ingest data in its raw form …
Open and comprehensive
• Store the data in its raw form:
• BEFORE
• Transforming
• Analyzing
• Manipulating
• Doing … anything … to it
CSV
ORC
Grok
Avro
Parquet
JSON
• This becomes your source of record you can
always go back to …
• Lifecycle policies allow you to shift it to warm and
cold storage.

22. Tiered storage to optimize price/performance
Lowest cost
• Tiered storage to optimize price/performance
• Amazon S3 Standard
• Amazon S3 Standard—Infrequent Access
• Amazon S3 One Zone—Infrequent Access
• Amazon Glacier
• Migrate between tiers based on lifecycle policies
• Store data at $0.023*/GB/month with Amazon S3
• Store data at $0.004*/GB/month with Amazon Glacier
* As of July, 2018
Amazon S3
Standard
Amazon S3 Standard
Infrequent Access
Amazon S3 One
Zone-IA
Amazon Glacier

23. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Datasets in the lake?
Raw datasets – immutable datasets that you can always go back
to
• Abstract out the complexities of how the data is stored
through the catalog and SerDes
Optimizing analytics and machine learning:
Curated datasets – query-optimized for consumption across wide
number of tools

24. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Raw data stored in data lake:
Preparation:
No rmalized
Partitio ned
Co mpressed
S to rage o ptimized
Extract – Transform – Load
Preparing raw data for consumption
Data lake on
AWS
Raw
ingestion
Curated
Datasets
Data Catalog
ETL

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Which tool should I use to analyze my
data?

26. How do I drive value?
Amazon SageMaker
AWS Deep Learning AMIs
Amazon Rekognition
Amazon Lex
AWS DeepLens
Amazon Comprehend
Amazon Translate
Amazon Transcribe
Amazon Polly
Amazon Athena
Amazon EMR
Amazon Redshift
Amazon Elasticsearch Service
Amazon Kinesis
Amazon QuickSight
AWS Direct Connect
AWS Snowball
AWS Snowmobile
AWS Database Migration Service
AWS IoT Core
Amazon Kinesis Data Firehose
Amazon Kinesis Data Streams
Amazon Kinesis Video Streams
Data lake
on AWS
Storage | Archival Storage | Data Catalog
AnalyticsMachine Learning
Real-time dataOn Premises movementdata movement

27. Different tools for different users … solving different problems
Business
reporting
Data scientists
Data engineer
IDE
Data
Catalog
Central
storage
SagemakerMachine Learning/Deep Learning

28. Amazon Athena – interactive analysis
Interactive query service to analyze data in Amazon S3 using standard SQL
No infrastructure to set up or manage and no data to load
Ability to run SQL queries on data archived in Amazon Glacier (coming soon)
$ SQL
Query instantly
Zero setup cost; just
point to Amazon S3
and start querying
Pay per query
Pay only for queries run;
save 30%–90% on per-
query costs through
compression
Open
ANSI SQL interface,
JDBC/ODBC drivers, multiple
formats, compression types,
and complex joins and data
types
Easy
Serverless: zero
infrastructure, zero
administration
Integrated with Amazon
QuickSight

29. Amazon EMR – big data processing
Analytics and ML at scale
19 open-source projects: Apache Hadoop, Spark, HBase, Presto, and more
Enterprise-grade security
$
Latest versions
Updated with the latest
open source frameworks
within 30 days of release
Low cost
Flexible billing with per-
second billing, Amazon
EC2 Spot, Reserved
Instances, and Auto
Scaling to reduce costs
50%-80%
Use Amazon S3 storage
Process data directly in
the Amazon S3 data lake
securely with high
performance using the
EMRFS connector
Easy
Launch fully managed
Hadoop & Spark in minutes;
no cluster setup, node
provisioning, cluster tuning
Data Lake
100110000100101011100
1010101110010101000
00111100101100101
010001100001

30. Hadoop/Spark Analytics on AWS
YARN (Hadoop ResourceManager)
NoSQLMachine
learning
Real-timeInteractiveScriptBatch
Data lake
on AWS
Amazon S3
Amazon EMR
Managed Hadoop/Spark
Object storage

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Fitting this into the Common Data Catalog
Amazon S3
Interactive Spark cluster
Amazon EMR
Amazon EMR
EMRFS
HDFS
Transient ETL job
Source of Truth
EMRFS
HDFS
Describes the data
MySQL DB
instance
Unifieddataview
AWS Glue
Data Catalog
Stores the data
…

32. Amazon Redshift – data warehousing
Fast, powerful, simple, and fully managed data warehouse at 1/10 the cost
Massively parallel, scale from gigabytes to petabytes
Fast at scale
Columnar storage
technology to improve I/O
efficiency and scale query
performance
$
Inexpensive
As low as $1,000 per
terabyte per year, 1/10 the
cost of traditional data
warehouse solutions; start
at $0.25 per hour
Open file formats Secure
Audit everything; encrypt
data end-to-end;
extensive certification and
compliance
Analyze optimized data
formats on the latest SSD,
and all open data formats in
Amazon S3

33. Data warehouse …
Amazon Redshift data warehouse
Relational data
Gigabytes to petabytes scale
Reporting and analysis
Schema defined prior to data load
AWS
Glue ETL
On Prem
Amazon QuickSight
Existing or new
BI tool
Amazon
Redshift
COPY

34. Complementary to EDW (not replacement) Data lake can be source for EDW
Schema on read (no predefined schemas) Schema on write (predefined schemas)
Structured/semi-structured/Unstructured data Structured data only
Fast ingestion of new data/content Time consuming to introduce new content
Data Science + Prediction/Advanced Analytics + BI use
cases
BI use cases
Data at low level of detail/granularity Data at summary/aggregated level of detail
Loosely defined SLAs Tight SLAs (production schedules)
Flexibility in tools (open source/tools for advanced
analytics)
Limited flexibility in tools (SQL only)
Elastic storage and compute capacity – decoupled
Explicitly sized environments, compute and storage
scaled in linearly
A data lake is not an enterprise data warehouse (EDW)
Data lake EDW

35. Data lakes extend the traditional data warehouse
Data warehouse
Business intelligence
OLTP ERP CRM LOB
• Relational and nonrelational data
• TBs–EBs scale
• Diverse analytical engines
• Low-cost storage & analytics
Devices Web Sensors Social
Data lake
Big data processing,
real-time, machine learning

36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Machine learning and data lake

37. Agility in machine learning
Amazon SageMaker
AWS Deep Learning AMIs
Amazon Rekognition
Amazon Lex
AWS DeepLens
Amazon Comprehend
Amazon Translate
Amazon Transcribe
Amazon Polly
Amazon Athena
Amazon EMR
Amazon Redshift
Amazon Elasticsearch Service
Amazon Kinesis
Amazon QuickSight
AWS Direct Connect
AWS Snowball
AWS Snowmobile
AWS Database Migration Service
AWS IoT Core
Amazon Kinesis Data Firehose
Amazon Kinesis Data Streams
Amazon Kinesis Video Streams
Data lake
on AWS
Storage | Archival Storage | Data Catalog
AnalyticsMachine Learning
Real-time dataOn-premises movementdata movement

38. Agility in machine learning – for all users
Application
Services
• Designed for application developers
• Solution-oriented prebuilt models available via apis
• Image analysis, text-to-speech, conversational UX
Platforms
• Designed for data scientists to address common needs
• Fully managed platform for model building
• Reduces the heavy lifting in model building & deployment
Frameworks
• Designed for data scientists to address advanced / emerging needs
• Provides maximum flexibility to develop on the leading AI frameworks
• Enables expert AI systems to be developed & deployed

39. Digital Globe – using ML to find the right data
Data Lake:
• 100 PB of data in cloud
• Optimize storage tiers
Solution:
• Optimize their data lake
storage, cut costs in half

41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
FINRA − data is central to our mission
Reconstruct the market from trillions of events
• Data from broker-dealers and exchanges
• Equities, options, fixed income
• Build a graph of market order events
Analyze the data looking for financial fraud
• Insider trading, layering, cross-product
manipulation, front running, & many more
• Looking for a needle in a haystack
4

42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
FINRA − from data puddles to data lake
Database1
Storage
Query/compute
Catalog
Database2
Storage
Query/compute
Catalog
Databasen
Storage
Query/compute
Catalog
Storage
Query/
compute
Catalog
EMR Spark LambdaEMR Presto EMR HBase
Herd Hive
Metastore
FINRA in data center FINRA in AWS
Scales Silo
Amazon
S3

43. Real-time
App state
or
materialized
view
Interactive
and
batch
Data lake
Amazon S3
Amazon Redshift
Amazon EMR
Presto
Hive
Pig
Spark
Amazon
ElastiCache
Amazon
DynamoDB
Amazon
RDS
Amazon
ES
AWS Lambda
Spark Streaming
on Amazon EMR
Applications
Amazon
Kinesis
KCL
Amazon
AI
Amazon
DynamoDB
Amazon
RDS
Change data capture
or export
Transactions
Stream
Files
Amazon Kinesis
Analytics
Amazon Athena
Amazon Kinesis
Firehose
Amazon ES

44. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Architectural Principles
Build decoupled systems
• Data → Store → Process → Store → Analyze → Answers
Use the right tool for the job
• Data structure, latency, throughput, access patterns
Leverage managed and serverless services
• Scalable/elastic, available, reliable, secure, no/low admin
Use log-centric design patterns
• Immutable logs (data lake), materialized views
Be cost-conscious
• Big data ≠ cost
Data Lakes and Data Warehouse compliments each other
AI/ML enable your applications

45. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Submit session feedback
1. Tap the Schedule icon.
2. Select the session you
attended.
3. Tap Session Evaluation to
submit your feedback.

46. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Thank you!