This document discusses building a modern data analytics architecture on AWS. It provides an overview of AWS services that can be used for ingesting, processing, storing, and analyzing large volumes of data in both real-time and batch scenarios. These include services like Amazon S3, Kinesis, EMR, Redshift, Athena, Elasticsearch, and Glue for ingesting, storing, processing, and querying data. Architectures shown include real-time data pipelines, data lakes, and batch ETL/ELT processes. Performance, cost effectiveness, and scalability benefits of AWS services are highlighted.

1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Russell Nash – AWS Solutions Architect, AWS
Building
A Modern Data Analytics
Architecture
on AWS
In partnership with:

2. SCALABLE FLEXIBLE MANAGEABLE
COST
EFFECTIVE

3. Ingest Serving
Speed (Real-time)
Scale (Batch)
Data analysts
Data scientists
Business users
Engagement platforms
Automation / events
Sources
Modern Data Architecture
AWS
Cloud Trail
AWS
IAM
Amazon
CloudWatch
AWS
KMS

4. Database Analytics Flat File
Processing
Real-time
Pipeline
Data Lake

5. Ingest Serving
Speed (Real-time)
Scale (Batch)
Data analysts
Data scientists
Business users
Engagement platforms
Automation / events
Sources
Database Analytics
Amazon
Redshift
Source
Database

6. MPP SQL Database
Optimised for Analytics
Gigabytes to Petabytes
Fully relational
Amazon
Redshift

8. ID Name
1 John Smith
2 Jane Jones
3 Peter Black
4 Pat Partridge
5 Sarah Cyan
6 Brian Snail
1 John Smith
4 Pat Partridge
2 Jane Jones
5 Sarah Cyan
3 Peter Black
6 Brian Snail

9. 1 John Smith
4 Pat Partridge
2 Jane Jones
5 Sarah Cyan
3 Peter Black
6 Brian Snail
SQL
SQL SQL SQLResults Results Results
Results

10. 160 GB
2 PB

11. Ingest Serving
Speed (Real-time)
Scale (Batch)
Data analysts
Data scientists
Business users
Engagement platforms
Automation / events
Sources
ETL
Amazon
Redshift
Source
Database
Database Analytics

12. AWS Database
Migration Service
Amazon
RedshiftSource
Database
ETL
Data Integration
Partners

13. Ingest Serving
Speed (Real-time)
Scale (Batch)
Data analysts
Data scientists
Business users
Engagement platforms
Automation / events
Sources
ELT
Amazon
Redshift
Amazon
Redshift
Source
Database
Database Analytics

14. https://aws.amazon.com/solutions/case-studies/boingo-wireless/

15. Database Analytics Flat File
Processing
Real-time
Pipeline
Data Lake

16. Ingest Serving
Speed (Real-time)
Scale (Batch)
Data analysts
Data scientists
Business users
Engagement platforms
Automation / events
Sources
Batch Processing
Flat
Files
Amazon
S3

17. Amazon
S3
Object Storage
Low Cost
Highly Scalable
11 9’s of durability

18. Ingest Serving
Speed (Real-time)
Scale (Batch)
Data analysts
Data scientists
Business users
Engagement platforms
Automation / events
Sources
Flat
Files
Amazon
S3
Batch Processing
AWS
Snowball
AWS
CLI & SDK

21. In pioneer days, they used oxen for heavy pulling,
and when one ox couldn’t budge a log,
they didn’t try to grow a bigger ox.
Grace Hopper

23. PIG
Infrastructure
Data Layer
Process Layer
Framework
Applications

24. PIG
SQL
Infrastructure
Data Layer
Process Layer
Framework
Applications

25. PIG
SQL
Amazon
EMR

26. PIG
SQL
Amazon
EMR
Amazon
S3
EMRFS

27. Amazon
EMR
• Managed Hadoop
• Optimized with S3
• Open Source Support

28. Compute Flexibility
Compute Memory Storage
Machine Learning
C4 Family
C3 Family
X1 Family
R3 Family
Interactive Analysis
D2 Family
I2 Family
Large HDFS
General
Batch Process
M4 Family
M3 Family

29. Cost & Time
# CPUs
Time
# CPUs
Time
Wall clock time: 1 hourWall clock time: 10 hours

31. Spot Price – M3.2XL
On-Demand Spot-Price
$0.08$0.75

32. Ingest Serving
Speed (Real-time)
Scale (Batch)
Data analysts
Data scientists
Business users
Engagement platforms
Automation / events
Sources
Flat
Files
Amazon
S3
Batch Processing
Amazon
EMR
Amazon
S3
AWS
Glue
AWS
Snowball
AWS
CLI & SDK

33. AWS
Glue
• Managed Transform Engine
• Job Scheduler
• Data Catalog
• Built on Apache Spark

35. Ingest Serving
Speed (Real-time)
Scale (Batch)
Data analysts
Data scientists
Business users
Engagement platforms
Automation / events
Sources
Flat
Files
Amazon
S3
Batch Processing
Amazon
EMR
Amazon
S3
AWS
Glue
Amazon
Redshift
Amazon
EMR
AWS
Snowball
AWS
CLI & SDK

36. PIG
SQL
Amazon
EMR
Amazon
S3
EMRFS
R

37. Ingest Serving
Speed (Real-time)
Scale (Batch)
Data analysts
Data scientists
Business users
Engagement platforms
Automation / events
Sources
Flat
Files
Amazon
S3
Batch Processing
Amazon
EMR
Amazon
S3
AWS
Glue
Amazon
Redshift
Amazon
EMR
Amazon
AthenaAWS
Snowball
AWS
CLI & SDK

38. Amazon
Athena
Query S3 data with SQL
Serverless
Instant Spin-Up
Pay per Query

39. Athena
S3

40. Comparison of SQL Processing engines
Amazon
Redshift
Amazon
Athena
Data Structure
Languages
Semi Semi
SQL, HiveQL SQL
Full
SQL
Data Store S3/HDFS S3 Local
SQL
Semi
SQL
S3/HDFS
Performance

41. Comparison of SQL Processing engines
Transformation
SQL Queries
For S3/HDFS
Fully Featured
SQL
Database
Use Case
Amazon
RedshiftAmazon
Athena
SQL
Serverless
SQL Queries
for S3

42. https://aws.amazon.com/solutions/case-studies/finra/

43. Database Analytics Flat File
Processing
Real-time
Pipeline
Data Lake

44. Ingest Serving
Speed (Real-time)
Scale (Batch)
Data analysts
Data scientists
Business users
Engagement platforms
Automation / events
Sources
Real-time Pipeline
Amazon
Kinesis
Machines
Devices
Mobile
Clickstream

45. Availability
Zone
Availability
Zone
Availability
Zone
Amazon Kinesis
Stream
AWS Lambda
KCL App
Amazon EMR
Streaming

46. Ingest Serving
Speed (Real-time)
Scale (Batch)
Data analysts
Data scientists
Business users
Engagement platforms
Automation / events
Sources
Amazon
Kinesis
AWS Lambda
Application
Amazon EMR
Streaming
S3
(Log)
Amazon
ElasticSearch
(Dashboard)
Real-time Pipeline

47. Amazon
Elasticsearch
• Search and Analytics
• Scalable
• Fully Managed
• Integrated – Logstash, Kibana

49. Ingest Serving
Speed (Real-time)
Scale (Batch)
Data analysts
Data scientists
Business users
Engagement platforms
Automation / events
Sources
Amazon
Kinesis
AWS Lambda
Application
Amazon EMR
Streaming
S3
(Logs)
Amazon
ElasticSearch
(Dashboards)
Amazon EMR
(Predictions)
ML
Amazon SNS
(Alerts)
Real-time Pipeline
Amazon
Redshift
(Analytics)

50. StreamAlert
https://medium.com/airbnb-engineering/streamalert-real-time-data-analysis-and-alerting-e8619e3e5043

51. Database Analytics Flat File
Processing
Real-time
Pipeline
Data Lake

52. Any data Any analysisData Lake

53. Ingest Serving
Speed (Real-time)
Scale (Batch)
Data analysts
Data scientists
Business users
Engagement platforms
Automation / events
Sources
Amazon
Kinesis
AWS Lambda
Application
Amazon EMR
Streaming
Amazon
EMR
Data Lake
Amazon
Redshift
ETL
Amazon
Athena
EC2
AWS
CLI & SDK
Amazon
S3
Amazon
EMR
Amazon
S3
AWS
Cloud Trail
AWS
IAM
Amazon
CloudWatch
AWS
KMS

54. Now available in the Mumbai region!
Amazon
Redshift
Amazon
EMR
Amazon
Kinesis
Amazon
Elasticsearch

55. Select Customers
The vast majority of Big Data use cases deployed in the cloud today run onAWS

56. New X1 Instance - Tons of Memory
• Large-scale, in-memory applications
• Intel® Xeon® E7 8880 v3 Haswell processors
• Up to 2TB of memory
• Up to 128 vCPUs per instance

57. Intel® Processor Technologies
Intel® AVX – Dramatically increases performance for highly parallel HPC workloads
such as life science engineering, data mining, financial analysis, media processing
Intel® AES-NI – Enhances security with new encryption instructions that reduce the
performance penalty associated with encrypting/decrypting data
Intel® Turbo Boost Technology – Increases computing power with performance that
adapts to spikes in workloads
Intel Transactional Synchronization (TSX) Extensions – Enables execution of
transactions that are independent to accelerate throughput
P state & C state control – provides granular performance tuning for cores and sleep
states to improve overall application performance

58. REGISTER NOW
http://amzn.to/2jFt11N
Complimentary labs are available only till 31 March 2017
Get hands on experience working with the AWS Technology.
Access the complimentary Big Data on AWS self-paced labs