AWS Cloud Kata 2014 | Jakarta - 2-3 Big Data

Jakarta
Big Data on AWS
Markku Lepistö
Principal Technology Evangelist
@markkulepisto

Does this Data make me look big?

Generation
Collection & storage
Analytics & computation
Collaboration & sharing

Getting your Data into AWS
Amazon S3
Corporate(Data(
Center(
• Console Upload
• FTP
• AWS Import Export
• S3 API
• Direct Connect
• Storage Gateway
• 3rd Party Commercial Apps
• Tsunami UDP
1

Services: Storage: Amazon S3
Deployment(&(Administra=on(
App(Services(
Compute( Storage(
Database(
Networking(
AWS(Global(Infrastructure(
Amazon Simple Storage Service (S3)
• Unlimited storage of objects of any type
• 99.999999999% durability, replicated across multiple facilities
• Cost effective storage, US$0.03/GB Month
• Granular access control and permissions over objects
• Encryption at rest using AES 256bit server side encryption
• Encryption in transit using HTTPS protocol
• High performance throughput supporting parallelized
upload or download
• Import or export data via physical device handling service
• Data remains in geographic location chosen

Write directly to a data source
Your%applica+on% Amazon S3
DynamoDB%
Any%other%data%
store%
Amazon S3
Amazon%EC2%%
2

Services: Database: Amazon DynamoDB
" Zero Admin NoSQL Service
" Unlimited Storage
" Provisioned Throughput
" Consistent <10ms response
" Durable on SSD
Compute( Storage(
Database(
Networking(

Scale or Autoscale DynamoDB Table Throughput

Queue, pre-process and then write to data source
Amazon%Simple%
Queue%Service%
(SQS)%
Amazon S3
DynamoDB%
Any%other%data%
store%
3

Aggregate and write to data source
Flume running
on EC2
Amazon S3
HDFS
Any%other%data%
store%
4

Choose depending upon design
Amazon%SQS%
Amazon S3
DynamoDB%
Any%SQL%or%NO%SQL%
Store%
Log%Aggrega+on%%
tools%

S3 as a single source of truth
S3
Courtesy http://techblog.netflix.com/2013/01/hadoop-platform-as-service-in-cloud.html

Generation
Collection storage
Analytics computation
Collaboration sharing

Hadoop based Analysis
Amazon%SQS%
Amazon S3
DynamoDB%
Any%SQL%or%NO%SQL%
Store%
Log%Aggrega+on%%
tools%
Amazon
EMR

What is Amazon Elastic MapReduce (EMR)?
EMR is Hadoop in the Cloud!

How does EMR work ?
EMR%Cluster
S3
Put the data
into S3
Choose: Hadoop distribution, # of
nodes, types of nodes, custom
configs, Hive/Pig/etc.
Get the output from
S3
Launch the cluster using the
EMR console, CLI, SDK, or
APIs
You can also store
everything in HDFS

What can you run on EMR…
S3
EMR Cluster

SQL based processing
Amazon%SQS%
Amazon S3
DynamoDB%
Any%SQL%or%NO%SQL%
Store%
Log%Aggrega+on%%
tools%
Amazon
EMR
Amazon
Redshift
Pre-processing
framework
Petabyte scale
Columnar Data -
warehouse

Services: Database: Amazon Redshift
Amazon Redshift
• Easily and rapidly analyze petabytes of data
• Fully managed data warehouse service
• Automated deployment and administration
• 1/10th the cost of traditional data warehouses
• $1000 / Terabyte / year
• Compatible with popular BI tools
Deployment((Administra=on(
App(Services(
Compute( Storage(
Database(
Networking(

Your choice of BI Tools on the cloud
Amazon%SQS%
Amazon S3
DynamoDB%
Any%SQL%or%NO%SQL%
Store%
Log%Aggrega+on%%
tools%
Amazon
EMR
Amazon
Redshift
Pre-processing
framework

Collaboration and Sharing insights
Amazon%SQS%
Amazon S3
DynamoDB%
Any%SQL%or%NO%SQL%
Store%
Log%Aggrega+on%%
tools%
Amazon
EMR
Amazon
Redshift

Sharing results and visualizations at scale
Amazon%SQS%
Amazon S3
DynamoDB%
Any%SQL%or%NO%SQL%
Store%
Log%Aggrega+on%%
tools%
Amazon
EMR
Amazon
Redshift
Visualization tools
Web App Server

Rinse and Repeat every day or hour

Rinse and Repeat
Amazon%SQS%
Amazon S3
DynamoDB%
Any%SQL%or%NO%SQL%
Store%
Log%Aggrega+on%%
tools%
Amazon
EMR
Amazon
Redshift
Business
Intelligence Tools
Visualization tools
GIS tools
Business
Intelligence Tools
GIS tools on
hadoop
Amazon data pipeline

The complete architecture
Amazon%SQS%
Amazon S3
DynamoDB%
Any%SQL%or%NO%SQL%
Store%
Log%Aggrega+on%%
tools%
Amazon
EMR
Amazon
Redshift
Business
Intelligence Tools
Visualization tools
GIS tools
Business
Intelligence Tools
GIS tools on
hadoop
Amazon data pipeline

is the exception rate
is the ad click-through
topics are trending
inventory remains
queries are slow
are the high scores
What right now?

HAPPENING NOW!
real-time == stream analytics

Ingest data streams
Store durably
Distribute
Scale out
Process as packets flow in

Realtime Analytics in the Cloud
Amazon%Kinesis%
Streaming*Data*Service*

Kinesis architecture
AZ AZ AZ
Durable, highly consistent storage replicates data
across three data centers (availability zones)
Amazon Web Services
Aggregate and
archive to S3
Millions of
sources producing
100s of terabytes
per hour
Front
End
Authentication
Authorization
Ordered stream
of events supports
multiple readers
Real-time
dashboards
and alarms
Machine learning
algorithms or
sliding window
analytics
Aggregate analysis
in Hadoop or a
data warehouse
Inexpensive: $0.028 per million puts

InBgame(
ac=vity(
Kinesis:(RealB=me(data(stream(of(inBgame(ac=vity(
Amazon(
Kinesis(
Clash(of(Clans(

KinesisBenabled(apps(on(EC2(
InBgame(
ac=vity(
Mul=ple(Kinesis(applica=ons:(Dashboards,(analy=cs(and(storage(
Clash(of(Clans(
RealB=me(clickstream(
processing(app(
Amazon(
Kinesis(

S3(and(Glacier:(Data(storage(and(long(term(archival(
S3( Aggregate(
sta=s=cs(
InBgame(
ac=vity(
EC2:(InBgame(
engagement(
trends(
dashboard(
Clash(of(Clans(
processing(app(
Amazon(
Kinesis(

BusinessBintelligence(
user(
EC2:(InBgame(
engagement(
trends(
dashboard(
InBgame(
ac=vity(
S3( Aggregate(
sta=s=cs(
Data(Warehouse:(BI(repor=ng(and(interac=ve(queries(
Clash(of(Clans(
EC2(
Data(
Warehouse(
processing(app(
Amazon(
Kinesis(

Glacier(
Clickstream(
archive(
EC2(
Data(
Warehouse(
EC2:(InBgame(
engagement(
trends(
dashboard(
Data(Warehouse:(BI(repor=ng(and(interac=ve(queries(
processing(app(
InBgame(
ac=vity(
S3(
Clash(of(Clans(
Aggregate(
sta=s=cs(
BusinessBintelligence(
user(
Amazon(
Kinesis(

Demo
Sliding Window Analytics ! Live Dashboard
S3 Storage ! Redshift Data Warehouse
Website
Clickstream
logs
Kinesis

Smart(Devices(
Powered(by(the(Cloud(

Smart?evices(

Smart?evices(
Arduino%Uno% Raspberry%Pi%
CPU( 20MHz(8bit( 700MHz(32bit(
Memory( 2(KB( 512(MB(
Storage( 32(KB( SD(card(

Camera(
Microphone(
Thermometer(
Distance(
GPS(
Gyroscope(
Accelerometer(
Actuator(
Relay(
Motor(
Manipulator(
Pressure( Switch(
Wheel(
Propeller(
Rotor(

Challenges(
Thousands(–(Millions(of(
Devices(/(Producers(

Challenges(
Users(/(Consumers(

Distributed(
Users(/(Consumers(

At(scale(
Users(/(Consumers(

Smart(Devices(
Unlimited(Storage(–(Memory(
((
Unlimited(Compute(–(Logic(

Camera(
Microphone(
Thermometer(
Distance(
GPS(
Gyroscope( Accelerometer(
Actuator(
Relay(
Motor(
Manipulator(
Pressure( Switch(
Wheel(
Propeller(
Rotor(

Dropcam(is(the(biggest(inbound(video(service(on(the(Web((
• More(data(uploaded(per(
minute(than(YouTube((
• Petabytes(of(data(
processed(every(month(
• Billions(of(mo=on(events(
detected(

The AWS Big Data Portfolio
COLLECT | STORE | ANALYZE | SHARE
Direct Connect S3
Import Export
S3 EC2
DynamoDB Redshift
Glacier
EMR
Data Pipeline
Kinesis
CloudFront

(((((((((((((((AWS(CBSDK(Experimental(
for((
ArduinoBstyle(IoT(Devices(

CBSDK(–(Na=ve(AWS(Libraries,(
Direct(Access(to(AWS(Services(from(Devices(

AcceleB
rometer(
MQTT(
Mosqui_o(MQTT(Broker(
MQTTBKinesis(Bridge(
AWS(SDK(
Amazon(Kinesis(
RealB=me(Streaming(
Data(Service(
AWS(APIs(
AWS(Elas=c(
Beanstalk(
Dashboard(

Amazon(SNS(
Alert(No=fica=on(
(5g(
Mobile(Push(
Spark(API(
Sound(Alarm(
(6g((
AWS(Elas=c(
Beanstalk(
Dashboard(

AWS Cloud Kata for Start-Ups and Developers
台北
6月25日

AWS Cloud Kata 2014 | Jakarta - 2-3 Big Data

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

En vedette

En vedette (11)

Similaire à AWS Cloud Kata 2014 | Jakarta - 2-3 Big Data

Similaire à AWS Cloud Kata 2014 | Jakarta - 2-3 Big Data (20)

Plus de Amazon Web Services

Plus de Amazon Web Services (20)

Dernier

Dernier (20)

AWS Cloud Kata 2014 | Jakarta - 2-3 Big Data