Amazon Kinesis Data Analytics gives us to tools to run SQL queries against active data streams. We'll look at how we can performance real-time log analytics and q build entire streaming applications using SQL, so that you can gain actionable insights promptly. Level: Intermediate Speaker: Bill Baldwin - Database Technical Evangelist, AWS

1. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Pop-up Loft
Analyzing Streams
Asser Moustafa
Data Warehousing Solutions Architect Specialist
aserm@amazon.com
Krishna Reddy
Technical Account Manager
reddradh@amazon.com

2. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Pop-up Loft
Agenda
• Introduction
– Case for Streaming Data
– Kinesis Service
• Kinesis Data Analytics Use Cases
• Streaming Analytics Coding Examples
• Kinesis Cost and Ecosystem
• Lab

3. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Pop-up Loft
Agenda
• Introduction
– Case for Streaming Data
– Kinesis Service
• Kinesis Data Analytics Use Cases
• Streaming Analytics Coding Examples
• Kinesis Cost and Ecosystem
• Lab

4. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Hourly server logs
Weekly or monthly bills
Daily web-site clickstream
Daily fraud reports
Real time metrics
Real time spending alerts/caps
Real time clickstream analysis
Real time detection
It’s All About the Pace
Batch Processing Stream Processing

5. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Why? Data loses value over time
Ingest data as it is generated
Analyze data in real time to get
insights immediately
Deliver data to in seconds instead
of hours

6. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Simple Pattern for Streaming Data
Continuously creates
data
Continuously writes
data to a stream
Can be almost anything
Data Producer
Durably stores data
Provides temporary
buffer that preps data
Supports very high-
throughput
Streaming Service
Continuously processes
data
Cleans, prepares, &
aggregates
Transforms data to
information
Data Consumer
Mobile Client Amazon Kinesis Amazon Kinesis app

7. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Kinesis
Amazon Kinesis
Data Streams
Amazon Kinesis
Data Analytics
Amazon Kinesis
Data Firehose
Build custom
applications that process
and analyze streaming
data
Easily process and
analyze streaming data
with standard SQL
Easily load streaming
data into AWS
Amazon Kinesis
Video Streams
Capture, process, and
store video streams for
analytics and machine
learning

8. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Kinesis Data Streams
• Easy administration and low cost
• Build real time applications with framework of choice
• Secure, durable storage

9. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Kinesis Data Analytics
• Powerful real time applications
• Easy to use, fully managed
• Automatic elasticity

10. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Kinesis Data Firehose
• Zero administration and seamless elasticity
• Direct-to-data store integration
• Serverless, continuous data transformations

11. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Kinesis Video Streams
• Zero administration and seamless elasticity
• Direct-to-data store integration

12. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Automatic ingestion Easy setup Write your own
Stream Data to Amazon Kinesis
Amazon
VPC Flow
Logs
Elastic Load
Balancing
Amazon
RDS
Amazon
CloudWatch
Logs
AWS
CloudTrail
Event Logs
Amazon
Pinpoint
Amazon API
Gateway
AWS IoT
events
AWS SDKs
Amazon
DynamoDB
Amazon
Kinesis Agent
Amazon
Kinesis
Producer
Library
As a proxy:
For change data capture:
Just a sample… many more ways stream data to Amazon Kinesis

13. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Pop-up Loft
Agenda
• Introduction
– Case for Streaming Data
– Kinesis Service
• Kinesis Data Analytics Use Cases
• Streaming Analytics Coding Examples
• Kinesis Cost and Ecosystem
• Lab

14. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Streaming
Ingest-
Transform-Load
Continuous
Metric
Generation
Actionable
Insights
Three Common Scenarios
Compute analytics as the data is generated
React to analytics based off of insights
Deliver data to analytics tools faster and
cheaper

15. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Example 1: Web Analytics and Leaderboards
Amazon
Kinesis Data
Analytics
AWS
Lambda
function
Amazon
Cognito
Lightweight JS
client code
Web Server on
Amazon EC2
Instance
OR
Amazon
DynamoDB
Table
Amazon
Kinesis Data
Streams
Compute top 10 usersIngest web app data Persist to feed live apps

16. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Example 2: Monitoring IoT Devices
IoT sensors AWS IoT
Amazon
RDS
MySQL DB
instance
Amazon
Kinesis
Data
Streams
Amazon
Kinesis
Data
Analytics
AWS
Lambda function
Compute avg temp
every 10 sec
Ingest sensor data
Persist time series
analytic to database

17. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Example 3: Ingest and deliver CloudTrail events
• CloudTrail provides continuous
account activity logging
• Events are sent in real time (to near
real time) to Kinesis Data Firehose or
Streams and persisted to S3
• Each event includes a timestamp, IAM
user, AWS service name, API call,
response, and more
AWS
CloudTrail
Amazon
CloudWatch
events trigger
Amazon S3
bucket for raw
data
Kinesis Data
Firehose

18. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Example 3 (co n t ): Analyzing CloudTrail Event Logs
AWS
CloudTrail
Amazon
CloudWatch
events trigger
Amazon
Kinesis
DataAnalytics
AWS
Lambda
function
Amazon S3
bucket for raw
data
Amazon S3
bucket for
processed data
Amazon
DynamoDB
Table(s)
Chart.JS
Dashboard
Compute
operational metrics
Ingest and deliver raw
log data
Deliver to a real time
dashboards and archival
Amazon Kinesis
Data Firehose
Amazon Kinesis
Data Firehose

19. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Pop-up Loft
Agenda
• Introduction
– Case for Streaming Data
– Kinesis Service
• Kinesis Data Analytics Use Cases
• Streaming Analytics Coding Examples
• Kinesis Cost and Ecosystem
• Lab

20. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Kinesis Data Analytics Applications
Easily write SQL code to process streaming data
Connect to streaming source
Continuously deliver SQL results

21. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How do I write streaming SQL? Easy!
Streams (in memory tables)
CREATE STREAM calls_per_ip_stream(
eventTimeStamp TIMESTAMP,
computationType VARCHAR(256),
category VARCHAR(1024),
subCategory VARCHAR(1024),
unit VARCHAR(256),
unitValue BIGINT
);

22. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How do I write streaming SQL? Easy!
Pumps (continuous query)
CREATE OR REPLACE PUMP calls_per_ip_pump AS
INSERT INTO calls_per_ip_stream
SELECT STREAM "eventTimestamp",
COUNT(*),
"sourceIPAddress"
FROM source_sql_stream_001 ctrail
GROUP BY "sourceIPAddress",
STEP(ctrail.ROWTIME BY INTERVAL '1' MINUTE),
STEP(ctrail."eventTimestamp" BY INTERVAL '1' MINUTE);

23. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How do we aggregate streaming data?
• Aggregations (count, sum, min,…) take granular real time data
and turn it into insights
• Data is continuously processed so you need to tell the
application when you want results
Windows!

24. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Window Types
• Sliding, tumbling, and custom windows
• Sliding windows (e.g. last 5 seconds)
• Tumbling windows are fixed size and grouped keys do not overlap (e.g.
every 5 seconds)

25. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Event, ingest, and processing time
• Event time is the timestamp assigned when the event occurred,
also called client-side time
• {"EVENT_TIME": "2018-06-13T14:11:05.766191", "TICKER": “AMZN", "PRICE": 43.65}
• Processing time is when your application reads and analyzes the
data (accessed via the operator ROWTIME)
…
GROUP BY "sourceIPAddress",
/* Trigger for results */
STEP(ctrail.ROWTIME BY INTERVAL '1' MINUTE),
/* A timestamp grouping key */
STEP(ctrail."eventTimestamp" BY INTERVAL '1' MINUTE);

26. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Late results
• An event is late if it arrives after the computation for which it
logically belongs to has been completed
• Your Kinesis Analytics application will produce an amendment
…
GROUP BY "sourceIPAddress",
/* Trigger for results */
STEP(ctrail.ROWTIME BY INTERVAL '1' MINUTE),
/* A timestamp grouping key */
STEP(ctrail."eventTimestamp" BY INTERVAL '1' MINUTE);

27. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Updating a database
• Perform inserts but on duplicate key update
• For DyanamoDB, here is the AWS Lambda code:
…
GROUP BY "sourceIPAddress",
/* Trigger for results */
STEP(ctrail.ROWTIME BY INTERVAL '1' MINUTE),
/* A timestamp grouping key */
STEP(ctrail."eventTimestamp" BY INTERVAL '1' MINUTE);

28. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Pop-up Loft
Agenda
• Introduction
– Case for Streaming Data
– Kinesis Service
• Kinesis Data Analytics Use Cases
• Streaming Analytics Coding Examples
• Kinesis Cost and Ecosystem
• Lab

29. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What does all this cost?
• All services used in the solution are pay as you
go
• All services used are serverless and have lower
devops expense
• This solution will cost the “average” customer
less than:
$100 per month

30. Lots of customer examples
1 billion events/wk from
connected devices | IoT
17 PB of game data per
season | Entertainment
80 billion ad
impressions/day, 30 ms
response time | Ad Tech
100 GB/day click streams
from 250+ sites |
Enterprise
50 billion ad
impressions/day sub-50
ms responses | Ad Tech
10 million events/day
| Retail
Amazon Kinesis as Databus -
Migrate from Kafka to Kinesis| Enterprise
Funnel all
production events
through Amazon
Kinesis

31. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Customer Case Studies

32. Integrate with your current solutions

33. Get help from partner systems integrators

34. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Pop-up Loft
Agenda
• Introduction
– Case for Streaming Data
– Kinesis Service
• Kinesis Data Analytics Use Cases
• Streaming Analytics Coding Examples
• Kinesis Cost and Ecosystem
• Lab

35. Pop-up Loft
Hands-on Lab: Analyzing Streams
https://github.com/wrbaldwin/da-week/
Create your first Amazon Kinesis Data Analytics application using
the AWS console. Visit the link below for instructions:

36. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved
Pop-up Loft
aws.amazon.com/activate
Everything and Anything Startups
Need to Get Started on AWS