Amazon Kinesis Streams was born of necessity, to enable Amazon Web Services to capture millions of individual metering records per second from AWS services and EC2 instances and aggregate these records by account identifier to produce a customer billing record in real time. Today, Kinesis is a foundational service which over a dozen AWS and Amazon retail services use to capture and process streaming data.
Since the launch of Kinesis Streams, Amazon has worked with customers as their stream processing needs have evolved, subsequently launching Kinesis Firehose—which enables customers to capture streaming data, perform in-line processing to clean, format, and deliver this data to service destinations, such as S3, Redshift, Elasticsearch, and Lambda, in minutes—and Kinesis Analytics—which enables customers to process streaming data in real time using ANSI SQL..
Today, Kinesis data streaming services are foundational for business critical workflows, providing customers with a new way to process big data and extract actionable insights. In this session, we offer an overview of Kinesis, Amazon’s data streaming platform—highlighting key architectural features—and explains how customers have architected their applications using Kinesis services for low-latency and extreme scale. We will also demonstrate how to build an end-to-end solution using the latest and greatest AWS Data and analytics services like Athena, Glue, Lambda and more.
8. Dilip Kumar, VP of Technology for Amazon Go, cites as an example
Amazon Go’s unique system of streaming data from hundreds of
cameras to track the shopping activities of customers. The
innovations his team concocted helped influence an AWS service
called Kinesis, which allows customers to stream video from
multiple devices to the Amazon cloud, where they can process it,
analyze it, and use it to further advance their machine learning
efforts.
9. Stream video from millions of devices
Easily build vision-enabled apps
Secure
Durable, searchable storage
Serverless
Amazon AI
Services
Apache
MxNet
TensorFlow
OpenCV
Custom
Video
Processing
Kinesis
Video
Streams
Amazon Kinesis Video Streaming
Capture, process, and store video streams for analytics/ machine learning
28. ”
“ • Redfin helps consumers buy and sell real
estate in 37 states
• Uses Amazon Kinesis to deliver house
comparison data in milliseconds
• Can scale to manage billions of records and
images
• Reaches nationwide audience with small
staff
Redfin: Real-time Home Recommendations
Redfin is a full-service residential real estate
company based in Seattle.
AWS freed us from the
operational aspect of managing
our infrastructure. That enabled
us to innovate faster so our
customers can buy and sell
houses easier and faster.
Yong Huang
Director, Big Data Analytics
”
“
35. 35
The Addressable Market is Massive
“The overall revenue pool from car data
monetization at global scale is estimated at
$450-750 billion by 2030”
McKinsey&Company, Advanced Industries September 2016
36. 36
The First Connected
Car Data Marketplace
A dedicated platform for safe and
simple exchange of connected car
data to enable an ecosystem of
services and applications
Car
Owner
Smart
Mobility
Smart
City
Fleets
Emergency
Retail
insurance
Energy
Data
Maintenance
Marketplace
37. 37
INSURANCE PARKING FINANCE ENERGY RETAIL MOBILITY
S T A N D A R T D A P I
CLOUD DATA LAKE
• Security
• Privacy
• Normalize
• Insights
• Billing
TSP TSP TSP OEM OEM OEM FLEET FLEET FLEET
DATAPROVIDERS
DATACONSUMERS
40. 40
Platform Requirements
• Ingesting 200M+ events per day
• 500K+ events/sec at peak
• Various consumers types for different use cases
• Persisting large amount of data easily and efficiently
• Real-time stream processing
41. 41
Kinesis Suite in Action with Otonomo
Kinesis Data Stream Kinesis Data AnalyticsKinesis Data Firehose
42. 42
Amazon Kinesis Suite at Otonomo in action
Kinesis Data
Streams
KinesisKinesis Data Analytics
Kinesis Data Firehose
Processed
Data
Kinesis Data
Streams
ETL (ECS cluster)
KCL / KPL
Lambda
Elastic Workers Pool
Lake Builder
(EMR cluster)
Gatekeeper
Processed Data
Structured &
Formatted
Data
Data Lake
Insight Engine
Insight Handlers
EMR
Analytics & ML
Athena
43. 43
Analytics
• Real Time Sub second Processing processing optimized for stream processing
• Kinesis Data Analytics
• Anomaly detection / Insights / K-Top analysis
• Easily run SQL / Windowing functions on stream
• Batch Analytics & ML on top S3 data lake using Athena / Spark / Containers
44. 44
Key Takeaways
• S3 for data lake
• Decoupling of storage and compute
• Scale independently
• Serverless + Managed
• Focus on core business and technology