SlideShare une entreprise Scribd logo
1  sur  50
Télécharger pour lire hors ligne
© 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Roy Ben-Alta
Principal Business Development Manager and Architect, Amazon Web Services
Sefi Itzkovich
VP Architecture, Otonomo
Amazon Kinesis – Building Serverless
real-time Solution
Overview
• Speed and Serverless Matters
• Amazon Kinesis Overview and Use Cases
• Otonomo Case Study
• Demo
• Q&A
© 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Timely Decisions Require New Data in Minutes
Source: Perishable insights, Mike Gualtieri, Forrester
Data loses value quickly over time
© 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Stream New Data in Seconds
Get actionable insights quickly
Streaming
Ingest video
& data as it’s
generated
Process or
deliver data
on the fly
Real-time
analytics/ML,
alerts, actions
© 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Most Common uses of Real-Time Streaming
Video Analytics Automation Data
Lakes
IoT Monitoring
and Analytics
Log
Analytics
© 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Speed Matters
© 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Speed Matters
Dilip Kumar, VP of Technology for Amazon Go, cites as an example
Amazon Go’s unique system of streaming data from hundreds of
cameras to track the shopping activities of customers. The
innovations his team concocted helped influence an AWS service
called Kinesis, which allows customers to stream video from
multiple devices to the Amazon cloud, where they can process it,
analyze it, and use it to further advance their machine learning
efforts.
Stream video from millions of devices
Easily build vision-enabled apps
Secure
Durable, searchable storage
Serverless
Amazon AI
Services
Apache
MxNet
TensorFlow
OpenCV
Custom
Video
Processing
Kinesis
Video
Streams
Amazon Kinesis Video Streaming
Capture, process, and store video streams for analytics/ machine learning
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
• Durable
• Continuous
• Fast
• Correct
• Reactive
• Reliable
Processing real-time, streaming data
What are the key requirements?
Ingest Transform Analyze React Persist
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Apache Kafka
Open Source
Robust Ecosystem
Requires Expertise In-house
Not Managed
Not Serverless
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Kinesis Data
Real-time
Serverless
Scalable
Secure
Cost-effective
EMR/Spark
Custom code
on EC2
Amazon S3
Amazon
Redshift
Splunk
Ingest,
store data
streams
Kinesis
Data
Streams
Kinesis
Data
Analytics
Aggregate,
filter,
enrich data
Kinesis
Data
Firehose
Egress
data
streams
AWS Lambda
Amazon
Elasticsearch
Kinesis Data Streaming Collect, process, and analyze data streams in real time
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
(Kafka OR Kinesis) OR (Kafka AND Kinesis)
• Apache Kafka or Amazon Kinesis
• Kafka To Kinesis Migration
• Kafka-Kinesis Connector Library
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Real-Time Streaming Common Solutions
CloudWatch logsS3 eventsAWS Metering Amazon.com’s catalog
© 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Streaming with Amazon Kinesis
Easily collect, process, and analyze video and data streams in real time
Capture, process, and
store video streams
Load data streams into
AWS data stores
Analyze data streams
with SQL
Capture, process, and
store data streams
Kinesis Video Streams Kinesis Data Streams Kinesis Data Firehose Kinesis Data Analytics
© 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Use Case 1: Clickstream Analytics
Example: Clickstream analytics
© 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Use Case 2: Real-time Analytics
Example: Analyze streaming social media data
© 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Use Case 3: IoT Stream Processing
Example: Sensors in tractor detect need for a spare part and automatically
place order
Finds tractors that need
maintenance and orders
replacement parts
© 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Kinesis Data Streams
Serverless
Elastically scalable
Low cost, pay for what you use
Managed encryption
AWS and open source integrations
Streaming ETL with Kinesis Streams and Firehose
Amazon S3
KDF with Lambda
Stream
AWS Lambda
Data Producers Stream Consumers
Custom analytics or transforms with the KCL
KDF
Stream
Data Producers Stream Consumers
Kinesis Client Library
Real time analytics Kinesis Data Analytics
KDF
Stream
Data Producers Stream Consumers
Kinesis Client Library
KDA
Amazon Kinesis Streams 3rd Party Connectors
Kinesis Data Firehose
Serverless
Zero administration
Pay for usage
Direct-to-data store integration
Serverless ETL using AWS Lambda
Serverless ETL with Kinesis Firehose
Firehose
Data Producers
Amazon S3
Amazon
Redshift
Amazon
Elasticsearch
Destinations
AWS Lambda
Inline Transform
Amazon
Athena
Kinesis Analytics
Serverless
Zero administration
Pay for usage
Sub 1-second processing latencies
Filter, aggregate, and enrich data using SQL
Programming skills not required
© 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Customer Examples
1 billion events per
week from
connected devices
Near-real-time
home valuation
(Zestimates)
Live clickstream
dashboards
refreshed under 10s
100 GB/day
clickstreams from
250+ sites
50 billion daily ad
impressions, sub-
50 ms responses
Online stylist
processing 10
million events/day
Migrated data bus
from Kafka to
Kinesis
Facilitate
communications
between 100+
microservices
Analyze billions of
network flows in
real-time
IoT predictive
analytics
”
“ • Redfin helps consumers buy and sell real
estate in 37 states
• Uses Amazon Kinesis to deliver house
comparison data in milliseconds
• Can scale to manage billions of records and
images
• Reaches nationwide audience with small
staff
Redfin: Real-time Home Recommendations
Redfin is a full-service residential real estate
company based in Seattle.
AWS freed us from the
operational aspect of managing
our infrastructure. That enabled
us to innovate faster so our
customers can buy and sell
houses easier and faster.
Yong Huang
Director, Big Data Analytics
”
“
© 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Redfin Architecture
Ingest Store Process Consume
Amazon S3
Data Lake
Amazon
EMR/Spark
Amazon
Redshift
Amazon
DynamoDB
Amazon
Kinesis
Business
Intelligence
Redfin
Website
Properties
Agents
Buyers
Data Insights
User profiles
Recommendations
Hot homes
Similar homes
Agent follow-up
Agent scorecard
Advertising metrics
© 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Thomson Reuters: Real-Time Dashboards
Thomson Reuters provides professionals with the
intelligence, technology, and human expertise they
need to find trusted answers.
“Using Amazon Kinesis, our
solution delivers new events to
user dashboards in less than
10 seconds.
Anders Fritz
Senior Manager, Product Innovation
”
• Ability to process up to 4,000 events per second,
anticipated to scale to 10,000 within one year
• Data pipeline accommodates twofold to threefold
traffic increases during breaking news
• No data loss or downtime since launch, thanks to
robust failover architecture
• Near-real-time availability of analytics data
• Simultaneous streaming and batching of data in one
solution
“
© 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Thomson Reuters Architecture
Kinesis Data
Streams
Kinesis Data
Firehose
NginxElastic Load
Balancing
Amazon S3
Amazon
Elasticsearch
Service
Kibana
dashboards
AWS
Lambda
Extract-
Transform-
Load (ETL)
Thomson Reuters
internal services
Real-time analytics layer
Serverless storage layer
© 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Otonomo Case Study
33
Fueling the Connected Car Revolution
Sefi Itzkovich, VP Architecture
March 14, 2018
34
Connected Autonomous
25GB
per hour
4,000GB
per hour
Automotive Data is Growing Exponentially
35
The Addressable Market is Massive
“The overall revenue pool from car data
monetization at global scale is estimated at
$450-750 billion by 2030”
McKinsey&Company, Advanced Industries September 2016
36
The First Connected
Car Data Marketplace
A dedicated platform for safe and
simple exchange of connected car
data to enable an ecosystem of
services and applications
Car
Owner
Smart
Mobility
Smart
City
Fleets
Emergency
Retail
insurance
Energy
Data
Maintenance
Marketplace
37
INSURANCE PARKING FINANCE ENERGY RETAIL MOBILITY
S T A N D A R T D A P I
CLOUD DATA LAKE
• Security
• Privacy
• Normalize
• Insights
• Billing
TSP TSP TSP OEM OEM OEM FLEET FLEET FLEET
DATAPROVIDERS
DATACONSUMERS
38
Example Application: Parking
Otonomo
Normalized, Aggregated
Multiple Platforms
No standard; Fragmented
39
Taking on the Tsunami of
Data Traffic
Kinesis Data Analytics
40
Platform Requirements
• Ingesting 200M+ events per day
• 500K+ events/sec at peak
• Various consumers types for different use cases
• Persisting large amount of data easily and efficiently
• Real-time stream processing
41
Kinesis Suite in Action with Otonomo
Kinesis Data Stream Kinesis Data AnalyticsKinesis Data Firehose
42
Amazon Kinesis Suite at Otonomo in action
Kinesis Data
Streams
KinesisKinesis Data Analytics
Kinesis Data Firehose
Processed
Data
Kinesis Data
Streams
ETL (ECS cluster)
KCL / KPL
Lambda
Elastic Workers Pool
Lake Builder
(EMR cluster)
Gatekeeper
Processed Data
Structured &
Formatted
Data
Data Lake
Insight Engine
Insight Handlers
EMR
Analytics & ML
Athena
43
Analytics
• Real Time Sub second Processing processing optimized for stream processing
• Kinesis Data Analytics
• Anomaly detection / Insights / K-Top analysis
• Easily run SQL / Windowing functions on stream
• Batch Analytics & ML on top S3 data lake using Athena / Spark / Containers
44
Key Takeaways
• S3 for data lake
• Decoupling of storage and compute
• Scale independently
• Serverless + Managed
• Focus on core business and technology
45
Thank You
Join the Ride!
careers@otonomo.io
© 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
AWS Tweets Demo
© 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
AWS Tweets Example
Record-level data
• What’s the overall sentiment today?
• What’s the sentiment trend now?
• What’s the most popular Language?
• What’s the Temp. affect on the tweet sentiment?
• Scale
• Highly availability
• Minimal Operational overhead
• Agile
• Cost Effective
Business Questions
Technical Requirements
© 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
ConsumeStore Process & AnalyzeIngest
Kinesis Data Streams
Kinesis Firehose
Delivery Streams
DynamoDB
AWS Lambda
Kinesis
Analytics
Raw Bucket
Parquet Bucket
Athena Redshift
Spectrum
QuickSight
SpeedLayerBatchLayer
Glue Data
Catalog
Spark/EMR Glue ETL
Real time
Web UI
© 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
ConsumeStore Process & AnalyzeIngest
Kinesis Data Streams
Kinesis Firehose
Delivery Streams
DynamoDB
AWS Lambda
Kinesis
Analytics
Raw Bucket
Parquet Bucket
Athena Redshift
Spectrum
QuickSight
SpeedLayerBatchLayer
Glue Data
Catalog
Spark/EMR Glue ETL
Real time
Web UI
© 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Thank You!

Contenu connexe

Tendances

Architecting for Real-Time Insights with Amazon Kinesis (ANT310) - AWS re:Inv...
Architecting for Real-Time Insights with Amazon Kinesis (ANT310) - AWS re:Inv...Architecting for Real-Time Insights with Amazon Kinesis (ANT310) - AWS re:Inv...
Architecting for Real-Time Insights with Amazon Kinesis (ANT310) - AWS re:Inv...Amazon Web Services
 
BDA305 Building Data Lakes and Analytics on AWS
BDA305 Building Data Lakes and Analytics on AWSBDA305 Building Data Lakes and Analytics on AWS
BDA305 Building Data Lakes and Analytics on AWSAmazon Web Services
 
Don’t Wait Until Tomorrow: From Batch to Streaming (ANT360) - AWS re:Invent 2018
Don’t Wait Until Tomorrow: From Batch to Streaming (ANT360) - AWS re:Invent 2018Don’t Wait Until Tomorrow: From Batch to Streaming (ANT360) - AWS re:Invent 2018
Don’t Wait Until Tomorrow: From Batch to Streaming (ANT360) - AWS re:Invent 2018Amazon Web Services
 
Machine Learning Inference at the Edge (IOT322-R1) - AWS re:Invent 2018
Machine Learning Inference at the Edge (IOT322-R1) - AWS re:Invent 2018Machine Learning Inference at the Edge (IOT322-R1) - AWS re:Invent 2018
Machine Learning Inference at the Edge (IOT322-R1) - AWS re:Invent 2018Amazon Web Services
 
BDA306 Building a Modern Data Warehouse: Deep Dive on Amazon Redshift
BDA306 Building a Modern Data Warehouse: Deep Dive on Amazon RedshiftBDA306 Building a Modern Data Warehouse: Deep Dive on Amazon Redshift
BDA306 Building a Modern Data Warehouse: Deep Dive on Amazon RedshiftAmazon Web Services
 
Real-Time Web Analytics with Amazon Kinesis Data Analytics (ADT401) - AWS re:...
Real-Time Web Analytics with Amazon Kinesis Data Analytics (ADT401) - AWS re:...Real-Time Web Analytics with Amazon Kinesis Data Analytics (ADT401) - AWS re:...
Real-Time Web Analytics with Amazon Kinesis Data Analytics (ADT401) - AWS re:...Amazon Web Services
 
Easy Rider: How ML, Serverless, and IoT Drive Mobility as a Service (AMT302) ...
Easy Rider: How ML, Serverless, and IoT Drive Mobility as a Service (AMT302) ...Easy Rider: How ML, Serverless, and IoT Drive Mobility as a Service (AMT302) ...
Easy Rider: How ML, Serverless, and IoT Drive Mobility as a Service (AMT302) ...Amazon Web Services
 
Data Privacy & Governance in the Age of Big Data: Deploy a De-Identified Data...
Data Privacy & Governance in the Age of Big Data: Deploy a De-Identified Data...Data Privacy & Governance in the Age of Big Data: Deploy a De-Identified Data...
Data Privacy & Governance in the Age of Big Data: Deploy a De-Identified Data...Amazon Web Services
 
BDA302 Building Intelligent Apps with AWS Machine Learning Language Services
BDA302 Building Intelligent Apps with AWS Machine Learning Language ServicesBDA302 Building Intelligent Apps with AWS Machine Learning Language Services
BDA302 Building Intelligent Apps with AWS Machine Learning Language ServicesAmazon Web Services
 
Get the Most out of Your Amazon Elasticsearch Service Domain (ANT334-R1) - AW...
Get the Most out of Your Amazon Elasticsearch Service Domain (ANT334-R1) - AW...Get the Most out of Your Amazon Elasticsearch Service Domain (ANT334-R1) - AW...
Get the Most out of Your Amazon Elasticsearch Service Domain (ANT334-R1) - AW...Amazon Web Services
 
Social Media Analytics with Amazon QuickSight (ANT370) - AWS re:Invent 2018
Social Media Analytics with Amazon QuickSight (ANT370) - AWS re:Invent 2018Social Media Analytics with Amazon QuickSight (ANT370) - AWS re:Invent 2018
Social Media Analytics with Amazon QuickSight (ANT370) - AWS re:Invent 2018Amazon Web Services
 
High Performance Data Streaming with Amazon Kinesis: Best Practices (ANT322-R...
High Performance Data Streaming with Amazon Kinesis: Best Practices (ANT322-R...High Performance Data Streaming with Amazon Kinesis: Best Practices (ANT322-R...
High Performance Data Streaming with Amazon Kinesis: Best Practices (ANT322-R...Amazon Web Services
 
Learn How You Can Accelerate Engineering Workloads with AppStream 2.0 (BAP318...
Learn How You Can Accelerate Engineering Workloads with AppStream 2.0 (BAP318...Learn How You Can Accelerate Engineering Workloads with AppStream 2.0 (BAP318...
Learn How You Can Accelerate Engineering Workloads with AppStream 2.0 (BAP318...Amazon Web Services
 
Leadership Session: AWS Semiconductor (MFG201-L) - AWS re:Invent 2018
Leadership Session: AWS Semiconductor (MFG201-L) - AWS re:Invent 2018Leadership Session: AWS Semiconductor (MFG201-L) - AWS re:Invent 2018
Leadership Session: AWS Semiconductor (MFG201-L) - AWS re:Invent 2018Amazon Web Services
 
SID301 Threat Detection and Mitigation
 SID301 Threat Detection and Mitigation SID301 Threat Detection and Mitigation
SID301 Threat Detection and MitigationAmazon Web Services
 
Best Practices to Secure Data Lake on AWS (ANT327) - AWS re:Invent 2018
Best Practices to Secure Data Lake on AWS (ANT327) - AWS re:Invent 2018Best Practices to Secure Data Lake on AWS (ANT327) - AWS re:Invent 2018
Best Practices to Secure Data Lake on AWS (ANT327) - AWS re:Invent 2018Amazon Web Services
 
Modern Data Architectures for Business Outcomes
Modern Data Architectures for Business OutcomesModern Data Architectures for Business Outcomes
Modern Data Architectures for Business OutcomesAmazon Web Services
 
SRV307 Applying AWS Purpose-Built Database Strategy: Match Your Workload to ...
 SRV307 Applying AWS Purpose-Built Database Strategy: Match Your Workload to ... SRV307 Applying AWS Purpose-Built Database Strategy: Match Your Workload to ...
SRV307 Applying AWS Purpose-Built Database Strategy: Match Your Workload to ...Amazon Web Services
 

Tendances (20)

Architecting for Real-Time Insights with Amazon Kinesis (ANT310) - AWS re:Inv...
Architecting for Real-Time Insights with Amazon Kinesis (ANT310) - AWS re:Inv...Architecting for Real-Time Insights with Amazon Kinesis (ANT310) - AWS re:Inv...
Architecting for Real-Time Insights with Amazon Kinesis (ANT310) - AWS re:Inv...
 
BDA305 Building Data Lakes and Analytics on AWS
BDA305 Building Data Lakes and Analytics on AWSBDA305 Building Data Lakes and Analytics on AWS
BDA305 Building Data Lakes and Analytics on AWS
 
Don’t Wait Until Tomorrow: From Batch to Streaming (ANT360) - AWS re:Invent 2018
Don’t Wait Until Tomorrow: From Batch to Streaming (ANT360) - AWS re:Invent 2018Don’t Wait Until Tomorrow: From Batch to Streaming (ANT360) - AWS re:Invent 2018
Don’t Wait Until Tomorrow: From Batch to Streaming (ANT360) - AWS re:Invent 2018
 
Machine Learning Inference at the Edge (IOT322-R1) - AWS re:Invent 2018
Machine Learning Inference at the Edge (IOT322-R1) - AWS re:Invent 2018Machine Learning Inference at the Edge (IOT322-R1) - AWS re:Invent 2018
Machine Learning Inference at the Edge (IOT322-R1) - AWS re:Invent 2018
 
BDA306 Building a Modern Data Warehouse: Deep Dive on Amazon Redshift
BDA306 Building a Modern Data Warehouse: Deep Dive on Amazon RedshiftBDA306 Building a Modern Data Warehouse: Deep Dive on Amazon Redshift
BDA306 Building a Modern Data Warehouse: Deep Dive on Amazon Redshift
 
Real-Time Web Analytics with Amazon Kinesis Data Analytics (ADT401) - AWS re:...
Real-Time Web Analytics with Amazon Kinesis Data Analytics (ADT401) - AWS re:...Real-Time Web Analytics with Amazon Kinesis Data Analytics (ADT401) - AWS re:...
Real-Time Web Analytics with Amazon Kinesis Data Analytics (ADT401) - AWS re:...
 
Easy Rider: How ML, Serverless, and IoT Drive Mobility as a Service (AMT302) ...
Easy Rider: How ML, Serverless, and IoT Drive Mobility as a Service (AMT302) ...Easy Rider: How ML, Serverless, and IoT Drive Mobility as a Service (AMT302) ...
Easy Rider: How ML, Serverless, and IoT Drive Mobility as a Service (AMT302) ...
 
Data Privacy & Governance in the Age of Big Data: Deploy a De-Identified Data...
Data Privacy & Governance in the Age of Big Data: Deploy a De-Identified Data...Data Privacy & Governance in the Age of Big Data: Deploy a De-Identified Data...
Data Privacy & Governance in the Age of Big Data: Deploy a De-Identified Data...
 
Airbnb - StreamAlert
Airbnb - StreamAlertAirbnb - StreamAlert
Airbnb - StreamAlert
 
BDA302 Building Intelligent Apps with AWS Machine Learning Language Services
BDA302 Building Intelligent Apps with AWS Machine Learning Language ServicesBDA302 Building Intelligent Apps with AWS Machine Learning Language Services
BDA302 Building Intelligent Apps with AWS Machine Learning Language Services
 
Get the Most out of Your Amazon Elasticsearch Service Domain (ANT334-R1) - AW...
Get the Most out of Your Amazon Elasticsearch Service Domain (ANT334-R1) - AW...Get the Most out of Your Amazon Elasticsearch Service Domain (ANT334-R1) - AW...
Get the Most out of Your Amazon Elasticsearch Service Domain (ANT334-R1) - AW...
 
SRV319 Amazon EC2 Foundations
SRV319 Amazon EC2 FoundationsSRV319 Amazon EC2 Foundations
SRV319 Amazon EC2 Foundations
 
Social Media Analytics with Amazon QuickSight (ANT370) - AWS re:Invent 2018
Social Media Analytics with Amazon QuickSight (ANT370) - AWS re:Invent 2018Social Media Analytics with Amazon QuickSight (ANT370) - AWS re:Invent 2018
Social Media Analytics with Amazon QuickSight (ANT370) - AWS re:Invent 2018
 
High Performance Data Streaming with Amazon Kinesis: Best Practices (ANT322-R...
High Performance Data Streaming with Amazon Kinesis: Best Practices (ANT322-R...High Performance Data Streaming with Amazon Kinesis: Best Practices (ANT322-R...
High Performance Data Streaming with Amazon Kinesis: Best Practices (ANT322-R...
 
Learn How You Can Accelerate Engineering Workloads with AppStream 2.0 (BAP318...
Learn How You Can Accelerate Engineering Workloads with AppStream 2.0 (BAP318...Learn How You Can Accelerate Engineering Workloads with AppStream 2.0 (BAP318...
Learn How You Can Accelerate Engineering Workloads with AppStream 2.0 (BAP318...
 
Leadership Session: AWS Semiconductor (MFG201-L) - AWS re:Invent 2018
Leadership Session: AWS Semiconductor (MFG201-L) - AWS re:Invent 2018Leadership Session: AWS Semiconductor (MFG201-L) - AWS re:Invent 2018
Leadership Session: AWS Semiconductor (MFG201-L) - AWS re:Invent 2018
 
SID301 Threat Detection and Mitigation
 SID301 Threat Detection and Mitigation SID301 Threat Detection and Mitigation
SID301 Threat Detection and Mitigation
 
Best Practices to Secure Data Lake on AWS (ANT327) - AWS re:Invent 2018
Best Practices to Secure Data Lake on AWS (ANT327) - AWS re:Invent 2018Best Practices to Secure Data Lake on AWS (ANT327) - AWS re:Invent 2018
Best Practices to Secure Data Lake on AWS (ANT327) - AWS re:Invent 2018
 
Modern Data Architectures for Business Outcomes
Modern Data Architectures for Business OutcomesModern Data Architectures for Business Outcomes
Modern Data Architectures for Business Outcomes
 
SRV307 Applying AWS Purpose-Built Database Strategy: Match Your Workload to ...
 SRV307 Applying AWS Purpose-Built Database Strategy: Match Your Workload to ... SRV307 Applying AWS Purpose-Built Database Strategy: Match Your Workload to ...
SRV307 Applying AWS Purpose-Built Database Strategy: Match Your Workload to ...
 

Similaire à Amazon Kinesis - Building Serverless real-time solution - Tel Aviv Summit 2018

How TrueCar Gains Actionable Insights with Splunk Cloud PPT
How TrueCar Gains Actionable Insights with Splunk Cloud PPTHow TrueCar Gains Actionable Insights with Splunk Cloud PPT
How TrueCar Gains Actionable Insights with Splunk Cloud PPTAmazon Web Services
 
Building a Data Lake in Amazon S3 & Amazon Glacier (STG401-R1) - AWS re:Inven...
Building a Data Lake in Amazon S3 & Amazon Glacier (STG401-R1) - AWS re:Inven...Building a Data Lake in Amazon S3 & Amazon Glacier (STG401-R1) - AWS re:Inven...
Building a Data Lake in Amazon S3 & Amazon Glacier (STG401-R1) - AWS re:Inven...Amazon Web Services
 
雲上打造資料湖 (Data Lake):智能化駕馭商機 (Level 300)
雲上打造資料湖 (Data Lake):智能化駕馭商機 (Level 300)雲上打造資料湖 (Data Lake):智能化駕馭商機 (Level 300)
雲上打造資料湖 (Data Lake):智能化駕馭商機 (Level 300)Amazon Web Services
 
Building a Real-Time Data Platform on AWS
Building a Real-Time Data Platform on AWSBuilding a Real-Time Data Platform on AWS
Building a Real-Time Data Platform on AWSInjae Kwak
 
From Batch to Streaming - How Amazon Flex Uses Real-time Analytics
From Batch to Streaming - How Amazon Flex Uses Real-time AnalyticsFrom Batch to Streaming - How Amazon Flex Uses Real-time Analytics
From Batch to Streaming - How Amazon Flex Uses Real-time AnalyticsAmazon Web Services
 
Get Started with Real-Time Streaming Data in Under 5 Minutes - AWS Online Tec...
Get Started with Real-Time Streaming Data in Under 5 Minutes - AWS Online Tec...Get Started with Real-Time Streaming Data in Under 5 Minutes - AWS Online Tec...
Get Started with Real-Time Streaming Data in Under 5 Minutes - AWS Online Tec...Amazon Web Services
 
Big Data Meets AI - Driving Insights and Adding Intelligence to Your Solutions
 Big Data Meets AI - Driving Insights and Adding Intelligence to Your Solutions Big Data Meets AI - Driving Insights and Adding Intelligence to Your Solutions
Big Data Meets AI - Driving Insights and Adding Intelligence to Your SolutionsAmazon Web Services
 
Euronext_AWS_talend_connect_paris_2018.pdf
Euronext_AWS_talend_connect_paris_2018.pdfEuronext_AWS_talend_connect_paris_2018.pdf
Euronext_AWS_talend_connect_paris_2018.pdfAmazon Web Services
 
Get to Know Your Customers - Build and Innovate with a Modern Data Architecture
Get to Know Your Customers - Build and Innovate with a Modern Data ArchitectureGet to Know Your Customers - Build and Innovate with a Modern Data Architecture
Get to Know Your Customers - Build and Innovate with a Modern Data ArchitectureAmazon Web Services
 
Aws Tools for Alexa Skills
Aws Tools for Alexa SkillsAws Tools for Alexa Skills
Aws Tools for Alexa SkillsBoaz Ziniman
 
Big Data@Scale_AWSPSSummit_Singapore
Big Data@Scale_AWSPSSummit_SingaporeBig Data@Scale_AWSPSSummit_Singapore
Big Data@Scale_AWSPSSummit_SingaporeAmazon Web Services
 
Modern Cloud Data Warehousing ft. Equinox Fitness Clubs: Optimize Analytics P...
Modern Cloud Data Warehousing ft. Equinox Fitness Clubs: Optimize Analytics P...Modern Cloud Data Warehousing ft. Equinox Fitness Clubs: Optimize Analytics P...
Modern Cloud Data Warehousing ft. Equinox Fitness Clubs: Optimize Analytics P...Amazon Web Services
 
Big Data on AWS - To infinity and beyond! - Tel Aviv Summit 2018
Big Data on AWS - To infinity and beyond! - Tel Aviv Summit 2018Big Data on AWS - To infinity and beyond! - Tel Aviv Summit 2018
Big Data on AWS - To infinity and beyond! - Tel Aviv Summit 2018Amazon Web Services
 
Better Business from Exploring Ideas - Modern Data Architectures on AWS
Better Business from Exploring Ideas - Modern Data Architectures on AWSBetter Business from Exploring Ideas - Modern Data Architectures on AWS
Better Business from Exploring Ideas - Modern Data Architectures on AWSAmazon Web Services
 
Need for Speed – Intro To Real-Time Data Streaming Analytics on AWS | AWS Sum...
Need for Speed – Intro To Real-Time Data Streaming Analytics on AWS | AWS Sum...Need for Speed – Intro To Real-Time Data Streaming Analytics on AWS | AWS Sum...
Need for Speed – Intro To Real-Time Data Streaming Analytics on AWS | AWS Sum...AWS Summits
 
Need for Speed – Intro To Real-Time Data Streaming Analytics on AWS | AWS Sum...
Need for Speed – Intro To Real-Time Data Streaming Analytics on AWS | AWS Sum...Need for Speed – Intro To Real-Time Data Streaming Analytics on AWS | AWS Sum...
Need for Speed – Intro To Real-Time Data Streaming Analytics on AWS | AWS Sum...Amazon Web Services
 
Emerging Trends in Big Data, Analytics, Machine Learning, and Internet-of-Thi...
Emerging Trends in Big Data, Analytics, Machine Learning, and Internet-of-Thi...Emerging Trends in Big Data, Analytics, Machine Learning, and Internet-of-Thi...
Emerging Trends in Big Data, Analytics, Machine Learning, and Internet-of-Thi...Michaela Bromfield
 
Building Data Lake on AWS | AWS Floor28
Building Data Lake on AWS | AWS Floor28Building Data Lake on AWS | AWS Floor28
Building Data Lake on AWS | AWS Floor28Amazon Web Services
 
AWS Floor 28 - Building Data lake on AWS
AWS Floor 28 - Building Data lake on AWSAWS Floor 28 - Building Data lake on AWS
AWS Floor 28 - Building Data lake on AWSAdir Sharabi
 

Similaire à Amazon Kinesis - Building Serverless real-time solution - Tel Aviv Summit 2018 (20)

How TrueCar Gains Actionable Insights with Splunk Cloud PPT
How TrueCar Gains Actionable Insights with Splunk Cloud PPTHow TrueCar Gains Actionable Insights with Splunk Cloud PPT
How TrueCar Gains Actionable Insights with Splunk Cloud PPT
 
Analysing Data in Real-time
Analysing Data in Real-timeAnalysing Data in Real-time
Analysing Data in Real-time
 
Building a Data Lake in Amazon S3 & Amazon Glacier (STG401-R1) - AWS re:Inven...
Building a Data Lake in Amazon S3 & Amazon Glacier (STG401-R1) - AWS re:Inven...Building a Data Lake in Amazon S3 & Amazon Glacier (STG401-R1) - AWS re:Inven...
Building a Data Lake in Amazon S3 & Amazon Glacier (STG401-R1) - AWS re:Inven...
 
雲上打造資料湖 (Data Lake):智能化駕馭商機 (Level 300)
雲上打造資料湖 (Data Lake):智能化駕馭商機 (Level 300)雲上打造資料湖 (Data Lake):智能化駕馭商機 (Level 300)
雲上打造資料湖 (Data Lake):智能化駕馭商機 (Level 300)
 
Building a Real-Time Data Platform on AWS
Building a Real-Time Data Platform on AWSBuilding a Real-Time Data Platform on AWS
Building a Real-Time Data Platform on AWS
 
From Batch to Streaming - How Amazon Flex Uses Real-time Analytics
From Batch to Streaming - How Amazon Flex Uses Real-time AnalyticsFrom Batch to Streaming - How Amazon Flex Uses Real-time Analytics
From Batch to Streaming - How Amazon Flex Uses Real-time Analytics
 
Get Started with Real-Time Streaming Data in Under 5 Minutes - AWS Online Tec...
Get Started with Real-Time Streaming Data in Under 5 Minutes - AWS Online Tec...Get Started with Real-Time Streaming Data in Under 5 Minutes - AWS Online Tec...
Get Started with Real-Time Streaming Data in Under 5 Minutes - AWS Online Tec...
 
Big Data Meets AI - Driving Insights and Adding Intelligence to Your Solutions
 Big Data Meets AI - Driving Insights and Adding Intelligence to Your Solutions Big Data Meets AI - Driving Insights and Adding Intelligence to Your Solutions
Big Data Meets AI - Driving Insights and Adding Intelligence to Your Solutions
 
Euronext_AWS_talend_connect_paris_2018.pdf
Euronext_AWS_talend_connect_paris_2018.pdfEuronext_AWS_talend_connect_paris_2018.pdf
Euronext_AWS_talend_connect_paris_2018.pdf
 
Get to Know Your Customers - Build and Innovate with a Modern Data Architecture
Get to Know Your Customers - Build and Innovate with a Modern Data ArchitectureGet to Know Your Customers - Build and Innovate with a Modern Data Architecture
Get to Know Your Customers - Build and Innovate with a Modern Data Architecture
 
Aws Tools for Alexa Skills
Aws Tools for Alexa SkillsAws Tools for Alexa Skills
Aws Tools for Alexa Skills
 
Big Data@Scale_AWSPSSummit_Singapore
Big Data@Scale_AWSPSSummit_SingaporeBig Data@Scale_AWSPSSummit_Singapore
Big Data@Scale_AWSPSSummit_Singapore
 
Modern Cloud Data Warehousing ft. Equinox Fitness Clubs: Optimize Analytics P...
Modern Cloud Data Warehousing ft. Equinox Fitness Clubs: Optimize Analytics P...Modern Cloud Data Warehousing ft. Equinox Fitness Clubs: Optimize Analytics P...
Modern Cloud Data Warehousing ft. Equinox Fitness Clubs: Optimize Analytics P...
 
Big Data on AWS - To infinity and beyond! - Tel Aviv Summit 2018
Big Data on AWS - To infinity and beyond! - Tel Aviv Summit 2018Big Data on AWS - To infinity and beyond! - Tel Aviv Summit 2018
Big Data on AWS - To infinity and beyond! - Tel Aviv Summit 2018
 
Better Business from Exploring Ideas - Modern Data Architectures on AWS
Better Business from Exploring Ideas - Modern Data Architectures on AWSBetter Business from Exploring Ideas - Modern Data Architectures on AWS
Better Business from Exploring Ideas - Modern Data Architectures on AWS
 
Need for Speed – Intro To Real-Time Data Streaming Analytics on AWS | AWS Sum...
Need for Speed – Intro To Real-Time Data Streaming Analytics on AWS | AWS Sum...Need for Speed – Intro To Real-Time Data Streaming Analytics on AWS | AWS Sum...
Need for Speed – Intro To Real-Time Data Streaming Analytics on AWS | AWS Sum...
 
Need for Speed – Intro To Real-Time Data Streaming Analytics on AWS | AWS Sum...
Need for Speed – Intro To Real-Time Data Streaming Analytics on AWS | AWS Sum...Need for Speed – Intro To Real-Time Data Streaming Analytics on AWS | AWS Sum...
Need for Speed – Intro To Real-Time Data Streaming Analytics on AWS | AWS Sum...
 
Emerging Trends in Big Data, Analytics, Machine Learning, and Internet-of-Thi...
Emerging Trends in Big Data, Analytics, Machine Learning, and Internet-of-Thi...Emerging Trends in Big Data, Analytics, Machine Learning, and Internet-of-Thi...
Emerging Trends in Big Data, Analytics, Machine Learning, and Internet-of-Thi...
 
Building Data Lake on AWS | AWS Floor28
Building Data Lake on AWS | AWS Floor28Building Data Lake on AWS | AWS Floor28
Building Data Lake on AWS | AWS Floor28
 
AWS Floor 28 - Building Data lake on AWS
AWS Floor 28 - Building Data lake on AWSAWS Floor 28 - Building Data lake on AWS
AWS Floor 28 - Building Data lake on AWS
 

Plus de Amazon Web Services

Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...Amazon Web Services
 
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...Amazon Web Services
 
Esegui pod serverless con Amazon EKS e AWS Fargate
Esegui pod serverless con Amazon EKS e AWS FargateEsegui pod serverless con Amazon EKS e AWS Fargate
Esegui pod serverless con Amazon EKS e AWS FargateAmazon Web Services
 
Costruire Applicazioni Moderne con AWS
Costruire Applicazioni Moderne con AWSCostruire Applicazioni Moderne con AWS
Costruire Applicazioni Moderne con AWSAmazon Web Services
 
Come spendere fino al 90% in meno con i container e le istanze spot
Come spendere fino al 90% in meno con i container e le istanze spot Come spendere fino al 90% in meno con i container e le istanze spot
Come spendere fino al 90% in meno con i container e le istanze spot Amazon Web Services
 
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...Amazon Web Services
 
OpsWorks Configuration Management: automatizza la gestione e i deployment del...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...OpsWorks Configuration Management: automatizza la gestione e i deployment del...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...Amazon Web Services
 
Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows WorkloadsMicrosoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows WorkloadsAmazon Web Services
 
Database Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatareDatabase Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatareAmazon Web Services
 
Crea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJSCrea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJSAmazon Web Services
 
API moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e webAPI moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e webAmazon Web Services
 
Database Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatareDatabase Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatareAmazon Web Services
 
Tools for building your MVP on AWS
Tools for building your MVP on AWSTools for building your MVP on AWS
Tools for building your MVP on AWSAmazon Web Services
 
How to Build a Winning Pitch Deck
How to Build a Winning Pitch DeckHow to Build a Winning Pitch Deck
How to Build a Winning Pitch DeckAmazon Web Services
 
Building a web application without servers
Building a web application without serversBuilding a web application without servers
Building a web application without serversAmazon Web Services
 
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...Amazon Web Services
 
Introduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container ServiceIntroduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container ServiceAmazon Web Services
 

Plus de Amazon Web Services (20)

Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
 
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
 
Esegui pod serverless con Amazon EKS e AWS Fargate
Esegui pod serverless con Amazon EKS e AWS FargateEsegui pod serverless con Amazon EKS e AWS Fargate
Esegui pod serverless con Amazon EKS e AWS Fargate
 
Costruire Applicazioni Moderne con AWS
Costruire Applicazioni Moderne con AWSCostruire Applicazioni Moderne con AWS
Costruire Applicazioni Moderne con AWS
 
Come spendere fino al 90% in meno con i container e le istanze spot
Come spendere fino al 90% in meno con i container e le istanze spot Come spendere fino al 90% in meno con i container e le istanze spot
Come spendere fino al 90% in meno con i container e le istanze spot
 
Open banking as a service
Open banking as a serviceOpen banking as a service
Open banking as a service
 
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
 
OpsWorks Configuration Management: automatizza la gestione e i deployment del...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...OpsWorks Configuration Management: automatizza la gestione e i deployment del...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...
 
Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows WorkloadsMicrosoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads
 
Computer Vision con AWS
Computer Vision con AWSComputer Vision con AWS
Computer Vision con AWS
 
Database Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatareDatabase Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatare
 
Crea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJSCrea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJS
 
API moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e webAPI moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e web
 
Database Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatareDatabase Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatare
 
Tools for building your MVP on AWS
Tools for building your MVP on AWSTools for building your MVP on AWS
Tools for building your MVP on AWS
 
How to Build a Winning Pitch Deck
How to Build a Winning Pitch DeckHow to Build a Winning Pitch Deck
How to Build a Winning Pitch Deck
 
Building a web application without servers
Building a web application without serversBuilding a web application without servers
Building a web application without servers
 
Fundraising Essentials
Fundraising EssentialsFundraising Essentials
Fundraising Essentials
 
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
 
Introduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container ServiceIntroduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container Service
 

Amazon Kinesis - Building Serverless real-time solution - Tel Aviv Summit 2018

  • 1. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved. Roy Ben-Alta Principal Business Development Manager and Architect, Amazon Web Services Sefi Itzkovich VP Architecture, Otonomo Amazon Kinesis – Building Serverless real-time Solution
  • 2. Overview • Speed and Serverless Matters • Amazon Kinesis Overview and Use Cases • Otonomo Case Study • Demo • Q&A
  • 3. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved. Timely Decisions Require New Data in Minutes Source: Perishable insights, Mike Gualtieri, Forrester Data loses value quickly over time
  • 4. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved. Stream New Data in Seconds Get actionable insights quickly Streaming Ingest video & data as it’s generated Process or deliver data on the fly Real-time analytics/ML, alerts, actions
  • 5. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved. Most Common uses of Real-Time Streaming Video Analytics Automation Data Lakes IoT Monitoring and Analytics Log Analytics
  • 6. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved. Speed Matters
  • 7. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved. Speed Matters
  • 8. Dilip Kumar, VP of Technology for Amazon Go, cites as an example Amazon Go’s unique system of streaming data from hundreds of cameras to track the shopping activities of customers. The innovations his team concocted helped influence an AWS service called Kinesis, which allows customers to stream video from multiple devices to the Amazon cloud, where they can process it, analyze it, and use it to further advance their machine learning efforts.
  • 9. Stream video from millions of devices Easily build vision-enabled apps Secure Durable, searchable storage Serverless Amazon AI Services Apache MxNet TensorFlow OpenCV Custom Video Processing Kinesis Video Streams Amazon Kinesis Video Streaming Capture, process, and store video streams for analytics/ machine learning
  • 10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. • Durable • Continuous • Fast • Correct • Reactive • Reliable Processing real-time, streaming data What are the key requirements? Ingest Transform Analyze React Persist
  • 11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Apache Kafka Open Source Robust Ecosystem Requires Expertise In-house Not Managed Not Serverless
  • 12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Kinesis Data Real-time Serverless Scalable Secure Cost-effective EMR/Spark Custom code on EC2 Amazon S3 Amazon Redshift Splunk Ingest, store data streams Kinesis Data Streams Kinesis Data Analytics Aggregate, filter, enrich data Kinesis Data Firehose Egress data streams AWS Lambda Amazon Elasticsearch Kinesis Data Streaming Collect, process, and analyze data streams in real time
  • 13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. (Kafka OR Kinesis) OR (Kafka AND Kinesis) • Apache Kafka or Amazon Kinesis • Kafka To Kinesis Migration • Kafka-Kinesis Connector Library
  • 14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Real-Time Streaming Common Solutions CloudWatch logsS3 eventsAWS Metering Amazon.com’s catalog
  • 15. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved. Streaming with Amazon Kinesis Easily collect, process, and analyze video and data streams in real time Capture, process, and store video streams Load data streams into AWS data stores Analyze data streams with SQL Capture, process, and store data streams Kinesis Video Streams Kinesis Data Streams Kinesis Data Firehose Kinesis Data Analytics
  • 16. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved. Use Case 1: Clickstream Analytics Example: Clickstream analytics
  • 17. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved. Use Case 2: Real-time Analytics Example: Analyze streaming social media data
  • 18. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved. Use Case 3: IoT Stream Processing Example: Sensors in tractor detect need for a spare part and automatically place order Finds tractors that need maintenance and orders replacement parts
  • 19. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved. Kinesis Data Streams Serverless Elastically scalable Low cost, pay for what you use Managed encryption AWS and open source integrations
  • 20. Streaming ETL with Kinesis Streams and Firehose Amazon S3 KDF with Lambda Stream AWS Lambda Data Producers Stream Consumers
  • 21. Custom analytics or transforms with the KCL KDF Stream Data Producers Stream Consumers Kinesis Client Library
  • 22. Real time analytics Kinesis Data Analytics KDF Stream Data Producers Stream Consumers Kinesis Client Library KDA
  • 23. Amazon Kinesis Streams 3rd Party Connectors
  • 24. Kinesis Data Firehose Serverless Zero administration Pay for usage Direct-to-data store integration Serverless ETL using AWS Lambda
  • 25. Serverless ETL with Kinesis Firehose Firehose Data Producers Amazon S3 Amazon Redshift Amazon Elasticsearch Destinations AWS Lambda Inline Transform Amazon Athena
  • 26. Kinesis Analytics Serverless Zero administration Pay for usage Sub 1-second processing latencies Filter, aggregate, and enrich data using SQL Programming skills not required
  • 27. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved. Customer Examples 1 billion events per week from connected devices Near-real-time home valuation (Zestimates) Live clickstream dashboards refreshed under 10s 100 GB/day clickstreams from 250+ sites 50 billion daily ad impressions, sub- 50 ms responses Online stylist processing 10 million events/day Migrated data bus from Kafka to Kinesis Facilitate communications between 100+ microservices Analyze billions of network flows in real-time IoT predictive analytics
  • 28. ” “ • Redfin helps consumers buy and sell real estate in 37 states • Uses Amazon Kinesis to deliver house comparison data in milliseconds • Can scale to manage billions of records and images • Reaches nationwide audience with small staff Redfin: Real-time Home Recommendations Redfin is a full-service residential real estate company based in Seattle. AWS freed us from the operational aspect of managing our infrastructure. That enabled us to innovate faster so our customers can buy and sell houses easier and faster. Yong Huang Director, Big Data Analytics ” “
  • 29. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved. Redfin Architecture Ingest Store Process Consume Amazon S3 Data Lake Amazon EMR/Spark Amazon Redshift Amazon DynamoDB Amazon Kinesis Business Intelligence Redfin Website Properties Agents Buyers Data Insights User profiles Recommendations Hot homes Similar homes Agent follow-up Agent scorecard Advertising metrics
  • 30. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved. Thomson Reuters: Real-Time Dashboards Thomson Reuters provides professionals with the intelligence, technology, and human expertise they need to find trusted answers. “Using Amazon Kinesis, our solution delivers new events to user dashboards in less than 10 seconds. Anders Fritz Senior Manager, Product Innovation ” • Ability to process up to 4,000 events per second, anticipated to scale to 10,000 within one year • Data pipeline accommodates twofold to threefold traffic increases during breaking news • No data loss or downtime since launch, thanks to robust failover architecture • Near-real-time availability of analytics data • Simultaneous streaming and batching of data in one solution “
  • 31. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved. Thomson Reuters Architecture Kinesis Data Streams Kinesis Data Firehose NginxElastic Load Balancing Amazon S3 Amazon Elasticsearch Service Kibana dashboards AWS Lambda Extract- Transform- Load (ETL) Thomson Reuters internal services Real-time analytics layer Serverless storage layer
  • 32. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved. Otonomo Case Study
  • 33. 33 Fueling the Connected Car Revolution Sefi Itzkovich, VP Architecture March 14, 2018
  • 34. 34 Connected Autonomous 25GB per hour 4,000GB per hour Automotive Data is Growing Exponentially
  • 35. 35 The Addressable Market is Massive “The overall revenue pool from car data monetization at global scale is estimated at $450-750 billion by 2030” McKinsey&Company, Advanced Industries September 2016
  • 36. 36 The First Connected Car Data Marketplace A dedicated platform for safe and simple exchange of connected car data to enable an ecosystem of services and applications Car Owner Smart Mobility Smart City Fleets Emergency Retail insurance Energy Data Maintenance Marketplace
  • 37. 37 INSURANCE PARKING FINANCE ENERGY RETAIL MOBILITY S T A N D A R T D A P I CLOUD DATA LAKE • Security • Privacy • Normalize • Insights • Billing TSP TSP TSP OEM OEM OEM FLEET FLEET FLEET DATAPROVIDERS DATACONSUMERS
  • 38. 38 Example Application: Parking Otonomo Normalized, Aggregated Multiple Platforms No standard; Fragmented
  • 39. 39 Taking on the Tsunami of Data Traffic Kinesis Data Analytics
  • 40. 40 Platform Requirements • Ingesting 200M+ events per day • 500K+ events/sec at peak • Various consumers types for different use cases • Persisting large amount of data easily and efficiently • Real-time stream processing
  • 41. 41 Kinesis Suite in Action with Otonomo Kinesis Data Stream Kinesis Data AnalyticsKinesis Data Firehose
  • 42. 42 Amazon Kinesis Suite at Otonomo in action Kinesis Data Streams KinesisKinesis Data Analytics Kinesis Data Firehose Processed Data Kinesis Data Streams ETL (ECS cluster) KCL / KPL Lambda Elastic Workers Pool Lake Builder (EMR cluster) Gatekeeper Processed Data Structured & Formatted Data Data Lake Insight Engine Insight Handlers EMR Analytics & ML Athena
  • 43. 43 Analytics • Real Time Sub second Processing processing optimized for stream processing • Kinesis Data Analytics • Anomaly detection / Insights / K-Top analysis • Easily run SQL / Windowing functions on stream • Batch Analytics & ML on top S3 data lake using Athena / Spark / Containers
  • 44. 44 Key Takeaways • S3 for data lake • Decoupling of storage and compute • Scale independently • Serverless + Managed • Focus on core business and technology
  • 45. 45 Thank You Join the Ride! careers@otonomo.io
  • 46. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved. AWS Tweets Demo
  • 47. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved. AWS Tweets Example Record-level data • What’s the overall sentiment today? • What’s the sentiment trend now? • What’s the most popular Language? • What’s the Temp. affect on the tweet sentiment? • Scale • Highly availability • Minimal Operational overhead • Agile • Cost Effective Business Questions Technical Requirements
  • 48. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved. ConsumeStore Process & AnalyzeIngest Kinesis Data Streams Kinesis Firehose Delivery Streams DynamoDB AWS Lambda Kinesis Analytics Raw Bucket Parquet Bucket Athena Redshift Spectrum QuickSight SpeedLayerBatchLayer Glue Data Catalog Spark/EMR Glue ETL Real time Web UI
  • 49. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved. ConsumeStore Process & AnalyzeIngest Kinesis Data Streams Kinesis Firehose Delivery Streams DynamoDB AWS Lambda Kinesis Analytics Raw Bucket Parquet Bucket Athena Redshift Spectrum QuickSight SpeedLayerBatchLayer Glue Data Catalog Spark/EMR Glue ETL Real time Web UI
  • 50. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved. Thank You!