SlideShare une entreprise Scribd logo
1  sur  30
Télécharger pour lire hors ligne
© 2016 Mesosphere, Inc. All Rights Reserved.
SCALABLE TIME
SERIES AND
STREAM
PROCESSING
FOR IOT
APPLICATIONS
1
Michael Hausenblas, Developer & Cloud Advocate | 2016-01-16
© 2015 Mesosphere, Inc. All Rights Reserved.
MOTIVATION
2
© 2015 Mesosphere, Inc. All Rights Reserved.
AIRLINES
3
© 2015 Mesosphere, Inc. All Rights Reserved.
LOGISTICS
4
© 2015 Mesosphere, Inc. All Rights Reserved.
HEALTH

CARE
5
© 2015 Mesosphere, Inc. All Rights Reserved.
TRADERS
6
© 2015 Mesosphere, Inc. All Rights Reserved.
FARMERS
7
© 2015 Mesosphere, Inc. All Rights Reserved.
CITIES
8
©2014,Wiredmagazine
© 2015 Mesosphere, Inc. All Rights Reserved.
YOU
9
© 2015 Mesosphere, Inc. All Rights Reserved.
THE

TOOLBOX
10
© 2015 Mesosphere, Inc. All Rights Reserved.
LET'S TALK ABOUT WORKLOADS* …
11*) kudos to Timothy St. Clair, @timothysc
batch streaming PaaS
MapReduce
© 2015 Mesosphere, Inc. All Rights Reserved.
• Apache Kafka
• ØMQ, RabbitMQ, Disque (Redis-based), etc.
• fluentd, Logstash, Flume
• Akka streams
• cloud-only: AWS SQS, Google Cloud Pub/Sub
• see also queues.io
MESSAGE QUEUES & ROUTERS
12
© 2015 Mesosphere, Inc. All Rights Reserved.
APACHE KAFKA
13
• High-throughput, distributed, persistent
publish-subscribe messaging system
• Originates from LinkedIn
• Typically used as buffer/de-coupling
layer in online stream processing
Message queues & routers
kafka.apache.org
© 2015 Mesosphere, Inc. All Rights Reserved.
FLUENTD
14
Message queues & routers
www.fluentd.org
© 2015 Mesosphere, Inc. All Rights Reserved.
STREAM PROCESSING PLATFORMS
15
• Apache Storm
• Apache Spark
• Apache Samza
• Apache Flink
• Concord
• cloud-only: AWS Kinesis, Google Cloud Dataflow
• see also my webinar on stream processing
© 2015 Mesosphere, Inc. All Rights Reserved.
APACHE STORM
16
• Distributed, fault-tolerant stream-
processing platform
• Guaranteed message processing
(replaying messages on failure)
• Concepts: tuples, streams, spouts, bolts,
topologies
Stream processing platforms
storm.apache.org
© 2015 Mesosphere, Inc. All Rights Reserved.
APACHE SPARK
17
Stream processing platforms
spark.apache.org
Spark SQL Spark Streaming
MLlib

(machine learning)
Spark core (RDD)
GraphX

(graph processing)
Mesos
Filesystem (local, HDFS, S3) or data store (HBase, Cassandra, Elasticsearch, etc.)
YARNStandalone
© 2015 Mesosphere, Inc. All Rights Reserved.
TIME SERIES DATASTORES
18
• InfluxDB
• OpenTSDB
• KairosDB
• Prometheus
• see also iot-a.info
© 2015 Mesosphere, Inc. All Rights Reserved.
OPENTSDB
19
• Distributed time series database on top HBase
• Store, index, query & plot metrics
• Extremely scalable
• Low-level monitoring
Time series datastores
opentsdb.net
© 2015 Mesosphere, Inc. All Rights Reserved.
INFLUXDB
20
• No-dependency, time series database written in Go
• SQLish query language (incl. regex, fan out)
• Single node or Raft-based distributed node mode
Time series datastores
influxdb.com
© 2015 Mesosphere, Inc. All Rights Reserved.
CHALLENGES
21
• Set up and operation of components
• Elasticity: static vs. dynamic partitioning
• Efficient usage of resources (TCO)
© 2015 Mesosphere, Inc. All Rights Reserved.
MEET THE
DATACENTER

OPERATING

SYSTEM

(DCOS)
22
© 2015 Mesosphere, Inc. All Rights Reserved.
LOCAL OS VS. DISTRIBUTED OS
23
http://bitly.com/os-vs-dcos
© 2015 Mesosphere, Inc. All Rights Reserved.
DCOS IS A DISTRIBUTED OPERATING SYSTEM
24
• local OS per node (+container enabled)
• scheduling (long-lived, batch)
• networking
• service discovery
• stateful services
• security
• monitoring, logging, debugging
© 2015 Mesosphere, Inc. All Rights Reserved. 25
© 2015 Mesosphere, Inc. All Rights Reserved.
BENEFITS
26
DCOS
• Run stateless services such as Web server or app
server and Big Data services like Kafka, Spark, or
Cassandra together on one cluster
• Dynamic partitioning of your cluster, depending on
your business requirements
• Increased utilization (10% → 80%++)
© 2015 Mesosphere, Inc. All Rights Reserved.
AN
EXAMPLE
27
© 2015 Mesosphere, Inc. All Rights Reserved. 28
https://mesosphere.com/blog/2015/11/18/dcos-time-series-demo
© 2015 Mesosphere, Inc. All Rights Reserved. 29
https://github.com/mesosphere/time-series-demo
© 2015 Mesosphere, Inc. All Rights Reserved.
Q & A
30
• @mhausenblas
• mhausenblas.info
• @mesosphere
• mesosphere.io/product
• mesosphere.com/infinity

Contenu connexe

En vedette

時系列データ3
時系列データ3時系列データ3
時系列データ3
graySpace999
 

En vedette (20)

Comzeit IoT Management Technology 紹介
Comzeit IoT Management Technology 紹介Comzeit IoT Management Technology 紹介
Comzeit IoT Management Technology 紹介
 
時系列データ3
時系列データ3時系列データ3
時系列データ3
 
Open nlp presentationss
Open nlp presentationssOpen nlp presentationss
Open nlp presentationss
 
Google voice
Google voice Google voice
Google voice
 
Natural Language Processing Tools for the Digital Humanities
Natural Language Processing Tools for the Digital HumanitiesNatural Language Processing Tools for the Digital Humanities
Natural Language Processing Tools for the Digital Humanities
 
Audio for VR
Audio for VRAudio for VR
Audio for VR
 
Speech recognition techniques
Speech recognition techniquesSpeech recognition techniques
Speech recognition techniques
 
Issues, Challenges and Perspectives of Digitization: the NLP Experience
Issues, Challenges and Perspectives of Digitization: the NLP ExperienceIssues, Challenges and Perspectives of Digitization: the NLP Experience
Issues, Challenges and Perspectives of Digitization: the NLP Experience
 
Personal Assistant Application Using Android
Personal Assistant Application Using AndroidPersonal Assistant Application Using Android
Personal Assistant Application Using Android
 
[db tech showcase Tokyo 2015] E35: Web, IoT, モバイル時代のデータベース、Apache Cassandraを学ぼう
[db tech showcase Tokyo 2015] E35: Web, IoT, モバイル時代のデータベース、Apache Cassandraを学ぼう[db tech showcase Tokyo 2015] E35: Web, IoT, モバイル時代のデータベース、Apache Cassandraを学ぼう
[db tech showcase Tokyo 2015] E35: Web, IoT, モバイル時代のデータベース、Apache Cassandraを学ぼう
 
Google voice
Google voice Google voice
Google voice
 
Internet of Things (IoT) and Google
Internet of Things (IoT) and GoogleInternet of Things (IoT) and Google
Internet of Things (IoT) and Google
 
Machine Intelligence Applications for IoT Slam Dec 1st 2016
Machine Intelligence Applications for IoT Slam Dec 1st 2016Machine Intelligence Applications for IoT Slam Dec 1st 2016
Machine Intelligence Applications for IoT Slam Dec 1st 2016
 
Watson Internet of Things Hexamite
Watson Internet of Things HexamiteWatson Internet of Things Hexamite
Watson Internet of Things Hexamite
 
Seminar
SeminarSeminar
Seminar
 
Cassandraのしくみ データの読み書き編
Cassandraのしくみ データの読み書き編Cassandraのしくみ データの読み書き編
Cassandraのしくみ データの読み書き編
 
Cassandra導入事例と現場視点での苦労したポイント cassandra summit2014jpn
Cassandra導入事例と現場視点での苦労したポイント cassandra summit2014jpnCassandra導入事例と現場視点での苦労したポイント cassandra summit2014jpn
Cassandra導入事例と現場視点での苦労したポイント cassandra summit2014jpn
 
Why Learn NLP or go on an NLP Training : Webinair
 Why Learn NLP or go on an NLP Training : Webinair Why Learn NLP or go on an NLP Training : Webinair
Why Learn NLP or go on an NLP Training : Webinair
 
COMP 4010 Lecture5 VR Audio and Tracking
COMP 4010 Lecture5 VR Audio and TrackingCOMP 4010 Lecture5 VR Audio and Tracking
COMP 4010 Lecture5 VR Audio and Tracking
 
SIRI: Future of Search
SIRI: Future of SearchSIRI: Future of Search
SIRI: Future of Search
 

Similaire à Michael Hausenblas- Scalable time series and stream processing for IoT applications

Similaire à Michael Hausenblas- Scalable time series and stream processing for IoT applications (20)

Mesos and the Architecture of the New Datacenter
Mesos and the Architecture of the New DatacenterMesos and the Architecture of the New Datacenter
Mesos and the Architecture of the New Datacenter
 
Microservices Manchester: Keynote. Microservices are so 2015, What's Next? By...
Microservices Manchester: Keynote. Microservices are so 2015, What's Next? By...Microservices Manchester: Keynote. Microservices are so 2015, What's Next? By...
Microservices Manchester: Keynote. Microservices are so 2015, What's Next? By...
 
Mesos, DC/OS and the Architecture of the New Datacenter
Mesos, DC/OS and the Architecture of the New DatacenterMesos, DC/OS and the Architecture of the New Datacenter
Mesos, DC/OS and the Architecture of the New Datacenter
 
Erlang User Conference 2016: Container Networking: A Field Report
Erlang User Conference 2016: Container Networking: A Field ReportErlang User Conference 2016: Container Networking: A Field Report
Erlang User Conference 2016: Container Networking: A Field Report
 
DCOS Presentation
DCOS PresentationDCOS Presentation
DCOS Presentation
 
Easy Docker Deployments with Mesosphere DCOS on Azure
Easy Docker Deployments with Mesosphere DCOS on AzureEasy Docker Deployments with Mesosphere DCOS on Azure
Easy Docker Deployments with Mesosphere DCOS on Azure
 
Scaling Like Twitter with Apache Mesos
Scaling Like Twitter with Apache MesosScaling Like Twitter with Apache Mesos
Scaling Like Twitter with Apache Mesos
 
Downtime is not an option - day 2 operations - Jörg Schad
Downtime is not an option - day 2 operations -  Jörg SchadDowntime is not an option - day 2 operations -  Jörg Schad
Downtime is not an option - day 2 operations - Jörg Schad
 
A Journey to Modern Apps with Containers, Microservices and Big Data
A Journey to Modern Apps with Containers, Microservices and Big DataA Journey to Modern Apps with Containers, Microservices and Big Data
A Journey to Modern Apps with Containers, Microservices and Big Data
 
Back to School - St. Louis Hadoop Meetup September 2016
Back to School - St. Louis Hadoop Meetup September 2016Back to School - St. Louis Hadoop Meetup September 2016
Back to School - St. Louis Hadoop Meetup September 2016
 
Kubernetes on DC/OS
Kubernetes on DC/OSKubernetes on DC/OS
Kubernetes on DC/OS
 
Kubernetes on Top of Mesos on Top of DCOS
Kubernetes on Top of Mesos on Top of DCOSKubernetes on Top of Mesos on Top of DCOS
Kubernetes on Top of Mesos on Top of DCOS
 
Mesosphere & Magnetic: Take the pain out of running complex and critical serv...
Mesosphere & Magnetic: Take the pain out of running complex and critical serv...Mesosphere & Magnetic: Take the pain out of running complex and critical serv...
Mesosphere & Magnetic: Take the pain out of running complex and critical serv...
 
Building the Glue for Service Discovery & Load Balancing Microservices
Building the Glue for Service Discovery & Load Balancing MicroservicesBuilding the Glue for Service Discovery & Load Balancing Microservices
Building the Glue for Service Discovery & Load Balancing Microservices
 
Mesosphere: Simplifying Streaming Analytics
Mesosphere: Simplifying Streaming AnalyticsMesosphere: Simplifying Streaming Analytics
Mesosphere: Simplifying Streaming Analytics
 
Cassandra summit 2015 - Simplifying Streaming Analytics
Cassandra summit 2015 - Simplifying Streaming AnalyticsCassandra summit 2015 - Simplifying Streaming Analytics
Cassandra summit 2015 - Simplifying Streaming Analytics
 
Journey to the Modern App with Containers, Microservices and Big Data
Journey to the Modern App with Containers, Microservices and Big DataJourney to the Modern App with Containers, Microservices and Big Data
Journey to the Modern App with Containers, Microservices and Big Data
 
Couchbase Connect 2016: Monitoring Production Deployments The Tools – LinkedIn
Couchbase Connect 2016: Monitoring Production Deployments The Tools – LinkedInCouchbase Connect 2016: Monitoring Production Deployments The Tools – LinkedIn
Couchbase Connect 2016: Monitoring Production Deployments The Tools – LinkedIn
 
Container Orchestration Wars
Container Orchestration WarsContainer Orchestration Wars
Container Orchestration Wars
 
Deploying Containers in Production and at Scale
Deploying Containers in Production and at ScaleDeploying Containers in Production and at Scale
Deploying Containers in Production and at Scale
 

Plus de WithTheBest

Engaging and sharing your VR experience
Engaging and sharing your VR experienceEngaging and sharing your VR experience
Engaging and sharing your VR experience
WithTheBest
 
How to survive the early days of VR as an Indie Studio
How to survive the early days of VR as an Indie StudioHow to survive the early days of VR as an Indie Studio
How to survive the early days of VR as an Indie Studio
WithTheBest
 
Unlocking Human Potential with Immersive Technology
Unlocking Human Potential with Immersive TechnologyUnlocking Human Potential with Immersive Technology
Unlocking Human Potential with Immersive Technology
WithTheBest
 

Plus de WithTheBest (20)

Riccardo Vittoria
Riccardo VittoriaRiccardo Vittoria
Riccardo Vittoria
 
Recreating history in virtual reality
Recreating history in virtual realityRecreating history in virtual reality
Recreating history in virtual reality
 
Engaging and sharing your VR experience
Engaging and sharing your VR experienceEngaging and sharing your VR experience
Engaging and sharing your VR experience
 
How to survive the early days of VR as an Indie Studio
How to survive the early days of VR as an Indie StudioHow to survive the early days of VR as an Indie Studio
How to survive the early days of VR as an Indie Studio
 
Mixed reality 101
Mixed reality 101 Mixed reality 101
Mixed reality 101
 
Unlocking Human Potential with Immersive Technology
Unlocking Human Potential with Immersive TechnologyUnlocking Human Potential with Immersive Technology
Unlocking Human Potential with Immersive Technology
 
Building your own video devices
Building your own video devicesBuilding your own video devices
Building your own video devices
 
Maximizing performance of 3 d user generated assets in unity
Maximizing performance of 3 d user generated assets in unityMaximizing performance of 3 d user generated assets in unity
Maximizing performance of 3 d user generated assets in unity
 
Wizdish rovr
Wizdish rovrWizdish rovr
Wizdish rovr
 
Haptics & amp; null space vr
Haptics & amp; null space vrHaptics & amp; null space vr
Haptics & amp; null space vr
 
How we use vr to break the laws of physics
How we use vr to break the laws of physicsHow we use vr to break the laws of physics
How we use vr to break the laws of physics
 
The Virtual Self
The Virtual Self The Virtual Self
The Virtual Self
 
You dont have to be mad to do VR and AR ... but it helps
You dont have to be mad to do VR and AR ... but it helpsYou dont have to be mad to do VR and AR ... but it helps
You dont have to be mad to do VR and AR ... but it helps
 
Omnivirt overview
Omnivirt overviewOmnivirt overview
Omnivirt overview
 
VR Interactions - Jason Jerald
VR Interactions - Jason JeraldVR Interactions - Jason Jerald
VR Interactions - Jason Jerald
 
Japheth Funding your startup - dating the devil
Japheth  Funding your startup - dating the devilJapheth  Funding your startup - dating the devil
Japheth Funding your startup - dating the devil
 
Transported vr the virtual reality platform for real estate
Transported vr the virtual reality platform for real estateTransported vr the virtual reality platform for real estate
Transported vr the virtual reality platform for real estate
 
Measuring Behavior in VR - Rob Merki Cognitive VR
Measuring Behavior in VR - Rob Merki Cognitive VRMeasuring Behavior in VR - Rob Merki Cognitive VR
Measuring Behavior in VR - Rob Merki Cognitive VR
 
Global demand for Mixed Realty (VR/AR) content is about to explode.
Global demand for Mixed Realty (VR/AR) content is about to explode. Global demand for Mixed Realty (VR/AR) content is about to explode.
Global demand for Mixed Realty (VR/AR) content is about to explode.
 
VR, a new technology over 40,000 years old
VR, a new technology over 40,000 years oldVR, a new technology over 40,000 years old
VR, a new technology over 40,000 years old
 

Dernier

Escorts Service Arekere ☎ 7737669865☎ Book Your One night Stand (Bangalore)
Escorts Service Arekere ☎ 7737669865☎ Book Your One night Stand (Bangalore)Escorts Service Arekere ☎ 7737669865☎ Book Your One night Stand (Bangalore)
Escorts Service Arekere ☎ 7737669865☎ Book Your One night Stand (Bangalore)
amitlee9823
 
➥🔝 7737669865 🔝▻ Muzaffarpur Call-girls in Women Seeking Men 🔝Muzaffarpur🔝 ...
➥🔝 7737669865 🔝▻ Muzaffarpur Call-girls in Women Seeking Men  🔝Muzaffarpur🔝  ...➥🔝 7737669865 🔝▻ Muzaffarpur Call-girls in Women Seeking Men  🔝Muzaffarpur🔝  ...
➥🔝 7737669865 🔝▻ Muzaffarpur Call-girls in Women Seeking Men 🔝Muzaffarpur🔝 ...
amitlee9823
 
Vip Mumbai Call Girls Kalyan Call On 9920725232 With Body to body massage wit...
Vip Mumbai Call Girls Kalyan Call On 9920725232 With Body to body massage wit...Vip Mumbai Call Girls Kalyan Call On 9920725232 With Body to body massage wit...
Vip Mumbai Call Girls Kalyan Call On 9920725232 With Body to body massage wit...
amitlee9823
 
Bommasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...
Bommasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...Bommasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...
Bommasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...
amitlee9823
 
怎样办理维多利亚大学毕业证(UVic毕业证书)成绩单留信认证
怎样办理维多利亚大学毕业证(UVic毕业证书)成绩单留信认证怎样办理维多利亚大学毕业证(UVic毕业证书)成绩单留信认证
怎样办理维多利亚大学毕业证(UVic毕业证书)成绩单留信认证
tufbav
 
➥🔝 7737669865 🔝▻ Deoghar Call-girls in Women Seeking Men 🔝Deoghar🔝 Escorts...
➥🔝 7737669865 🔝▻ Deoghar Call-girls in Women Seeking Men  🔝Deoghar🔝   Escorts...➥🔝 7737669865 🔝▻ Deoghar Call-girls in Women Seeking Men  🔝Deoghar🔝   Escorts...
➥🔝 7737669865 🔝▻ Deoghar Call-girls in Women Seeking Men 🔝Deoghar🔝 Escorts...
amitlee9823
 
➥🔝 7737669865 🔝▻ kakinada Call-girls in Women Seeking Men 🔝kakinada🔝 Escor...
➥🔝 7737669865 🔝▻ kakinada Call-girls in Women Seeking Men  🔝kakinada🔝   Escor...➥🔝 7737669865 🔝▻ kakinada Call-girls in Women Seeking Men  🔝kakinada🔝   Escor...
➥🔝 7737669865 🔝▻ kakinada Call-girls in Women Seeking Men 🔝kakinada🔝 Escor...
amitlee9823
 
(👉Ridhima)👉VIP Model Call Girls Mulund ( Mumbai) Call ON 9967824496 Starting ...
(👉Ridhima)👉VIP Model Call Girls Mulund ( Mumbai) Call ON 9967824496 Starting ...(👉Ridhima)👉VIP Model Call Girls Mulund ( Mumbai) Call ON 9967824496 Starting ...
(👉Ridhima)👉VIP Model Call Girls Mulund ( Mumbai) Call ON 9967824496 Starting ...
motiram463
 
Vip Mumbai Call Girls Andheri East Call On 9920725232 With Body to body massa...
Vip Mumbai Call Girls Andheri East Call On 9920725232 With Body to body massa...Vip Mumbai Call Girls Andheri East Call On 9920725232 With Body to body massa...
Vip Mumbai Call Girls Andheri East Call On 9920725232 With Body to body massa...
amitlee9823
 
➥🔝 7737669865 🔝▻ Vijayawada Call-girls in Women Seeking Men 🔝Vijayawada🔝 E...
➥🔝 7737669865 🔝▻ Vijayawada Call-girls in Women Seeking Men  🔝Vijayawada🔝   E...➥🔝 7737669865 🔝▻ Vijayawada Call-girls in Women Seeking Men  🔝Vijayawada🔝   E...
➥🔝 7737669865 🔝▻ Vijayawada Call-girls in Women Seeking Men 🔝Vijayawada🔝 E...
amitlee9823
 
Makarba ( Call Girls ) Ahmedabad ✔ 6297143586 ✔ Hot Model With Sexy Bhabi Rea...
Makarba ( Call Girls ) Ahmedabad ✔ 6297143586 ✔ Hot Model With Sexy Bhabi Rea...Makarba ( Call Girls ) Ahmedabad ✔ 6297143586 ✔ Hot Model With Sexy Bhabi Rea...
Makarba ( Call Girls ) Ahmedabad ✔ 6297143586 ✔ Hot Model With Sexy Bhabi Rea...
Naicy mandal
 

Dernier (20)

SM-N975F esquematico completo - reparación.pdf
SM-N975F esquematico completo - reparación.pdfSM-N975F esquematico completo - reparación.pdf
SM-N975F esquematico completo - reparación.pdf
 
Get Premium Pimple Saudagar Call Girls (8005736733) 24x7 Rate 15999 with A/c ...
Get Premium Pimple Saudagar Call Girls (8005736733) 24x7 Rate 15999 with A/c ...Get Premium Pimple Saudagar Call Girls (8005736733) 24x7 Rate 15999 with A/c ...
Get Premium Pimple Saudagar Call Girls (8005736733) 24x7 Rate 15999 with A/c ...
 
Escorts Service Arekere ☎ 7737669865☎ Book Your One night Stand (Bangalore)
Escorts Service Arekere ☎ 7737669865☎ Book Your One night Stand (Bangalore)Escorts Service Arekere ☎ 7737669865☎ Book Your One night Stand (Bangalore)
Escorts Service Arekere ☎ 7737669865☎ Book Your One night Stand (Bangalore)
 
➥🔝 7737669865 🔝▻ Muzaffarpur Call-girls in Women Seeking Men 🔝Muzaffarpur🔝 ...
➥🔝 7737669865 🔝▻ Muzaffarpur Call-girls in Women Seeking Men  🔝Muzaffarpur🔝  ...➥🔝 7737669865 🔝▻ Muzaffarpur Call-girls in Women Seeking Men  🔝Muzaffarpur🔝  ...
➥🔝 7737669865 🔝▻ Muzaffarpur Call-girls in Women Seeking Men 🔝Muzaffarpur🔝 ...
 
Top Rated Pune Call Girls Katraj ⟟ 6297143586 ⟟ Call Me For Genuine Sex Serv...
Top Rated  Pune Call Girls Katraj ⟟ 6297143586 ⟟ Call Me For Genuine Sex Serv...Top Rated  Pune Call Girls Katraj ⟟ 6297143586 ⟟ Call Me For Genuine Sex Serv...
Top Rated Pune Call Girls Katraj ⟟ 6297143586 ⟟ Call Me For Genuine Sex Serv...
 
Vip Mumbai Call Girls Kalyan Call On 9920725232 With Body to body massage wit...
Vip Mumbai Call Girls Kalyan Call On 9920725232 With Body to body massage wit...Vip Mumbai Call Girls Kalyan Call On 9920725232 With Body to body massage wit...
Vip Mumbai Call Girls Kalyan Call On 9920725232 With Body to body massage wit...
 
Bommasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...
Bommasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...Bommasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...
Bommasandra Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore...
 
Top Rated Pune Call Girls Chakan ⟟ 6297143586 ⟟ Call Me For Genuine Sex Serv...
Top Rated  Pune Call Girls Chakan ⟟ 6297143586 ⟟ Call Me For Genuine Sex Serv...Top Rated  Pune Call Girls Chakan ⟟ 6297143586 ⟟ Call Me For Genuine Sex Serv...
Top Rated Pune Call Girls Chakan ⟟ 6297143586 ⟟ Call Me For Genuine Sex Serv...
 
(ISHITA) Call Girls Service Aurangabad Call Now 8617697112 Aurangabad Escorts...
(ISHITA) Call Girls Service Aurangabad Call Now 8617697112 Aurangabad Escorts...(ISHITA) Call Girls Service Aurangabad Call Now 8617697112 Aurangabad Escorts...
(ISHITA) Call Girls Service Aurangabad Call Now 8617697112 Aurangabad Escorts...
 
怎样办理维多利亚大学毕业证(UVic毕业证书)成绩单留信认证
怎样办理维多利亚大学毕业证(UVic毕业证书)成绩单留信认证怎样办理维多利亚大学毕业证(UVic毕业证书)成绩单留信认证
怎样办理维多利亚大学毕业证(UVic毕业证书)成绩单留信认证
 
Pooja 9892124323, Call girls Services and Mumbai Escort Service Near Hotel Th...
Pooja 9892124323, Call girls Services and Mumbai Escort Service Near Hotel Th...Pooja 9892124323, Call girls Services and Mumbai Escort Service Near Hotel Th...
Pooja 9892124323, Call girls Services and Mumbai Escort Service Near Hotel Th...
 
➥🔝 7737669865 🔝▻ Deoghar Call-girls in Women Seeking Men 🔝Deoghar🔝 Escorts...
➥🔝 7737669865 🔝▻ Deoghar Call-girls in Women Seeking Men  🔝Deoghar🔝   Escorts...➥🔝 7737669865 🔝▻ Deoghar Call-girls in Women Seeking Men  🔝Deoghar🔝   Escorts...
➥🔝 7737669865 🔝▻ Deoghar Call-girls in Women Seeking Men 🔝Deoghar🔝 Escorts...
 
➥🔝 7737669865 🔝▻ kakinada Call-girls in Women Seeking Men 🔝kakinada🔝 Escor...
➥🔝 7737669865 🔝▻ kakinada Call-girls in Women Seeking Men  🔝kakinada🔝   Escor...➥🔝 7737669865 🔝▻ kakinada Call-girls in Women Seeking Men  🔝kakinada🔝   Escor...
➥🔝 7737669865 🔝▻ kakinada Call-girls in Women Seeking Men 🔝kakinada🔝 Escor...
 
(👉Ridhima)👉VIP Model Call Girls Mulund ( Mumbai) Call ON 9967824496 Starting ...
(👉Ridhima)👉VIP Model Call Girls Mulund ( Mumbai) Call ON 9967824496 Starting ...(👉Ridhima)👉VIP Model Call Girls Mulund ( Mumbai) Call ON 9967824496 Starting ...
(👉Ridhima)👉VIP Model Call Girls Mulund ( Mumbai) Call ON 9967824496 Starting ...
 
Book Sex Workers Available Pune Call Girls Yerwada 6297143586 Call Hot India...
Book Sex Workers Available Pune Call Girls Yerwada  6297143586 Call Hot India...Book Sex Workers Available Pune Call Girls Yerwada  6297143586 Call Hot India...
Book Sex Workers Available Pune Call Girls Yerwada 6297143586 Call Hot India...
 
Vip Mumbai Call Girls Andheri East Call On 9920725232 With Body to body massa...
Vip Mumbai Call Girls Andheri East Call On 9920725232 With Body to body massa...Vip Mumbai Call Girls Andheri East Call On 9920725232 With Body to body massa...
Vip Mumbai Call Girls Andheri East Call On 9920725232 With Body to body massa...
 
Call Girls in Vashi Escorts Services - 7738631006
Call Girls in Vashi Escorts Services - 7738631006Call Girls in Vashi Escorts Services - 7738631006
Call Girls in Vashi Escorts Services - 7738631006
 
➥🔝 7737669865 🔝▻ Vijayawada Call-girls in Women Seeking Men 🔝Vijayawada🔝 E...
➥🔝 7737669865 🔝▻ Vijayawada Call-girls in Women Seeking Men  🔝Vijayawada🔝   E...➥🔝 7737669865 🔝▻ Vijayawada Call-girls in Women Seeking Men  🔝Vijayawada🔝   E...
➥🔝 7737669865 🔝▻ Vijayawada Call-girls in Women Seeking Men 🔝Vijayawada🔝 E...
 
HLH PPT.ppt very important topic to discuss
HLH PPT.ppt very important topic to discussHLH PPT.ppt very important topic to discuss
HLH PPT.ppt very important topic to discuss
 
Makarba ( Call Girls ) Ahmedabad ✔ 6297143586 ✔ Hot Model With Sexy Bhabi Rea...
Makarba ( Call Girls ) Ahmedabad ✔ 6297143586 ✔ Hot Model With Sexy Bhabi Rea...Makarba ( Call Girls ) Ahmedabad ✔ 6297143586 ✔ Hot Model With Sexy Bhabi Rea...
Makarba ( Call Girls ) Ahmedabad ✔ 6297143586 ✔ Hot Model With Sexy Bhabi Rea...
 

Michael Hausenblas- Scalable time series and stream processing for IoT applications

  • 1. © 2016 Mesosphere, Inc. All Rights Reserved. SCALABLE TIME SERIES AND STREAM PROCESSING FOR IOT APPLICATIONS 1 Michael Hausenblas, Developer & Cloud Advocate | 2016-01-16
  • 2. © 2015 Mesosphere, Inc. All Rights Reserved. MOTIVATION 2
  • 3. © 2015 Mesosphere, Inc. All Rights Reserved. AIRLINES 3
  • 4. © 2015 Mesosphere, Inc. All Rights Reserved. LOGISTICS 4
  • 5. © 2015 Mesosphere, Inc. All Rights Reserved. HEALTH
 CARE 5
  • 6. © 2015 Mesosphere, Inc. All Rights Reserved. TRADERS 6
  • 7. © 2015 Mesosphere, Inc. All Rights Reserved. FARMERS 7
  • 8. © 2015 Mesosphere, Inc. All Rights Reserved. CITIES 8 ©2014,Wiredmagazine
  • 9. © 2015 Mesosphere, Inc. All Rights Reserved. YOU 9
  • 10. © 2015 Mesosphere, Inc. All Rights Reserved. THE
 TOOLBOX 10
  • 11. © 2015 Mesosphere, Inc. All Rights Reserved. LET'S TALK ABOUT WORKLOADS* … 11*) kudos to Timothy St. Clair, @timothysc batch streaming PaaS MapReduce
  • 12. © 2015 Mesosphere, Inc. All Rights Reserved. • Apache Kafka • ØMQ, RabbitMQ, Disque (Redis-based), etc. • fluentd, Logstash, Flume • Akka streams • cloud-only: AWS SQS, Google Cloud Pub/Sub • see also queues.io MESSAGE QUEUES & ROUTERS 12
  • 13. © 2015 Mesosphere, Inc. All Rights Reserved. APACHE KAFKA 13 • High-throughput, distributed, persistent publish-subscribe messaging system • Originates from LinkedIn • Typically used as buffer/de-coupling layer in online stream processing Message queues & routers kafka.apache.org
  • 14. © 2015 Mesosphere, Inc. All Rights Reserved. FLUENTD 14 Message queues & routers www.fluentd.org
  • 15. © 2015 Mesosphere, Inc. All Rights Reserved. STREAM PROCESSING PLATFORMS 15 • Apache Storm • Apache Spark • Apache Samza • Apache Flink • Concord • cloud-only: AWS Kinesis, Google Cloud Dataflow • see also my webinar on stream processing
  • 16. © 2015 Mesosphere, Inc. All Rights Reserved. APACHE STORM 16 • Distributed, fault-tolerant stream- processing platform • Guaranteed message processing (replaying messages on failure) • Concepts: tuples, streams, spouts, bolts, topologies Stream processing platforms storm.apache.org
  • 17. © 2015 Mesosphere, Inc. All Rights Reserved. APACHE SPARK 17 Stream processing platforms spark.apache.org Spark SQL Spark Streaming MLlib
 (machine learning) Spark core (RDD) GraphX
 (graph processing) Mesos Filesystem (local, HDFS, S3) or data store (HBase, Cassandra, Elasticsearch, etc.) YARNStandalone
  • 18. © 2015 Mesosphere, Inc. All Rights Reserved. TIME SERIES DATASTORES 18 • InfluxDB • OpenTSDB • KairosDB • Prometheus • see also iot-a.info
  • 19. © 2015 Mesosphere, Inc. All Rights Reserved. OPENTSDB 19 • Distributed time series database on top HBase • Store, index, query & plot metrics • Extremely scalable • Low-level monitoring Time series datastores opentsdb.net
  • 20. © 2015 Mesosphere, Inc. All Rights Reserved. INFLUXDB 20 • No-dependency, time series database written in Go • SQLish query language (incl. regex, fan out) • Single node or Raft-based distributed node mode Time series datastores influxdb.com
  • 21. © 2015 Mesosphere, Inc. All Rights Reserved. CHALLENGES 21 • Set up and operation of components • Elasticity: static vs. dynamic partitioning • Efficient usage of resources (TCO)
  • 22. © 2015 Mesosphere, Inc. All Rights Reserved. MEET THE DATACENTER
 OPERATING
 SYSTEM
 (DCOS) 22
  • 23. © 2015 Mesosphere, Inc. All Rights Reserved. LOCAL OS VS. DISTRIBUTED OS 23 http://bitly.com/os-vs-dcos
  • 24. © 2015 Mesosphere, Inc. All Rights Reserved. DCOS IS A DISTRIBUTED OPERATING SYSTEM 24 • local OS per node (+container enabled) • scheduling (long-lived, batch) • networking • service discovery • stateful services • security • monitoring, logging, debugging
  • 25. © 2015 Mesosphere, Inc. All Rights Reserved. 25
  • 26. © 2015 Mesosphere, Inc. All Rights Reserved. BENEFITS 26 DCOS • Run stateless services such as Web server or app server and Big Data services like Kafka, Spark, or Cassandra together on one cluster • Dynamic partitioning of your cluster, depending on your business requirements • Increased utilization (10% → 80%++)
  • 27. © 2015 Mesosphere, Inc. All Rights Reserved. AN EXAMPLE 27
  • 28. © 2015 Mesosphere, Inc. All Rights Reserved. 28 https://mesosphere.com/blog/2015/11/18/dcos-time-series-demo
  • 29. © 2015 Mesosphere, Inc. All Rights Reserved. 29 https://github.com/mesosphere/time-series-demo
  • 30. © 2015 Mesosphere, Inc. All Rights Reserved. Q & A 30 • @mhausenblas • mhausenblas.info • @mesosphere • mesosphere.io/product • mesosphere.com/infinity