IoT Sensor Analytics with Python, Jupyter, TensorFlow, Keras, Apache Kafka, KSQL and MQTT

1Confidential
IoT Sensor Analytics with
Apache Kafka, KSQL, TensorFlow and MQTT
Kai Waehner
Technology Evangelist
kontakt@kai-waehner.de
LinkedIn
@KaiWaehner
www.confluent.io
www.kai-waehner.de
Kafka-Native End-to-End IoT Data Integration and Processing

3
Agenda
1) IoT Use Cases
2) MQTT Standard
3) Apache Kafka Ecosystem
4) TensorFlow for IoT Scenarios
5) End-to-End IoT Integration Architecture(s)
6) IoT Data Processing
7) Live Demo: End-to-End Sensor Analytics

4
Agenda
1) IoT Use Cases
2) MQTT Standard

6
Connected Intelligence (Cars, Machines, Robots, …)

8
Smart Retail and Customer 360

9
Intelligent Applications (Early Part Scrapping, Predictive Maintenance, …)

10
?
Architecture (High Level)
Kafka BrokerKafka BrokerStreaming
Platform
Connect
w/ MQTT
connector
GatewayDevicesDevicesDevicesDevice
Device Tracking
(Real Time)
Predictive
Maintenance
(Near Real Time)
Log Analytics
(Batch)
Edge Data Center / Cloud
How to integrate?

13
Agenda
1) IoT Use Cases
2) MQTT Standard

14
MQTT - Publish / subscribe messaging protocol
• Built on top of TCP/IP for constrained devices and unreliable networks
• Many (open source) broker implementations
• Many client libraries
• IoT-specific features for bad network / connectivity
• Widely used (mostly IoT, but also web and mobile apps via MQTT over WebSockets)

17
MQTT Architecture (large scale)
Load
Balancer
MQTT
Server 1
MQTT
Server 2
MQTT
Server 3
MQTT
Server 4
topic: [deviceid]/car
...
Processor
1
Processor
2
Processor
3
Processor
4

18
MQTT Trade-Offs
Pros
• Lightweight
• Simple API
• Built for poor connectivity / high latency scenario
• Many client connections (tens of thousands per MQTT server)
Cons
• Queuing, not stream processing
• Can’t handle usage surges (no buffering)
• No high scalability (true for most MQTT brokers)
• Very asynchronous processing (often offline for long time)
• No good integration to the rest of the enterprise
• No reprocessing of events

19
Agenda
1) IoT Use Cases
2) MQTT Standard

20
Apache Kafka – The Rise of a Streaming Platform
The Log ConnectorsConnectors
Producer Consumer
Streaming Engine

23
Apache Kafka == Distributed Commit Log with Replication

25
Apache Kafka at Scale
https://conferences.oreilly.com/strata/strata-ca/public/schedule/detail/63921
https://qconlondon.com/london2018/presentation/cloud-native-and-scalable-kafka-architecture
(2018)
(2018)

26
Kafka Trade-Offs (from IoT perspective)
Pros
• Stream processing, not just queuing
• High throughput
• Large scale
• High availability
• Long term storage and buffering
• Reprocessing of events
• Good integration to the rest of the enterprise
Cons
• Not built for tens of thousands connections
• Requires stable network and good infrastructure
• No IoT-specific features like keep alive, last will or testament

27
(De facto) Standards for Processing IoT Data
A Match Made In Heaven
+ =

28
Agenda
1) IoT Use Cases
2) MQTT Standard

29
TensorFlow
TensorFlow is an open source software library for high
performance numerical computation. Its flexible architecture
allows easy deployment of computation across a variety of
platforms (CPUs, GPUs, TPUs), and from desktops to clusters of
servers to mobile and edge devices. Originally developed by
researchers and engineers from the Google Brain team within
Google’s AI organization, it comes with strong support for
machine learning and deep learning and the flexible
numerical computation core is used across many other scientific
domains.
https://www.tensorflow.org/

30
The First Analytic Models
How to deploy the models
in production?
…real-time processing?
…at scale?
…24/7 zero downtime?

31
Hidden Technical Debt in Machine Learning Systems
https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf

32
Apache Kafka’s Open Source Ecosystem as Infrastructure for ML

33
Apache Kafka’s Open Ecosystem as Infrastructure for ML
Kafka
Streams
Kafka
Connect
Rest Proxy
Schema Registry
Go/.NET /Python
Kafka Producer
KSQL
Kafka
Streams

37
Replayability — a log never forgets!
Time
Model B Model XModel A
Producer
Distributed Commit Log
Different models with same data
Different ML frameworks
AutoML compatible
A/B testing
Google Cloud Storage HDFS

38
Analytic Model (Autoencoder for Anomaly Detection)

39
Model Deployment #1: RPC Communication to do Model Inference
Streams
Input Event
Prediction
Request
Response
Model Serving
TensorFlow Serving
gRPC

40
Model deployment #2: Model interference natively in the App
Streams
Input Event
Prediction

41
Agenda
1) IoT Use Cases
2) MQTT Standard

42
?
Architecture (High Level)
Platform
Connect
w/ MQTT
connector
Device Tracking
(Real Time)
Predictive
Maintenance
(Near Real Time)
Log Analytics
(Batch)
How to integrate?

43
?
Architecture (High Level) – Machine Learning Perspective
Platform
Connect
w/ MQTT
connector
Edge Analytics
Real Time
Model Serving
Predictive
Maintenance
Near Real Time
Model Serving
Model Training
(Batch)

46
Kafka-Native Integration Options between MQTT and Apache Kafka
Kafka Connect
MQTT Proxy
REST Proxy

47
Kafka Connect
MQTT Proxy
REST Proxy

49
?
Integration with Kafka Connect (Source and Sink)
Kafka BrokerKafka BrokerKafka Broker
MQTT
Broker
Connect
w/ MQTT
connector
Connect
w/ MQTT
connectorMQTT
DevicesDevicesDevicesDevice
Kafka
Consumer
MQTT Broker
Persistent + offers MQTT-specific features
Consumes push data from IoT devices
Kafka Connect
Kafka Consumer + Kafka Producer under the hood
Pull-based (at own pace, without overwhelming the source or getting overwhelmed by the source)
Out-of-the-box scalability and integration features (like connectors, converters, SMTs)
?
Connect
w/ MQTT
connector
Connect
w/ MQTT
connector

56
Kafka Connect
MQTT Proxy
REST Proxy

57
MQTT Proxy
Kafka BrokerKafka BrokerKafka Broker
MQTT
ProxyMQTT
DevicesDevicesDevicesDevices
Kafka
Consumer
MQTT Proxy
MQTT is push-based
Horizontally scalable
Consumes push data from IoT devices and forwards it to Kafka Broker at low-latency
Kafka Producer under the hood
No MQTT Broker needed
Kafka Broker
Source of truth
Responsible for persistence, high availability, reliability

59
Kafka Connect
MQTT Proxy
REST Proxy

60
Confluent REST Proxy
REST Proxy
IoT Applicatons
Native Kafka
Applications
(Java, C, Go, …)
REST / HTTP(S)
TCP
The „simple alternative“ for IoT
• Simple and understood
• HTTP(S) Proxy à Push-based
• Security ”easier”
• Scalable with standard load balancer
(still synchronous HTTP)
• Not for very high throughput
• Implement Kafka Connect features in
your client app

62
Agenda
1) IoT Use Cases
2) MQTT Standard

6363
Processing Options for MQTT Data with Apache Kafka
Streams
Kafka native vs. additional big data cluster and technology
(or others, you name it …)

6464
IoT Data Processing
Kafka Client
Batch
System
AnalyticsKafka Cluster Kafka Connect
Kafka Streams
/
KSQL
MQTT Device
Kafka Ecosystem
Other Components
Real Time
System
All Data
Alerting
Process
Data
Continuously
Forward
Processed
Data
On premise DC / CloudAt the edge

6868
KSQL – Continuous Queries for Streaming ETL / Anomaly Detection
CREATE STREAM vip_actions AS
SELECT userid, page, action FROM clickstream c
LEFT JOIN users u ON c.userid = u.user_id
WHERE u.level = 'Platinum';
CREATE TABLE possible_fraud AS
SELECT card_number, count(*)
FROM authorization_attempts
WINDOW TUMBLING (SIZE 5 MINUTES)
GROUP BY card_number
HAVING count(*) > 3;

6969
Agenda
1) IoT Use Cases
2) MQTT Standard

7070
KSQL and Deep Learning (Auto Encoder) for Anomaly Detection
MQTT
Proxy
Elastic
search
Grafana
Kafka
Cluster
Kafka
Connect
KSQL
Car Sensors
Kafka Ecosystem
Other Components
Real Time
Emergency
System
All Data
PotentialDefect
Apply
Analytic
Model
Filter
Anomalies
On premise DCAt the edge
5858
KSQL and Deep Learning (Auto Encoder) for Anomaly Detection
MQTT
Proxy
Elastic
search
Grafana
Kafka
Cluster
Kafka
Connect
KSQL
Car Sensors
Kafka Ecosystem
Other Components
Real Time
Emergency
System
All Data
PotentialDefect
Apply
Analytic
Model
Filter
Anomalies
On premise DCAt the edge

7171
Model Training with Python, KSQL, TensorFlow, Keras and Jupyter
https://github.com/kaiwaehner/python-jupyter-apache-kafka-ksql-tensorflow-keras

7272
Model Deployment with Apache Kafka, KSQL and TensorFlow
“CREATE STREAM AnomalyDetection AS
SELECT sensor_id, detectAnomaly(sensor_values)
FROM car_engine;“
User Defined Function (UDF)

73
Live Demo
End-to-End Sensor Analytics…
Python, Jupyter Notebook, TensorFlow, Keras, Apache Kafka, KSQL and MQTT

74
Model Training with Python, KSQL, TensorFlow, Keras and Jupyter
https://github.com/kaiwaehner/python-jupyter-apache-kafka-ksql-tensorflow-keras

75
Deep Learning UDF for KSQL for Streaming Anomaly Detection of MQTT IoT Sensor Data
https://github.com/kaiwaehner/ksql-udf-deep-learning-mqtt-iot

77
Kai Waehner
Technology Evangelist
kontakt@kai-waehner.de
@KaiWaehner
www.kai-waehner.de
www.confluent.io
LinkedIn
Questions? Feedback?
Please contact me!
Come to our booth
to find out more about
Kafka and Confluent

IoT Sensor Analytics with Python, Jupyter, TensorFlow, Keras, Apache Kafka, KSQL and MQTT

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à IoT Sensor Analytics with Python, Jupyter, TensorFlow, Keras, Apache Kafka, KSQL and MQTT

Similaire à IoT Sensor Analytics with Python, Jupyter, TensorFlow, Keras, Apache Kafka, KSQL and MQTT (20)

Plus de Kai Wähner

Plus de Kai Wähner (20)

Dernier

Dernier (20)

IoT Sensor Analytics with Python, Jupyter, TensorFlow, Keras, Apache Kafka, KSQL and MQTT