Building a reliable and scalable IoT platform with MongoDB and HiveMQ

Building a Scalable and Reliable
IoT Platform with HiveMQ and MongoDB
Dr. Christian Kurze
Principal Solutions Architect
MongoDB
Dominik Obermaier
CTO and Co-Founder
HiveMQ

How to optimize your Business with IoT?
Typical IoT Challenges
HiveMQ and MongoDB Solution
Fleet Management Demo
What you will
learn today
Q&A

Or: Why should you care?
How to optimize your Business with IoT?

Digital Services are the Key Differentiator in Global Competition
Intensifying Competition ... … Eroding Margins ...
… leading to the need to rethink
current business models
8 5
41
24
18
28
53
24
Asia-Paciﬁc
Americas
Europe
Rest of World
1998 2019 General industrial machinery
Already commoditized
Future price
premium at stake
Special purpose machinery
Risk of
hardware
commoditization
McKinsey: IIoT platforms: The technology stack as value driver in industrial equipment and machinery
Services
Software
Machinery
1998 2019
Production of machinery
by region (%)
Indicative Value Pools

Digital Services are the Key Differentiator in Global Competition
Intensifying Competition ... … Eroding Margins ...
… leading to the need to rethink
current business models

Leaders gain >15%
positive impact on cost
and revenue via IoT
McKinsey: Best Practices Separating IoT Leaders from Laggards

Develop new IoT Products and Services
e.g. apps, ﬂeet management
Business
Strategies
Drive Sales and Service Efficiency
e.g. monitoring, ﬁeld services, maintenance, staffing
Optimize Business Operations
e.g. manufacturing, supply chain, R&D

Devices with Sensors and Actuators
e.g. cars, buildings, equipment, wearables
Device Enablement Platforms
e.g. obtaining, importing, and processing data using standard protocols
IT
Initiatives
Business Applications
e.g. customer- and/or device-facing functionality, dashboards, mobile apps
Cloud and Edge Computing
e.g. for new workloads and cost optimizations

A Digital Twin can represent
almost everything:
Machines, Processes, Places,
Buildings, Cities, even
Humans like You and Me.

Global leaders leverage IoT powered by MongoDB
Source of Industries: IHS Markit

What is the core of IoT & Digital Twins?
R&D
Production
Operation
Maintenance
Decommissioning
Describe
Predict
Recommend
Combines all data across the
whole lifecycle of a product
Outlives the physical product in
order to optimize business
processes and unlock new
revenue streams
Product Life-Cycle
Information Life-Cycle

Only ~30% of relevant IoT solutions are in
company-wide roll-out.
Delivering IoT at scale requires the ability to
extract, interpret, and harmonize data from
disparate systems that were not designed to
work together.
McKinsey: Best Practices Separating IoT Leaders from Laggards

Or: Why does not everyone beneﬁt?
Typical IoT Challenges

Web Technology used today is
built for the
Internet of Humans,
NOT for the
Internet of Things

Many Different Types of Data
● Device data arrives in different
formats (JSON, AVRO, Protobuf,
custom binary formats)
● Often in time series data
● Data agnostic message brokers
used to distribute data into the
backend
● Relational databases are sometimes
not well suited for IoT data

Responsiveness of Systems
● Low latency is critical for many IoT
use cases
● End users expect responsive IoT
applications
● Unreliable cellular networks can
have a signiﬁcant impact on
responsiveness
BMW Case Study: bit.ly/bmw-casestudy

Scalability
● IoT solutions need to
scale to accommodate
growth (100s - 1,000,000s
of devices)
● Scale-up and scale-down
to accommodate spikes

MQTT Broker Cluster
IoT Devices
Extension
Extension
Extension
Enterprise Integration
● IoT data needs to be
integrated into
enterprise systems
● Device to Cloud and
Cloud to Device data
integration

On-premise Self hosted Managed Service
Deployment Agnostic
Atlas

Many different types of data
Top IoT
Challenges
Responsiveness of systems
Enterprise integration
Deployment agnostic
Scalability to thousands & millions of devices

Or: How to tackle the typical challenges?
HiveMQ & MongoDB Solution

IoT Simpliﬁed: Same Pattern for any Application
Sensors & Actuators
Wireless communication over
industry standard protocols
Edge Gateway
Offline Storage,
Local Processing
Streaming & Routing
Standard Protocols and Tools,
e.g. MQTT, Kafka
Data Storage
Hot & Cold Data for
Real-Time & Batch Access
Dashboards
Visual Insights
Applications
User-Facing Applications &
Automations
Advanced Analytics &
Machine Learning
Gaining Insights into Data,
Predict & Act

HiveMQ’s MQTT Broker for Communication
Sensors & Actuators
Edge Gateway
Offline Storage,
Local Processing
Streaming & Routing
e.g. MQTT, Kafka
Data Storage
Hot & Cold Data for
Dashboards
Visual Insights
Applications
Automations
Machine Learning
Predict & Act

Publish/subscribe based architecture
Easy
(I)IoT Messaging Protocol
Minimal Overhead
Designed for reliable communication
over unreliable channels
Binary
Data agnostic
What is
MQTT?

Logistics
Tele-
communication
IoT Messaging
Middleware
MQTT Use Cases
Connected
Car
IIoT /
Industry 4.0

HiveMQ Enterprise MQTT Platform

Building new digital products
Improving customer experience
Creating more efficient operations
and insights
Avoiding data loss with efficient
system
Our Customers
…and more
...and more

Challenge: Responsiveness of Systems &
Deployment Agnostic
Challenge: Scalability to thousands & millions of devicesChallenge: Scalability to thousands & millions of devices
Challenge: Enterprise Integration
Tackling IoT Challenges
World-class scalable MQTT
● Masterless architecture, Auto healing, elastic
scaling
● Private or Public deployments
● K8s, OpenShift, AWS, Azure, GCP
● Ideal for multi-cloud
Reliable Cloud Native Architecture
● Visibility for operations team
● Live debug of individual clients
● Trace recording for playback on message
sequences
Real-time Monitoring Across Device Fleets
● Data integration with existing enterprise
systems
● Integration with other MQTT clients and
broker
● Off-the-shelf integrations and custom
extensions
Extension Framework and Marketplace
● Scales to 10 million connections and
more
● MQTT 5 fully supported
● Full hybrid support of 3.1.1 and 3.1

HiveMQ & MongoDB on-premises
IoT Devices
Extension
Extension
Extension

HiveMQ Cloud & MongoDB Atlas
Extension
Extension
Extension
IoT Devices
Atlas

MongoDB’s Data Platform for IoT
Sensors & Actuators
Edge Gateway
Offline Storage,
Local Processing
Streaming & Routing
e.g. MQTT, Kafka
Data Storage
Hot & Cold Data for
Dashboards
Visual Insights
Applications
Automations
Machine Learning
Predict & Act

MongoDB Atlas: End-2-End Data Platform for IoT
Primary Secondary Secondary
High Volume Real Time Operational Data
Analytical Analytical
Real Time Analytical Data
Data Lake, offline, queryable archive
Lucene based Text Search
Sharding and Replica Sets
Multi-Cloud Data Platform
OneQueryLanguage,API,andSQL
Sensors & Actuators
Edge Gateway
Offline Storage,
Local Processing
Streaming & Routing
e.g. MQTT, Kafka
Mobile Database
Edge to Cloud Sync
Native Visualizations
Applications & Microservices
Advanced Analytics
Triggers & Events
Reporting

Beneﬁts by Using One Single Data Platform
Fast Time-to-Market
One database query language for all
platforms, incl. the data lake
One codebase independent of
deployment strategy, Optimization
done once
Transfer of resources between
different teams
Lower TCO
Training (dev & ops) to be done once
Efficiency of scale for operational costs (one team vs.
platform-dedicated teams)
Time-to-Value for new features is the same on all
platforms
One agreement across all platforms,
same support team
High Security
Secure by default on different cloud providers
Reusable security concept on multiple
platforms
Encryption (At Rest, In Use, In Flight),
Authentication/Authorization, Auditing
Low Operational Effort
Same operations tooling and APIs
independent of platform
Same scaling approach and reusable
integrations across all platforms
Technical Support by the same Service
Engineers independent of platform

Modelling a Basic “Thing”
Unique Identiﬁer
Title, Description
Creation, Modiﬁcation Date
Base Thing
Relational Schema

Enriching a “Thing” Requires more Tables
Unique Identiﬁer
Title, Description
Base Thing
Properties & Schema
Actions & Input / Output
Events & Schema / Subscription /
Cancellation
Relational Schema

Further Enrichment Creates Complex Schemas
Unique Identiﬁer
Title, Description
Base Thing
Properties & Schema
Actions & Input / Output
Events & Schema / Subscription /
Cancellation
Translations
Security Schemes
Form Representation
User-Deﬁned Data ? Relational Schema

Strict Schema vs. Flexible Data Model
id title description
12345-WoTLamp-1234 My Lamp A lamp in the room
67890-WoTLamp-1234 My Other Lamp Another lamp in the room
id thing_id title readOnly writeOnly
4711-p1 12345-WoTLamp-1234 status false false
id thing_id title safe idempodent
4711-a1 12345-WoTLamp-1234 toggle false false
id thing_id title readOnly writeOnly
4711-e1 12345-WoTLamp-1234 overheating false false
Thing
Property
Action
Event
{
"id": "123456-WoTLamp-1234",
"title": "My Lamp",
"description": "A lamp in the room",
"properties": {
"status": {
"type": "string",
"readOnly" : false,
"writeOnly" : false
}
},
"actions": {
"toggle": {
"safe": false,
"idempodent": false
}
},
"events": {
"overheating": {
"data": {
"type": "string",
"readOnly" : false,
"writeOnly" : false
}
}
}
}
Relational Example Flexible Document Model

Extensible at Runtime
{
"id": "123456-WoTLamp-1234",
"title": "My Lamp",
+ "properties": { ... },
"actions": {
"toggle": {
"safe": false,
"idempotent": false
}
},
+ "events": { ... }
}
{
"id": "123456-WoTLamp-1234",
"title": "My Lamp",
"securityDefinitions": {
"basic_sc": {
"scheme": "basic",
"in": "header"
}
},
+ "properties": { ... },
"actions": {
"toggle": {
"safe": false,
"idempodent": false,
"forms": [{
"op": "invokeaction",
"href": "https://mylamp.example.com/toggle",
"contentType": "application/json"
}]
}
},
+ "events": { ... }
}
Adding Security
and Action
Information
Schema-Free based on well-deﬁned standards like
JSON-LD and W3C’s Web of Things Vocabulary

What about Timeseries?
Schema Design Pattern: Bucketing
https://www.mongodb.com/collateral/time-series-best-practices

Telediagnostics: The Future of Mercedes Benz Services
Learn more about the project in Madalin Broscaru’s MongoDB.live presentation:
Telediagnostics@Mercedes Benz powered by MongoDB
… the Vehicle Data
Conditioning (VDC) where
these technical vehicle events
are processed …
CAC
Retail
Customer
… and the follow-up processes are
triggered with real-time
recommendations for actions.
Vehicles are transmitting regularly
status and health data into ...
Aggregated
Quality Analysis

Telediagnostics: The Future of Mercedes Benz Services
{
.............
"schema": "3.1.0"
"createdAt": {..},
.............
"vehicleIdentData": {
"chassisNumber": "WDD24708A5432J63",
"countryCode": "4f3490b14e238a5f",
"modelSeries": "f16ad22d42064811",
"modelType": "2196868af1c70d74",
"modelYear": "5e01ac15d73c3e4a",
"steering": "4a3424fe6411461c"
},
"basicData": {
"mileage": {..},
"batteries": [..],
"tanks": [..],
"tiresPressure": [..],
},
"controlUnits": [..],
"affectedFunctions": [..],
"vehicleClusterMessagesData": [..],
"maintenanceData": {...},
..............
}
"controlUnits": [
{
"ecuId": {..}
"name": "a927e49b0549f00f71",
"detailsHardware": {},
"detailsSoftware": {},
"dtc": [
{
"code": "B214F73",
...
"failureText": "81957650ea",
"environmentalData": {...}
},
...
]
},
...
]

Or: How does a solution look like? Show me the architecture and the code!
Fleet Management Demo

IoT is broad - let’s look into a specific example
Source of Industries: IHS Markit
Fleet Management
Covers fleets of commercial vehicles, fork lifts,
trains, goods, literally anything that forms a fleet
and needs to be monitored and actively managed.
Major Challenges
High distribution of fleet, unstable network
connections, multitude of device types and data
structures, scaling from 100’s to 1,000’s to 1,000,000s
Typical Benefits
Reduction of shipping costs, goods arrive in time, less
outages and damages, reaction times in minutes
instead of hours, CO2 reporting and reduction, higher
fleet utilization, compliance with legal requirements

Getting Started with MongoDB & HiveMQ
MongoDB Atlas
Free Tier Cluster
Retention ½ Day
HiveMQ
vehicles/trucks/truck-XXXXX
MQTT
Subscriber
Python
Trucks on tour
sending location,
speed, break time
MQTT
Subscriber
Python
Truck Simulator:
- >9000 Warehouses across Germany
- Trucks travel between random warehouses
- Every second, the current location, speed, speed
limit and break information is sent
Real-Time Position of Trucks:
- Subscription to MQTT topics
for live visualization
Analysis of Trucks:
- Visualization of historical
data based on truck routes

Or: What about X, Y and Z?
Q&A

Get Started Today!
MongoDB Atlas
https://cloud.mongodb.com
HiveMQ
https://hivemq.com/
We love to hear your feedback!
Dr. Christian Kurze | Principal Solutions Architect | christian.kurze@mongodb.com
Dominik Obermaier | CTO and Co-Founder | dominik.obermaier@hivemq.com

Building a reliable and scalable IoT platform with MongoDB and HiveMQ

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Building a reliable and scalable IoT platform with MongoDB and HiveMQ

Similar to Building a reliable and scalable IoT platform with MongoDB and HiveMQ (20)

More from Dominik Obermaier

More from Dominik Obermaier (16)

Recently uploaded

Recently uploaded (20)

Building a reliable and scalable IoT platform with MongoDB and HiveMQ