Azure IoT and data basics

Basics on Azure IoT and data
processing
ECSCIA, European Centre of Supply Chain Information
Architecture

Architecture
The intention with this presentation is to describe the basic building
blocks in the Microsoft Azure IoT landscape on a level that allows for
understanding of concepts before proceeding into discussions usage
scenarios in supply chains.
The presentation holds snapshot references to products that are in
continuous change.

The IoT chain starts with
some kid of sensing device…
…..doing some kind of
processing and
communication
In documentation on Azure IoT, this is often represented by a
Raspberry PI running Windows IoT core, equipped with sensors and
hooked up to the internet using Wi-Fi.
It´s a cheap and (very) easy way to get prototyping started
Architecture
https://developer.microsoft.com/en-us/windows/iot
https://www.microsoft.com/en-us/software-download/windows10IoTCore
https://docs.microsoft.com/en-us/windows/iot-core/tutorials/rpi

Devices sends that data on a
journey to “somewhere”
…..that journey usually
passes a gateway in a “cloud”
on the internet
In Azure IoT, that gateway is “Azure IoT hub”
Architecture
Sensor reads data

Azure IoT hub provides reliability and security to the connection
between devices and the cloud .
It allows for registering and monitoring devices on individual level.
It is where the Device and the module twin lives.
The hub scales to support (very) many devices and (very) many events
Architecture

Direct…
HTTP
MQTT
AMQPS
TLS
Protocols like…
Custom protocol
Devices capable of
connecting through
IP
Devices not capable
of connecting
directly to internet
Field gateway
Other
Protocol
gateway
AMQPS
AMQPS
Azure IoT hub
Architecture
…and indirect connections
MQTT
Custom protocol
For depth & details , study “IoT solutions in Microsoft Azure IoT
suite” (Scott Klein) and https://docs.microsoft.com/en-us/azure/iot-
hub/about-iot-hub

Act“Hot path”
“Cold path” Act
Architecture

Setting up an IoT hub
…...A Microsoft
Azure subscription
Need to have http://portal.azure.comThen go to
Create a hub and assign it
Name
Pricing and scale tier
Resource group
Location
…………
Architecture

Two central objects in the IoT hub model are
“Device” and “Message”
The hub knows things about devices
The hub receives messages from devices
The hub sends messages to devices
Architecture

Cloud to device settings
Default TTL (Time To Live) : How long a message is available to the
Feedback retention time : How long he hub will retain feedback
Maximum Delivery Count : Number of times the hub will attempt to deliver a message
Architecture

Device to cloud settings
Partitions : Number of partitions for events
Event hub compatible name : Specifies name used when reading from event hub
Event hub compatible endpoint : Endpoint used when reading from event hub
Retention time : Time the hub will maintain device to cloud events
Consumer groups : Used by applications to extract data from the hub
Architecture

IoT Hub
Event Hub
Two way communication.
File upload
Very large (“unlimited”) number of
devices
Configurable routing of messages
Device level events
One way communication.
No file upload
High but limited number of devices
Aggregated metrics
Coded routing of messages
Architecture
https://docs.microsoft.com/en-us/azure/iot-hub/iot-hub-compare-event-hubs

Consumer
Reads data from the IoT Hub or the
event Hub
Consumer group
A opening to extract data from the
hub. Provides a consumer a own view
of the event stream
Architecture

Azure IoT Protocol gateway
Adapter for a variety of protocols.
Allows for use of custom protocols
Architecture

Device management Device lifecycle
Plan
Provision
ConfigureMonitor
Retire
Architecture
https://docs.microsoft.com/en-us/azure/iot-hub/iot-hub-device-management-overview

Plan Prepare metadata scheme (tags and properties) stored in the device. Allows for bulk
device management operations
Provision Provision (onboards) new devices to the IoT hub
Configure Configuration (and firmware) updates/settings to devices (in bulk)
Monitor Monitor the state (“health”) of devices
Retire Decommission (possibly replace) devices
Architecture
Plan
Provision
ConfigureMonitor
Retire

Monitoring categories
Device identity operations : Errors on the device identity registry
Device to cloud communication : Errors device to cloud
Cloud to device communication : Errors cloud to device
Connections : Errors when device connects/disconnects with the hub
Architecture

Device twins are representations
of the device instance stored in
JSON documents in the Azure IoT
hub. The documents hold
configuration and state of the
real world device
Device instances are the real
devices existing in the physical
world
Architecture

Module twins are JSON
documents storing state and
configuration of a module
A module is a individual
connection to the IoT Hub. A
device might have multiple
sensors, each might be assigned
to a separate module.
Architecture

Azure IoT SDK
Architecture
Device explorer
The Device explorer allows for managing devices and monitoring messages from device to
cloud and from cloud to device.
It´s a stand alone, client tool that connects to the IoT hub.
https://github.com/Azure/azure-iot-sdk-csharp

Architecture
Device explorer IoT Hub
Connection string
(Primary key)
Is fetched
from
Azure portal
IoT Hub
Using
Connects
to
Shared access policies

Architecture
Device explorer
IoT Hub
Connection string
Device
Is registered
in
Using
Device
Assigns
to

Architecture
Goes beyond this presentations. Read a book or go surfing for topics like:
“deviceClient.CreateFromConnectionstring”:
https://docs.microsoft.com/en-
us/dotnet/api/microsoft.azure.devices.client.deviceclient.createfromconnectionstring?view=azure-dotnet
“deviceClient.SendEventAsync” :
https://docs.microsoft.com/en-
us/dotnet/api/microsoft.azure.devices.client.deviceclient.sendeventasync?view=azure-dotnet
“UploadToBlobAsync”
Etc, etc…..
The client programming stuff….

Links
• https://docs.microsoft.com/en-us/azure/iot-hub/iot-hub-device-
management-overview
• Azure training modules: https://docs.microsoft.com/en-
us/learn/browse/?products=azure
• https://azure.microsoft.com/en-us/overview/iot/products/
• https://discover.microsoft.com/azure-iot-building-solutions-dev-
guide/
Architecture

Architecture
Document
Azure blob storage
Document
Event
Device to cloud payload
Event
Retention : 1-7 days
IoT hub
Retention : Long

Architecture
Ok, so now streams of data are flowing
in from devices to the hub….
…..maybe some “Azure Stream Analytics”?
Event Event Event Event Event Event
IoT hub
……What to do with those streams?....

Architecture
Azure stream analytics
Real time
analysis
Event
Azure stream analytics
Micro batch
processing

Architecture
Azure stream analytics job
Azure portal
Azure stream
analytics job
Configure a
Inputs Metrics OutputsDefine
Alerts
Events

Architecture
It´s just the same old….
Ingest Prepare Analyze Publish Consume
Collect
data
Transform
data to be
usable in
the
following
context(s)
Process
data to
find
meaning in
it
Make data
available
Use data

Architecture
Inputs
Outputs
Inputs specify where the Azure stream
analytics job fetches data from
https://docs.microsoft.com/en-us/azure/stream-
analytics/stream-analytics-add-inputs
IoT Hub
Event Hub
Azure Blob Storage
Data streams
Reference data
Outputs specify where the Azure stream
analytics puts data to
https://docs.microsoft.com/en-us/azure/stream-
analytics/stream-analytics-define-outputs
Azure data lake store
Azure SQL Database
Azure Blob Storage
Event Hub
Power BI
Table storage
Service bus queues
Service bus topics
Azure Cosmos DB
Azure Functions

Architecture
Streaming Unit
A way of quantifying stream analytics
workload/throughput.
Combination of CPU, I/O and memory.
https://docs.microsoft.com/en-us/azure/stream-analytics/stream-analytics-streaming-unit-consumption

Architecture
Event 1 Event 2 Event 3 Event 4
Events can arrive out of order
(relative their timestamp)
Events can arrive late
(relative their timestamp)
Rules can be set up in Azure Stream
Analytics to deal with that

Architecture
Storing for further processing
Event
Azure Stream Analytics Azure Data Lake Store

Architecture
Input OutputStream Analytics Query
A Stream Analytics Query…
“Code similar to SQL
Operating on the input
stream and outputting
to the Output stream”

Architecture
Input OutputStream Analytics Query
Window
Tumbling : Fix size, none overlapping
“Windows” are built in technologies for aggregation of events
Hopping : Fix size, overlapping
Sliding: Fix size, Moving start point

Architecture
Now we have data
streams…
…Other cloud data
sources…
…and other on prem
data sources…
…And we want to
bring them
together…
Azure Data Factory

Architecture
Azure Data Factory
“A data integration service”
Activity
Actions to be performed
on data
Pipeline
A chain of activities
Dataset
A reference to the
data the activity
operates on
Linked Service
Definition of a
connection to an
external data source
…defined/Glued together using JSON…

Linked serviceLinked service
Architecture
Input
dataset
Output
dataset
Pipeline
“A pipeline is a logical
grouping of activities that
together perform a task”
Activity
Activity
Activity
https://docs.microsoft.com/en-us/azure/data-factory/concepts-pipelines-activities

Architecture
Azure Pipelines are set up in azure Data
factories. The pipeline is kind of a Digital
production line, containing “equipment”
represented as activities refining and moving
digital goods
Pipeline
Linked serviceLinked service
Input
dataset
Output
dataset

Architecture
Azure portal
Azure Data
factory
Configure a
Create a Pipeline By selecting Datastore
Linked Service
Creates a..

Architecture
Visual studio
Create a
Azure Data
Factory project
Create a Linked Service
Create a
Template
From a..
Table
(Data set)
Template
From a..
Create a
Table
(Dataset)
Template
From a..

Architecture
https://docs.microsoft.com/en-us/azure/data-factory/concepts-pipelines-activities
Activity
Data Movement activity
Data transformation activity
Control activity

Architecture
“Compute” Linked service
A “compute” linked service is a service that processes / transforms data. It
can be called from the data factory
HD Insight Cluster
Azure Batch
Azure Machine Learning
Azure Data Lake analytics
Azure SQL, Azure SQL Datawarehouse, SQL server stored procedures
Azure Data Bricks
https://docs.microsoft.com/en-us/azure/data-factory/compute-linked-services

Architecture
Azure portal
Azure Data
factory
Create a
“Compute”
Linked service

Architecture
Monitoring Metrics
https://docs.microsoft.com/en-us/azure/data-factory/monitor-visually
events and activities in the Data Factory can be monitored, metrics
analyzed and alerts set
Alerts

Architecture
Azure Data Lake Store
(Generation 2)
https://azure.microsoft.com/en-us/services/storage/data-lake-storage/
https://azure.microsoft.com/en-us/blog/under-the-hood-performance-scale-security-
for-cloud-analytics-with-adls-gen2/
Hierarchical file storage
Blob storage
(REST) APIs
Azure Data Lake Store
Simple version : It´ resides in the cloud and you can
Use it to store limitless amounts of (big) data in any
size and any type. It´s optimized for analytics.
The line between
Hierarchical file
storage
and BLOB storage
is blurred in
ADLS Generation
2.

Architecture
Azure Data Lake Store scales up (and down) to hold
very, very large data volumes

Architecture
Azure portal
Azure Data Lake
Store
Create a
Storage accountCreate a
Hierarchical
namespace
Enable

Architecture
Programmatic access : Azure Data Lake Store and Visual Studio
Goes beyond this PPT,
Check out :
https://www.nuget.org/packages/Microsoft.Azure.Management.DataLake.Store
https://www.nuget.org/packages/Microsoft.Azure.Management.DataLake.StoreUploader/1.0.1-preview
https://www.nuget.org/packages/Microsoft.Rest.ClientRuntime.Azure.Authentication
https://github.com/Azure-Samples/data-lake-store-adls-dot-net-get-started

Architecture
Basic access control: IP range filters
IP
IP
IP
IP
IP
IP
IP
IP
IP
IP
ADSL

Architecture
Identification and authorization
Azure Active Directory
Identify
ADSL
Role based
access
Data access
RBAC POSIX ACL
https://docs.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal
ACL: Access Control List.
Specifies permissions for
users and groups

Architecture
Azure Identity and access management
https://azure.microsoft.com/en-us/product-categories/identity/
Azure Active Directory
Azure Active Directory B2C
Azure Active Directory Domain services
https://azure.microsoft.com/en-us/services/active-directory/
https://azure.microsoft.com/en-us/services/active-directory-b2c/
https://azure.microsoft.com/en-us/services/active-directory-ds/

Architecture
Looking at a speedometer in one vehicle at a specific moment in time provides
information to support decisions to adjust speed at that moment
Looking at collected, historical information from the vehicle supports decisions
to adjust driver behavior, define routes, set service cycles for the vehicle etc.
Looking at collected, historical information from multiple vehicle allows for
understanding the whole traffic system which supports decision on a holistic
level.
To build understanding from data sensed in systems of significant size &
frequency requires an ability to do analytics processing on very large sets of
data
Azure Data Lake Analytics
HDInsight

Architecture
U-SQL
Goes beyond this PPT, tale a look at:
https://docs.microsoft.com/en-us/azure/data-lake-analytics/data-lake-analytics-u-sql-get-started
https://saveenr.gitbooks.io/usql-tutorial/content/
“kind of C# and T-SQL
brought together for
analytical processing”

Architecture
Azure portal
Create a
Azure Data Lake
Account
https://docs.microsoft.com/en-us/azure/data-lake-analytics/data-lake-analytics-get-started-portal
Create a U-SQL Job
Submit a U-SQL Job

Architecture
Programmatic access : Azure Data Lake Analytics and Visual Studio
Goes beyond this PPT,
Check out :
https://docs.microsoft.com/en-us/azure/data-lake-analytics/data-lake-analytics-data-lake-tools-install
https://docs.microsoft.com/en-us/azure/data-lake-analytics/data-lake-analytics-data-lake-tools-get-started
https://www.nuget.org/packages/Microsoft.Azure.Management.DataLake.Analytics

Architecture
HDInsight
A cloud based Hadoop implementation.
HDInsight provides Hadoop capabilities
Without the hassle of setting up
Hadoop on prem.
Spark
Hive
Storm
Kafka
……..

Architecture
Azure Machine Learning
2let the machine take (propose) decisions based
on statistics2
Data
Machine learning
model
Machine learning
web service
Define/tune
algorithm
Apply/use
algorithm

Architecture
Azure portal
Create a
Azure Machine
Learning
Workspace
Launch
Machine
Learning Studio
Upload Data
Publish
Model
(webservice)
Build an “Experiment”

Try it out….Sign up for a free tier workspace
Architecture
https://azure.microsoft.com/en-us/free/machine-
learning/search/?&OCID=AID2000118_SEM_tfDDMLFR&MarinID=tfDDMLFR_340807131738
_%2Bazure%20%2Bmachine%20%2Blearning_b_c__66077640120_kwd-
73504516228&lnkd=Google_Azure_Brand&dclid=CPK69pHh6OMCFd7Fsgod4l8HJg
https://studio.azureml.net/
Web service API sample
https://studio.azureml.net/apihelp/workspaces/1e3e94c0e29346838b1b154c7ffd8c58/webs
ervices/fc37773e2aec4a58be09d5593bd68024/endpoints/1623f0d70c074507b314e83891ef
2411/score

Architecture
https://azure.microsoft.com/en-in/services/iot-central/
Azure IoT central
AzureIoTcentral.com
Kind of wraps it all up in a simpler UI

Azure IoT SDKs
Architecture
https://github.com/Azure/azure-iot-
sdks?&ocid=mkto_eml_EM497281A1LA1&mkt_tok=eyJpIjoiWlRnM01tSTFOV1poWlRkaCIsInQiOiJVT2p1Rzg
wbWVERzdiY2FvRVwvVU5mWk80XC9HZGludng4bllFT1JPYk12eThFNFhUaG1KN0Y3MW56ZXplTEtvUmZUWG
MyRlNnQ0szdHJ2T1VpcFRSNWlGaEtjUTErVVVsXC9oYzZXSVdtMXBOVUVGS3ZhWGRDbE1JZlhGN0ZvOVZVTkZ
xT2ZEQTZhQW9vRW1yZkNjTndSUXc9PSJ9

Azure IoT and data basics

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Azure IoT and data basics

Similaire à Azure IoT and data basics (20)

Plus de Stefan Harrstedt

Plus de Stefan Harrstedt (20)

Dernier

Dernier (20)

Azure IoT and data basics