5. What We Need
• An integrated data solution that will be:
– Able to process events from external sources
– Able to walk data through different pipelines
– Fast and responsive
– Big-Data ready
6. In Other Words
Consume
BI Dashboards Applications
Process
ETL Aggregations Computation Analysis Querying
Persist
Hadoop SQL NoSQL
Ingest
Structured Data Un-Structured Data
7. Microsoft Azure Services for
IoT and BigData
Devices Device Connectivity Storage Analytics Presentation & Action
Event Hubs SQL Database
Machine
Learning
App Service
IoT Hub
Table/Blob
Storage
Stream Analytics Power BI
External Data
Sources
DocumentDB HDInsight
Notification
Hubs
External Data
Sources
Data Factory Mobile Apps
BizTalk Services
{ }
8. Microsoft Azure Services for
IoT and BigData
Devices Device Connectivity Storage Analytics Presentation & Action
Event Hubs SQL Database
Machine
Learning
App Service
IoT Hub
Table/Blob
Storage
Stream Analytics Power BI
External Data
Sources
DocumentDB HDInsight
Notification
Hubs
External Data
Sources
Data Factory Mobile Apps
BizTalk Services
{ }
9. Field
Gateway
Device
Connectivity & Management
Analytics &
Operationalized Insights
IoT & Data Processing
PatternsDevices
RTOS,Linux,Windows,Android,iOS
Protocol
Adaptation
Batch Analytics & Visualizations
Azure HDInsight, AzureML, Power BI,
Azure Data Factory
Hot Path Analytics
Azure Stream Analytics, Azure HDInsight Storm
Hot Path Business Logic
Service Fabric & Actor Framework
Cloud Gateway
Event Hubs
&
IoT Hub
Field
Gateway
Protocol
Adaptation
14. Azure Stream Analytics
• Automatic recovery
• Monitoring and alerting
• Scale on demand
• Managed Cloud Service
• Each unit handles 1MB/s
• Can scale up to 1GB/s
• SQL like language
• temporal windowing
semantics
• support for reference data
15. Tumbling Windows
• How many vehicles enter each toll booth
every 5 minutes?
SELECT TollId, COUNT(*) FROM EntryStream
GROUP BY TollId, TumblingWindow(minute,5)
17. What is Azure Data Factory?
Azure Data Factory is a managed service to
produce trusted information from data stored in
the cloud and on-premises. Easily create,
orchestrate and schedule highly-available, fault
tolerant work flows to move and transform
your data at scale.
18. Evolving Approaches to
Analytics
ETL Tool
(SSIS, etc)
EDW
(SQL Svr, Teradata, etc)
Extract
Original
Data
Load
Transformed
Data
Transform
BI Tools
Ingest
Original
Data
Scale-out
Storage &
Compute
(HDFS, Blob
Storage, etc)
Transform & Load
Data Marts
Data Lake(s)
Dashboards
Apps
Streaming data
19. Data Factory Concepts
Call Log
Files
Customer
Table
Customer
Churn
Table
Customer
Call
Details
Customers
Likely to
Churn
21. DocumentDB and Azure Data
Services
fully managed, scalable, queryable, schema free JSON
document database service for modern applications
transactional processing
rich query
managed as a service
elastic scale
internet accessible http/rest
schema-free data model
arbitrary data formats
25. Azure IoT Hub vs Event Hub
Area IoT Hub Event Hubs
Communication
patterns
Device-to-Cloud
Cloud-to-Device
event ingress (device-to-cloud)
Device protocol
support
AMQP, AMQP over WebSockets, HTTP, and
MQTT
AMQP, AMQP over WebSockets, and HTTP
Security Per-device identity and revocable access
control
Event Hubs-wide shared access policies, with
limited revocation support
Operations
monitoring
Rich set of device identity management Only aggregate metrics
File upload File notification endpoint for workflow
integration
Manually request files from devices
Scale Millions of simultaneously connected
devices
Limited number of simultaneous connections--
up to 5,000 AMQP connections
Device SDKs C, Node.js, Java, .NET, Python Java, .NET
C and Node.js in preview
27. Resources
• Azure IoT Dev center
– http://www.azure.com/iotdev
• Azure Services
– https://azure.microsoft.com/en-us/services/event-hubs
– http://azure.microsoft.com/en-us/services/iot-hub
– https://azure.microsoft.com/en-us/services/stream-analytics
– https://azure.microsoft.com/en-us/services/data-factory
– https://azure.microsoft.com/en-us/services/documentdb
– https://azure.microsoft.com/en-us/services/hdinsight
• Microsoft and IoT
– https://www.microsoft.com/en-us/cloud-platform/internet-of-things
– https://blogs.microsoft.com/iot/
• My Info
– @IdoFlatow // idof@sela.co.il // http://www.idoflatow.net/downloads
Notes de l'éditeur
Key goal of slide:
IoT as you know is a hot area these days and there are a number of players that claim to be active in this space…. And they tend to focus on specific elements you see in this diagram.
Microsoft has the most comprehensive portfolio of cloud services that customers need to develop and deploy end-to-end IoT solutions.
Customers are adopting these services and are successfully deploying their solutions today (reference Rockwell, ThyssenKrupp)
Talk track [Short Version for Sam’s Leadership Session]:
As we think about Azure IoT services, Microsoft has the most comprehensive portfolio of cloud services that customers need to develop and deploy end-to-end IoT solutions
Ranging from devices that produce data, to connecting them to the cloud storage, and driving analytics to gain valuable business insights that allows enterprises to take actions
Talk track [Long Version Chris’ Breakout Session]:
As we think about Azure IoT services, there are a collection of capabilities involved.
First there are Producers. These can be basic sensors, small form factor devices, traditional computer systems, or even complex assets made up of a number of data sources.
Next we have the Connect Devices capabilities on the ingress level within and around Azure. The primary destination is Service Bus & Event Hubs, but this relies on client agent technology either at the edge device level or within a field or cloud gateway. We also have capabilities for other external data sources o provide data
As data is ingressed to Azure, there are various Storage options there can be a number of destinations engaged. Traditional database technology, table or blob, or even more complex destinations like Document DB are possible. External or third party technologies can also be used. This is where the flexibility and agility of a platform shows its strength, This is where analysts like Gartner are forming opinions about just how robust our platform can be.
As this data is processed in Azure, there are a number of capabilities that can be utilized. Machine Learning, HD Insight, Stream Analytics are examples of tools that can analytics the data in various ways.
Finally the concept of Take Actions uses Azure services. Data may populate a LOB portal, be pushed to apps, or presented in analytics and productivity tools. These are all ways that the data gets out of these architecture points to allow organizations to use analysis to change / transform their business.
Through all of these areas, there is the possibility of utilizing existing investments either within your Azure environment, or elsewhere.
Key goal of slide:
IoT as you know is a hot area these days and there are a number of players that claim to be active in this space…. And they tend to focus on specific elements you see in this diagram.
Microsoft has the most comprehensive portfolio of cloud services that customers need to develop and deploy end-to-end IoT solutions.
Customers are adopting these services and are successfully deploying their solutions today (reference Rockwell, ThyssenKrupp)
Talk track [Short Version for Sam’s Leadership Session]:
As we think about Azure IoT services, Microsoft has the most comprehensive portfolio of cloud services that customers need to develop and deploy end-to-end IoT solutions
Ranging from devices that produce data, to connecting them to the cloud storage, and driving analytics to gain valuable business insights that allows enterprises to take actions
Talk track [Long Version Chris’ Breakout Session]:
As we think about Azure IoT services, there are a collection of capabilities involved.
First there are Producers. These can be basic sensors, small form factor devices, traditional computer systems, or even complex assets made up of a number of data sources.
Next we have the Connect Devices capabilities on the ingress level within and around Azure. The primary destination is Service Bus & Event Hubs, but this relies on client agent technology either at the edge device level or within a field or cloud gateway. We also have capabilities for other external data sources o provide data
As data is ingressed to Azure, there are various Storage options there can be a number of destinations engaged. Traditional database technology, table or blob, or even more complex destinations like Document DB are possible. External or third party technologies can also be used. This is where the flexibility and agility of a platform shows its strength, This is where analysts like Gartner are forming opinions about just how robust our platform can be.
As this data is processed in Azure, there are a number of capabilities that can be utilized. Machine Learning, HD Insight, Stream Analytics are examples of tools that can analytics the data in various ways.
Finally the concept of Take Actions uses Azure services. Data may populate a LOB portal, be pushed to apps, or presented in analytics and productivity tools. These are all ways that the data gets out of these architecture points to allow organizations to use analysis to change / transform their business.
Through all of these areas, there is the possibility of utilizing existing investments either within your Azure environment, or elsewhere.