The document discusses using an IoT analytics platform for measurement and knowledge extraction from big data in IoT. It covers: 1. The business opportunity of IoT with over 10 billion connected devices generating data that can improve understanding and decision making. 2. An IoT analytics platform that ingests, processes, stores, analyzes and publishes/visualizes data from diverse sources. It discusses modules for integration, processing, machine learning, APIs, dashboards and more. 3. Two use cases - energy segmentation using clustering algorithms on consumption data, and traceability in distribution using real-time monitoring of deliveries.

1. From measurement to knowledge
Big data and real time analytics in the IoT
http://twitter.com/SOFIA2_Platform
http://about.sofia2.com
plataformasofia2@indra.es
http://sofia2.com

2. 2
Raquel López
IoT Expert Manager
Sofia2 IoT Platform
Presentation
raquel-lopez-alarcon
rlalarcon@minsait.com
@rlalarcon

3. 3
1. Business Approach to IoT
2. IoT Analytics Platform
3. The Technical Part….
We will talk about…

4. 4
Business Approach to IoT
Section 1

5. 5
IoT as a Business Opportunity
More than
10 billion of
connected
devices
Plus 10-20
billions to add in
the next 10 years
Only 1% o fthe
data is actually
used (as
alerts, event
triggering….)
 Devices generate a continuous flow of data whose acquisition
and analysis will allow us a better understanding of reality and
an improvement and faster decision making.
 We will have a volume of data without coming, which will allow us
to extract an extensive knowledge of the client and with this to
offer the customer custom solutions to measure and participate
in their day to day
Internet Of Things
terminology was used
for the first time on 1999
referring to identifiable
objects (things) and its
vistual representation in
an infrastructure.
Business Processes Transformation
New Business Models

6. 6
Why using IoT?
The Goal
Making better and faster business decisions by the combination of real-time process and
analysis of the information.

7. 7
IoT Mainstays
Integration & Connectivity Process and Analytics
IoT Flow Social flow General
SystemsDevices
…
RRSSAPIs G.Analytics PIWIK
Y.Finance …
“Interoperability between IoT systems is critical. Of the total
potential economic value the IoT enables, interoperability is
required for 40 percent on average and for nearly 60
percent in some settings.” McKinsey
“Currently, most IoT data are not used. For example, on an
oil rig that has 30,000 sensors, only 1 percent of the data
are examined. That’s because this information is used
mostly to detect and control anomalies—not for optimization
and prediction, which provide the greatest value.” McKinsey

8. 8
Types of IoT Analytics
Descriptive
Predictive
Prescriptive
Pushes and Alerts
Embedded Models
Streaming
Realtime Analysis
What question to answer? What method to apply?
Combining both perspectives we can get many applicable use-cases

9. 9
IoT Analytics usecases
Forbes

10. 10
IoT analytics platform
Section 2

11. 11
IoT Platform Capabilities

12. 12
IoT Bussiness Solution Referential Model

13. 13
Administration, Security, Monitoring, Control
Business apps
Custom apps
Sensors and devices
Storage Publish
People
Third Party
Systems
INGEST
Bulk, Event Ingestion
PROCESS
Preparation, Anaytics
VISUALIZE
IoT Platform Information Flow

14. 14
IoT Broker
 Light communication
Protocols: REST,
MQTT, WS, etc…
 Development APIs &
SDKs
 Real time processing.
Rules and flows
 Semantic model
Real Time
Repository
Big Data Repository
Staging Area
Notebooks:
• Algorithm
Execution
• Schedulling
• Data Scientist
collaborative
environment
• Exports and
Visualization.
Machine Learning:
• Algorithm
Execution
• Parametrization
• Publication
• Machine Learning
inference
API Manager:
Service Publication
and lifecycle
management
Dashboards &
synoptics
Diifferent options to
configure
dashboards for
realtime & analytics
visualization of the
data
Open Data
Portal:
Opendata
publication in
different standard
formats.
Data Links:
ODBC & JDBC
Access to data
Data Flow
 Visual ETL style
model.
 Bacth ingestión.
 Monitorization
 Stadistics & graphics
 Heterogeneous
sources of information
IoT Platform Functional Capabilities

15. 15
Platform Modules
Discovery
Apps & VerticalIoT Flow
Social flow
General
SOURCES INGEST & PROCESS STORE ANALYZE PUBLISH & VIEW
SystemsSDKs
Java Javascript
Python Android iOS
.Net R C(C++ …
Devices
…
RRSSAPIs
G.Analytics PIWIK
Y.Finance …
IoT
Gateway
IoT
Broker
Ontology
Stream
Process
Sofia2 Storage
Real Time Database
Notebooks API
Manager
DataLink
HTML5 & API
MANAGEMENT
CONFIG
DB
Sofia2
Control Panel
Security
Historical Database
Staging Area ODBC
JDBC
Dashboards
Ontology
APIs
VerticalSystemsAppsExplotationVisual
Plugins
REST
Ext APIs
Open Data
Portal
Synoptics
CEP
Ontology
Flows
MQTT
… APIs
Rules
Bots
ML Engine
SPACES
DEVICES
Devices
SEMANTIC
Custom
Integration
DataFlow
BigData Ingestion

16. 16
The technical part…
Section 3

17. 17
Energy Segmentation
Context / complexity
• Electricity consumption information retrieval
• Segments for the organization of consumption patterns / Period of the
customer set.
• Consumer recovery and trend analysis.
Technological solution
• Collection and storage of sources: Ingestion in HDFS.
• Data exploration: Understanding its distribution, quality, volumetry.
They are data of hourly curves of 6 months.
• Cleaning of data and selection of relevant information.
• Indicators generation by CUP and Period.
• Inference of new data to complete dataset.
• Application of K-means algorithm and elbow technique for selection
of K.
• Analysis of results

18. 18
Energy Segmentation
Apps & VerticalIoT Flow
Social flow
General
SOURCES INGEST & PROCESS STORE ANALYZE PUBLISH & VIEW
SystemsSDKs
Java Javascript
Python Android iOS
.Net R C(C++ …
Devices
…
RRSSAPIs
G.Analytics PIWIK
Y.Finance …
Stream
Process
Sofia2 Storage
Real Time Database
Notebooks API
Manager
HTML5 & API
MANAGEMENT
CONFIG
DB
Sofia2
Control Panel
Historical Database
Staging Area
JDBC
Dashboards
Ontology
APIs
VerticalSystemsApps
Ext APIs
CEP
Ontology
Flows
APIs
Rules
ML Engine
SPACES
DEVICES
SEMANTIC
DataFlow
1
2
3 4
5

19. 19
INGEST / STORE ANALYZE PUBLISH & VIEW
Capabilities
Visual ETL modelling. Java extensible. Integrated in Sofia2
Control Panel
(HTML5)
Debug and Preview. Sources & destinations:
Kafka, Hadoop, JMS,
JDBC, AmazonS3…
Alerts and
Notifications
Deployment integrated
on Control Panel.
18 available processors
for ETL modelling: Split,
hasher, remover, merger…
Charts and execution
statistics visualization
Energy Segmentation

20. 20
Energy Segmentation
INGEST / STORE ANALYZE PUBLISH & VIEW
Optimized for and agile and real time access to the
information.
Real Time Data Base
Optimized not the storage and management of big
Data, not so often accessed.
Historical Data Base
Capabilities
One repository per each
moment on the lifecycle
of the information.
Support to different
technologies depending
on the information access
pattern.
Standard support and SQL
& NO-SQL databases
Access to the information
with optimized
response times.
Horizontal Scalable. Different repositories
are integrated between
them and with other
modules of the platform
Each entity of the data model have a
configured time-window, in order to
identify the data stored as historic or real-
time information
Information will be persisted in the real-
time database until it would be
automatically moved to the historical
database.
Any information (real-time or historical)
will be accessible from any of the other
modules of the platform: Integration,
Machine Learning, APIManager
Differential
Staging Repository
For structured, semi-structured and unstructured
data, as a temporary holder in case of transform
processess requiring persistence.

21. 21
Energy Segmentation
INGEST / STORE ANALYZE PUBLISH & VIEW
Capabilities
Interpreter Management
(sh, jdbc, md, angular…)
Shared and
multiuser environment
Data Scientist IDE.
Multilenguaje execution
on the same notebook
(SparkSQL, R, Hive, Scala,
Python)
Instant visualization of
results with
incorporated charts
Invocable from
other platform
moduels such as
Realtime
Processing Module.
Schedulled execution of Web environment
integrated in the
Control Panel of the
platform
Export and
visualization

22. 22
Energy Segmentation
INGEST / STORE ANALYZE PUBLISH & VIEW
Capabilities
Configurable Gadgets
with integration and
querying capabilities to
the Repositories
Configurable
Dashboards reusing
gadgets, with drilldown
and cascade
dependencies, URL
exposition and CSS
customization
.
Dashboards

23. 23
Energy Segmentation
YouTube
Sofia2 IoT Platform
• Demos and viewers
• Sofia2 presentations
• Workshops and
tutorials
• Dashboards
• Technical support
• Solutions built on
Sofia2

24. 24
Traceability in distribution: Executive Dashboards
Context / complexity
• Solution for automatic verification of delivery punctuality, next point arrival
forecast update and temperature ranges verification during stores supply.
• Monitoring and registration of Arrival time to store
• Early warning + Expected arrival time notification to the next store
• Traceability of goods delivered and collected at each store
• Online monitoring of delivery operations
Technological solution
• Real Time Management & Reporting system.
• Black-box (GPS & Mobile Comms.) + Beacons integrated with The Route
Planning system.
• Sofia2 as core for device integration, data analytics, calculations, events
detection and notifications.
Results
• Greater efficiency in distribution, store unloading and platform
loading
• Avoid management conflicts with transportation providers and
internal customers or franchisees
• Compliance with temperature transport regulations
• Reduction in the loss of means and its control, as well as of the
goods in transit
• Improvements in route planning and contracting suppliers
• Improve efficiency in all internal processes of the value chain:
• Verification of load units sent in the correct vehicle
• Control of in route stops + openings control
• Verification of load units left and loaded at each stop
• Selective inventories of accumulated media in stores and
warehouses
• Truthful and timely information for the management of
incidences of distribution or lack of means
a
a
ab b
b
c c
c
(tª) (tª)

25. 25
Traceability in distribution: Executive Dashboards
Apps & VerticalIoT Flow
Social flow
General
SOURCES INGEST & PROCESS STORE ANALYZE PUBLISH & VIEW
SystemsSDKs
Java Javascript
Python Android iOS
.Net R C(C++ …
Devices
…
RRSSAPIs
G.Analytics PIWIK
Y.Finance …
IoT
Gateway
IoT
Broker
Ontology
Stream
Process
Sofia2 Storage
Real Time Database
Notebooks API
Manager
HTML5 & API
MANAGEMENT
CONFIG
DB
Sofia2
Control Panel
Security
Historical Database
Dashboards
Ontology
APIs
VerticalSystemsApps
Plugins
REST
Ext APIs
CEP
Ontology
Flows
MQTT
… APIs
Rules
SPACES
DEVICES
Devices
SEMANTIC
1 2
3
4

26. 26
Traceability in distribution: Executive Dashboards
INGEST / PROCESS & STORE ANALYZE PUBLISH & VIEW
Capabilities
Communication
connectors for different
client’s programming
language.
Development APIs for
the client side.
Security capable in
terms of communication,
authentication and
authorization.
Light communication
protocols (REST, OPC,
MODBUS, WebSockets,
MQTT, WS, JMS, AMQP…)
Real-time processing and
validation of the
interchanged information.
Homogenization of the
information received,
according to the data model
defined (protocol
independent)
Java Extensible (APIs,
Protocols, Plugins)
Auditing of integration
activity
Integrated in Sofia2
Control Panel (HTML5)

27. 27
Traceability in distribution: Executive Dashboards
Capabilities
Real time processing of
the information received
Different coding
languages supported
(Groovy, R, Python).
Integrated with other
modules of the platform
such as repositories or
notebooks.
Capabilities for simple
IFTTT, CEP or visual
definition of flows
Alerts, notifications,
enrichment of
information…
Managed and monitoried
from Sofia2 Control
Panel (HTML5)
INGEST / PROCESS & STORE ANALYZE PUBLISH & VIEW

28. 28
Traceability in distribution: Executive Dashboards
INGEST / STORE ANALYZE PUBLISH & VIEW
Capabilities
Integrated Security
(authenticación,
autorization,
encription…)
Data can be published
independently of the
repository storing the
information (real time
or historical)
Custom Query
commands, APIs clone,
HTML5 CRUDs
generated automatically
Integration with third
Party APIs including
Open Data or
WebServices to be
Apified
.
APIs Lifecycle
managemen (Created,,
in Process of
development, published,
dDeprecated, Removed)
Throtling control and
auditing of the activity

29. 29
Traceability in distribution: Executive Dashboards
YouTube
Sofia2 IoT Platform
• Demos and viewers
• Sofia2 presentations
• Workshops and
tutorials
• Dashboards
• Technical support
• Solutions built on
Sofia2

30. 30

31. 31