The kenyote presentation from Predictive Analytics World entitled "Advanced Analytics for Any Data at Real-Time Speed" Dan Potter, CMO from Datawatch, presents a new approach to prepare, incorporate, enrich and visualize streaming data for advanced visual analysis is essential for making timelier, high-impact business decisions in tough competitive markets.
3. NASDAQ: DWCH
Pioneer in real-time visual data discovery and self-service data
preparation
Global operations and support
US, UK, Germany, France, Australia, Singapore, Philippines
Extensive global customer base
93 of the Fortune 100
12 of the 15 largest financial institutions
Embedded and resold by leading vendors
About Datawatch
4. IoT and Industrial Analytics
50 billion devices will be interconnected by 2020
- Cisco
By 2017, over 50% of analytics implementations will make use of
event data streams generated from instrumented machines,
applications and/or individuals
- Gartner
Real-time visualization leads to more opportunities, greater
output, and lower costs
- Aberdeen Group
IoT companies attracted more than $1 billion in venture capital
- Forbes
8. 6 Requirements for IoT Visualization
Data Discovery
Streaming Data Visualization
Time Series Data
Predictive & Advanced Analytics
Data Preparation
Real-time Geospatial & Location
9. #1 Data Discovery, not just Dashboards
• Easy for users to author,
customize and share
• Interactive exploration &
visually filter results
• Quickly identify
anomalies and outliers
with large or in-motion
datasets
• Rich palette of
visualizations for static
and time series data
10. #2 Streaming Data Visualization
Database Distributed or
Hybrid Database
In-Memory
Database
Streaming Analytics
Faster Speed, Faster Insights
11. Data at Rest
Limitations of Traditional BI
Database Distributed or
Hybrid Database
In-Memory
Database
Streaming Analytics
12. Data at Rest
Streaming Data Discovery
Database Distributed or
Hybrid Database
In-Memory
Database
Streaming Analytics
13. Streaming Data Discovery
Database Distributed or
Hybrid Database
In-Memory
Database
Streaming Data
Alert! Steam turbine stress
level over threshold
How does this compare to
intra-day? What is mean
time to failure?
What is likely to occur?
Act! Schedule shut down
14. Streaming Data Visualization
• Connect directly to data in motion
• Hosted IoT platforms
• Complex Event Processing & MQ
• Data model optimized for both caching and persistence
• High density visuals with rendering in milliseconds
Monitor Analyze Take Action
15. #3 Time Series Data
• Traditional BI only looks at buckets of
time
• Day, week, month, year
• Sensor data is a continuous and has
different requirements
• Second, millisecond, nanosecond
• Time windows
• Time slices
• Playback
• Complete situational awareness
• Now (streaming)
• Intra-day
• Historic
16. #4 Predictive & Advanced Analytics
• Enrich streaming OT data
with “what is likely to occur”
• Predictive models based on
historic data patterns
• Many use cases in IoT (e.g.
predictive maintenance,
smart logistics, clinical
pattern detection etc.)
• Leverage commercial and
open source solutions
18. Modeled and
transformed
for analysis
#5 Data Preparation
• Sensor and machine data often in multi-structured format
• Need to transform, enrich and prepare data
• Almost no metadata
• For example, wave form visualization from JSON arrays
stored in MongoDB and streaming
Log Files
HTML,
XML JSON
PDFs
20. 6 Requirements for IoT Visualization
Visual Data Discovery
Streaming Data Visualization
Time Series Data
Predictive & Advanced Analytics
Data Preparation
Real-time Geospatial & Location
New
Analytic
Approach
Required
21. The Next Wave of Business
Transformation
Source: Industrial Analytics: The Next Wave of Business Transformation
Gartner, March 2014
22. ROI for Real-Time Data Visualization
Growth in New Pipeline
Increase in Cash Generated
Greater Operational Cost Reduction
“Real-Time Data Visualization” October 2013
23. Customer Use Case
• Fortune 500 oil & gas exploration and production company
• Moving to real-time streaming visualization
• From 24 hour latency moving data overnight from OSIsoft Pi to Oracle Warehouse
feeding dashboards
• To real-time, streaming data discovery connecting directly to OSIsoft Pi server
• Initial goal is to reduce steam cost by 3-5% ($M) in year 1
Pi Server
IoT companies attracted more than $1 billion in venture capital in 2013
Time Series
Conflation
Streaming visualization
Alerts
Integration with CEP and event processing
JSON file formats
MQTT
Scada and Pi Server – OSIsoft
Predictive – R and Pytohn
Would you cross the street based on yesterday’s news
An in-memory, OLAP-based StreamCube is associated with each graphical display object. The system processes new data as it arrives, selects the subset of important data, recalculates the relevant sections of the model and refreshes the associated parts of the display immediately. The parts of the model and the display that are not affected by the new data are not touched. This is faster and more efficient than conventional data visualization tools that operate on batch-loaded snapshots of data, run less frequently, and then recalculate the model and rebuild the display for each iteration.
Can be used in combination with time series transforms
Time Series
Conflation
Streaming visualization
Alerts
Integration with CEP and event processing
JSON file formats
MQTT
Scada and Pi Server – OSIsoft
Predictive – R and Pytohn
Source: Aberdeen Group, “Real-Time Data Visualization,” October 2013.
