Hortonworks Data In Motion Series Part 4

Harnessing Data-in-Motion with
Hortonworks DataFlow
A Paradigm Shift to Business as Usual:
Real World Use Cases of Real-Time
DataFlows in Record Time
Anna Yong
Product Marketing
Haimo Liu
Product Manager

2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Agenda
Overview of Hortonworks DataFlow (HDF)
How HDF transforms data movement – months to minutes
HDF Use Cases
Real-World HDF Use Cases
Apache NiFi Case Studies from Hadoop Summit
Other Apache NiFi/HDF Use Cases

Connected Data Platforms

4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Payment
Tracking
Due
Diligence
Social
Mapping
Product
Design
M & A
Call
Analysis
Machine
Data
Defect
Detecting
Factory
Yields
Customer
Support
Basket
Analysis
Segments
Customer
Retention
Sentiment
Analysis
Optimize
Inventories
Supply
Chain
Cross-
Sell
Vendor
Scorecards
Ad
Placement
Cyber
Security
Disaster
Mitigation
Investment
Planning
Ad
Placement
Risk
Modeling
Proactive
Repair
Inventory
Predictions
Next
Product Recs
OPEX
Reduction
Historical
Records
Mainframe
Offloads
Device
Data
Ingest
Rapid
Reporting
Digital
Protection
Data
as a
Service
Fraud
Prevention
Public
Data
Capture
INNOVATE
RENOVATE
E X PLO R E O PTIMIZE TR A NS FO R M
ACTIVE
ARCHIVE
ETL
ONBOARD
DATA
ENRICHMENT
DATA
DISCOVERY
SINGLE
VIEW
PREDICTIVE
ANALYTICS

 Constrained
 High-latency
 Localized context
 Hybrid – cloud / on-premises
 Low-latency
 Global context
Core
Infrastructure
Hortonworks DataFlow Manages Data in Motion
Regional
InfrastructureSources

Hortonworks DataFlow Manages Data in Motion
Core
 Constrained
 High-latency
 Localized context
 Hybrid – cloud / on-premises
 Low-latency
 Global context
Regional
Infrastructure

451 Analyst Report on Stream Processing and Streaming Integration
http://hortonworks.com/info/value-streaming-integration/

Dataflow Management

Problems Today: Timely Access to Data and Decisions
http://diginomica.com/2016/04/22/royal-mail-starts-to-deliver-on-hortonworks-data-in-motion-promise
“HDF helps us to streamline the flow
of data and build models and
visualisations quickly, so that my team
can work iteratively with business
colleagues on building solutions
that work for the business.“
Royal Mail

HDP
HORTONWORKS
DATA PLATFORM
Powered by Apache Hadoop
HDF Makes Big Data Ingest Easy
Complicated, messy, and takes weeks to
months to move the right data into Hadoop
HDP
HORTONWORKS
DATA PLATFORM
Streamlined, Efficient, Easy
HDP
HORTONWORKS
DATA PLATFORM
Powered by Apache Hadoop

Hortonworks DataFlow, Powered by Apache NiFi. Demo Time

Create a live dataflow in minutes
How would that change your business?

Add processor for data intake. Time: 1 minute
1 Drag and drop processor from top menu

Choose the specific processor
2 Choose one of the processors – currently 170+ available

Example: Pick Twitter Processor

Configure the processor. Time: 2 minutes
3
4
Select processor and choose
option to Configure
Adjust
parameters as
required

Another processor for data output. Time: 1 minute
5
6 Filter for and select a “Put” processor
Drag and drop processor from top menu

Configure second processor. Time: 1 minute
7 Configure 2nd processor

Connect processors, configure connection. 2 minutes
Configure Connection8
Note: Sample Flow is different from previous example of PutHDFS. This dataflow is PutFile. Same concepts apply.

Click Start to Begin Processing. Time total: 7 minutes
9 Click start “play” to being processing
(will run continuously until you select stop)

HDF Use Cases

Core
Infrastructure
Hortonworks DataFlow Use Cases
Regional
Dataflow Management
• On-ramp into Hadoop
• Log Collection / Splunk Optimization
• Cyber Security
• IoT Ingestion
• Deliver data into stream processing engines
Real-time Event Processing
(Kafka, Storm)
Move data between
from on-prem and
cloud environments

Optimize Log Analytics with Content Based Routing

IoT Data Ingestion
Constrained
High-Latency
Localized Context
Hybrid – Cloud/On-Premise
Low-Latency
Global Context
Resolves real world connectivity and transmission issues often overlooked by assuming connectivity is always perfect

Enterprise Data Movement and Hybrid Cloud
• Seamlessly fuse dataflows between data centers
• Data center to data center,
• Remote location to data center,
• Data center to cloud
HDF
Between Data Centers
HDF
HDF
Remote to Data Center
HDF
HDF
HDF
HDF
Between Data Centers & Cloud
HDF

© Hortonworks Inc. 2011
Stream Processing
Page 26
Data Acquisition
Edge Processing
Real Time Stream Analytics
Rapid Application Development
IoT
ANALYTICS
CLOUD

Royal Mail’s Journey
Satisfied customers
with better service
levels & improved
retention
Actionable intelligence improves
customer experience
 Data management transformed to deliver
specific, actionable insights to line of
business departments
 Analysis delivered within days & weeks
rather than months per project deadlines
 Churn modelling project identified customers
at risk by vertical sector in order to take
preventative action
 Improved accuracy of delivery times for
business customers & highlighted trends
related to volumes of mail expected
 Governance & compliance simplified due to
central data platform
Satisfied customers
with better service
levels & improved
retention
P R E D I C T I V E
A N A L Y T I C S
A C T I V E
A R C H I V E
S I N G L E
V I E W
A C T I V E
A R C H I V E
S I N G L E
V I E W
E T L
O N B O A R D
D A T A
E N R I C H M E N T
P R E D I C T I V E
A N A L Y T I C S
D A T A
E N R I C H M E N T
Parcel
distribution
Customer
Acquisition
S I N G L E
V I E W
Customer
Support
Inventory
Predictions
Investment
Planning
Data-as-a-
Service
Public data
capture
Rapid
reporting
EDW offload
OPEX
reduction
Innovate
Renovate
New Data
Products

Open Energi Uses HDF for Electricity Demand Response
As a result of its investment in Hortonworks
Dataflow, Open Energi is Already:
 Reducing costs thanks to 10-15% less data
being transmitted across a mobile
network
 Creating a full transparent trail for data
provenance that Open Energi can share
with customers
 Enabling line of business teams to
contribute to building dataflow rules and
processes
 Standardizing the output of data across
various end point devices
Open Energi: hortonworks.com/blog/data-fuel-open-energi-virtual-power-station-hortonworks-dataflow/

Prescient Traveler transformed the travel risk
management market
$0.5MM
Savings in development costs due
to Hortonworks HDF
700%
Improvement in analyst
productivity in determining
actual threats
49,000
Number of data sources currently
being analyzed to identify threats

Centrica’s Journey
1.3 Million
Smart Meters
EDW
Offload
Mobile App for
Customer Sites
Ingest 300
GB per Day
Product
Cross-Sell
Building a Data-Driven
Energy Utility Business
Self-service analytics for 3
million customers in UK & North
America
HDP and HDF simplify IT estate
Ingest of 300 GB/day
rationalizes maintenance work
Personalized customer
communications replaced
impersonal up-sell messages
Legacy EDWs decommisioned
Innovate
Renovate
Smart, Efficient
Homes
D A T A
D I S C O V E R Y
D A T A
E N R I C H M E N T
P R E D I C T I V E
A N A L Y T I C S
S I N G L E
V I E W
A C T I V E
A R C H I V E
E T L
O N B O A R D
SINGLE
VIEW
S I N G L E
V I E W
P R E D I C T I V E
A N A L Y T I C S
On-site customer
data capture
Optimized
engineering
schedule
Tailored
servicing
Customer
sentiment

Apache NiFi Case Studies:
Hadoop Summit San Jose

From Zero to DataFlow: http://www.slideshare.net/HadoopSummit/from-zero-to-data-flow-in-hours-with-apache-nifi-64032731

Make Streaming Analytics Work For You
http://www.slideshare.net/HadoopSummit/make-streaming-analytics-work-for-you-the-devil-is-in-the-details

Hadoop Summit Keynote: Apache Metron
Ingest log data into their cyber security data lake
https://youtu.be/Nffx8SKn7l4?t=1h37m50s

Hadoop Summit Keynote: Improving Customer Experience
https://youtu.be/BY_0HB9uyXQ

Other HDF/Apache NiFi
Use Cases

Data Hacks & Demos Keynote: Retail Simulation at Hadoop Summit
Live Voting, Electronic Conversation, Real-Time Facial Recognition
Intro, Demo 1, Demo 2, Demo 3, Demo 4.
https://www.youtube.com/watch?v=BY_0HB9uyXQ&feature=youtu.be&t=49m10s

GENIVI Alliance: Open Source, In-Vehicle Infotainment Software
The GENIVI Alliance is a nonprofit industry
alliance committed to driving the broad adoption
of specified, open source, In-Vehicle Infotainment
software.

More Use Cases
Using Apache NiFi to read children’s books” https://twitter.com/KayLerch/status/721455415456882689

Yet More Use Cases
https://twitter.com/4everfusiongal/status/7351585225
39855872
https://www.linkedin.com/pulse/making-rain-apache-nifi-
jeremy-dyer?trk=prof-post
www.linkedin.com/hp/update/6138493082129149952
https://community.hortonworks.com/articles/30636/how-to-simulate-a-
sales-executive-with-hdf.html

Even More Use Cases
https://community.hortonworks.com/articles/47854/accessing-facebook-
page-data-from-apache-nifi.html
https://community.hortonworks.com/content/kbentry/32605/runn
ing-nifi-on-raspberry-pi-best-practices.html
Accessing Facebook Data

And Even More Use Cases
https://community.hortonworks.com/articles/20318/visualize-patients-
complaints-to-their-doctors-usi.html
Visualize patients' complaints to their doctors
using NiFi and Solr/Banana
http://hortonworks.com/blog/qualcomm-hortonworks-showcase-
connected-car-platform-tu-automotive-detroit/
Connected Car

Questions? Hortonworks Community Connection:
Data Ingestion and Streaming
https://community.hortonworks.com/
Contact Us:
http://hortonworks.com/contact-us/

Hortonworks Data In Motion Series Part 4

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (6)

Similar to Hortonworks Data In Motion Series Part 4

Similar to Hortonworks Data In Motion Series Part 4 (20)

More from Hortonworks

More from Hortonworks (20)

Recently uploaded

Recently uploaded (20)

Hortonworks Data In Motion Series Part 4

Editor's Notes