NiFi Best Practices for the Enterprise

1. NiFi Best Practices for the
Enterprise
August 8, 2017
Future of Data – New Jersey
Hosted by Honeywell
Greg Keys
Solutions Engineer @ Hortonworks
Agenda
 NiFi Quick Overview
 NiFi Comparison with Other Technology
Best Practices: People, Process, Assets

2. © Hortonworks Inc. 2011 – 2016. All Rights Reserved2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
NiFi Quick Overview

3. © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Hortonworks Data Flow (HDF)
NiFi-like UI for
building stream
analytics
Reusable schema
registry across all
components
Data in Motion Data at Rest

4. © Hortonworks Inc. 2011 – 2016. All Rights Reserved
NiFi Crash Course
Other concepts
• template = code
• processor group = encapsulation
• remote processor group = communication across
NiFi clusters
• expression language = dynamic
• provenance = data lineage
• audit = history of data state for all
• replay = playback data in queue

5. © Hortonworks Inc. 2011 – 2016. All Rights Reserved
NiFi Big Picture Pattern: Diverse Flows from One Tool

6. © Hortonworks Inc. 2011 – 2016. All Rights Reserved6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
NiFi Capability Sweetspots
Comparison to other technologies

7. © Hortonworks Inc. 2011 – 2016. All Rights Reserved
NiFi vs HDF Streaming Analytics Manager (SAM)
NiFi: Flow Management SAM: Stream Processing
batch, microbatch or streaming data
small (KB) to large (GB)
parse, filter, transform, enrich, reformat
route, merge, prioritization, back
pressure, and edge intelligence
diverse (structured, semi-, un-)
diverse (e.g. text, video analytics)
minimal
streaming (real-time flow)
small (KB, MB) per message
minimal
minimal (mostly routing and merging)
typically Kafka or live stream
analytics and event processing
rich and powerful, live data
Move and process diverse data Analytics and alerting on live data
speed
content size
content manipulation
dataflow management
data sources
analytics
KafkaCommon Pattern:
sweet spot
Both have easy UIs to build complex data movement processing
computation

8. © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Apache
NiFi/MiNiFi
NiFI Compared to Other Technologies
ETL Tools
(Informatica,
etc)
Enterprise
Service Bus
(Mulesoft,
Fuse, etc)
Message
Queue
(Kafka, JMS,
etc)
NiFi gets you
80% there
NiFi gets you
80% there
NiFi as a single tool has
a very wide sweetspot
“Swiss Army Knife of
Data Movement”
Deliver and process wide
range of data to the
business
(sensor, logs, relational,
restful/json, video …)
NiFi is
complimentary
Event Processing & Streaming Analytics:
previous slide

9. © Hortonworks Inc. 2011 – 2016. All Rights Reserved
NiFi vs ETL Tools
ETL Tools
ETL on structured & semi-structured
(schema registry for structured)
Diverse sources and targets (e.g. Rest
APIs, PDFs, .xls) including RDBMS
All-encompassing flows: no data prep with
other tool; can join/merge flows; lookup, etc
rapid development
flow management capabilities
(backpressure, prioritization, etc)
Not designed for large/complex joins &
aggregations, log-based CDC (vs field-
based), or industrial-strength cleansing
• Tailored for RDBMS
• ETL based on schema / data modelling
• Highly efficient, optimized performance
sweet spots
Must prepare and model data beforehand
Not designed for dataflow problems and
diverse types of data
NiFi gets you 80% there

10. © Hortonworks Inc. 2011 – 2016. All Rights Reserved
NiFi vs Enterprise Service Bus
Enterprise Service Bus
Easy robust integrations from diverse
sources to diverse targets
(e.g. Salesforce to Hive; Oracle to Mongo, sFTP
to file system)
Easy file conversions (e.g. XML to JSON)
Easy protocol conversions (route on
attribute, e.g. FTP to Rest)
Easy cross data center communication
(remote processor)
Not a comprehensive single abstraction
layer: no code in applications, minimal
process choreography, SOA stuff, etc
• Single abstraction layer for content
movement throughout enterprise
• Specialized tools for each component of
abstraction (e.g. process orchestration,
protocol binding)
• Highly efficient, optimized performance
sweet spots
Industrial-strength focus on single
abstraction layer prevents flexibility and
agile dev of NiFi data flows (platform stack
vs Swiss Army knife)
NiFi gets you 80% there

11. © Hortonworks Inc. 2011 – 2016. All Rights Reserved
NiFi vs Message Queue
Message Queue
General and flexible data movement
tool
Great for acquiring and processing
data before delivering to queue
Not designed to be a message queue
(i.e. asynchronous durable handoff to
receiver of message)
• Low latency content transfer between
sender and receiver
• Asynchronous: sender releases
message without receiver taking it
• Durable: message persists until
receiver consumes it
• Highly efficient, optimized performance
sweet spots
Not designed to be a data flow solution
NiFi and message queues are complimentary

12. © Hortonworks Inc. 2011 – 2016. All Rights Reserved12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
NiFi Best Practices
People, process, assets

13. © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Scaling NiFi across the enterprise
1) think NiFi like software projects
For NiFi to scale across the enterprise and over time:
think people and process
COE
Program Mgmt SDLC Ops
2) establish a COE to align stakeholders and provide direction
plan execute
Standards
admin & monitor
strategize
align stakeholders
Challenges:
- multitenancy
- reuse /
efficiencies
- platform sanity

14. © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Enterprise People, Process, Assets
Program Mgmt SDLC Ops
Standards
COE
key player key asset
enterprise
architect
program
mgmt
ops
strategy
leadersecurity risk/
compliance
business
leaders
capabilities/value
needs
education plan
success metrics
data flow summary
• owner
• source(s)
• frequency
• volume
• target(s)
• goal of flow
principles / standards
reusable components
patterns
failure response procedure
Key points
• Similar to what you already know
• Education of diverse stakeholders is
key to applying this to NiFi
• Use agile for process and assets:
start small/simple
iterate, learn, advance … repeat
• Involve business leaders closely:
NiFi (like all tech) ultimately is about
providing business value and should be
driven by this

15. © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Standards and SDLC examples
SDLC
Standards
principles / standards
reusable components
patterns
enterprise
architect
https://hortonworks.com/blog/enterprise-nifi-implementing-
reusable-components-software-development-lifecycle/
Strive to
• leverage NiFi sweetspots
• use reusable components when possible (including Schema Registry!)
• use Expression Language to make processor as flexible as possible
• N- retry error handling template
• store raw to HDFS template
principles / standards
reusable components
SDLC & reuse
patterns
Pattern: N- retry error handling Anti-pattern
• infinite retry loop on long-lasting error and incoming flow file
(e.g. HTTPRequest: url endpoint is down)
Consequence:
• continuous retry with no email alert
• possible backward propagation of backpressure
processor
N times
PutEmail
> N
PutFile
HDF 3.1 Big Improvements Coming!

16. © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Best Practices: One NiFi Cluster or Many?
Challenges:
- multitenancy
- reuse /
efficiencies
- platform sanity
NiFI
NiFi
NiFi
NiFi
NiFi
NiFi
NiFi
business groups:
• less independent
• smaller
• semi-autonomous
• larger
challenges for single cluster
• smaller • larger
Separate clusters make more
sense with larger, semi-
autonomous business groups
COE still has governance
role over multiple clusters
/ business groups
NiFi
Key
business group
NiFi cluster