What is “dataflow?” — the process and tooling around gathering necessary information and getting it into a useful form to make insights available. Dataflow needs change rapidly — what was noise yesterday may be crucial data today, an API endpoint changes, or a service switches from producing CSV to JSON or Avro. In addition, developers may need to design a flow in a sandbox and deploy to QA or production — and those database passwords aren’t the same (hopefully). Learn about Apache NiFi — a robust and secure framework for dataflow development and monitoring. Abstract: Identifying, collecting, securing, filtering, prioritizing, transforming, and transporting abstract data is a challenge faced by every organization. Apache NiFi and MiNiFi allow developers to create and refine dataflows with ease and ensure that their critical content is routed, transformed, validated, and delivered across global networks. Learn how the framework enables rapid development of flows, live monitoring and auditing, data protection and sharing. From IoT and machine interaction to log collection, NiFi can scale to meet the needs of your organization. Able to handle both small event messages and “big data” on the scale of terabytes per day, NiFi will provide a platform which lets both engineers and non-technical domain experts collaborate to solve the ingest and storage problems that have plagued enterprises. Expected prior knowledge / intended audience: developers and data flow managers should be interested in learning about and improving their dataflow problems. The intended audience does not need experience in designing and modifying data flows. Takeaways: Attendees will gain an understanding of dataflow concepts, data management processes, and flow management (including versioning, rollbacks, promotion between deployment environments, and various backing implementations). Current uses: I am a committer and PMC member for the Apache NiFi, MiNiFi, and NiFi Registry projects and help numerous users deploy these tools to collect data from an incredibly diverse array of endpoints, aggregate, prioritize, filter, transform, and secure this data, and generate actionable insight from it. Current users of these platforms include many Fortune 100 companies, governments, startups, and individual users across fields like telecommunications, finance, healthcare, automotive, aerospace, and oil & gas, with use cases like fraud detection, logistics management, supply chain management, machine learning, IoT gateway, connected vehicles, smart grids, etc.

1. 1 © Hortonworks Inc. 2011–2018. All rights reserved
Dataflow Management From Edge to
Core with Apache NiFi
Andy LoPresto | @yolopey
Sr. Member of Technical Staff at Hortonworks, Apache NiFi PMC & Committer
11 October 2018 Dataworks Summit Singapore

2. 2 © Hortonworks Inc. 2011–2018. All rights reserved
Gauging Audience Familiarity with NiFi
“What’s a NeeFee?”
No experience with dataflow
No experience with NiFi
“I can pick this up pretty quickly”
Some experience with dataflow
Some experience with NiFi
“I refactored the Ambari
integration endpoint to
allow for mutual
authentication TLS during
my coffee break”
Forgotten more about NiFi
than most of us will ever
know

3. 3 © Hortonworks Inc. 2011–2018. All rights reserved
Agenda
• What is dataflow and what are the challenges?
• Apache NiFi
• Apache MiNiFi
• Apache NiFi Registry
• Complementary Tools
• Community
• All slides provided online, so no need to transcribe

4. 4 © Hortonworks Inc. 2011–2018. All rights reserved
What Is Dataflow?

5. 5 © Hortonworks Inc. 2011–2018. All rights reserved
What Is Dataflow?
• Moving some content from A to B
• Content could be any bytes
• Logs
• HTTP
• XML
• CSV
• Images
• Video
• Telemetry
Producers A.K.A
Things
Anything
AND
Everything
Internet!
Consumers
• User
• Storage
• System
• …More Things

6. © Hortonworks Inc. 2011–2018. All rights reserved
Moving Data Effectively Is Hard
“Data Pipeline” https://xkcd.com/2054/

7. © Hortonworks Inc. 2011–2018. All rights reserved
• Standards
• Formats
• Protocols
• Veracity
• Validity
• Schemas
• Partitioning/Bun
dling
Data
Dataflow Challenges in 3 Categories
Infrastructure
• “Exactly Once”
Delivery
• Ensuring
Security
• Overcoming
Security
• Credential
Management
• Network
People
• Compliance
• “That
[person|team|g
roup]”
• Consumers
Change
• Requirements
Change
• “Exactly Once”
Delivery

8. © Hortonworks Inc. 2011–2018. All rights reserved
Raise your hand if you want to maintain Python scripts for the rest of your life
Let’s Connect Lots of As to Bs to As to Cs to Bs to Δs to Cs to ϕs

9. 9 © Hortonworks Inc. 2011–2018. All rights reserved
Apache NiFi

10. © Hortonworks Inc. 2011–2018. All rights reserved
• Guaranteed delivery
• Data buffering
• Backpressure
• Pressure release
• Prioritized queuing
• Flow specific QoS
• Latency vs. throughput
• Loss tolerance
Key Features
Apache NiFi
• Data provenance
• Supports push and pull models
• Recovery/recording
a rolling log of fine-grained history
• Visual command and control
• Flow templates
• Pluggable, multi-tenant security
• Designed for extension
• Clustering

11. © Hortonworks Inc. 2011–2018. All rights reserved
Flowfiles Are Like HTTP Data
HTTP Data FlowFile
HTTP/1.1 200 OK
Date: Sun, 10 Oct 2010 23:26:07 GMT
Server: Apache/2.2.8 (CentOS) OpenSSL/0.9.8g
Last-Modified: Sun, 26 Sep 2010 22:04:35 GMT
ETag: "45b6-834-49130cc1182c0"
Accept-Ranges: bytes
Content-Length: 13
Connection: close
Content-Type: text/html
Hello world!
Standard FlowFile Attributes
Key: 'entryDate’ Value: 'Fri Jun 17 17:15:04 EDT 2016'
Key: 'lineageStartDate’ Value: 'Fri Jun 17 17:15:04 EDT 2016'
Key: 'fileSize’ Value: '23609'
FlowFile Attribute Map Content
Key: 'filename’ Value: '15650246997242'
Key: 'path’ Value: './’
Binary Content *
Header
Content

12. © Hortonworks Inc. 2011–2018. All rights reserved
User Interface
Less of this… … more of this

13. © Hortonworks Inc. 2011–2018. All rights reserved
Deeper Ecosystem Integration: 274+ Processors,
57 Controller Services
Hash
Extract
Merge
Duplicate
Scan
GeoEnrich
Replace
ConvertSplit
Translate
Route Content
Route Context
Route Text
Control Rate
Distribute Load
Generate Table Fetch
Jolt Transform JSON
Prioritized Delivery
Encrypt
Tail
Evaluate
Execute
All Apache project logos are trademarks of the ASF and the respective projects.
Fetch
HTTP
Syslog
Email
HTML
Image
HL7
FTP
UDP
XML
SFTP
AMQP
WebSocket
Parse Records Convert Records

14. 22 © Hortonworks Inc. 2011–2018. All rights reserved
Apache MiNiFi

15. 23 © Hortonworks Inc. 2011–2018. All rights reserved
IoT Challenges
• Limited computing capability
• Limited power/network
• Restricted software library/platform
availability
• No UI
• Physically inaccessible
• Not frequently updated
• Competing standards/protocols
• Scalability
• Privacy & Security
@_lennart

16. © Hortonworks Inc. 2011–2018. All rights reserved27
• NiFi is designed to “own the box”
• NiFi 0.7.x started up in about 10-15 minutes on RP3 (593 MB)
• NiFi 1.x started up in about 30 minutes on RP3 (760 MB)
• 33 new processors
• Rewrite for multi tenant authorization
• Complete UI overhaul
So Why Do We Need a Different Solution?

17. © Hortonworks Inc. 2011–2018. All rights reserved28
• Get the key parts of NiFi close to where data begins and provide bidirectional
communication
• NiFi lives in the data center — give it an enterprise server or a cluster of them
• MiNiFi lives as close to where data is born and is a guest on that device or system
• IoT
• Connected car
• Legacy hardware
Apache NiFi Subproject: MiNiFi

18. © Hortonworks Inc. 2011–2018. All rights reserved30
• MiNiFi Java (v0.5.0)
• Modified version of NiFi
• No UI
• YAML configuration
• Reduced processor count
• 63+ by default, more
available with
additional NARs
• MiNiFi C++ (v0.5.0)
• Written from scratch
• 33 processors by default
• Bi-directional site-to-site & provenance data
Flavors of MiNiFi

19. © Hortonworks Inc. 2011–2018. All rights reserved32
• NiFi
• Design flows
• Aggregate data from many
sources
• Perform routing/analysis/SEP
• MiNiFi
• Receive flows
• Collect data
• Send for processing
How Does MiNiFi Interact with NiFi?

20. © Hortonworks Inc. 2011–2018. All rights reserved33
• We’ve been imagining EDGE to CORE as a bi-directional linear system
• Let’s expand
that to the real
world
Let’s Add Dimensionality

21. © Hortonworks Inc. 2011–2018. All rights reserved34
• Data tagging/provenance
• Governance from edge (geopolitical
restrictions)
• Security (encryption, certificate-based
authentication)
• Low latency (immediate reactions &
decision-making)
What Does MiNiFi provide? Connected Car Reference Platform Box
Tuner + DSRC CardConnectivity Card

22. © Hortonworks Inc. 2011–2018. All rights reserved37
• Site-to-Site
• NiFi protocol
• Two implementations
• Raw socket
• HTTP(S)
• Secured with mutual authentication TLS
• HTTP(S), (S)FTP, JMS, Syslog, File, Email, Process
MiNiFi Exfil

23. 38 © Hortonworks Inc. 2011–2018. All rights reserved
Apache NiFi Registry

24. 39 © Hortonworks Inc. 2011–2018. All rights reserved
Flow Development Lifecycle (FDLC)
• Origins of NiFi
• Operator Experience
• MC data, don’t drop, mitigate temporarily
• Version Control
• Environment Promotion

25. 40 © Hortonworks Inc. 2011–2018. All rights reserved
Operator Experience

26. © Hortonworks Inc. 2011–2018. All rights reserved41
• Shows previous values (user,
time changed)
• Sensitive values are always
encrypted at rest and never
returned via the API
Component Property History

27. 42 © Hortonworks Inc. 2011–2018. All rights reserved
Exporting Flows
• XML templates
• Copying flow.xml.gz
between systems

28. 43 © Hortonworks Inc. 2011–2018. All rights reserved
Challenges
• Templates
• Updates/replacement
• Sensitive property replacement
• Flow.xml.gz migration
• Key synchronization
• Environment promotion
• Approval processes
• Verifiability

29. 44 © Hortonworks Inc. 2011–2018. All rights reserved
Template Replacement
• Export a new version of template
• Transfer (somehow)
• Verify?
• Import onto canvas side-by-side existing
flow
• Stop processors
• Empty queues
• Reconnect queues
• Start
• Pray?

30. 45 © Hortonworks Inc. 2011–2018. All rights reserved
Template Replacement

31. © Hortonworks Inc. 2011–2018. All rights reserved46
• Previously, flows were exported via XML
templates
• Didn’t contain sensitive values
• Couldn’t be updated in-place
• No tracking system
• NiFi Registry brings asset management
as first-class citizen to NiFi
• Flows can be versioned
Introducing Apache NiFi Registry 0.3.0
NiFi Registry for Dataflows

32. © Hortonworks Inc. 2011–2018. All rights reserved47
• Connect multiple NiFi instances
to a NiFi Registry instance
• Communicate between multiple
NiFi Registry instances
• via multiple Registry Clients
• via NiFi CLI
Flows Can Be Promoted Between Environments

33. © Hortonworks Inc. 2011–2018. All rights reserved48
• Git-backed persistence
• Share flows via GitHub, etc.
• Commit hooks
• Register a hook & action
• “When a new version of the
flow is committed to QA
Registry, email the QA team
and post in the QA Deploy
Slack channel”
• Pluggable DB implementations
Extensibility

34. 49 © Hortonworks Inc. 2011–2018. All rights reserved
Demo

35. © Hortonworks Inc. 2011–2018. All rights reserved50
• Install nifi-registry
• $ mvn clean install
• $ ./bin/nifi-registry.sh
start
• Browse to http://localhost:18080
Create Registry

36. 51 © Hortonworks Inc. 2011–2018. All rights reserved
Create Bucket

37. © Hortonworks Inc. 2011–2018. All rights reserved52
Connect to NiFi

38. © Hortonworks Inc. 2011–2018. All rights reserved53
Create Process Group

39. © Hortonworks Inc. 2011–2018. All rights reserved54
Commit Version

40. © Hortonworks Inc. 2011–2018. All rights reserved55
View Flow in Registry

41. © Hortonworks Inc. 2011–2018. All rights reserved56
Import New Instance into NiFi

42. © Hortonworks Inc. 2011–2018. All rights reserved57
Modify the Original Flow

43. © Hortonworks Inc. 2011–2018. All rights reserved58
See Local Changes Before Committing

44. © Hortonworks Inc. 2011–2018. All rights reserved59
Commit

45. © Hortonworks Inc. 2011–2018. All rights reserved60
Update New Instance from Registry

46. 61 © Hortonworks Inc. 2011–2018. All rights reserved
Complementary Tools

47. © Hortonworks Inc. 2011–2018. All rights reserved62
• NiFi Toolkit
• NiPyAPI
• MiNiFi Converter Toolkit
Complementary Tools

48. 63 © Hortonworks Inc. 2011–2018. All rights reserved
NiFi Toolkit
• TLS Toolkit
• Generates, signs, and packages
keys and certificates for NiFi
services (node/cluster, clients)
• Encrypt Config
• Protects sensitive configuration
values like passwords
• CLI
• Interacts with NiFi & NiFi
Registry to operate on flows

49. 64 © Hortonworks Inc. 2011–2018. All rights reserved
NiPyAPI
• Python wrapper around NiFi REST API
• Community-provided by Daniel Chaffelson
• Exposes common operations for automation, batch processing, recursion, etc.
dev_bucket = nipyapi.versioning.get_registry_bucket(dev_bucket_name)
dev_ver_flow = nipyapi.versioning.get_flow_in_bucket(
dev_bucket.identifier,
identifier=dev_ver_flow_name
)
dev_export = nipyapi.versioning.export_flow_version(
bucket_id=dev_bucket.identifier,
flow_id=dev_ver_flow.identifier,
mode='yaml'
)

50. 65 © Hortonworks Inc. 2011–2018. All rights reserved
MiNiFi Converter Toolkit
• Save as template from NiFi
• Run $ ./bin/config.sh transform
template.xml config.yml
• MiNiFi flow ready to run

51. 66 © Hortonworks Inc. 2011–2018. All rights reserved
Community

52. © Hortonworks Inc. 2011–2018. All rights reserved67
• FDLC with Apache NiFi, Kevin
Doran
• NiPyAPI Docs, Daniel
Chaffelson
• DevOps Tips, Tim Spann
• Automate Workflow, Pierre
Villard
More Resources

53. © Hortonworks Inc. 2011–2018. All rights reserved68
• NiFi 1.8.0 — … Oct 2018 (170+ Jiras)
• Jetty, DB improvements
• Auto load-balancing queues
• TLS Toolkit w/ external CA
• Record processor improvements
• MiNiFi C++ 0.5.0 — 6 June 2018
• MiNiFi Java 0.5.0 — 7 July 2018
• NiFi Registry 0.3.0 — 25 Sept 2018
New Announcements

54. © Hortonworks Inc. 2011–2018. All rights reserved69
Community Health

55. © Hortonworks Inc. 2011–2018. All rights reserved70
Apache NiFi site
https://nifi.apache.org
Subproject MiNiFi site
https://nifi.apache.org/minifi/
Subscribe to and collaborate at
dev@nifi.apache.org
users@nifi.apache.org
Submit Ideas or Issues
https://issues.apache.org/jira/browse/NIFI
Follow us on Twitter
@apachenifi
Learn More and Join Us

56. 72 © Hortonworks Inc. 2011–2018. All rights reserved
Thank you
alopresto@hortonworks.com | alopresto@apache.org | @yolopey
github.com/alopresto/slides