Data Ingestion with Apache NiFi and MiNiFi

•

3 j'aime•765 vues

This document discusses data ingestion using Apache NiFi and MiNiFi. Apache NiFi is a dataflow system that allows for reliable and secure transfer of data between systems. It is used for ingesting data from sources into analytic platforms and preparing data through actions like format conversion and parsing. Apache MiNiFi is a subproject that deploys smaller agents to collect data at the edge and push it to NiFi in the data center. The document provides an overview of NiFi's capabilities like guaranteed delivery, buffering, security, and clustering support.

Technologie

Data Ingestion with
Apache NiFi and MiNiFi
Lucian Neghina
Big Data Architect
by Developer for Developers

➔ Periodically batch ingestion
➔ Incremental ingestion
➔ Over 100 servers
➔ No access to the servers
➔ Push access only
➔ Networking failure
➔ Secure communication
Challenges

Apache NiFi
NiFi is based on Flow Based Programming
➔ Data agnostic
➔ Guaranteed data delivery
➔ Data Buffering / Back-Pressure
➔ Prioritized queuing
➔ Secure (SSL, HTTPS, encrypted content...)
➔ Data provenance
➔ Support push and pull models
➔ Designed for extension
➔ Clustering

Apache NiFi
What is NiFi used for?
➔ Reliable and secure transfer of data between systems
➔ Delivery of data from sources to analytic platforms
➔ Enrichment and preparation of data:
◆ Conversion between formats
◆ Extraction /Parsing
◆ Routing decisions
What is NiFi NOT used for?
➔ Distributed computation
➔ Complex Event Processing
➔ Long term data storage

Apache MiNiFi
Subproject of Apache NiFi
➔ Agent model to collect data at the edge
➔ Smaller footprint
➔ Design & Deploy
➔ Warm re-deploys
NiFi lives in the data center
MiNiFi lives as close to where data is born

System Architecture
Physical Store
...
Data Center
Server Cluster

Thank you
● NiFi Template
https://github.com/nflucian/nifi/raw/master/DemoNiFi.xml
● twitter: @nflucian
● github: github.com/nflucian

Recommandé

Data ingestion and distribution with apache NiFiLev Brailovskiy

Nifi workshopYifeng Jiang

Apache NiFi Meetup - Introduction to NiFi RegistryBryan Bende

Joe witt may2015_kafka_nyc_apachenifi-overviewJoseph Witt

NifiJulio Castro

Apache Nifi - Custom Processor thotasrinath

Joe Witt presentation on Apache NiFiMark Kerzner

What will be new in Apache NiFi 1.2.0Koji Kawamura

Recommandé

Data ingestion and distribution with apache NiFiLev Brailovskiy

Nifi workshopYifeng Jiang

Apache NiFi Meetup - Introduction to NiFi RegistryBryan Bende

Joe witt may2015_kafka_nyc_apachenifi-overviewJoseph Witt

NifiJulio Castro

Apache Nifi - Custom Processor thotasrinath

Joe Witt presentation on Apache NiFiMark Kerzner

What will be new in Apache NiFi 1.2.0Koji Kawamura

Big Data Day LA 2016/ Big Data Track - Building scalable enterprise data flow...Data Con LA

Integrating Apache NiFi and Apache Apex Apache Apex

Taking DataFlow Management to the Edge with Apache NiFi/MiNiFiBryan Bende

HDF Powered by Apache NiFi IntroductionMilind Pandit

Hortonworks Data in Motion Webinar Series Part 7 Apache Kafka Nifi Better Tog...Hortonworks

Introduction to data flow management using apache nifiAnshuman Ghosh

Running Apache NiFi with Apache Spark : Integration OptionsTimothy Spann

Flink and NiFi, Two Stars in the Apache Big Data ConstellationMatthew Ring

AWS User Group Meetup Berlin - Kay Lerch on Apache NiFi (2016-04-19)Kay Lerch

Beyond Messaging Enterprise Dataflow powered by Apache NiFiIsheeta Sanghi

Setting Up InfluxDB for IoT by David G SimmonsInfluxData

Integrating Apache Spark and NiFi for Data LakesDataWorks Summit/Hadoop Summit

Log Analytics OptimizationIsheeta Sanghi

IOT, Streaming Analytics and Machine Learning DataWorks Summit/Hadoop Summit

Dataflow with Apache NiFi - Apache NiFi Meetup - 2016 Hadoop Summit - San JoseAldrin Piri

InfluxDB 2.0 Client Libraries by Noah CrowleyInfluxData

Integrating NiFi and FlinkBryan Bende

Confluent kafka meetupseattle jan2017Nitin Kumar

You Can't Search Without DataBryan Bende

Extending The Yahoo Streaming Benchmark to Apache ApexApache Apex

AIDevWorldApacheNiFi101Timothy Spann

Integração de Dados com Apache NIFI - Marco Garcia CetaxMarco Garcia

Contenu connexe

Tendances

Big Data Day LA 2016/ Big Data Track - Building scalable enterprise data flow...Data Con LA

Integrating Apache NiFi and Apache Apex Apache Apex

Taking DataFlow Management to the Edge with Apache NiFi/MiNiFiBryan Bende

HDF Powered by Apache NiFi IntroductionMilind Pandit

Hortonworks Data in Motion Webinar Series Part 7 Apache Kafka Nifi Better Tog...Hortonworks

Introduction to data flow management using apache nifiAnshuman Ghosh

Running Apache NiFi with Apache Spark : Integration OptionsTimothy Spann

Flink and NiFi, Two Stars in the Apache Big Data ConstellationMatthew Ring

AWS User Group Meetup Berlin - Kay Lerch on Apache NiFi (2016-04-19)Kay Lerch

Beyond Messaging Enterprise Dataflow powered by Apache NiFiIsheeta Sanghi

Setting Up InfluxDB for IoT by David G SimmonsInfluxData

Integrating Apache Spark and NiFi for Data LakesDataWorks Summit/Hadoop Summit

Log Analytics OptimizationIsheeta Sanghi

IOT, Streaming Analytics and Machine Learning DataWorks Summit/Hadoop Summit

Dataflow with Apache NiFi - Apache NiFi Meetup - 2016 Hadoop Summit - San JoseAldrin Piri

InfluxDB 2.0 Client Libraries by Noah CrowleyInfluxData

Integrating NiFi and FlinkBryan Bende

Confluent kafka meetupseattle jan2017Nitin Kumar

You Can't Search Without DataBryan Bende

Extending The Yahoo Streaming Benchmark to Apache ApexApache Apex

Tendances (20)

Big Data Day LA 2016/ Big Data Track - Building scalable enterprise data flow...

Integrating Apache NiFi and Apache Apex

Taking DataFlow Management to the Edge with Apache NiFi/MiNiFi

HDF Powered by Apache NiFi Introduction

Hortonworks Data in Motion Webinar Series Part 7 Apache Kafka Nifi Better Tog...

Introduction to data flow management using apache nifi

Running Apache NiFi with Apache Spark : Integration Options

Flink and NiFi, Two Stars in the Apache Big Data Constellation

AWS User Group Meetup Berlin - Kay Lerch on Apache NiFi (2016-04-19)

Beyond Messaging Enterprise Dataflow powered by Apache NiFi

Setting Up InfluxDB for IoT by David G Simmons

Integrating Apache Spark and NiFi for Data Lakes

Log Analytics Optimization

IOT, Streaming Analytics and Machine Learning

Dataflow with Apache NiFi - Apache NiFi Meetup - 2016 Hadoop Summit - San Jose

InfluxDB 2.0 Client Libraries by Noah Crowley

Integrating NiFi and Flink

Confluent kafka meetupseattle jan2017

You Can't Search Without Data

Extending The Yahoo Streaming Benchmark to Apache Apex

Similaire à Data Ingestion with Apache NiFi and MiNiFi

AIDevWorldApacheNiFi101Timothy Spann

Integração de Dados com Apache NIFI - Marco Garcia CetaxMarco Garcia

Automate your data flows with Apache NIFIAdam Doyle

Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFiDataWorks Summit

Introduction to Apache NiFi dws19 DWS - DC 2019Timothy Spann

Using Apache NiFi with Apache Pulsar for Fast Data On-RampTimothy Spann

OSSNA Building Modern Data Streaming AppsTimothy Spann

Timothy Spann [StreamNative] | Using FLaNK with InfluxDB for EdgeAI IoT at Sc...InfluxData

Using FLiP with influxdb for EdgeAI IoT at ScaleTimothy Spann

Moving to hyper-converged? Don’t forget about data protection.Veritas Technologies LLC

Budapest Data/ML - Building Modern Data Streaming Apps with NiFi, Flink and K...Timothy Spann

OpenSource and the Cloud ApacheCon.pptxlohitvijayarenu

Big Data Day LA 2015 - Always-on Ingestion for Data at Scale by Arvind Prabha...Data Con LA

Conf42Python -Using Apache NiFi, Apache Kafka, RisingWave, and Apache Iceberg...Timothy Spann

Lightning Fast Analytics with Hive LLAP and DruidDataWorks Summit

xGem Data Stream ProcessingJorge Hirtz

Devfest uk & ireland using apache nifi with apache pulsar for fast data on-r...Timothy Spann

Unconference Round Table NotesTimothy Spann

Machine Data 101Splunk

Big Data for Security - DNS AnalyticsMarco Casassa Mont

Similaire à Data Ingestion with Apache NiFi and MiNiFi (20)

AIDevWorldApacheNiFi101

Integração de Dados com Apache NIFI - Marco Garcia Cetax

Automate your data flows with Apache NIFI

Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi

Introduction to Apache NiFi dws19 DWS - DC 2019

Using Apache NiFi with Apache Pulsar for Fast Data On-Ramp

OSSNA Building Modern Data Streaming Apps

Timothy Spann [StreamNative] | Using FLaNK with InfluxDB for EdgeAI IoT at Sc...

Using FLiP with influxdb for EdgeAI IoT at Scale

Moving to hyper-converged? Don’t forget about data protection.

Budapest Data/ML - Building Modern Data Streaming Apps with NiFi, Flink and K...

OpenSource and the Cloud ApacheCon.pptx

Big Data Day LA 2015 - Always-on Ingestion for Data at Scale by Arvind Prabha...

Conf42Python -Using Apache NiFi, Apache Kafka, RisingWave, and Apache Iceberg...

Lightning Fast Analytics with Hive LLAP and Druid

xGem Data Stream Processing

Devfest uk & ireland using apache nifi with apache pulsar for fast data on-r...

Unconference Round Table Notes

Machine Data 101

Big Data for Security - DNS Analytics

Dernier

Injustice - Developers Among Us (SciFiDevCon 2024)Allon Mureinik

IAC 2024 - IA Fast Track to Search Focused AI SolutionsEnterprise Knowledge

Boost PC performance: How more available memory can improve productivityPrincipled Technologies

Scaling API-first – The story of a global engineering organizationRadu Cotescu

Histor y of HAM Radio presentation slidevu2urc

Google AI Hackathon: LLM based Evaluator for RAGSujit Pal

Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...Neo4j

Data Cloud, More than a CDP by Matt RobisonAnna Loughnan Colquhoun

08448380779 Call Girls In Diplomatic Enclave Women Seeking MenDelhi Call girls

[2024]Digital Global Overview Report 2024 Meltwater.pdfhans926745

🐬 The future of MySQL is Postgres 🐘RTylerCroy

Salesforce Community Group Quito, Salesforce 101Paola De la Torre

Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...gurkirankumar98700

08448380779 Call Girls In Greater Kailash - I Women Seeking MenDelhi Call girls

The Codex of Business Writing Software for Real-World Solutions 2.pptxMalak Abu Hammad

Maximizing Board Effectiveness 2024 Webinar.pptxOnBoard

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...HostedbyConfluent

The 7 Things I Know About Cyber Security After 25 Years | April 2024Rafal Los

A Call to Action for Generative AI in 2024Results

WhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure servicePooja Nehwal

Dernier (20)

Injustice - Developers Among Us (SciFiDevCon 2024)

IAC 2024 - IA Fast Track to Search Focused AI Solutions

Boost PC performance: How more available memory can improve productivity

Scaling API-first – The story of a global engineering organization

Histor y of HAM Radio presentation slide

Google AI Hackathon: LLM based Evaluator for RAG

Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...

Data Cloud, More than a CDP by Matt Robison

08448380779 Call Girls In Diplomatic Enclave Women Seeking Men

[2024]Digital Global Overview Report 2024 Meltwater.pdf

🐬 The future of MySQL is Postgres 🐘

Salesforce Community Group Quito, Salesforce 101

Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...

08448380779 Call Girls In Greater Kailash - I Women Seeking Men

The Codex of Business Writing Software for Real-World Solutions 2.pptx

Maximizing Board Effectiveness 2024 Webinar.pptx

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...

The 7 Things I Know About Cyber Security After 25 Years | April 2024

A Call to Action for Generative AI in 2024

WhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure service

Data Ingestion with Apache NiFi and MiNiFi

1. Data Ingestion with Apache NiFi and MiNiFi Lucian Neghina Big Data Architect by Developer for Developers

2. ➔ Periodically batch ingestion ➔ Incremental ingestion ➔ Over 100 servers ➔ No access to the servers ➔ Push access only ➔ Networking failure ➔ Secure communication Challenges

3. Data Flow

4. Simplistic view of Data Flow

5. Realistic view of Data Flow

6. Apache NiFi NiFi is based on Flow Based Programming ➔ Data agnostic ➔ Guaranteed data delivery ➔ Data Buffering / Back-Pressure ➔ Prioritized queuing ➔ Secure (SSL, HTTPS, encrypted content...) ➔ Data provenance ➔ Support push and pull models ➔ Designed for extension ➔ Clustering

7. Apache NiFi What is NiFi used for? ➔ Reliable and secure transfer of data between systems ➔ Delivery of data from sources to analytic platforms ➔ Enrichment and preparation of data: ◆ Conversion between formats ◆ Extraction /Parsing ◆ Routing decisions What is NiFi NOT used for? ➔ Distributed computation ➔ Complex Event Processing ➔ Long term data storage

8. Apache MiNiFi Subproject of Apache NiFi ➔ Agent model to collect data at the edge ➔ Smaller footprint ➔ Design & Deploy ➔ Warm re-deploys NiFi lives in the data center MiNiFi lives as close to where data is born

9. System Architecture Physical Store ... Data Center Server Cluster

10. Data Flow - Server-side (NiFi)

11. Data Flow - Agent-side (MiNiFi)

12. Thank you ● NiFi Template https://github.com/nflucian/nifi/raw/master/DemoNiFi.xml ● twitter: @nflucian ● github: github.com/nflucian