2. Agenda
1. What is a Data Pipeline ?
2. Components of a Data pipeline.
3. Traditional Data Flows and issues
4. Introduction to Apache Airflow
5. Features
6. Core Components
7. Key Components
8. Demo
3. What is a Data Pipeline
Data Pipeline is a set of data processing elements connected in series, where the
output of one element is the input of the next one. The elements of the pipeline are
often executed in parallel or in time-sliced fashion. The name โpipelineโ come from
a rough analogy with physical plumbing.
โ Modern data pipelines are used to ingest & process vast volumes of data in
real time.
โ Real time processing of data as opposed to traditional ETL / batch modes.
4. Common Components of a data pipeline
Typical parts of a data pipeline
โ Data Ingestion
โ Filtering
โ Processing
โ Querying of the data
โ Data warehousing
โ Reprocessing capabilities
Typical Requirements
โ Scalability
โ Billions of messages and terabytes of
data 24 /7
โ Availability and redundancy
โ Across physical Locations
โ Latency
โ Real time / Batch
โ Platform support
5. Traditional data flow model
Webclients Reporting
Apps
Public Rest API Billing System
Microservices
OLTP
DB
Report
DB
Metrics
DB
$ curl api.example.com | filter.py | psql
Analytics
6. Messy data flow model ( 6 / 12 months later)
web clients reporting
Apps
Public Rest API Billing System
Microservices
OLTP
DB
Report
DB
Metrics
DB
Analytics
External
cloud
Doc
Store
DWH
7. Apache Airflow Introduction
โ Apache Airflow is a way to programatically author, schedule and monitor
workflows
โ Developed in Python and is open source.
โ Workflows are configured as Python code.
โ It uses python as the programming language, where in we can enrich the quality
of data pipelines by using python inbuilt libraries.
โ Has multiple hooks and operators for handling BigData ecosystem components, (
Hive, Sqoop etc.. ) and DB hooks for RDBMS and Other NOSQL databases.
8. Features
โ Cron replacement
โ Fault tolerant.
โ Dependency rules.
โ Beautiful UI.
โ Handle task failures.
โ Python Code.
โ Report / Alert on failures.
โ Monitor your pipelines from the WebUI.
โ And etc..
9. Core Components
โ Webserver - Apache Airflow WebUI.
โ Scheduler - Responsible for scheduling your jobs.
โ Executor - bound to the scheduler , determine the worker process that
executes the the schedule task. ( Sequential , LocalExecutor, CeleryExecutor)
โ Worker - Process that execute the task , determined by the executor.
โ Metadatabase - Database were all the metadata related to your jobs are stored
10. Key Concepts
โ DAG - Directed Acyclic graph . the graphical representation of your data
pipeline
โ Operator - describes a single task in your data pipeline
โ Task - An instance of operator task.
โ Workflow - DAG + Operator + Task
11. Overview
โ What is a DAG?
โ What is an Operator?
โ Operator relationships and Bitshift composition
โ How the scheduler works?
โ What is a Workflow ?
12. DAG ( Directed Acyclic Graph)
Simple DAG where we could imagine that
Task 1 - downloading the data.
Task 2 - Sending the data for processing.
Task 3 - monitoring the data processing.
Task 4 - generating the report.
Task 5 - Sending the email to the DAG owner or intended recipients.
Task 1 Task 2 Task 3 Task 4 Task 5
14. Operators
While DAG describes how to run a workflow , Operator defines what actually gets
done.
โ Operator describes a single task in a workflow.
โ Operators should be idempotent. ( it should produce the same result
irrespective of how many times it is executed.
โ Retry Automatically in case of Failure.
15. Different Operators
โ Bash Operator
โ Executes a bash command
โ Python Operator
โ Calls an Arbitrary python function
โ Email Operator
โ Sends an Email
โ Mysql Operator, SQLite Operator, Postgres Operator.
โ Executes the SQL commands
โ <Custom Operators> Inheriting from the BaseOperator
16. Types of Operators
There are 3 types of operators
โ Action Operators
โ Perform an action ( Bash operator, Python Operator , Email Operator)
โ Transfer Operators
โ Moving data from one system to another ( PrestoToMySQL operator, SFTP operator
โ Sensor Operators
โ Waiting for the data to arrive at the default location.
17. Important Properties
โ DAGโs are defined in Python files placed into Airflows DAG_FOLDER
โ dag_id serves as a unique identifier for your DAG.
โ description the description of your DAG.
โ start_date - tell when your DAG should start.
โ schedule_interval - define how often your DAG runs.
โ depend_on_past - run the next DAGRun if the previous one is completed
successfully.
โ default_args - a dictionary of variables to be used as constructor keyword
parameter when initializing operators
