This document discusses AWS Data Pipeline, which is a service that helps reliably process and move data between AWS services and on-premises sources at specified intervals. It describes how AWS Data Pipeline can be used to build a data pipeline that collects data from different sources, performs analytics using EMR, and generates weekly reports. The key components of AWS Data Pipeline include the pipeline definition, pipeline, task runner, schedules, and pipeline components like instances and attempts. It also provides an example of using AWS Data Pipeline to import and export data from DynamoDB.

2. AWS Architect Certification Training www.edureka.co/cloudcomputing
Agenda
01
Need for Data
Pipeline
02
What is
AWS Data Pipeline?
03
AWS Data Pipeline
components
04
Demo on AWS
Data Pipeline

3. Copyright © 2018, edureka and/or its affiliates. All rights reserved.
Need of Data Pipeline

4. AWS Architect Certification Training www.edureka.co/cloudcomputing
A Hypothetical Example
Goal 1
Improve business by
targeting content
Goal 2
Manage application
efficiently
Goal 3
Improve business faster
but at cheaper rate

5. AWS Architect Certification Training www.edureka.co/cloudcomputing
Problem Statement
Huge amount of data in different formats – so processing,
storing & migrating data becomes complex
Real-time data for registered users
Webserver logs for potential users
Demographic data & login credentials
Sensor data & 3rd party datasets
DynamoDB
Amazon S3
Amazon RDS
Amazon S3

6. AWS Architect Certification Training www.edureka.co/cloudcomputing
Solution
Feasible Solution Optimal Solution
Analyse the data & convert from
unstructured to structured format
Use a data pipeline which handles
processing, visualisations & migration

7. Copyright © 2018, edureka and/or its affiliates. All rights reserved.
AWS Data Pipeline

8. AWS Architect Certification Training www.edureka.co/cloudcomputing
AWS Data Pipeline
AWS Data Pipeline is a web service that helps you reliably process and move
data between different AWS compute and storage services, as well as on-
premises data sources, at specified intervals.
AWS Data Pipeline
Task: Copy log
files
Daily task
Task: Launch
data analysis
Weekly task
EC2 Instance S3 Bucket Amazon EMR

9. AWS Architect Certification Training www.edureka.co/cloudcomputing
AWS Data Pipeline
Data Stores (O/P)Data Stores (I/P) Compute Resources
DATA DATA

10. AWS Architect Certification Training www.edureka.co/cloudcomputing
Example: Launch Data Analysis
Collect data from different data sources, perform EMR
analysis & generate weekly reports
Event data from
DynamoDB
Bulk data from S3
Daily EMR
analytics
Daily EMR results
Weekly summary report Weekly report in
Redshift

11. AWS Architect Certification Training www.edureka.co/cloudcomputing
Benefits of AWS Data PipelineBenefits of AWS Data Pipeline
Provides a drag & drop
console
Built on distributed,
reliable infrastructure
Supports scheduling
& error handling
Distribute work to one
machine or many
Inexpensive to use
Full control over the
computational resources

12. Copyright © 2018, edureka and/or its affiliates. All rights reserved.
AWS Data Pipeline Components

13. AWS Architect Certification Training www.edureka.co/cloudcomputing
Components of AWS Data Pipeline
Pipeline definition
Pipeline
Task Runner
Specifies business logic of data management
Data Nodes Activities Schedules
PreconditionsResourcesActions

14. AWS Architect Certification Training www.edureka.co/cloudcomputing
Components of AWS Data Pipeline
Pipeline definition
Pipeline
Task Runner
Schedules & runs the tasks to perform
defined activities
Pipeline
Components
Instances Attempts

15. AWS Architect Certification Training www.edureka.co/cloudcomputing
Components of AWS Data Pipeline
Pipeline definition
Pipeline
Task Runner
Polls AWS Data Pipeline for tasks & then
performs those tasks
Pipeline schedules
tasks
Task runner
polls tasks
Retries
remaining
Task runner
reports when
task is done
Did task
succeed?
Task ends
Yes No
No
Yes

16. Copyright © 2018, edureka and/or its affiliates. All rights reserved.
Demo – Import & export DynamoDB
data

17. AWS Architect Certification Training www.edureka.co/cloudcomputing