Presenter - Dr Sandeep Deshmukh, Committer Apache Apex, DataTorrent engineer Abstract: Ingesting and extracting data from Hadoop can be a frustrating, time consuming activity for many enterprises. Apache Apex Data Ingestion is a standalone big data application that simplifies the collection, aggregation and movement of large amounts of data to and from Hadoop for a more efficient data processing pipeline. Apache Apex Data Ingestion makes configuring and running Hadoop data ingestion and data extraction a point and click process enabling a smooth, easy path to your Hadoop-based big data project. In this series of talks, we would cover how Hadoop Ingestion is made easy using Apache Apex. The third talk in this series would focus on ingesting unbounded data from Kafka to JDBC with couple of processing operators -Transform and enrichment.

1. Meetup Series on
Hadoop Ingestion Made Easy
Part 3: Ingestion of Unbounded Data
(Kafka -> Enrich -> Transform -> JDBC)
Dr. Sandeep Deshmukh
Committer@Apache Apex
Engineer@DataTorrent
Yogi Devendra
Committer@Apache Apex
Engineer@DataTorrent

2. 2
What is Ingestion
Data ingestion
• A process of obtaining, importing, and analyzing data for later use or storage
in a database
Big Data Ingestion
• Discovering the data sources
• Importing the data
• Processing data to produce intermediate data
• Sending data out to durable data stores
ETL + Big Data => Data ingestion

3. 3
Requirements
• Read from stream
• Kafka
• JMS
• JDBC
• ….
• Enrich
• Transform
• Store
• JDBC

4. 4
Use Case
Kafka
Reader
Enrich Transform
JDBC
Output
● Kafka 0.9
● Multi-topic
● Multi-partition
● Enrich State
using City
● Mark a
transaction as
HIGH or LOW
value
● Store in
DB
Parallel Partition can scale to large volume

5. 5 5
Let us walk-through the code and appreciate how simple it is to
use Apex
● Code walk-through for an data ingestion pipeline using
○ Operators/Modules in Malhar
○ Drag-n-drop user interface
● Validate
○ Partitioning
○ Scalability
Let us create and run the app

6. 6
Resources
• http://apex.apache.org/
• Learn more: http://apex.apache.org/docs.html
• Subscribe - http://apex.apache.org/community.html
• Download - http://apex.apache.org/downloads.html
• Follow @ApacheApex - https://twitter.com/apacheapex
• Meetups – http://www.meetup.com/pro/apacheapex/
• More examples: https://github.com/DataTorrent/examples
• Slideshare: http://www.slideshare.net/ApacheApex/presentations
• https://www.youtube.com/results?search_query=apache+apex
• Free Enterprise License for Startups -
https://www.datatorrent.com/product/startup-accelerator/

7. Q&A
7