On September 17, 2014 at the NJ Big Data Palooza MeetUp, Kishore Veleti, Big Data Engineer at Knowledgent, presented on Stream Processing with Big Data using Apache Kafka.This presentation includes the content he covered during the event, including an overview of Kafka terminology and processes.

1. Stream Processing with Big Data
Learn Apache Kafka
Kishore Veleti
Big Data Engineer
©2014 Knowledgent Group Inc. All Rights Reserved

2. • Big Data Engineer at Knowledgent
• Background in enterprise application development using
Hadoop stack, Java, PHP
• Worked in Healthcare, Banking, and Social Media
Applications
• Passionate in sharing knowledge
©2014 Knowledgent 2 Group Inc. All Rights Reserved
About Me

3. Tutorial
©2014 Knowledgent 3 Group Inc. All Rights Reserved

4. ©2014 Knowledgent 4 Group Inc. All Rights Reserved
We will discuss:
•What is Apache Kafka?
• Apache Kafka Terminology
• Apache Kafka – about Topic & Partition
• Apache Kafka hands-on

5. What is Apache Kafka?
• Apache Kafka is a publish-subscribe messaging system
implemented as a distributed commit log
©2014 Knowledgent 5 Group Inc. All Rights Reserved
• It is written in Java/Scala
• Built by LinkedIn to process activity stream data from their
website

6. • All the messages in Kafka are real-time
• There are many subscribers to a message
• Kafka persists messages to the disk
• Messages are retained for a specific time period
• Subscribers/clients store the state of their reads
• Easy to replay messages
©2014 Knowledgent 6 Group Inc. All Rights Reserved
What is Apache Kafka?

7. Apache Kafka Terminology
©2014 Knowledgent 7 Group Inc. All Rights Reserved
• Message: A datum to send
• Topic: Kafka maintains messages in categories called “topics”
• Partition: A logical division of a topic
• Producer: An API to publish messages to Kafka topic
• Broker: A server
• Cluster: Kafka cluster comprises one or more brokers
• Consumer: API to consume published messages and process further
• Replication: Kafka replicates log for each partition across servers

8. Apache Kafka Terminology &
Big Picture
Message Topic Partition Producer Broker Consumer
At a high level, producers send messages over the network to
the Kafka cluster.
Kafka cluster in turn serves them up to consumers.
©2014 Knowledgent 8 Group Inc. All Rights Reserved

9. Apache Kafka Terminology &
Big Picture
Message Topic Partition Producer Broker Consumer
Let’s do a hands-on exercise of Kafka with knowledge we’ve
learned until now
©2014 Knowledgent 9 Group Inc. All Rights Reserved

10. Apache Kafka: About Topic and
Partition
Message Topic Partition Producer Broker Consumer
In Kafka for each topic a partition log is maintained.
Each partition is an ordered, immutable sequence of messages that is
appended to
Each message in the partition is assigned a sequential id number called the
offset
©2014 Knowledgent 10 Group Inc. All Rights Reserved
Partition 1
Writes
Partition 2
Partition 3

11. Apache Kafka: About Topic and
Partition
Message Topic Partition Producer Broker Consumer
In Kafka, a Producer is an API to publish messages to topic
©2014 Knowledgent 11 Group Inc. All Rights Reserved

12. Apache Kafka: About Topic and
Partition
Message Topic Partition Producer Broker Consumer
In Kafka, a Consumer is an API to consume messages from topics
©2014 Knowledgent 12 Group Inc. All Rights Reserved

13. Apache Kafka Terminology &
Big Picture
Message Topic Partition Producer Broker Consumer
Let’s do a hands-on exercise of Kafka with knowledge we’ve
learned until now
©2014 Knowledgent 13 Group Inc. All Rights Reserved

14. Apache Kafka Use Cases
©2014 Knowledgent 14 Group Inc. All Rights Reserved
• Trading Systems
- Risk Identification in real-time
• Change Data Capture
- Capturing the changed data into data lake environment
• Online Gaming
- Identifying top scorers of a game

15. Thank you!
Questions?
©2014 Knowledgent 15 Group Inc. All Rights Reserved