Hello, kafka! (an introduction to apache kafka)

Hello, Kafka!
(An Introduction to Apache Kafka)
Timothy Spann - Principal DataFlow Field Engineer
July-2021
https://www.meetup.com/futureofdata-boston/
@PaasDev

© 2021 Cloudera, Inc. All rights reserved. 3
Welcome to Future of Data - Virtual - 15/July/2021
@PaasDev
https://www.meetup.com/futureofdata-princeton/
https://www.meetup.com/futureofdata-newyork/
https://www.meetup.com/futureofdata-philadelphia/
https://www.meetup.com/futureofdata-boston/
From Big Data to AI to Streaming to Containers to
Cloud to Analytics to Cloud Storage to Fast Data to
Machine Learning to Microservices to ...

https://github.com/tspannhw https://www.datainmotion.dev/

CLOUDERA DATAFLOW DATA-IN-MOTION PLATFORM

AGENDA
● What is Event Streaming?
● What is Apache Kafka?
● What Can You Do With Apache Kafka?
● An Introduction to Apache Kafka
● Demos
● Q&A
● Raﬄe
● Closing Remarks

What is Event Streaming?
Events are data points that are delivered in a stream.
In Event Streaming we work with data in motion often from systems that continuously
produce data such as logs, IIoT devices, distributed applications, live orders, CDC from
production databases, stock data, temperature, weather feeds, sensors, time series data and
more.
Events have data and various timestamps that let us know things like creation date/time,
processing date/time and more.
https://en.wikipedia.org/wiki/Event-driven_architecture

What is Event Streaming?
OVERVIEW
A comprehensive edge-to-cloud
real-time streaming data platform.
Cloudera Dataflow (CDF) is a scalable, real-time streaming data platform that ingests, curates, and analyzes data for key
insights and immediate actionable intelligence. DataFlow addresses the following challenges:
● Processing real-time data streaming at high volume and high scale
● Tracking data provenance and lineage of streaming data
● Managing and monitoring edge applications and streaming sources
● Gaining real-time insights and actionable intelligence from streaming data

WHAT IS REAL-TIME?

What is Apache Kafka?
– Distributed: horizontally scalable (just like Hadoop!)
– Partitioned: the data is split-up and distributed across the brokers
– Replicated: allows for automatic failover
– Unique: Kafka does not track the consumption of messages (the consumers
do)
– Fast: designed from the ground up with a focus on performance and
throughput
– Kafka was built at Linkedin in 2011
– Open sourced as an Apache project

Yes, Franz, It’s Kafka
Let’s do a metamorphosis on your data. Don’t fear changing data.
You don’t need to be a brilliant writer to stream
data.
Franz Kafka was a German-speaking
Bohemian novelist and short-story writer,
widely regarded as one of the major figures of
20th-century literature. His work fuses
elements of realism and the fantastic.
Wikipedia

What is Can You Do With Apache Kafka?
• Web site activity: track page views, searches, etc. in real time
• Events & log aggregation: particularly in distributed systems where messages
come from multiple sources
• Monitoring and metrics: aggregate statistics from distributed applications and
build a dashboard application
• Stream processing: process raw data, clean it up, and forward it on to another
topic or messaging system
• Real-time data ingestion: fast processing of a very large volume of messages

KAFKA TERMINOLOGY
• Kafka is a publish/subscribe messaging system comprised of the
following components:
– Topic: a message feed
– Producer: a process that publishes messages to a topic
– Consumer: a process that subscribes to a topic and processes its messages
– Broker: a server in a Kafka cluster

Apache Kafka
• Highly reliable distributed
messaging system
• Decouple applications, enables
many-to-many patterns
• Publish-Subscribe semantics
• Horizontal scalability
• Eﬃcient implementation to
operate at speed with big data
volumes
• Organized by topic to support
several use cases
Source
System
Source
System
Source
System
Kafka
Fraud
Detection
Security
Systems
Real-Time
Monitoring
Source
System
Source
System
Source
System
Fraud
Detection
Security
Systems
Real-Time
Monitoring
Many-To-Many
Publish-Subscribe
Point-To-Point
Request-Response

KAFKA COMPONENTS
Kafka Cluster
producer
producer
producer
consumer
consumer
consumer
brokers
Kafka uses ZooKeeper to coordinate
brokers with consumers

Kafka: Anatomy of a Topic
Partition 0 Partition 1 Partition 2
0 0 0
1 1 1
2 2 2
3 3 3
4 4 4
5 5 5
6 6 6
7 7 7
8 8 8
9 9 9
10 10
11 11
12
Writes
Old
New
APACHE
KAFKA

Kafka: Under the Hood
Broker 1
Topic-1
Partition-0
Zookeeper
Stores Information about cluster
status and consumer offsets
APACHE
KAFKA
Broker 2
Topic-1
Partition-1
Broker 3
Topic-1
Partition-2
producer
consumer
Kafka
Cluster
producer
consumer
consumer

Kafka Basics
Kafka has 4 core APIs
1. Producer API
2. Consumer API
3. Streams API
4. Connector API
Anatomy of a Kafka Topic
Kafka Consumers

OVERVIEW OF TOPICS
• A topic is a name assigned to a feed to which messages are published
– A topic in Kafka is partitioned
• Each partition is an ordered, immutable sequence of messages
– it is continually appended to
– each message is assigned a sequential id called an offset
• Messages are retained for a conﬁgurable amount of time (24 hours, 7
days, etc.)
• Each consumer retains its own offset in the partition
– allows the consumer to go back and re-read messages without retaining the
message
– the offset is the only metadata that the consumer retains
– different consumers maintain their own offset

PUBLISHING MESSAGES
producer
message_a
message_b
message_c
message_d
message_e
message_f
1. A producer publishes messages to a topic
2. The producer decides which
partition to send each message to
offset -> 0 1 2 3 4
Partition 0 message_b message_f
Partition 1 message_a message_c message_
e
Partition 2 message_d
Old New
3. New messages are written to the
end of the partition
consumer
4. A consumer fetches messages from a
partition by specifying an offset

LEADER AND FOLLOWERS
Broker 1
my_topic
Partition-1 (follower)
Broker 2
my_topic
Partition-1 (leader)
Broker 3
my_topic
Partition-1 (follower)
producer
consumer
producer
consumer
consumer
The leader handles all
read and write requests

CONSUMING MESSAGES
• Messages are consumed in Kafka by a consumer group
• Each individual consumer is labeled with a group name
• Each message in a topic is sent to one consumer in the group
• In other words, messages are consumed at the group level, not at the individual
consumer level
– This allows for fault tolerance and scalability of consumers
• This design allows for both queue and publish-subscribe models:
– If you need a queue behavior, then simply place all consumers into the same group
– If you need a publish-subscribe model, then create multiple consumer groups that
subscribe to a topic

CONSUMER GROUPS
Kafka Cluster
Broker 1
my_topic: Partition-0
Broker 2
Consumer Group A
consumer consumer
consumer consumer
Consumer Group B
consumer consumer
consumer consumer
consumer
message_1
Each message is
consumed by one
consumer per
group

THE CONSUMER OFFSET
It is up to the consumer to maintain its offset in the partition (stored in a
special topic named __consumer_offsets)
0 1 2 3 4 5 6 7 8 9 10 11 12
Messages a b c d e f g h i j
• This has several key benefits, including:
• performance: there is no back-and-forth acknowledging of message consumption
• simplicity: the consumer only has to maintain a single integer value for its state, which can
be easily stored and shared between consumers (if a failure occurs)
• re-consume messages: it becomes trivial for a consumer to re-consume messages
consumer
offset

MESSAGE DELIVERY GUARANTEES
• Kafka guarantees at-least-once delivery by default
• At-most-once delivery is possibly by disabling retries on the producer (when a
commit fails)
• Exactly-once delivery is possible (with clever coordination of your consumers and
the consumer offset)
• Other guarantees:
– Messages in a partition are stored in the order that they were sent by the publisher
– Each partition is consumed by exactly one consumer in the group
– That consumer is the only reader in the group of that partition in the group
– Messages are consumed in order
– Messages committed to the log are not lost for up to N-1 broker failures

IN-SYNC REPLICAS
• Kafka replicates the messages in each partition across multiple brokers
• New messages are always appended to the leader
• A follower that keeps up is called an ISR, or in-sync replica, which means:
• A message is considered committed when all ISRs have a copy of the
message

How does Kafka preserve message order
⬢ Partition algorithm is ﬁxed (hash on key)
⬢ Stored as a log sequential write to a ﬁle
⬢ Consume in order based on offset

How does Kafka prevent data loss
⬢ Replicate, replicate, replicate
⬢ Acknowledge you got the message
⬢ Keep it even after it is consumed

AT-MOST-ONCE

AT-LEAST-ONCE

EXACTLY-ONCE

Demo Time
Produce Messages
Consume Messages
View Messages in SMM
Show Details of Brokers, Topics, Consumer Groups, Producers, Partitions

Download these assets today

TH N Y U

Hello, kafka! (an introduction to apache kafka)

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Hello, kafka! (an introduction to apache kafka)

Similar to Hello, kafka! (an introduction to apache kafka) (20)

More from Timothy Spann

More from Timothy Spann (20)

Recently uploaded

Recently uploaded (20)

Hello, kafka! (an introduction to apache kafka)