In the financial industry, losing data is unacceptable. Financial firms are adopting Kafka for their critical applications. Kafka provides the low latency, high throughput, high availability, and scale that these applications require. But can it also provide complete reliability? As a system architect, when asked “Can you guarantee that we will always get every transaction,” you want to be able to say “Yes” with total confidence.
In this session, we will go over everything that happens to a message – from producer to consumer, and pinpoint all the places where data can be lost – if you are not careful. You will learn how developers and operation teams can work together to build a bulletproof data pipeline with Kafka. And if you need proof that you built a reliable system – we’ll show you how you can build the system to prove this too.
3. “If data is the lifeblood of high technology, Apache
Kafka is the circulatory system”
--Todd Palino
Kafka SRE @ LinkedIn
4. If Kafka is a critical piece of our pipeline
Can we be 100% sure that our data will get there?
Can we lose messages?
How do we verify?
Who’s fault is it?
5. Distributed Systems
Things Fail
Systems are designed to
tolerate failure
We must expect failures
and design our code and
configure our systems to
handle them
6. Network
Broker MachineClient Machine
Data Flow
Kafka Client
Broker
O/S Socket Buffer
NIC
NIC
Page Cache
Disk
Application Thread
O/S Socket Buffer
async
callback
✗
✗
✗
✗
✗
✗
data
ack /
exception
7. Client Machine
Kafka Client
O/S Socket Buffer
NIC
Application Thread
✗
✗
Broker Machine
Broker
NIC
Page Cache
Disk
O/S Socket Buffer
miss
✗
✗
✗
Network
Data Flow
✗
data
offsets
ZK
Kafka✗
8. Replication is your friend
Kafka protects against failures by replicating data
The unit of replication is the partition
One replica is designated as the Leader
Follower replicas fetch data from the leader
The leader holds the list of “in-sync” replicas
10. ISR
• 2 things make a replica in-sync
– Lag behind leader
• replica.lag.time.max.ms – replica that didn’t fetch or is behind
• replica.lag.max.messages – will go away has gone away in 0.9
– Connection to Zookeeper
11. Terminology
• Acked
– Producers will not retry sending.
– Depends on producer setting
• Committed
– Consumers can read.
– Only when message got to all
ISR.
• replica.lag.time.max.ms
– how long can a dead replica
prevent consumers from
reading?
12. Replication
• Acks = all
– only waits for in-sync replicas to reply.
Replica 3
100
Replica 2
100
Replica 1
100
Time
13. • Replica 3 stopped replicating for some reason
Replication
Replica 3
100
Replica 2
100
101
Replica 1
100
101
Time
Acked in acks = all
“committed”
Acked in acks = 1
but not
“committed”
17. Replication
• If Replica 2 or 3 come back online before the leader, you can will lose
data.
Replica 3
100
Replica 2
100
101
Replica 1
100
101
102
103
104Time
All those are
“acked” and
“committed”
18. So what to do
• Disable Unclean Leader Election
– unclean.leader.election.enable = false
• Set replication factor
– default.replication.factor = 3
• Set minimum ISRs
– min.insync.replicas = 2
19. Warning
• min.insync.replicas is applied at the topic-level.
• Must alter the topic configuration manually if created
before the server level change
• Must manually alter the topic < 0.9.0 (KAFKA-2114)
24. Producer Internals
• Producer sends batches of messages to a buffer
M3
Application
Thread
Application
Thread
Application
Thread
send()
M2 M1 M0
Batch 3
Batch 2
Batch 1
Fail
? response
retry
Update Future
callback
drain
Metadata or
Exception
25. Basics
• Durability can be configured with the producer
configuration request.required.acks
– 0 The message is written to the network (buffer)
– 1 The message is written to the leader
– all The producer gets an ack after all ISRs receive the data; the
message is committed
• Make sure producer doesn’t just throws messages away!
– block.on.buffer.full = true
26. “New” Producer
• All calls are non-blocking async
• 2 Options for checking for failures:
– Immediately block for response: send().get()
– Do followup work in Callback, close producer after error threshold
• Be careful about buffering these failures. Future work? KAFKA-1955
• Don’t forget to close the producer! producer.close() will block until in-flight txns
complete
• retries (producer config) defaults to 0
• message.send.max.retries (server config) defaults to 3
• In flight requests could lead to message re-ordering
27.
28. Consumer
• Three choices for Consumer API
– Simple Consumer
– High Level Consumer (ZookeeperConsumer)
– New KafkaConsumer
29. New Consumer – attempt #1
props.put("enable.auto.commit", "true");
props.put("auto.commit.interval.ms", "10000");
KafkaConsumer<String, String> consumer = new KafkaConsumer<String, String>(props);
consumer.subscribe(Arrays.asList("foo", "bar"));
while (true) {
ConsumerRecords<String, String> records = consumer.poll(100);
for (ConsumerRecord<String, String> record : records) {
processAndUpdateDB(record);
}
}
What if we crash
after 8 seconds?
Commit automatically every
10 seconds
30. New Consumer – attempt #2
props.put("enable.auto.commit", "false");
KafkaConsumer<String, String> consumer = new KafkaConsumer<String, String>(props);
consumer.subscribe(Arrays.asList("foo", "bar"));
while (true) {
ConsumerRecords<String, String> records = consumer.poll(100);
for (ConsumerRecord<String, String> record : records) {
processAndUpdateDB(record);
consumer.commitSync();
What are you really
committing?
31. New Consumer – attempt #3
props.put("enable.auto.commit", "false");
KafkaConsumer<String, String> consumer = new KafkaConsumer<String, String>(props);
consumer.subscribe(Arrays.asList("foo", "bar"));
while (true) {
ConsumerRecords<String, String> records = consumer.poll(100);
for (ConsumerRecord<String, String> record : records) {
processAndUpdateDB(record);
TopicPartition tp = new TopicPartition(record.topic(), record.partition());
OffsetAndMetadata oam = new OffsetAndMetadata(record.offset() +1);
consumer.commitSync(Collections.singletonMap(tp,oam));
Is this fast enough?
32. New Consumer – attempt #4
props.put("enable.auto.commit", "false");
KafkaConsumer<String, String> consumer = new KafkaConsumer<String, String>(props);
consumer.subscribe(Arrays.asList("foo", "bar"));
int counter = 0;
while (true) {
ConsumerRecords<String, String> records = consumer.poll(500);
for (ConsumerRecord<String, String> record : records) {
counter ++;
processAndUpdateDB(record);
if (counter % 100 == 0) {
TopicPartition tp = new TopicPartition(record.topic(), record.partition());
OffsetAndMetadata oam = new OffsetAndMetadata(record.offset() + 1);
consumer.commitSync(Collections.singletonMap(tp, oam));
36. Rebalance Listener
public class MyRebalanceListener implements ConsumerRebalanceListener {
@Override
public void onPartitionsAssigned(Collection<TopicPartition> partitions) {
}
@Override
public void onPartitionsRevoked(Collection<TopicPartition> partitions) {
commitOffsets();
}
}
consumer.subscribe(Arrays.asList("foo", "bar"), new MyRebalanceListener());
Careful! This method will need
to know the topic, partition and
offset of last record you got
37. At Least Once Consuming
1. Commit your own offsets - Set autocommit.enable =
false
2. Use Rebalance Listener to limit duplicates
3. Make sure you commit only what you are done processing
4. Note: New consumer is single threaded – one consumer
per thread.
38. Exactly Once Semantics
• At most once is easy
• At least once is not bad either – commit after 100% sure
data is safe
• Exactly once is tricky
– Commit data and offsets in one transaction
– Idempotent producer
39. Using External Store
• Don’t use commitSync()
• Implement your own “commit” that saves both data and
offsets to external store.
• Use the RebalanceListener to find the correct offset
40. Seeking right offset
public class SaveOffsetsOnRebalance implements ConsumerRebalanceListener {
private Consumer<?,?> consumer;
public void onPartitionsRevoked(Collection<TopicPartition> partitions) {
// save the offsets in an external store using some custom code not described here
for (TopicPartition partition : partitions)
saveOffsetInExternalStore(consumer.position(partition));
}
public void onPartitionsAssigned(Collection<TopicPartition> partitions) {
// read the offsets from an external store using some custom code not described here
for (TopicPartition partition : partitions)
consumer.seek(partition, readOffsetFromExternalStore(partition));
}
}
41. Monitoring for Data Loss
• Monitor for producer errors – watch the retry numbers
• Monitor consumer lag – MaxLag or via offsets
• Standard schema:
– Each message should contain timestamp and originating service and
host
• Each producer can report message counts and offsets to a
special topic
• “Monitoring consumer” reports message counts to another
special topic
• “Important consumers” also report message counts
• Reconcile the results
Low Level Diagram: Not talking about producer / consumer design yet…maybe this is too low-level though
Show diagram of network send -> os socket -> NIC -> ---- NIC -> Os socket buffer -> socket -> internal message flow / socket server -> response back to client -> how writes get persisted to disk including os buffers, async write etc
Then overlay places where things can fail.
Low Level Diagram: Not talking about producer / consumer design yet…maybe this is too low-level though
Show diagram of network send -> os socket -> NIC -> ---- NIC -> Os socket buffer -> socket -> internal message flow / socket server -> response back to client -> how writes get persisted to disk including os buffers, async write etc
Then overlay places where things can fail.
Highlight boxes with different color
Kafka exposes it’s binary TCP protocl via a Java api which is what we’ll be discussing here.
So everything in the box is what’s happening inside the producer. Generally speaking, you have an application thread, or threads that take individual messages and “send” them to Kafka. What happens under the covers is that these messages are batched up where possible in order to amortize the overhead of the send, stored in a buffer and communicated over to kafka. After Kafka has completed it’s work, a response is returned back for each message. This happens asynchronously, using Java’s concurrent API.
This response is comprised of either an exception or a metadata record. If the metadata is returned, which contains the offset, partition and topic, then things are good and we continue processing. However, if an error has returned the producer will automatically retry the failed message, up to a configurable # or amount of time. When this exception occurs and we have retries enabled, these retries actually just go right back to the start of the batches being prepared to send back to Kafka.
Commit every 10 seconds, but we don’t really have any control over what’s processed, and this can lead to duplicates
If you are doing too much work commits don’t count as heartbeat ->
So lets so we have auto-commit enabled, and we are chugging along, and counting on the consumer to commit our offsets for us. This is great because we don’t have to code anything, and don’t have think about the frequency of commits and the impact that might have on our throughput. Life is good. But now we’ve lost a thread or a process. And we don’t really know where we are in the processing, Because the last auto-commit committed stuff that we hadn’t actually written to disk.
So now we’re in a situation where we think we’ve read all of our data but we will have gaps in data. Note the same risk applies if we lose a partition or broker and get a new leader. OR