1. Intelligent, Automatic Restarts for
Unhealthy Kafka Consumers on
Kubernetes
Chris Shepherd

2. ● Cloudflare has been using Kafka in production since 2014.
● 14 distinct Kafka clusters, across multiple data centers.
● Running roughly 330 nodes.
● Some consumer groups handling 50m messages per second.
● Over 1 trillion messages processed!
Kafka at Cloudflare

3. Kafka applications on Kubernetes

4. Keeping Kubernetes
applications healthy

5. ● Liveness probes: Ensures that an application within a container is
live and operational. The kubelet uses liveness probes to know
when to restart a container.
● Readiness probes: Used to determine when a container is ready to
start accepting traffic. A Pod is considered ready when all of its
containers are ready.
● Startup probes: Replace liveness checks on slow starting
containers to avoid them being killed before they are up & running.
Kubernetes health checks

6. Kubernetes health checks

7. ● Kafka applications don’t typically accept HTTP traffic, so readiness
probes are redundant. For this reason we configure only liveness
checks on them.
● These checks are usually configured to start after a brief period
(e.g. 10s) and are executed at intervals to check if the application is
healthy (e.g. 15s).
● We also usually tolerate a reasonable amount of failed health
checks before restarting the pod (e.g. 3 times).
Health checks for Kafka applications

8. A naive approach to Kafka
health checks

9. A naive approach

10. Why it doesn’t always work

11. Noisy Oncall

12. A first iteration on smart
health checks

13. ● This uses two values:
● Current (latest) offset - last message sent to the
topic
● Committed offset - last message processed by
consumer
● Check consumer is moving forwards by ensuring:
● Latest offset is changing - receiving new
messages
● Offsets are committed - consumer processing
new messages
A new approach!

14. Smart Health Check

15. Why it didn’t work for us

16. Our final approach
● Each replica only keeps track of it’s own
partitions offsets
● Sarama library has the functionality to
observe when a rebalancing happens
● We remove keys from the in-memory
offset map so it only includes the
relevant partition values
for {
select {
case message, ok := <-claim.Messages(): // <-- Message received
// Store latest received offset in-memory
offsetMap[message.Partition] = message.Offset
// Handle message
handleMessage(ctx, message)
// Commit message offset
session.MarkMessage(message, "")
case <-session.Context().Done(): // <-- Rebalance happened
// Remove rebalanced partition from in-memory map
delete(offsetMap, claim.Partition())
}
}

17. Another Gotcha!

18. ● Helpful debug logging:
● Partition being consumed
● Offsets being committed
● Duration of health checks
● Metrics for visibility on health check
performance is vital
Visibility is important!

19. ● Without proper thought, “dumb” health checks can lead to a false sense of security
that a service is running as expected even when it’s not.
● Good health checks can often be the difference between engineers being
called out to fix trivial issues and a service which is self-healing
● Besides initial teething problems, we have reduced the number of PagerDuty
alerts in our team by 50% since using smart health checks
● It’s important to think about specific behavior of the service and decide what
being unhealthy means in each instance, instead of just ensuring that
dependent services are connected.
Takeaways

20. Thank you!
chris-shepherd1993
Chris Shepherd Senior Systems Engineer
Check out the Cloudflare blog for more information on our smart health checks -
Intelligent, automatic restarts for unhealthy Kafka consumers