"Kafka often finds itself as the backbone of a company’s systems, but the failure modes and signals leading to those failures are not always well understood. Chaos Engineering espouses empiricism, experimentation over testing, and verification over validation. We can prime a Kafka cluster as a Chaos Experiment by putting it under a controlled load test called a ‘squeeze test’. This session gives attendees confidence in the steps needed to build experiments to prove Kafka cluster(s) can fulfill the needs of the business. We start by demonstrating how to build a “steady state” hypothesis based on cluster sizing, best practices and expected usage, monitoring configuration, and perfunctory performance testing. We then develop an example hypothesis that as the load on the Kafka cluster increases towards the tipping point we will receive monitoring alerts/signals for key metrics. Attendees learn in detail how real world events were varied for the experiment, including design goals, hard trade-offs, and safety mechanisms necessary for the load tool to adhere to Chaos Engineering principles. We show how the results were analyzed to support or debunk the hypothesis. Finally, we lay out the next steps for attendees’ Chaos Engineering journey."

1. Don’t Let Kafka Be A Cluster:
Kafka Chaos Experimentation
Justin Fetherolf

2. Welcome!
VERICA | CONTINUOUS VERIFICATION
● What is Chaos Engineering?
● Why should we care?
● Experiment design
● Experiment results
● What’s next?

3. What is Chaos
Engineering?
VERICA | CONTINUOUS VERIFICATION

4. What does it all mean?
● Chaos Engineering
○ “the facilitation of experiments to uncover systemic weaknesses.”
● Experiment
○ “an operation or procedure carried out under controlled conditions in
order to discover an unknown effect, to test or establish a hypothesis, or
to illustrate a known law”
● Use experiments to create new knowledge
○ Tests make assertions about known properties
● Experiments verify behavior; not validate
VERICA | CONTINUOUS VERIFICATION

5. Chaos Engineering Principals
● Define “steady state”
● Form a hypothesis
● Introduce variables
● Attempt to disprove hypothesis
VERICA | CONTINUOUS VERIFICATION

6. Advanced Principles
● Build hypothesis around steady-state behavior
● Vary real-world events
● Run experiments in production
● Automate experiments to run continuously
● Minimize blast radius
VERICA | CONTINUOUS VERIFICATION

7. Why should
we care about
Chaos Engineering?
VERICA | CONTINUOUS VERIFICATION

8. Complex Systems
● Businesses require capabilities/properties/features
● Requires complexity from systems
● Can’t avoid complexity
● Embrace and navigate complexity
● As complexity increases, can’t maintain mental
model
VERICA | CONTINUOUS VERIFICATION

9. ● Kafka sits at the core of our businesses
● Kafka is a complex system
● More complex systems built on top of Kafka
● Cloud infrastructure isn’t always what we expect
● Know the safety margins of our systems
VERICA | CONTINUOUS VERIFICATION
Chaofka?

10. Our Kafka
Experiment
VERICA | CONTINUOUS VERIFICATION

11. Steady State
● Cluster
○ 5-node EKS w/ 1 broker per node - 5-broker Kafka cluster
○ t2.xlarge instance types
■ “moderate” network - 83.4 - 107.3 MiB/s
■ 20 GB “gp2” EBS volumes - 128 MiB/s
○ Metrics to Prometheus/Grafana
● Batch style workload; ~3 min @ 2.5 MiB/s every 5 min
○ ~5 million messages produced and consumed
○ 3 partitions, 3 replicas
VERICA | CONTINUOUS VERIFICATION

12. Steady State Metrics
VERICA | CONTINUOUS VERIFICATION

13. Hypothesis
“As the load on the Kafka cluster increases, the standard workload
can continue to successfully process each batch of messages
before the next batch begins.”
● How do we measure this?
○ Monitoring
■ Message/data rates
■ CPU/Memory/Net/Disk usage
○ Application status
VERICA | CONTINUOUS VERIFICATION

14. Introducing Variables
● How do we increase load? Enter Horus!
○ Scalable & configurable
○ Safety features
■ Halting could be triggered by
● Cluster or client metrics
● Other conditions
● Manual intervention
VERICA | CONTINUOUS VERIFICATION

15. Load Scaling Configuration
VERICA | CONTINUOUS VERIFICATION
● 4 distinct, increasing client sets; 15 minutes each
● 5 partition, 5 replica topic
● 10 - 40 producers; 10 step
○ 500 msg/s; 1024 byte/msg
● 7 consumer groups; 3 consumers each
● 4.88 - 19.5 MiB/s total production traffic
● 34.18 - 136.72 MiB/s total consumer traffic
● Increased replication traffic

16. Experiment Results
VERICA | CONTINUOUS VERIFICATION

17. What’s Next?
VERICA | CONTINUOUS VERIFICATION

18. The Future!
● Context sensitive
○ One size does not fit all
● Start small
● Start in non-production environment
● Minimize blast radius
● Unleash the Chaos!
VERICA | CONTINUOUS VERIFICATION

19. References and Resources
● Rosenthal, Casey and Jones, Nora. Chaos Engineering: System Resiliency in
Practice. 1st ed., O’Reilly, 2020.
● Hausmann, Steffen. “Best practices for right-sizing your Apache Kafka
clusters to optimize performance and cost.” AWS Big Data Blog, 17 Mar. 2022,
https://aws.amazon.com/blogs/big-data/best-practices-for-right-sizing-your-
apache-kafka-clusters-to-optimize-performance-and-cost/
● https://principlesofchaos.org/
● https://www.verica.io
● https://www.thevoid.community/
VERICA | CONTINUOUS VERIFICATION

20. Justin Fetherolf
Sr. Software Engineer
https://www.verica.io