Don’t Let Kafka Be A Cluster: Kafka Chaos Experimentation with Justin Fetherolf
Don’t Let Kafka Be A Cluster: Kafka Chaos Experimentation with Justin Fetherolf
"Kafka often finds itself as the backbone of a company’s systems, but the failure modes and signals leading to those failures are not always well understood. Chaos Engineering espouses empiricism, experimentation over testing, and verification over validation. We can prime a Kafka cluster as a Chaos Experiment by putting it under a controlled load test called a ‘squeeze test’. This session gives attendees confidence in the steps needed to build experiments to prove Kafka cluster(s) can fulfill the needs of the business. We start by demonstrating how to build a “steady state” hypothesis based on cluster sizing, best practices and expected usage, monitoring configuration, and perfunctory performance testing. We then develop an example hypothesis that as the load on the Kafka cluster increases towards the tipping point we will receive monitoring alerts/signals for key metrics. Attendees learn in detail how real world events were varied for the experiment, including design goals, hard trade-offs, and safety mechanisms necessary for the load tool to adhere to Chaos Engineering principles. We show how the results were analyzed to support or debunk the hypothesis. Finally, we lay out the next steps for attendees’ Chaos Engineering journey."
Recommandé
Contenu connexe
Similaire à Don’t Let Kafka Be A Cluster: Kafka Chaos Experimentation with Justin Fetherolf
Similaire à Don’t Let Kafka Be A Cluster: Kafka Chaos Experimentation with Justin Fetherolf(20)
Plus de HostedbyConfluent
Plus de HostedbyConfluent(20)
Dernier
Dernier(20)
![[2023] Putting the R! in R&D.pdf](https://cdn.slidesharecdn.com/ss_thumbnails/2023puttingtherinrd-231122234738-294b2909-thumbnail.jpg?width=768&fit=bounds)
Don’t Let Kafka Be A Cluster: Kafka Chaos Experimentation with Justin Fetherolf
- 1. Don’t Let Kafka Be A Cluster: Kafka Chaos Experimentation Justin Fetherolf
- 2. Welcome! VERICA | CONTINUOUS VERIFICATION ● What is Chaos Engineering? ● Why should we care? ● Experiment design ● Experiment results ● What’s next?
- 3. What is Chaos Engineering? VERICA | CONTINUOUS VERIFICATION
- 4. What does it all mean? ● Chaos Engineering ○ “the facilitation of experiments to uncover systemic weaknesses.” ● Experiment ○ “an operation or procedure carried out under controlled conditions in order to discover an unknown effect, to test or establish a hypothesis, or to illustrate a known law” ● Use experiments to create new knowledge ○ Tests make assertions about known properties ● Experiments verify behavior; not validate VERICA | CONTINUOUS VERIFICATION
- 5. Chaos Engineering Principals ● Define “steady state” ● Form a hypothesis ● Introduce variables ● Attempt to disprove hypothesis VERICA | CONTINUOUS VERIFICATION
- 6. Advanced Principles ● Build hypothesis around steady-state behavior ● Vary real-world events ● Run experiments in production ● Automate experiments to run continuously ● Minimize blast radius VERICA | CONTINUOUS VERIFICATION
- 7. Why should we care about Chaos Engineering? VERICA | CONTINUOUS VERIFICATION
- 8. Complex Systems ● Businesses require capabilities/properties/features ● Requires complexity from systems ● Can’t avoid complexity ● Embrace and navigate complexity ● As complexity increases, can’t maintain mental model VERICA | CONTINUOUS VERIFICATION
- 9. ● Kafka sits at the core of our businesses ● Kafka is a complex system ● More complex systems built on top of Kafka ● Cloud infrastructure isn’t always what we expect ● Know the safety margins of our systems VERICA | CONTINUOUS VERIFICATION Chaofka?
- 10. Our Kafka Experiment VERICA | CONTINUOUS VERIFICATION
- 11. Steady State ● Cluster ○ 5-node EKS w/ 1 broker per node - 5-broker Kafka cluster ○ t2.xlarge instance types ■ “moderate” network - 83.4 - 107.3 MiB/s ■ 20 GB “gp2” EBS volumes - 128 MiB/s ○ Metrics to Prometheus/Grafana ● Batch style workload; ~3 min @ 2.5 MiB/s every 5 min ○ ~5 million messages produced and consumed ○ 3 partitions, 3 replicas VERICA | CONTINUOUS VERIFICATION
- 12. Steady State Metrics VERICA | CONTINUOUS VERIFICATION
- 13. Hypothesis “As the load on the Kafka cluster increases, the standard workload can continue to successfully process each batch of messages before the next batch begins.” ● How do we measure this? ○ Monitoring ■ Message/data rates ■ CPU/Memory/Net/Disk usage ○ Application status VERICA | CONTINUOUS VERIFICATION
- 14. Introducing Variables ● How do we increase load? Enter Horus! ○ Scalable & configurable ○ Safety features ■ Halting could be triggered by ● Cluster or client metrics ● Other conditions ● Manual intervention VERICA | CONTINUOUS VERIFICATION
- 15. Load Scaling Configuration VERICA | CONTINUOUS VERIFICATION ● 4 distinct, increasing client sets; 15 minutes each ● 5 partition, 5 replica topic ● 10 - 40 producers; 10 step ○ 500 msg/s; 1024 byte/msg ● 7 consumer groups; 3 consumers each ● 4.88 - 19.5 MiB/s total production traffic ● 34.18 - 136.72 MiB/s total consumer traffic ● Increased replication traffic
- 16. Experiment Results VERICA | CONTINUOUS VERIFICATION
- 17. What’s Next? VERICA | CONTINUOUS VERIFICATION
- 18. The Future! ● Context sensitive ○ One size does not fit all ● Start small ● Start in non-production environment ● Minimize blast radius ● Unleash the Chaos! VERICA | CONTINUOUS VERIFICATION
- 19. References and Resources ● Rosenthal, Casey and Jones, Nora. Chaos Engineering: System Resiliency in Practice. 1st ed., O’Reilly, 2020. ● Hausmann, Steffen. “Best practices for right-sizing your Apache Kafka clusters to optimize performance and cost.” AWS Big Data Blog, 17 Mar. 2022, https://aws.amazon.com/blogs/big-data/best-practices-for-right-sizing-your- apache-kafka-clusters-to-optimize-performance-and-cost/ ● https://principlesofchaos.org/ ● https://www.verica.io ● https://www.thevoid.community/ VERICA | CONTINUOUS VERIFICATION
- 20. Justin Fetherolf Sr. Software Engineer https://www.verica.io