Unveiling the Inner Workings of Apache Kafka® with Flamegraphs with Christo Lolov & Divij Vaidya
Signaler
HostedbyConfluentSuivre
2 Jun 2023•0 j'aime•353 vues
Unveiling the Inner Workings of Apache Kafka® with Flamegraphs with Christo Lolov & Divij Vaidya
2 Jun 2023•0 j'aime•353 vues
Télécharger pour lire hors ligne
Signaler
Technologie
Apache Kafka® is known for its scalability, reliability, and performance, but understanding how it works and identifying performance bottlenecks can be challenging. In this talk, we will explore the use of flamegraphs as a tool for understanding the internals of Apache Kafka and for identifying performance issues. Flamegraphs are a visualization technique that allows you to see the relative usage of CPU and memory by different functions in a program. They provide a clear and concise view of the call stack, making it easier to understand how different components of the system are interacting and where the hot spots are. We will cover the basics of how to generate and interpret flamegraphs, and we will walk through a number of real-world examples of using them to troubleshoot performance issues in Apache Kafka® . These examples will include cases where flamegraphs helped to identify issues such as CPU and memory contention, inefficient code paths, and suboptimal configuration choices. By the end of this talk, you will have a better understanding of how to use flamegraphs to understand the inner workings of Apache Kafka® and to identify performance bottlenecks. You will also have a set of practical tips and techniques for using flamegraphs to troubleshoot performance issues in your own Kafka deployments.
HostedbyConfluentSuivre
Recommandé
AWS Lambda Powertools walkthrough.pdfHeitor Lessa
apidays Helsinki & North 2023 - Interface Contracts in Event-driven Microfron...apidays
20 ways event-driven architectures can improve your development - Copy.pptxJames Beswick
AWS reInvent 2022 reCap AI/ML and DataChris Fregly
Better Together: Delivering Graph Value with AWS & Neo4j - Antony Prasad Thev...Neo4j
How to build a FullStack Airline Ticketing Web App.pdfAmazon Web Services
Contenu connexe
Similaire à Unveiling the Inner Workings of Apache Kafka® with Flamegraphs with Christo Lolov & Divij Vaidya
Advanced VPC Design and New Capabilities for Amazon VPC (NET303) - AWS re:Inv...Amazon Web Services
Next generation intelligent data lakes, powered by GraphQL & AWS AppSync - MA...Amazon Web Services
Better Together: Delivering Graph Value with AWS & Neo4j - Antony Prasad The...Neo4j
20190318 Amazon EC2 スポットインスタンス再入門Amazon Web Services Japan
AWS Meetup Brussels 3rd Sep 2019 Simplify Frontend Apps with Serverless BackendsPatrick Sard
kreuzwerker AWS Modernizing Legacy Operations with Containerized Solutions 20...kreuzwerker GmbH
Similaire à Unveiling the Inner Workings of Apache Kafka® with Flamegraphs with Christo Lolov & Divij Vaidya(20)
Plus de HostedbyConfluent
Chill, Distill, No Overkill: Best Practices to Stress Test Kafka with Siva Ku...HostedbyConfluent
Manage Consistent Configurations Across Multiple Kafka Environments with Naga...HostedbyConfluent
The Cost of Kafka’s High Availability on Cloud with Geetha AnneHostedbyConfluent
Case-Study: Building Real-Time Applications at Scale-Cyclist Crash Detection ...HostedbyConfluent
Migrating Your System to the Different Kafka Platforms Available wtih Bhavesh...HostedbyConfluent
Eliminating the Double Write Problem in Apache Kafka Using the Outbox Pattern...HostedbyConfluent
Dernier
roomos_webinar_280923_v2.pptxThousandEyes
"Stateful app as an efficient way to build dispatching for riders and drivers...Fwdays
Recommendation Modeling with Impression Data at NetflixJiangwei Pan
Webhook Testing StrategyDimpy Adhikary
Product Research Presentation-Maidy Veloso.pptxMaidyVeloso
Take Control of Podcasting thanks to Open Source and Podcasting 2.0🎙 Benjamin Bellamy
Unveiling the Inner Workings of Apache Kafka® with Flamegraphs with Christo Lolov & Divij Vaidya
- 1. © 2023, Amazon Web Services, Inc. or its affiliates. Unveiling the Inner Workings of Apache Kafka with Flamegraphs Christo Lolov and Divij Vaidya
- 2. © 2023, Amazon Web Services, Inc. or its affiliates. © 2023, Amazon Web Services, Inc. or its affiliates. Introduction 2 Divij Vaidya (He/Him) Software Engineer @ Amazon Web Services (MSK) 10+ yrs. building databases and data related technologies PMC member/committer @ Apache TinkerPop Apache Kafka enthusiast ✨ #LiveToEat Christo Lolov (He/Him) Software Engineer @ Amazon Web Services (MSK) 4 years at Amazon, 2 at AWS MSK Avid hiker and beginner Spanish learner
- 3. © 2023, Amazon Web Services, Inc. or its affiliates. © 2023, Amazon Web Services, Inc. or its affiliates. Agenda 3 Introduction to flame graphs Journey inside Kafka broker* Case Study ⁉ Questions *flamegraphs available at: https://github.com/divijvaidya/flamegraph-samples/tree/main/kafka/kafka-summit-2023
- 4. © 2023, Amazon Web Services, Inc. or its affiliates. © 2023, Amazon Web Services, Inc. or its affiliates. Important clarifications 4 Focus on CPU flame graphs Focus on JVM languages Used https://github.com/async-profiler/async-profiler
- 5. © 2023, Amazon Web Services, Inc. or its affiliates. © 2023, Amazon Web Services, Inc. or its affiliates. Example flame graph 5
- 6. © 2023, Amazon Web Services, Inc. or its affiliates. © 2023, Amazon Web Services, Inc. or its affiliates. What is profiling? 6 Periodically gathering state of a running program
- 7. © 2023, Amazon Web Services, Inc. or its affiliates. © 2023, Amazon Web Services, Inc. or its affiliates. What is a stack trace? 7 source - https://issues.apache.org/jira/browse/KAFKA-14967
- 8. © 2023, Amazon Web Services, Inc. or its affiliates. © 2023, Amazon Web Services, Inc. or its affiliates. Why a flame graph? 8 Present profiling information in a more understandable form
- 9. © 2023, Amazon Web Services, Inc. or its affiliates. © 2023, Amazon Web Services, Inc. or its affiliates. Overall picture 9 State Profiler Flame graph
- 10. © 2023, Amazon Web Services, Inc. or its affiliates. Flame graphs are a visualization of hierarchical data, created to visualize stack traces of profiled software so that the most frequent code-paths to be identified quickly and accurately Brendan Gregg Source: https://www.brendangregg.com/flamegraphs.html 10 Inventor of flame graphs
- 11. © 2023, Amazon Web Services, Inc. or its affiliates. © 2023, Amazon Web Services, Inc. or its affiliates. Example flame graph 11
- 12. © 2023, Amazon Web Services, Inc. or its affiliates. © 2023, Amazon Web Services, Inc. or its affiliates. Example flame graph 13
- 13. © 2023, Amazon Web Services, Inc. or its affiliates. © 2023, Amazon Web Services, Inc. or its affiliates. Example flame graph 14
- 14. © 2023, Amazon Web Services, Inc. or its affiliates. © 2023, Amazon Web Services, Inc. or its affiliates. Example flame graph 15
- 15. © 2023, Amazon Web Services, Inc. or its affiliates. © 2023, Amazon Web Services, Inc. or its affiliates. Example flame graph 16
- 16. © 2023, Amazon Web Services, Inc. or its affiliates. © 2023, Amazon Web Services, Inc. or its affiliates. Under the hood of Kafka broker 17 🔎
- 17. © 2023, Amazon Web Services, Inc. or its affiliates. © 2023, Amazon Web Services, Inc. or its affiliates. Apache Kafka: High level components 18 Producer(s) Broker(s) Consumer(s)
- 18. © 2023, Amazon Web Services, Inc. or its affiliates. Test setup Linux EC2 - M5.large JDK 17 Kafka 3.4.0 1 broker 19 4 producers 4 consumers Uncompressed No authZ/authN No SSL OpenMessaging Workload
- 19. © 2023, Amazon Web Services, Inc. or its affiliates. Flamegraph: Broker 20
- 20. © 2023, Amazon Web Services, Inc. or its affiliates. Flamegraph: Broker (cleaned) 21
- 21. © 2023, Amazon Web Services, Inc. or its affiliates. Broker Thread Pools 22
- 22. © 2023, Amazon Web Services, Inc. or its affiliates. Detour: Thread pools in Kafka 23 Acceptor thread(s) (1) Network thread(s) (N) Handler thread(s) (M) Broker Java process Accepts & configures new connections Read requests from socket(s) Request Queue (P) P = queued.max.requests N = num.network.threads M = num.io.threads Processes requests OS Socket Response Queue
- 23. © 2023, Amazon Web Services, Inc. or its affiliates. Broker Thread Pools 24
- 24. © 2023, Amazon Web Services, Inc. or its affiliates. Network Threads 25
- 25. © 2023, Amazon Web Services, Inc. or its affiliates. Network Threads - breakdown 26
- 26. © 2023, Amazon Web Services, Inc. or its affiliates. Network Threads - processCompletedSends 27
- 27. © 2023, Amazon Web Services, Inc. or its affiliates. Network Threads - processCompletedReceives 28
- 28. © 2023, Amazon Web Services, Inc. or its affiliates. Network Threads - poll 29
- 29. © 2023, Amazon Web Services, Inc. or its affiliates. Flamegraph: Broker Thread Pools 30
- 30. © 2023, Amazon Web Services, Inc. or its affiliates. Flamegraph: Request Handler Threads 31
- 31. © 2023, Amazon Web Services, Inc. or its affiliates. Flamegraph: Produce API 32
- 32. © 2023, Amazon Web Services, Inc. or its affiliates. Flamegraph: OffsetCommit API 33
- 33. © 2023, Amazon Web Services, Inc. or its affiliates. © 2023, Amazon Web Services, Inc. or its affiliates. Case Study 34
- 34. © 2023, Amazon Web Services, Inc. or its affiliates. Comparison: compressed vs. non-compressed 35 Uncompressed workload Compressed workload
- 35. © 2023, Amazon Web Services, Inc. or its affiliates. Case study: Reduce memory allocation 36 Kafka 3.4 PR-13135
- 36. © 2023, Amazon Web Services, Inc. or its affiliates. Case study: Reduce memory allocation 37
- 37. © 2023, Amazon Web Services, Inc. or its affiliates. Case study: Reduce memory allocation 38
- 38. © 2023, Amazon Web Services, Inc. or its affiliates. © 2023, Amazon Web Services, Inc. or its affiliates. Recap 39
- 39. © 2023, Amazon Web Services, Inc. or its affiliates. 40 We learnt about… ❌ 📈 🔥 ✅ ❌ 🤯 🔎
- 40. © 2023, Amazon Web Services, Inc. or its affiliates. Thank you! © 2023, Amazon Web Services, Inc. or its affiliates. Christo Lolov 📧 christololov@gmail.com Divij Vaidya 📧 divijv@apache.org @divijvaidya
- 41. © 2023, Amazon Web Services, Inc. or its affiliates. © 2023, Amazon Web Services, Inc. or its affiliates. Questions 42
- 42. © 2023, Amazon Web Services, Inc. or its affiliates. Flamegraph: OffsetCommit API 43
- 43. © 2023, Amazon Web Services, Inc. or its affiliates. Which thread pool perform Authorization? 44