Nielsen Presents: Fun with Kafka, Spark and Offset Management
•
3 j'aime•472 vues
1) The document discusses how Nielsen moved from using Kafka's offset manager to store offsets to using an RDS database for offset storage. This provided more consistent committing of offsets and allowed subscribing to topics with different offset strategies. 2) Key aspects of the new approach included building an offsets map with topic and partition information, getting current offsets from the database, and implementing a "two phase commit" to the database and Kafka in a consistent way. 3) Benefits of the new offset storage approach included faster restart times, improved monitoring, better disaster recovery, and unified configurations.
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à Nielsen Presents: Fun with Kafka, Spark and Offset Management
Similaire à Nielsen Presents: Fun with Kafka, Spark and Offset Management (20)
Dernier
Dernier (20)
Nielsen Presents: Fun with Kafka, Spark and Offset Management
- 1. Nielsen Presents: Fun with Kafka, Spark and Offset Management or how we stopped worrying and started to enjoy offset management
- 2. Agenda
- 3. Agenda • How we used to work • What we wanted to do and why • What we ended up doing • Extra Benefits
- 4. • Simona • Big Data Engineer at Nielsen Marketing Cloud • Data lover • Concert Goer • Japan enthusiast whoami
- 5. How we used to work • Kafka Spark Consumer • Offsets Strategies • Kafka Offset Manager • TEAM A vs TEAM B Flows
- 6. Our Different Flows TEAM RON TEAM ITAI Micro Batch Length 4-6 minutes 1 hour Output to Commit Kafka Offsets Kafka Offsets, File names
- 7. Defining Starting Offsets • Kafka Offset Manager • earliest • latest • XML
- 8. Defining Starting Offsets • Kafka Offset Manager • earliest • latest • XML
- 9. Defining Starting Offsets • Kafka Offset Manager • earliest • latest • XML
- 10. Defining Starting Offsets • Kafka Offset Manager • earliest • latest • XML
- 11. Defining Starting Offsets • Kafka Offset Manager • earliest • latest • XML
- 12. Defining Starting Offsets • Kafka Offset Manager • earliest • latest • XML
- 13. kafka offset manager • Self made • Offsets stored in kafka • Everything is against brokers
- 14. What we wanted to do and why • JUST A SIMPLE UPGRADE • Two main goals to achieve with our infrastructure • Some of the problems we encountered
- 15. Just a Simple Upgrade ● New Kafka Consumer API ● New consumer config ● Subscribing to Kafka
- 16. SOME OF THE PROBLEMS WE ENCOUNTERED Committing offsets to Kafka fails Longer timeouts then! Spark graceful shutdown Spark Kafka aSync Commit ……?
- 17. what we ended up doing • RDS Offset Store • Consistent Committing • Subscribing to topics (Offsets Strategies) • Unified infrastructure • Timeouts
- 18. RDS Offset Store • Table Structure • Constraints and upserts • Triggers
- 19. RDS Offset Store
- 20. RDS Offset Store
- 21. RDS Offset Store
- 23. Building the Offsets Map ● What is the offsets Map[TopicPartition,Long] ● Getting the number of partitions ● Constructing the offsets map
- 32. Getting Current Offsets ● Each map represents a row ● Reduce rows into the offsets map
- 33. “Two Phase” Commit ● What is a “two phase” commit? ● Why do we need it? ● How did we end up implementing it?
- 40. Extra benefits • Restart Time • Monitoring • Disaster Recovery • Unified Configurations
- 41. Committing to Kafka ● Why? ● Two options of committing to Kafka
- 42. New Offsets Strategies • DataBase • Earliest • Latest