Building Stream Infrastructure across Multiple Data Centers with Apache Kafka
•
13 j'aime•7,605 vues
To manage the ever-increasing volume and velocity of data within your company, you have successfully made the transition from single machines and one-off solutions to large distributed stream infrastructures in your data center, powered by Apache Kafka. But what if one data center is not enough? I will describe building resilient data pipelines with Apache Kafka that span multiple data centers and points of presence, and provide an overview of best practices and common patterns while covering key areas such as architecture guidelines, data replication, and mirroring as well as disaster scenarios and failure handling.
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à Building Stream Infrastructure across Multiple Data Centers with Apache Kafka
Similaire à Building Stream Infrastructure across Multiple Data Centers with Apache Kafka (20)
Plus de Guozhang Wang
Plus de Guozhang Wang (13)
Dernier
Dernier (20)
Building Stream Infrastructure across Multiple Data Centers with Apache Kafka
- 1. When One Data Center is not Enough Guozhang Wang Strata San Jose, 2016 Building large-scale stream infrastructure across multiple data centers with Apache Kafka
- 2. 2 • Why across Data Centers? • Design patterns for Multi-DC • Kafka for Multi-DC • Conclusion Agenda
- 3. 3 Why across Data Centers?
- 4. 4 Why across Data Centers • Catastrophic / expected failures • Routine maintenance • Geo-locality (Example: CDNs)
- 5. 5 Why NOT across Data Centers • Low bandwidth (10Mbps - 1Gbps) • High latency (50ms - 450ms) • Much More $$$
- 6. 6 Why NOT across Data Centers • … is hard and expensive
- 7. 7 Why NOT across Data Centers • … is hard and expensive • … with real-time writes? Harder
- 8. 8 Why NOT across Data Centers • … is hard and expensive • … with real-time writes? Harder • … consistently? Oh My!
- 9. 9 Consistency • Weak • Eventual • Strong Latency Guarantee
- 10. 10 Weak No Consistency • Now you see my writes, now you don’t • Best effort only, data can be stale • Examples: think of “caches”, VoIP
- 11. 11 Eventual Consistency • You will see my writes, … eventually • May need to resolve conflicts (manually) • Examples: think of “emails”, SMTP
- 12. 12 Strong Consistency • You get what you write, for sure • External > Sequential > Causal (Session) • Examples: RDBMS, file systems
- 13. 13 • LAN: consistency over latency • WAN: latency over consistency Latency vs. Consistency
- 14. 14 • Why across Data Centers? • Design patterns for Multi-DC • Kafka for Multi-DC • Conclusion Agenda
- 15. 15 Option I: Don’t do it • Bunkerize the single data center • Expect data loss at failures • Examples: ??
- 16. 16 Option II: Primary with Hot Standby • Failover to hot standby (maybe inconsistent) • Window of data loss at failures • Examples: MySQL binlog
- 17. 17 Option III: Active-Active • Accepts writes in multi-DC • Resolve conflicts (strong / week consistency) • Examples: Amazon DynamoDB (vector clock) Google Spanner (2PC), Mesa (Paxos)
- 18. 18 Ordering is the Key!
- 19. 19 Ordering is Key • Vector clocks: partial ordering • Paxos, 2PC: global ordering • Log shipping: logical ordering (per-partition)
- 21. 21 Apache Kafka • A distributed messaging system ..that store messages as a log!
- 22. 22 Store Messages as a Log 4 5 5 7 8 9 10 11 12... Producer Write Consumer1 Reads (offset 7) Consumer2 Reads (offset 10) Messages 3
- 23. 23 Partition the Log across Machines Topic 1 Topic 2 Partitions Producers Producers Consumers Consumers Brokers
- 24. 24 ACK mode Latency On Failures “no" no network delay some data loss “leader" 1 network roundtrip a few data loss “all" ~2 network roundtrips no data loss Configurable ISR Commits
- 25. 25 • Why across Data Centers? • Design patterns for Multi-DC • Kafka for Multi-DC • Conclusion Agenda
- 26. 26 Option I: Active-Passive Replication Kafka local producers consumer consumer DC 1 MirrorMaker DC 2 Kafka replica
- 27. 27 Option I: Active-Passive Replication • Async- replication across DC • May lose data on failover • Example: ETL to data warehouse / HDFS Kafka local producers consumer consumer DC 1 MirrorMaker DC 2 Kafka replica
- 28. 28 Option II: Active-Active Replication Kafka local Kafka aggregate Kafka aggregate producers producers consumer consumer MirrorMaker Kafka local on DC1 failure DC 1 DC 2
- 29. 29 Option II: Active-Active Replication • Global view on agg. cluster • Require offsets to resume • Example: store materialization, index updates Kafka local Kafka agg Kafka agg producers producers consumer consumer MirrorMaker Kafka local on DC1 failure DC 1 DC 2
- 30. 30 • Offsets not identical between Kafka clusters • Duplicates during failover • Partition selection may be different • Solutions • Resume from log end offset (suitable for real-time apps) • Resume from a timestamp (ListOffsets, offset index: KIP-33) Caveats: offsets across DCs
- 31. 31 Option III: Deploy across DCs Kafka producers producers consumer consumer DC 1 DC 2
- 32. 32 Option III: Deploy across DCs • Multi-tenancy support • Security (0.9) • Quota Management (0.9) • Latency optimization • Rack-aware partition assignment (0.10) • Read affinity (future?) Kafka producers producers consumer consumer DC 1 DC 2
- 33. 33 • Same region: essentially same network • asymmetric partitioning is rare, low latency • Need at least 3 DCs for Zookeeper • Reserved instance to reduce churns • EIP for external clients, private IPs for internal communication • Reserved instance, local storage Example: EC2 multi-AZ Deployment
- 34. 34 Take-aways • Multi-DC: trade-off between latency and consistency • Kafka: replicated log streams for multihoming
- 35. Thank you Guozhang | guozhang@confluent.io | @guozhangwang Meet Confluent in booth #838 Confluent University ~ Kafka training ~ confluent.io/training Join the Stream Data Hackathon Apr 25, SF kafka-summit.org/hackathon/ Download Apache Kafka & Confluent Platform confluent.io/download