Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

Disaster Recovery with MirrorMaker 2.0 (Ryanne Dolan, Cloudera) Kafka Summit London 2019

6 206 vues

Publié le

Increasingly, organizations are relying on Kafka for mission critical use-cases where high availability and fast recovery times are essential. In particular, enterprise operators need the ability to quickly migrate applications between clusters in order to maintain business continuity during outages. In many cases, out-of-order or missing records are entirely unacceptable. MirrorMaker is a popular tool for replicating topics between clusters, but it has proven inadequate for these enterprise multi-cluster environments. Here we present MirrorMaker 2.0, an upcoming all-new replication engine designed specifically to provide disaster recovery and high availability for Kafka. We describe various replication topologies and recovery strategies using MirrorMaker 2.0 and associated tooling.

Publié dans : Technologie
  • You can hardly find a student who enjoys writing a college papers. Among all the other tasks they get assigned in college, writing essays is one of the most difficult assignments. Fortunately for students, there are many offers nowadays which help to make this process easier. The best service which can help you is ⇒ www.WritePaper.info ⇐
       Répondre 
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici
  • Writing a good research paper isn't easy and it's the fruit of hard work. For help you can check writing expert. Check out, please ⇒ www.HelpWriting.net ⇐ I think they are the best
       Répondre 
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici
  • When is MM2 getting released?
       Répondre 
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici

Disaster Recovery with MirrorMaker 2.0 (Ryanne Dolan, Cloudera) Kafka Summit London 2019

  1. 1. © Cloudera, Inc. All rights reserved. DISASTER RECOVERY WITH MIRRORMAKER 2.0 Ryanne Dolan, 14th May 2019
  2. 2. © Cloudera, Inc. All rights reserved. Kafka Producer Producer Producer Producer Producer Consumer Consumer Consumer Consumer Consumer Clients connect directly to Kafka. Step 0: Your Kafka today. Producer
  3. 3. © Cloudera, Inc. All rights reserved. Kafka 1 Kafka 2 Load balancer Producer Producer Producer Producer Producer Consumer Consumer Consumer Consumer Consumer Producers can use VIPs, load balancers, and health checks to failover transparently. Step 1: Add a second cluster Producer Consumer Consumer Consumer Consumer Consumer Two independent consumer groups.
  4. 4. © Cloudera, Inc. All rights reserved. Kafka 1 Kafka 2 Load balancer Producer Producer Producer Producer Producer Consumer Consumer Consumer Consumer Consumer Disaster strikes. Can still ingest. Producer Consumer Consumer Consumer Consumer Consumer
  5. 5. © Cloudera, Inc. All rights reserved. More clusters, more availability.
  6. 6. © Cloudera, Inc. All rights reserved. What about data loss?
  7. 7. © Cloudera, Inc. All rights reserved. Kafka 1 Kafka 2 Gateway Producer Producer Producer Producer Producer Consumer Consumer Consumer Consumer Consumer Records sent to both clusters, but with different order and offsets. Step 2: Dual ingest Producer R1 R1 R1 R1
  8. 8. © Cloudera, Inc. All rights reserved. Kafka 1 Kafka 2 Gateway Producer Producer Producer Producer Producer Consumer Consumer Consumer Consumer Consumer Step 2: Dual ingest Producer R1 Where to resume processing? R1
  9. 9. © Cloudera, Inc. All rights reserved. Introducing: MirrorMaker 2.0
  10. 10. © Cloudera, Inc. All rights reserved. primary secondary Load balancer Producer Producer Producer Producer Producer Consumer Consumer Consumer Consumer Consumer Producers send to either cluster. Consumers can be migrated between primary and secondary clusters. Active/standby with MirrorMaker 2.0 Producer MM2
  11. 11. © Cloudera, Inc. All rights reserved. primary secondary Load balancer Producer Producer Producer Producer Producer Consumer Consumer Consumer Consumer Consumer Data, offset syncs, and consumer checkpoints. Active/standby with MirrorMaker 2.0 Producer
  12. 12. © Cloudera, Inc. All rights reserved. primary secondary Load balancer Producer Producer Producer Producer Producer Consumer Consumer Consumer Consumer Consumer Migrate consumers to backup cluster. Primary down: migrate consumers. Producer MM2
  13. 13. © Cloudera, Inc. All rights reserved. 13© Cloudera, Inc. All rights reserved. primary secondary Replicated topics are renamed according to ReplicationPolicy. <source>.<topic> topic1 topic2 secondary.topic1 secondary.topic2 topic1 topic2 primary.topic1 primary.topic2 Remote topics
  14. 14. © Cloudera, Inc. All rights reserved. 14© Cloudera, Inc. All rights reserved. Checkpoint stream replicates consumer group state. topic=primary.topic1 partition=4 group=consumer-group-2 upstreamOffset=100 offset=102 Checkpoints secondary primary.checkpoints.internal
  15. 15. © Cloudera, Inc. All rights reserved. 15© Cloudera, Inc. All rights reserved. Offset sync stream maps offsets between mirrored clusters. topic=primary.topic1 partition=4 upstreamOffset=100 downstreamOffset=102 Offset Syncs secondary primary.offset-syncs.internal
  16. 16. © Cloudera, Inc. All rights reserved. 16© Cloudera, Inc. All rights reserved. Cross-cluster offset translation Translate offsets between clusters via RemoteClusterUtils Map<TopicPartition, Long> newOffsets = RemoteClusterUtils.translateOffsets( newClusterProperties, oldClusterName, consumerGroupId); consumer.seek(newOffsets); ● offset translation based on checkpoints in new cluster ● no connection to old cluster required
  17. 17. © Cloudera, Inc. All rights reserved. 17© Cloudera, Inc. All rights reserved. MirrorMaker 2.0 configuration file # mm2.properties clusters = primary, secondary primary.bootstrap.servers = … secondary.bootstrap.servers = … primary->secondary.enabled = true secondary->primary.enabled = true
  18. 18. © Cloudera, Inc. All rights reserved. primary secondary Load balancer Producer Producer Producer Producer Producer Primary permanently lost? Recover from secondary. Producer primary-2
  19. 19. © Cloudera, Inc. All rights reserved. 19© Cloudera, Inc. All rights reserved. MirrorMaker 2.0 configuration file # mm2.properties clusters = secondary, primary-2 secondary.bootstrap.servers = … primary-2.bootstrap.servers = … primary-2->secondary.enabled = true secondary->primary-2.enabled = true
  20. 20. © Cloudera, Inc. All rights reserved. 20© Cloudera, Inc. All rights reserved. primary-2 secondary Lost primary topics can be recovered from remote topics on secondary cluster. topic1 topic2 secondary.topic1 secondary.topic2 secondary.primary.topic1 secondary.primary.topic2 topic1 topic2 primary.topic1 primary.topic2 primary-2.topic1 primary-2.topic2 Recovered topics
  21. 21. © Cloudera, Inc. All rights reserved. 21© Cloudera, Inc. All rights reserved. primary-2 secondary topic1 topic2 secondary.topic1 secondary.topic2 secondary.primary.topic1 secondary.primary.topic2 topic1 topic2 primary.topic1 primary.topic2 primary-2.topic1 primary-2.topic2 Recovered topics data from old primary data from old primary
  22. 22. © Cloudera, Inc. All rights reserved. primary secondary Load balancer Producer Producer Producer Producer Producer Consumer Consumer Consumer Consumer Consumer Producers send to either cluster. Active/active with MirrorMaker 2.0 Producer MM2 Consumer Consumer Consumer Consumer Consumer Consumers in both clusters.
  23. 23. © Cloudera, Inc. All rights reserved. primary secondary Load balancer Producer Producer Producer Producer Producer Consumer Consumer Consumer Consumer Consumer Cross-cluster consumer groups Producer MM2 Consumer Consumer Consumer Consumer Consumer R1 R1 R1
  24. 24. © Cloudera, Inc. All rights reserved. primary secondary Load balancer Producer Producer Producer Producer Producer Consumer Consumer Consumer Consumer Consumer Cross-cluster consumer groups Producer MM2 Consumer Consumer Consumer Consumer Consumer R1 R1 R1 R2 R2 R2 Only one consumer processes each record.
  25. 25. © Cloudera, Inc. All rights reserved. primary secondary Load balancer Producer Producer Producer Producer Producer Consumer Consumer Consumer Consumer Consumer Effectively one big consumer group Producer MM2 Consumer Consumer Consumer Consumer Consumer R1 R1 R1 R2 R2 R2 Only one consumer processes each record.
  26. 26. © Cloudera, Inc. All rights reserved. us-eastus-west Cross-data center replication (XDCR) us-north Producer LB LB LBLBs prefer their local DC, but can failover to other DCs ConsumerConsumerConsumer Consumers see records from all DCs.
  27. 27. © Cloudera, Inc. All rights reserved. Producers failover to another data center. us-northus-west us-east Producer LB LB LB ConsumerConsumerConsumer
  28. 28. © Cloudera, Inc. All rights reserved. Entire DC permanently lost? Stand up a new Kafka cluster. us-northus-west us-east Producer LB LB LB ConsumerConsumerConsumer
  29. 29. © Cloudera, Inc. All rights reserved. New us-west-2 cluster can be recovered from other DCs. us-northus-west-2 us-east Producer LB LB LB ConsumerConsumer
  30. 30. © Cloudera, Inc. All rights reserved. New cluster back online. us-northus-west-2 us-east Producer LB LB LB ConsumerConsumerConsumer
  31. 31. © Cloudera, Inc. All rights reserved. Reference architecture: XDCR with active/active AZs DC-1 DC-2 AZ-1 AZ-2 AZ-3 AZ-4 K1 K2 K3 K4 LB LB
  32. 32. © Cloudera, Inc. All rights reserved. Producers send to nearby DC DC-1 DC-2 K1 K2 K3 K4 LB LB P P
  33. 33. © Cloudera, Inc. All rights reserved. Consumers in each zone. Same group.id. DC-1 DC-2 AZ-1 AZ-2 AZ-3 AZ-4 K1 K2 K3 K4 LB LB C C C C
  34. 34. © Cloudera, Inc. All rights reserved. Effectively two big consumer groups. DC-1 DC-2 AZ-1 AZ-2 AZ-3 AZ-4 K1 K2 K3 K4 LB LB C C C C
  35. 35. © Cloudera, Inc. All rights reserved. Record processed by one consumer in each DC. DC-1 DC-2 AZ-1 AZ-2 AZ-3 AZ-4 K1 K2 K3 K4 LB LB C C C C P R1 R1 R1
  36. 36. © Cloudera, Inc. All rights reserved. HA: ingest and process with any single K cluster. DC-1 DC-2 AZ-1 AZ-2 AZ-3 AZ-4 K1 K2 K3 K4 LB LB C C C C P R2 R2
  37. 37. © Cloudera, Inc. All rights reserved. K1 catches up. Records are not re-processed in AZ-1. DC-1 DC-2 AZ-1 AZ-2 AZ-3 AZ-4 K1 K2 K3 K4 LB LB C C C C R2 P R2
  38. 38. © Cloudera, Inc. All rights reserved. K3 catches up. Records processed in DC-2 for first time. DC-1 DC-2 AZ-1 AZ-2 AZ-3 AZ-4 K1 K2 K3 K4 LB LB C C C C R2 R2 P R2
  39. 39. © Cloudera, Inc. All rights reserved. K4 catches up. Records are not re-processed in AZ-4. DC-1 DC-2 AZ-1 AZ-2 AZ-3 AZ-4 K1 K2 K3 K4 LB LB C C C C R2 R2 P R2
  40. 40. © Cloudera, Inc. All rights reserved. MM2 config # mm2.properties clusters = K1, K2, K3, K4 K1.bootstrap.servers = … # active/active w/in DC-1 K1->K2.enabled = true K2->K1.enabled = true # active/active w/in DC-2 K3->K4.enabled = true K4->K3.enabled = true # XDCR b/t DCs K1->K3.enabled = true K3->K1.enabled = true
  41. 41. © Cloudera, Inc. All rights reserved. Consumer config • Same consumer group.id across all AZs, DCs. • DC-1 subscription: • topic1 • K3.topic1 • K3.K4.topic1 • DC-2 subscription: • topic1 • K1.topic1 • K1.K2.topic1 Alternatively: .*.topic1
  42. 42. © Cloudera, Inc. All rights reserved. THANK YOU

×