From migrations between Apache Kafka clusters to multi-region deployments across datacenters, the introduction of MirrorMaker2 has expanded the possibilities for Apache Kafka deployments and use cases. In this session you will learn about patterns, best practices, and learnings compiled from running MirrorMaker2 in production at every scale.

1. A Look into the Mirror
Patterns and Best Practices for MirrorMaker2
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. 1
Cliff Gilmore
Principal Solutions Architect – Streaming Data

2. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. 2
Agenda
• MirrorMaker2 Overview
• Live Migration
• Active Passive
• Hub and Spoke / Aggregation
• Active Active
• Capacity Planning
• Best Practices

3. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. 3
Key MirrorMaker2 Features
• Multi-cluster data replication engine based on Kafka Connect framework
• Detects new topics and partitions in self-discovery mode
• Automatically synchronizes topic configuration & ACLs
• Supports “active/active” cluster pairs, as well as any number of active clusters
• Provides new metrics including end-to-end replication latency across multiple clusters
• Emits the offsets required to migrate consumers between clusters and tooling for offset
translation (in Apache Kafka 2.7, offset sync is also automated)
• Configuration file for specifying multiple clusters and replication flows in one place

4. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. 4
MirrorMaker2 Core Concepts
• Connectors
• MirrorSourceConnector
• MirrorSinkConnector
• MirrorCheckpointConnector
• MirrorHeartbeatConnector
• Replication Policies
• Controls naming of replicated topics
• DefaultReplicationPolicy
• <source>.<topic> convention
• CustomReplicationPolicy
• Create your own policy based on preferences (IE, no prefix for migration/active-passive)
• Configuration Parameters
• topics lets you control which topics are replicated
• Groups lets you control which consumer groups have offsets replicated
• More configuration for options, connectivity and topology defined at
• https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0

5. Live Migrations with MM2
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. 5

6. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. 6
Live Migration Pattern (Initial State)
Source
Cluster
Destination
Cluster
Consumer
Producers

7. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. 7
Live Migration Pattern (Enable Replication)
Source
Cluster
Destination
Cluster
Consumer
Producers
Kafka Connect
MirrorSourceConnector
MirrorCheckpointConnector
MirrorHeartBeatConnector
Replication for
Topics and Offsets
Replication for
Topics and Offsets
Source Cluster
Heartbeats
Topic Names stay consistent via CustomReplicationPolicy

8. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. 8
Live Migration Pattern (Stop Consumer, Update Offsets)
Source
Cluster
Destination
Cluster
Producers
Kafka Connect
MirrorSourceConnector
MirrorCheckpointConnector
MirrorHeartBeatConnector
Replication for
Topics and Offsets
Replication for
Topics and Offsets
Source Cluster
Heartbeats
Note: OffsetSync can be Automated with the MirrorMaker2 version part of Apache Kafka 2.7 release
RemoteClusterUtils to
update offsets

9. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. 9
Live Migration Pattern (Start Consumer on Destination)
Source
Cluster
Destination
Cluster
Producers
Kafka Connect
MirrorSourceConnector
MirrorCheckpointConnector
MirrorHeartBeatConnector
Replication for
Topics and Offsets
Replication for
Topics and Offsets
Source Cluster
Heartbeats
Consumer

10. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. 10
Live Migration Pattern (Point Producers to Destination)
Source
Cluster
Destination
Cluster
Kafka Connect
MirrorSourceConnector
MirrorCheckpointConnector
MirrorHeartBeatConnector
Replication for
Topics and Offsets
Replication for
Topics and Offsets
Source Cluster
Heartbeats
Consumer
Producers

11. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. 11
Live Migration Pattern (Shut Down Source/Connect)
Destination
Cluster
Consumer
Producers
• The source cluster can now be safely shut
down
• This leaves us with our clients migrated to
the new cluster
• Very little delay is introduced with just the
restart time of each topic’s producers and
consumers being the disruption

12. Active – Passive Replication with DR Failover/Back
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. 12

13. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. 13
Active-Passive (Normal State)
Source
Cluster
Destination
Cluster
Consumer
Producers
Kafka Connect
MirrorSourceConnector
MirrorCheckpointConnector
MirrorHeartBeatConnector
Replication for
Topics and Offsets
Replication for
Topics and Offsets
Source Cluster
Heartbeats
Datacenter/Region #1 Datacenter/Region #2
topic1
topic1
region1.topic1

14. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. 14
Active-Passive (Cluster Failure, Bounce clients)
Source
Cluster
Destination
Cluster
Consumer
Producers
Kafka Connect
MirrorSourceConnector
MirrorCheckpointConnector
MirrorHeartBeatConnector
Datacenter/Region #1 Datacenter/Region #2
topic1
region1.topic1
RemoteClusterUtils to
update offsets

15. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. 15
Active-Passive (Datacenter Failure)
Source
Cluster
Destination
Cluster
Consumer
Producers
Kafka Connect
MirrorSourceConnector
MirrorCheckpointConnector
MirrorHeartBeatConnector
Datacenter/Region #1 Datacenter/Region #2
Copy of
Producers
Consumers with
same group ID
topic1
region1.topic1
RemoteClusterUtils to
update offsets

16. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. 16
Active-Passive (Reverse Replication)
Source
Cluster
Destination
Cluster
Kafka Connect
MirrorSourceConnector
MirrorCheckpointConnector
MirrorHeartBeatConnector
Datacenter/Region #1 Datacenter/Region #2
Copy of
Producers
Consumers with
same group ID
topic1
region1.topic1
topic1
region2.topic1

17. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. 17
Active-Passive (Reverse Replication)
Source
Cluster
Destination
Cluster
Kafka Connect
MirrorSinkConnector
MirrorCheckpointConnector
MirrorHeartBeatConnector
Datacenter/Region #1 Datacenter/Region #2
Copy of
Producers
Consumers with
same group ID
topic1
region1.topic1
topic1
region2.topic1
Replication for
Topics and Offsets
Replication for
Topics and Offsets
Dest Cluster
Heartbeats
RemoteClusterUtils
to
update offsets

18. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. 18
Active-Passive (Failback Producer/Consumer)
Source
Cluster
Destination
Cluster
Kafka Connect
MirrorSinkConnector
MirrorCheckpointConnector
MirrorHeartBeatConnector
Datacenter/Region #1 Datacenter/Region #2
topic1
region1.topic1
topic1
region2.topic1
Replication for
Topics and Offsets
Replication for
Topics and Offsets
Dest Cluster
Heartbeats
Consumer
Producers

19. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. 19
Active-Passive (Resume Normal Replication)
Source
Cluster
Destination
Cluster
Consumer
Producers
Kafka Connect
MirrorSourceConnector
MirrorCheckpointConnector
MirrorHeartBeatConnector
Replication for
Topics and Offsets
Replication for
Topics and Offsets
Source Cluster
Heartbeats
Datacenter/Region #1 Datacenter/Region #2
topic1
region2.topic1
topic1
region1.topic1

20. Hub and Spoke aka Aggregation Pattern
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. 20

21. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. 21
Hub and Spoke/Aggregation
Destination
Cluster
Consumer
Producers
Kafka Connect
MirrorSourceConnector
Replication for
Topics
Consumer
Producers
Consumer
Producers
Consumer
Local Datacenter/Region #2
Local Datacenter/Region #1
Local Datacenter/Region #3
Central Datacenter/Region
region1.topic1
region2.topic1
region3.topic1

22. Active – Active also useful for Cloud Bridging
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. 22

23. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. 23
Active-Active
OnPrem
Cluster
Cloud
Cluster
Consumers
Producers
Kafka Connect
MirrorSourceConnector
MirrorCheckpointConnector
MirrorHeartBeatConnector
Datacenter
Cloud Region
topic1
cloud.topic1 topic1
datacenter.topic1
Producers
Consumers
Kafka Connect
MirrorSourceConnector
MirrorCheckpointConnector
MirrorHeartBeatConnector

24. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. 24
Capacity Planning Considerations
• Run Kafka Connect in distributed mode
• Mainly size based on network throughput
• Always consider the impact of a downed worker in capacity planning
• Remember to measure impact on source cluster (extra consumer)
• Follow general task to CPU core ratio of 8-16 tasks per core
• Avoid SMTs and type conversion
• Use CPU/Network optimized instances/VMs/machines as disk or high amounts of RAM are not needed
• Increase the Heap size to something between 6-12GB

25. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. 25
Best Practices / Observations
• Keep partitions per connect task ratio <=10:1
• There can be issues with offset replication when > 10:1
• https://issues.apache.org/jira/browse/KAFKA-12558
• Set consumer.task.id in the connect configuration manually
• This allows for client quotas to be set to control replication max throughput for balancing with other
producers/consumers
• Monitor task status vs Connect REST interface
• Monitor JMX metrics to ensure latency and throughput are as expected
• You can’t measure consumer lag due to the way MM2 consumes from the source assign() vs subscribe()
• Negative lag on replication destinations can occur if replication is pending or behind and offset sync is
enabled

26. © 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. 26
Monitoring Metrics
• MirrorSourceConnector
• record-count
• record-age-ms
• record-age-ms-min
• record-age-ms-max
• record-age-ms-avg
• replication-latency-ms
• replication-latency-ms-min
• replication-latency-ms-max
• replication-latency-ms-avg
• byte-rate
• MirrorCheckpointConnector
• checkpoint-latency-ms
• checkpoint-latency-ms-min
• checkpoint-latency-ms-max
• checkpoint-latency-ms-avg

27. Thanks!
Twitter : @cwgdata
LinkedIN: https://www.linkedin.com/in/cliffgilmore
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved. 27