Twitter has one of the largest Kafka fleets in the world, handling hundreds of millions of events per second. In order to operate Kafka Connect at this scale, we've had to get creative. In this talk we'll present some of the problems we've run into with Kafka Connect, and how we've engineered around them.

1. Connect @
Twitter Scale
Ryanne Dolan @dolanRyanne
Jordan Bull @jordangbull

2. 2
Project: CloudShuttle
Kafka
BigQuery
DC 1
Cache
MySQL
DC 2
Manhattan
Cloud Region 1
PubSub
BigQuery
Cloud Region 2
PubSub
?

3. 3
Connectors
PubSubSource PubSubSink
ManhattanSink
NighthawkSink
TwemcacheSink
KinesisSource
KinesisSink
MySQLSource
MirrorSink
Kafka
MirrorSource

4. Text Boxes
4
A Brief History of
Connect @Twitter

5. 5
First pass: one Connect
cluster per Kafka cluster
Kafka Connect GCP PubSub

6. 6
Enter the PubSub
Source Connector
Kafka Connect GCP PubSub

7. 7
Challenges with the
Topology
● Operational Burden
● Capacity Modeling
● Large Blast Radius

8. 8
Our Solution
Kafka Connect GCP PubSub
PubSub Source
PubSub Sink

9. Text Boxes
9
Rejectected
Alternatives

10. 10
Megacluster
Main Kafka

11. 11
Miniclusters

12. Text Boxes
12
Operating
Connect as
a Service

13. Code on background
13
Configuration
cps.streamingPull.enabled: true
cps.streamingPull.flowControlMessages: 10000
cps.streamingPull.parallelStreams: 4
cps.streamingPull.maxAckExtensionMs: 10000
cps.streamingPull.maxMsPerAckExtension: 10000
kafka.record.headers: true
kafka.partition.scheme: kafka_partitioner
header.converter:
org.apache.kafka.connect.storage.StringConverter
key.converter:
org.apache.kafka.connect.storage.StringConverter
value.converter:
org.apache.kafka.connect.converters.ByteArrayConverter
cps.streamingPull.enabled: true
cps.streamingPull.flowControlMessages: 10000
cps.streamingPull.parallelStreams: 4
cps.streamingPull.maxAckExtensionMs: 10000
cps.streamingPull.maxMsPerAckExtension: 10000
kafka.record.headers: true
kafka.partition.scheme: kafka_partitioner
header.converter:
org.apache.kafka.connect.storage.StringConverter
key.converter:
org.apache.kafka.connect.storage.StringConverter
value.converter:
org.apache.kafka.connect.converters.ByteArrayConverter
cps.streamingPull.enabled: true
cps.streamingPull.flowControlMessages: 10000
cps.streamingPull.parallelStreams: 4
cps.streamingPull.maxAckExtensionMs: 10000
cps.streamingPull.maxMsPerAckExtension: 10000
kafka.record.headers: true
kafka.partition.scheme: kafka_partitioner
header.converter:
org.apache.kafka.connect.storage.StringConverter
key.converter:
org.apache.kafka.connect.storage.StringConverter
value.converter:
org.apache.kafka.connect.converters.ByteArrayConverter
cps.streamingPull.enabled: true
cps.streamingPull.flowControlMessages: 10000
cps.streamingPull.parallelStreams: 4
cps.streamingPull.maxAckExtensionMs: 10000
cps.streamingPull.maxMsPerAckExtension: 10000
kafka.record.headers: true
kafka.partition.scheme: kafka_partitioner
header.converter:
org.apache.kafka.connect.storage.StringConverter
key.converter:
org.apache.kafka.connect.storage.StringConverter
value.converter:
org.apache.kafka.connect.converters.ByteArrayConverter
cps.streamingPull.enabled: true
cps.streamingPull.flowControlMessages: 10000
cps.streamingPull.parallelStreams: 4
cps.streamingPull.maxAckExtensionMs: 10000
cps.streamingPull.maxMsPerAckExtension: 10000
kafka.record.headers: true
kafka.partition.scheme: kafka_partitioner
header.converter:
cps.streamingPull.enabled: true
cps.streamingPull.flowControlMessages: 10000

14. 14
Monitoring SLOs

15. 15
SLO: Latency
Kafka Connect Destination
T1 T2

16. 16
SLO: Delivery
Kafka Connect Destination
%

17. 17
Additional Alarms
● Failed Tasks
● Config Load Errors

18. 18
KIPs!
● IdentityReplicationPolicy (382) ✅
● Record Rate Limiting (731)
● Delete Committed Records (762)
● Latency Metrics (767)

19. Thank You
19
Ryanne Dolan @dolanRyanne
Jordan Bull @jordangbull