SlideShare une entreprise Scribd logo
1  sur  76
Télécharger pour lire hors ligne
Bringing Kafka Without ZooKeeper
Into Production
Colin McCabe
Principal Engineer
Confluent
About Me
2
I work on Apache Kafka at
Confluent.
Kafka Committer & PMC Member
3
● KRaft Architecture
● Deploying KRaft
● Rolling KRaft Clusters
● Troubleshooting KRaft Clusters
● Upgrading from ZooKeeper
● Roadmap
● Q&A
Table of Contents
KRaft Architecture
4
Removing ZooKeeper
5
Current
ZK-Based
Architecture
New
KRaft-Based
Architecture
How Kafka Uses ZooKeeper
6
Stores most persistent
metadata in ZooKeeper
● Topics
● Partitions
● Configurations
● Quotas
● ACLs
Also uses ZooKeeper for
coordinating cluster
membership.
/brokers/ids/0
/brokers/ids/1
/brokers/topics/foo
/brokers/topics/foo/partitions/0/state
/brokers/topics/foo/partitions/1/state
/brokers/topics/foo/partitions/2/state
...
Controller Startup with ZK
7
Controller Startup with ZK
8
● One broker wins the
election in ZooKeeper to
be the controller
Load full
metadata
Controller Startup with ZK
9
Controller Startup with ZK
10
● UpdateMetadataRequest
● LeaderAndIsrRequest
All
partitions
Problems with ZK-based Controller Startup
11
● Have to load all metadata synchronously on startup:
○ O(num_partitions), O(num_brokers)
○ Controller is unavailable during this time
■ Cold start: cluster unavailable.
■ Controller restart: admin ops and ISR changes unavailable
● Have to send all metadata to all brokers on startup
3 minutes
How KRaft Replaces ZooKeeper
12
● Instead of ZooKeeper, we have an internal
topic named __cluster_metadata
○ Replicated with KRaft
○ Single partition
○ The leader is the active controller
● KRaft: Raft for Kafka
○ The Raft protocol implemented
in Kafka
■ Records committed by a
majority of nodes
○ Self-managed quorum
■ Doesn’t rely on an external
system for leader election
Metadata Records
13
● Binary records containing
metadata
○ KRPC format
○ Auto-generated from
protocol schemas
○ Can also be translated
into JSON for human
readability
● Two ways to evolve format
○ New record versions
○ Tagged fields
● Some records are deltas that
apply changes to existing
state
{
"type": "REGISTER_BROKER_RECORD",
"version": 0,
"data": {
"brokerId": 1,
"incarnationId": "P3UFsWoNR-erL9PK98YLsA",
"brokerEpoch": 0,
"endPoints": [
{
"name": "PLAINTEXT",
"host": "localhost",
"port": 9092,
"securityProtocol": 0
}
],
"features": [],
"rack": null
}
}
Metadata as An Ordered Log
14
TopicRecord(name=foo, id=rtkInsMkQPiEBj6uz67rrQ)
PartitionRecord(id=rtkInsMkQPiEBj6uz67rrQ, index=0, …)
PartitionRecord(id=rtkInsMkQPiEBj6uz67rrQ, index=1, …)
PartitionRecord(id=rtkInsMkQPiEBj6uz67rrQ, index=2, …)
ConfigRecord(name=num.io.threads, value=...)
RegisterBrokerRecord(id=4, endpoints=…, …)
10434
10435
10436
10437
10438
10439
2
1
Controller Startup with KRaft
15
● The blue nodes are
designated KRaft
controllers
Controller Startup with KRaft
16
● The KRaft controllers will
elect a single leader
● The leader will have all
previously committed
records
Controller Startup with KRaft
17
● The newly elected KRaft
controller is ready
immediately
● Brokers fetch only the
metadata they need
● New brokers and brokers that
are behind fetch snapshots
Controller Failover
18
ZK Mode KRaft Mode
• Win controller
election
• Load all topics
and partitions
• Send
LeaderAndIsr +
UpdateMetadata
to all brokers
• Win KRaft
election
• Start handling
requests from
brokers
Rolling Nodes
19
ZK Mode KRaft Mode
Restarted node
begins with no
metadata. Must
wait for full
metadata update
RPCs from
controller.
Restarted node
consults its local
metadata cache,
transfers only what
it doesn’t have,
based on
metadata offset.
Calling an Admin API with ZK
20
● External client connects to
a random broker
Calling an Admin API with ZK
21
● Broker alters ZooKeeper
● Example: create
znodes for new topic
Calling an Admin API with ZK
22
● Znode watch triggers
● Controller loads changes
Calling an Admin API with ZK
23
● Controller pushes out
changes to brokers
● Incremental LeaderAndIsr
● Incremental UpdateMetadata
ZK-based Programming Model
24
Controller Thread
ZooKeeper
Create
/brokers/topics/foo
Request Handler
/brokers/topics
Changed
List
/brokers/topics
Compute
changes
List
/brokers/topics/foo
Problems
25
● What if the controller fails to send an update to a specific broker?
○ Metadata divergence
○ No easy way to know what we have and what we don’t have
● Difficult programming model
○ Multiple writers to ZooKeeper
○ Can’t assume your cache is up-to-date!
● ZK is the bottleneck for all admin operations
○ Often 1 admin operation leads to many ZK operations
○ Blocking
Calling an Admin API with KRaft
26
● External client connects to
a random broker
Calling an Admin API with KRaft
27
● Broker forwards the
request to the active
controller for processing
Calling an Admin API with KRaft
28
● Controller creates
metadata records and
persists them to
__cluster_metadata
Calling an Admin API with KRaft
29
● Once the records have
been committed to the
metadata log, the active
controller returns the
result to the forwarding
broker, which returns it to
the external client
Calling an Admin API with KRaft
30
● Brokers are continuously
fetching metadata from the
active controller.
● They become aware of the
admin changes by reading
the new metadata records.
KRaft-based Programming Model
31
Raft Quorum
Controller
CreateTopicsRequest
Request Handler
Write new
records
Last stable
offset advances
CreateTopicsResponse
KRaft-based Programming Model: Pipelining
32
Raft Quorum
Controller
CreateTopicsRequest
Request Handler
Write new
records
Last stable
offset advances
CreateTopicsResponse
Admin Operations in KRaft
33
● Pull-based metadata propagation model can recover from RPC send
failures easily
● Simpler programming model: single-writer
● Pipelining means that we can have multiple metadata operations in
flight at once
Deploying KRaft
34
Getting KRaft
35
● The latest Apache (and Confluent) releases support both ZK mode and
KRaft mode
○ KRaft mode is supported from the same code base – not a branch
● KRaft mode is not yet recommended for production
○ In Apache 2.8, KRaft was in early access
○ As of Apache 3.0, KRaft is in preview
○ Definitely use the latest version if you are trying it out – many bugs
have been fixed
● ZooKeeper mode will be supported for a while
○ I’ll discuss the release plans in more detail in the Roadmap section of
this talk
Creating a New KRaft Cluster
● First two steps are
new!
36
$ kafka-storage.sh random-uuid
MX1HbXq8TFymY2SYR1hGYg
$ kafka-storage.sh format 
-c ./config/kraft/controller.properties 
--cluster-id MX1HbXq8TFymY2SYR1hGYg
Formatting /tmp/kraft-controller-logs
● Generate a new random uuid with kafka-storage tool
● On each node, use the kafka-storage tool to format storage directories
● Start all nodes
The Role of Formatting
37
● In ZK mode, brokers auto-format storage directories if they are blank on
startup.
● In KRaft mode, we must format each broker and controller before starting
the nodes.
● Why not auto-format?
○ Avoid admin mistakes or filesystem issues that could lose data
■ If a volume is not mounted, the directory may appear empty
■ The same reasons why databases and filesystems require a
formatting step.
○ We need a way to bootstrap security-related dynamic configurations
that the quorum and the brokers need to function
■ SCRAM configurations
■ Dynamic SSL configurations
■ Metadata version
Bootstrapping
38
ZK Mode KRaft Mode
• Start up
ZooKeeper
cluster
• Set SCRAM
users, dynamic
security configs,
etc. in ZK
• Start brokers
• Generate
cluster ID
• Run format tool on all
nodes, specifying
cluster ID, SCRAM,
security configs,
metadata version, etc.
• Start nodes
KRaft Controllers
● A small pool of nodes that we designate ahead of time
○ Hot standbys for the active controller
○ Single log directory for __cluster_metadata
● Sizing is very similar to ZooKeeper
○ Typically 3 controller nodes
○ May use more to provide more resilience
○ Just like with ZK, need to keep a majority of nodes alive
● Can be configured by copying over broker configuration file
○ Set node.roles to controller
○ Set node.id
39
4
0
Combined Mode
Controller processes
in the same JVM as
broker processes
Shared IDs, shared
JVMs, shared nodes
Can have single JVM
Kafka cluster
Deploying KRaft Controllers
Separate Mode
Controller
processes on
separate nodes
Different IDs,
different JVMs,
different nodes
Better isolation,
easier rolls
Separate Mode with
Kubernetes
Bin-Packing
Controllers in
separate k8s pods
Different IDs,
different JVMs,
maybe shared
nodes
Some Considerations for Deploying Controllers
41
● Isolation
● Avoid co-location in busy clusters
● Try to avoid situations where
traffic from outside disrupts
internals
● High Availability
● Try to not to have multiple
controllers in a single failure
domain
● In clusters which span 3
availability zones, put a controller
in each zone
Controller
New Controller Endpoint
42
Broker
Controller
EXTERNAL INTERBROKER
CONTROLLER
● Only exposed on
controllers
● Not an advertised
listener or broker listener
● Should not be accessible
externally
Configuring KRaft Clusters
● node.id
○ Replaces broker.id on both brokers and controllers.
● process.roles
○ broker
○ controller
○ broker,controller
● controller.quorum.voters
○ 1@controller1.9093,2@controller2:9093,3@controller3:9093
○ Statically configured
● controller.listener.names
○ Tell brokers and other controllers how to connect to the controller
○ These are not advertised listeners and cannot appear in
“advertised.listeners” on the broker
○ On the controller they appear in “listeners”
43
Rolling KRaft Clusters
44
Rolling KRaft Controllers
45
● Much lighter roll process
○ No cluster-destabilizing full metadata updates to send
● KRaft controllers can be rolled very quickly compared to brokers
○ They do not manage lots of data directories
○ Make sure that monitoring software can keep up
● KRaft controller software can be upgraded on a separate
schedule
○ Forwards and backwards compatibility with alternate broker
versions
Monitoring Rolls
46
● ZK Mode
○ Under-replicated partitions
○ ActiveControllerCount
○ LeaderAndIsr request time
○ UpdateMetadata request time
○ ZooKeeperExpiresPerSec
● KRaft Mode
○ Under-replicated partitions
■ Shows broker health
○ ActiveControllerCount
○ MetadataOffset
○ SnapshotLag
○ SnapshotSizeBytes
○ MetadataCommitRate
○ MetadataCommitLatency
○ Current Raft state (follower,
leader, candidate, observer)
■ Shows controller health
○ FencedBrokerCount
○ ActiveBrokerCount
Creating New Partitions During Cluster Rolls in ZK
47
● In ZK mode, controller “forgets
about” brokers that are
temporary down during a roll
● In a 3-node cluster, if one node is
rolling, we don’t have enough
nodes to create a partition with
3x replication.
● Must run with 4 nodes, even if
we don’t need all 4.
Rolling!
Replica 1 Replica 2 Replica 3
New
Partition
48
● In KRaft mode, the controller
remembers brokers that are
temporary down during a roll
● In KRaft mode, those brokers
enter “fenced state”
○ Can place new replicas on
these nodes if needed
○ Brokers will be unfenced
once they come up
● Many small clusters can now use
3 nodes rather than 4.
Rolling!
Replica 1 Replica 2 Replica 3
New
Partition
Creating New Partitions During Cluster Rolls in KRaft
Handling Version Skew
49
● During the roll, old and new software versions
must co-exist
● Older brokers can’t understand newer APIs and
protocols
● inter.broker.protocol solves this in the ZK world
○ Controls what RPC protocols brokers use
when communicating with each other
○ Controls what features brokers support
● Examples
○ Whether to use AlterPartitions for ISRs
○ Whether to use topic IDs
○ Whether to use the new version of some API
(ListOffsets, FetchRequest, etc.)
KAFKA_2_4_IV0,
KAFKA_2_4_IV1,
KAFKA_2_5_IV0,
KAFKA_2_6_IV0,
KAFKA_2_7_IV0,
KAFKA_2_7_IV1,
KAFKA_2_7_IV2,
KAFKA_2_8_IV0,
KAFKA_2_8_IV1,
KAFKA_3_0_IV0,
KAFKA_3_0_IV1,
KAFKA_3_1_IV0,
KAFKA_3_2_IV0
Problems with Inter Broker Protocol in ZK
50
● inter.broker.protocol version must be manually configured
○ If it is left out of the configuration file, it defaults to the latest
version, which is probably not what the user wants
● Because inter.broker.protocol is statically configured, it can’t be
changed without restarting all the nodes
○ To upgrade to a new version AND get the new features requires a
“double roll” in ZK mode
● No downgrade support
○ It is safe to downgrade between some pairs of versions, but not
others.
Introducing metadata.version
51
● In KRaft mode, inter.broker.protocol is replaced by metadata.version
● Each new inter.broker.protocol version has a corresponding
metadata.version
● metadata.version is dynamically configured
○ Does not require a roll to change
○ It is changed by invoking a controller API.
○ “Guard rails”
■ The controller will refuse to do the upgrade if some brokers
have not been rolled
● Supports downgrade!
Downgrading metadata.version
52
● Like upgrade, downgrade can be done dynamically from the command
line
● Two kinds of downgrades
○ Safe: no metadata will be lost by the downgrade
○ Unsafe: some metadata may be cleared during the downgrade
● Unsafe downgrades require the operator to provide an override flag
Troubleshooting KRaft Clusters
53
Troubleshooting KRaft Clusters
54
● In KRaft, __cluster_metadata replaces ZooKeeper as the store of
record
● It’s often helpful to examine the metadata log to see what
happened
● All batches come with timestamps
● Offsets are uniform across the cluster
○ A specific record will have the same offset on each
controller and broker
● Several tools for looking at metadata logs
○ kafka-dump-log
○ kafka-metadata-shell
kafka-dump-log
55
$ ./bin/kafka-dump-log.sh 
--cluster-metadata-decoder 
--files /tmp/logs/__cluster_metadata-0/00000000000000000000.log
Dumping /tmp/logs/__cluster_metadata-0/00000000000000000000.log
Starting offset: 0
baseOffset: 0 lastOffset: 0 count: 1 baseSequence: -1
[...]
| offset: 0 CreateTime: 1650857270775 keySize: 4 valueSize: 19
sequence: -1 headerKeys: [] controlType: LEADER_CHANGE(2)
baseOffset: 1 lastOffset: 1 count: 1 baseSequence: -1
[...]{"type":"REGISTER_BROKER_RECORD","version":0,"data":{"...
Interactive Metadata Shell
56
● Replaces zookeeper-shell in KRaft clusters
● Data sources
○ Snapshot
○ Running controller cluster
● Reads __cluster_metadata log entries into memory
● Constructs a “virtual filesystem” with the cluster’s information
● Commands available: ls, pwd, cd, find, etc.
kafka-metadata-shell
57
$ ./bin/kafka-metadata-shell.sh --snapshot 
/tmp/logs/__cluster_metadata-0/00000000000000000000.log
Loading...
Starting...
[ Kafka Metadata Shell ]
>> ls
brokers configs local metadataQuorum topicIds topics
>> cat /brokers/1/registration
RegisterBrokerRecord(brokerId=1,
incarnationId=tHo3Z8dYSuONV5hA82BVug, brokerEpoch=0,
endPoints=[BrokerEndpoint(name='PLAINTEXT', host='localhost',
port=9092, securityProtocol=0)], features=[], rack=null, fenced=true)
Monitoring the Quorum
58
● Metrics
○ MetadataOffset
○ SnapshotLag
○ SnapshotSizeBytes
○ MetadataCommitRate
○ MetadataCommitLatency
○ Current Raft state (follower,
leader, candidate, observer)
■ Important for controller
health
● DescribeQuorum RPC
○ Leader ID
○ Leader epoch
○ High water mark
○ Current voters
■ Controllers
○ Current observers
■ Brokers
■ Possible metadata shell
instances, etc.
○ Log end offset of all followers
Upgrading from ZooKeeper Mode
59
Upgrading Clusters from ZK Mode
60
ZK mode ZK mode ZK mode
● Initially: all metadata in
ZooKeeper.
● All brokers in ZK mode.
Upgrading Clusters from ZK Mode
61
61
ZK mode ZK mode ZK mode
KRaft
controller
● We add a quorum of new
KRaft controllers to the
cluster
KRaft
controller
KRaft
controller
Upgrading Clusters from ZK Mode
62
62
ZK mode ZK mode ZK mode
● The KRaft controllers elect
a leader from their ranks
and force that leader to be
the cluster leader in ZK.
KRaft
controller
KRaft
controller
KRaft
controller
Upgrading Clusters from ZK Mode
63
63
ZK mode ZK mode ZK mode
● The KRaft controller loads
all metadata from ZK, and
sends out the appropriate
LeaderAndIsr, UMRs.
● Future metadata changes
go into the KRaft quorum.
KRaft
controller
KRaft
controller
KRaft
controller
metadata
Upgrading Clusters from ZK Mode
64
64
ZK mode ZK mode
KRaft
mode
● Now brokers can be rolled
one by one.
● KRaft controller will still
send LeaderAndIsr, etc. to
un-upgraded brokers.
KRaft
controller
KRaft
controller
KRaft
controller
Upgrading Clusters from ZK Mode
65
65
ZK mode
KRaft
mode
KRaft
mode
● This relies on brokers never
directly communicating
with ZK (except to register
themselves)
● KIP-591 Forwarding
KRaft
controller
KRaft
controller
KRaft
controller
Upgrading Clusters from ZK Mode
66
66
KRaft
mode
KRaft
mode
KRaft
mode
● Once all brokers have been
rolled, ZK can be removed.
KRaft
controller
KRaft
controller
KRaft
controller
Roadmap
67
KRaft Project Phases
1. Design
2. Initial Implementation
3. Available for testing
4. Available for production
5. Bridge release
6. ZK removed
68
KRaft Project Phases
69
1. Design
2. Initial Implementation
3. Available for testing
4. Available for production
5. Bridge release
6. ZK removed
Design Phase
70
● During the design phase, we had many upstream discussions about the
goals of the KRaft project (called the KIP-500 project at the time) and how
we would evolve the project to meet them.
● September 2019: vote for “KIP-500: Replace ZooKeeper with a
Self-Managed Metadata Quorum” passes
● June 2020: vote for “KIP-591: Redirect Zookeeper Mutation Protocols to
The Controller” passes (some revisions will be made later)
● August 2020: vote for “KIP-595: A Raft Protocol for the Quorum” passes
● September 2020: vote for “KIP-631: The Quorum-Based Kafka Controller”
passes
● October 2020: vote for “KIP-630: Kafka Raft Snapshots” passes
● Several other minor KIPs filled gaps in our admin APIs which were being
filled by direct ZK access
Initial Implementation Phase
71
● During this phase, we put the ideas from the design phase into practice.
● The implementation was pretty unstable during this time period – many
big changes and fixes were happening all the time.
● There were many minor changes to remove ZK dependencies as well.
● Late 2020: KIP-500 work started in a branch.
● Early 2021: all code merged to trunk
○ New controller code
○ New self-managed Raft quorum implementation
● April 2021: Apache 2.8 released
○ First Apache release with KRaft
○ KRaft in Early Access
● September 2021: Apache 3.0 release
○ First release with KRaft snapshot support
○ Added reassignment and EOS support as well
○ Preview
Testing Phase
72
● This is where we are now.
● KRaft is in preview mode
○ Not recommended for production
○ Available for testing
● January 2022: Apache Kafka 3.1 released
○ Many bug fixes, new tests
○ Some new KRaft metrics
● May 2022: Apache Kafka 3.2 released (soon)
○ New KRaft-based authorizer
Production Phase
73
● Coming soon!
● During this phase, we will recommend KRaft for production use.
○ Also known as “going GA”
● There may still be some feature gaps
○ SCRAM
○ JBOD support
○ Delegation token support
● Most importantly, we will not have support for upgrading from ZK yet in
this phase.
○ Therefore, in this phase, KRaft will be useful for new clusters, but not
for existing ones
Bridge Release
74
● A bridge release is a release that
supports upgrading from ZK mode
to KRaft mode.
○ As described in the section on
upgrading from ZK, this
requires brokers to channel
their admin changes through
the controller, rather than
mutating ZK directly.
● We will make multiple bridge
releases
● We will work hard to close all the
remaining feature gaps during this
phase, so that there are no longer
any users stuck on ZK.
Last Phase: Removing ZK Support
75
● Eventually, we will want to remove ZK support so that we no longer have
to maintain two controller implementations
● Before we can remove ZK support, we have to deprecate it, and then
make a new major release
● Timeline for full removal is TBD
Thank You!
Colin McCabe
cmccabe@apache.org

Contenu connexe

Tendances

Automate Your Kafka Cluster with Kubernetes Custom Resources
Automate Your Kafka Cluster with Kubernetes Custom Resources Automate Your Kafka Cluster with Kubernetes Custom Resources
Automate Your Kafka Cluster with Kubernetes Custom Resources confluent
 
Autoscaling Flink with Reactive Mode
Autoscaling Flink with Reactive ModeAutoscaling Flink with Reactive Mode
Autoscaling Flink with Reactive ModeFlink Forward
 
Cruise Control: Effortless management of Kafka clusters
Cruise Control: Effortless management of Kafka clustersCruise Control: Effortless management of Kafka clusters
Cruise Control: Effortless management of Kafka clustersPrateek Maheshwari
 
Performance Tuning RocksDB for Kafka Streams’ State Stores
Performance Tuning RocksDB for Kafka Streams’ State StoresPerformance Tuning RocksDB for Kafka Streams’ State Stores
Performance Tuning RocksDB for Kafka Streams’ State Storesconfluent
 
Getting up to speed with Kafka Connect: from the basics to the latest feature...
Getting up to speed with Kafka Connect: from the basics to the latest feature...Getting up to speed with Kafka Connect: from the basics to the latest feature...
Getting up to speed with Kafka Connect: from the basics to the latest feature...HostedbyConfluent
 
CockroachDB: Architecture of a Geo-Distributed SQL Database
CockroachDB: Architecture of a Geo-Distributed SQL DatabaseCockroachDB: Architecture of a Geo-Distributed SQL Database
CockroachDB: Architecture of a Geo-Distributed SQL DatabaseC4Media
 
Storing 16 Bytes at Scale
Storing 16 Bytes at ScaleStoring 16 Bytes at Scale
Storing 16 Bytes at ScaleFabian Reinartz
 
From Postgres to Event-Driven: using docker-compose to build CDC pipelines in...
From Postgres to Event-Driven: using docker-compose to build CDC pipelines in...From Postgres to Event-Driven: using docker-compose to build CDC pipelines in...
From Postgres to Event-Driven: using docker-compose to build CDC pipelines in...confluent
 
Kafka 101 and Developer Best Practices
Kafka 101 and Developer Best PracticesKafka 101 and Developer Best Practices
Kafka 101 and Developer Best Practicesconfluent
 
Introduction to Kafka connect
Introduction to Kafka connectIntroduction to Kafka connect
Introduction to Kafka connectKnoldus Inc.
 
Apache Kafka Architecture & Fundamentals Explained
Apache Kafka Architecture & Fundamentals ExplainedApache Kafka Architecture & Fundamentals Explained
Apache Kafka Architecture & Fundamentals Explainedconfluent
 
ksqlDB: A Stream-Relational Database System
ksqlDB: A Stream-Relational Database SystemksqlDB: A Stream-Relational Database System
ksqlDB: A Stream-Relational Database Systemconfluent
 
Securing Kafka
Securing Kafka Securing Kafka
Securing Kafka confluent
 
Tuning Apache Kafka Connectors for Flink.pptx
Tuning Apache Kafka Connectors for Flink.pptxTuning Apache Kafka Connectors for Flink.pptx
Tuning Apache Kafka Connectors for Flink.pptxFlink Forward
 
Getting up to speed with MirrorMaker 2 | Mickael Maison, IBM and Ryanne Dolan...
Getting up to speed with MirrorMaker 2 | Mickael Maison, IBM and Ryanne Dolan...Getting up to speed with MirrorMaker 2 | Mickael Maison, IBM and Ryanne Dolan...
Getting up to speed with MirrorMaker 2 | Mickael Maison, IBM and Ryanne Dolan...HostedbyConfluent
 
Performance Tuning RocksDB for Kafka Streams' State Stores (Dhruba Borthakur,...
Performance Tuning RocksDB for Kafka Streams' State Stores (Dhruba Borthakur,...Performance Tuning RocksDB for Kafka Streams' State Stores (Dhruba Borthakur,...
Performance Tuning RocksDB for Kafka Streams' State Stores (Dhruba Borthakur,...confluent
 
Grafana Mimir and VictoriaMetrics_ Performance Tests.pptx
Grafana Mimir and VictoriaMetrics_ Performance Tests.pptxGrafana Mimir and VictoriaMetrics_ Performance Tests.pptx
Grafana Mimir and VictoriaMetrics_ Performance Tests.pptxRomanKhavronenko
 
Building Event Driven Architectures with Kafka and Cloud Events (Dan Rosanova...
Building Event Driven Architectures with Kafka and Cloud Events (Dan Rosanova...Building Event Driven Architectures with Kafka and Cloud Events (Dan Rosanova...
Building Event Driven Architectures with Kafka and Cloud Events (Dan Rosanova...confluent
 

Tendances (20)

Automate Your Kafka Cluster with Kubernetes Custom Resources
Automate Your Kafka Cluster with Kubernetes Custom Resources Automate Your Kafka Cluster with Kubernetes Custom Resources
Automate Your Kafka Cluster with Kubernetes Custom Resources
 
Autoscaling Flink with Reactive Mode
Autoscaling Flink with Reactive ModeAutoscaling Flink with Reactive Mode
Autoscaling Flink with Reactive Mode
 
Cruise Control: Effortless management of Kafka clusters
Cruise Control: Effortless management of Kafka clustersCruise Control: Effortless management of Kafka clusters
Cruise Control: Effortless management of Kafka clusters
 
Performance Tuning RocksDB for Kafka Streams’ State Stores
Performance Tuning RocksDB for Kafka Streams’ State StoresPerformance Tuning RocksDB for Kafka Streams’ State Stores
Performance Tuning RocksDB for Kafka Streams’ State Stores
 
Kafka presentation
Kafka presentationKafka presentation
Kafka presentation
 
Getting up to speed with Kafka Connect: from the basics to the latest feature...
Getting up to speed with Kafka Connect: from the basics to the latest feature...Getting up to speed with Kafka Connect: from the basics to the latest feature...
Getting up to speed with Kafka Connect: from the basics to the latest feature...
 
CockroachDB: Architecture of a Geo-Distributed SQL Database
CockroachDB: Architecture of a Geo-Distributed SQL DatabaseCockroachDB: Architecture of a Geo-Distributed SQL Database
CockroachDB: Architecture of a Geo-Distributed SQL Database
 
Storing 16 Bytes at Scale
Storing 16 Bytes at ScaleStoring 16 Bytes at Scale
Storing 16 Bytes at Scale
 
From Postgres to Event-Driven: using docker-compose to build CDC pipelines in...
From Postgres to Event-Driven: using docker-compose to build CDC pipelines in...From Postgres to Event-Driven: using docker-compose to build CDC pipelines in...
From Postgres to Event-Driven: using docker-compose to build CDC pipelines in...
 
Kafka 101 and Developer Best Practices
Kafka 101 and Developer Best PracticesKafka 101 and Developer Best Practices
Kafka 101 and Developer Best Practices
 
Introduction to Kafka connect
Introduction to Kafka connectIntroduction to Kafka connect
Introduction to Kafka connect
 
Apache Kafka Architecture & Fundamentals Explained
Apache Kafka Architecture & Fundamentals ExplainedApache Kafka Architecture & Fundamentals Explained
Apache Kafka Architecture & Fundamentals Explained
 
ksqlDB: A Stream-Relational Database System
ksqlDB: A Stream-Relational Database SystemksqlDB: A Stream-Relational Database System
ksqlDB: A Stream-Relational Database System
 
Securing Kafka
Securing Kafka Securing Kafka
Securing Kafka
 
Tuning Apache Kafka Connectors for Flink.pptx
Tuning Apache Kafka Connectors for Flink.pptxTuning Apache Kafka Connectors for Flink.pptx
Tuning Apache Kafka Connectors for Flink.pptx
 
Getting up to speed with MirrorMaker 2 | Mickael Maison, IBM and Ryanne Dolan...
Getting up to speed with MirrorMaker 2 | Mickael Maison, IBM and Ryanne Dolan...Getting up to speed with MirrorMaker 2 | Mickael Maison, IBM and Ryanne Dolan...
Getting up to speed with MirrorMaker 2 | Mickael Maison, IBM and Ryanne Dolan...
 
Performance Tuning RocksDB for Kafka Streams' State Stores (Dhruba Borthakur,...
Performance Tuning RocksDB for Kafka Streams' State Stores (Dhruba Borthakur,...Performance Tuning RocksDB for Kafka Streams' State Stores (Dhruba Borthakur,...
Performance Tuning RocksDB for Kafka Streams' State Stores (Dhruba Borthakur,...
 
Grafana Mimir and VictoriaMetrics_ Performance Tests.pptx
Grafana Mimir and VictoriaMetrics_ Performance Tests.pptxGrafana Mimir and VictoriaMetrics_ Performance Tests.pptx
Grafana Mimir and VictoriaMetrics_ Performance Tests.pptx
 
Building Event Driven Architectures with Kafka and Cloud Events (Dan Rosanova...
Building Event Driven Architectures with Kafka and Cloud Events (Dan Rosanova...Building Event Driven Architectures with Kafka and Cloud Events (Dan Rosanova...
Building Event Driven Architectures with Kafka and Cloud Events (Dan Rosanova...
 
Galera Cluster DDL and Schema Upgrades 220217
Galera Cluster DDL and Schema Upgrades 220217Galera Cluster DDL and Schema Upgrades 220217
Galera Cluster DDL and Schema Upgrades 220217
 

Similaire à Bringing Kafka Without Zookeeper Into Production with Colin McCabe | Kafka Summit London 2022

Kafka’s New Control Plane: The Quorum Controller | Colin McCabe, Confluent
Kafka’s New Control Plane: The Quorum Controller | Colin McCabe, ConfluentKafka’s New Control Plane: The Quorum Controller | Colin McCabe, Confluent
Kafka’s New Control Plane: The Quorum Controller | Colin McCabe, ConfluentHostedbyConfluent
 
Disaster Recovery Options Running Apache Kafka in Kubernetes with Rema Subra...
 Disaster Recovery Options Running Apache Kafka in Kubernetes with Rema Subra... Disaster Recovery Options Running Apache Kafka in Kubernetes with Rema Subra...
Disaster Recovery Options Running Apache Kafka in Kubernetes with Rema Subra...HostedbyConfluent
 
Kafka Needs no Keeper( Jason Gustafson & Colin McCabe, Confluent) Kafka Summi...
Kafka Needs no Keeper( Jason Gustafson & Colin McCabe, Confluent) Kafka Summi...Kafka Needs no Keeper( Jason Gustafson & Colin McCabe, Confluent) Kafka Summi...
Kafka Needs no Keeper( Jason Gustafson & Colin McCabe, Confluent) Kafka Summi...confluent
 
Scaling Up Logging and Metrics
Scaling Up Logging and MetricsScaling Up Logging and Metrics
Scaling Up Logging and MetricsRicardo Lourenço
 
Kafka Needs No Keeper
Kafka Needs No KeeperKafka Needs No Keeper
Kafka Needs No KeeperC4Media
 
OpenTelemetry For Operators
OpenTelemetry For OperatorsOpenTelemetry For Operators
OpenTelemetry For OperatorsKevin Brockhoff
 
Kubernetes @ Squarespace (SRE Portland Meetup October 2017)
Kubernetes @ Squarespace (SRE Portland Meetup October 2017)Kubernetes @ Squarespace (SRE Portland Meetup October 2017)
Kubernetes @ Squarespace (SRE Portland Meetup October 2017)Kevin Lynch
 
Clusternaut: Orchestrating  Percona XtraDB Cluster with Kubernetes
Clusternaut:  Orchestrating  Percona XtraDB Cluster with KubernetesClusternaut:  Orchestrating  Percona XtraDB Cluster with Kubernetes
Clusternaut: Orchestrating  Percona XtraDB Cluster with KubernetesRaghavendra Prabhu
 
Tips & Tricks for Apache Kafka®
Tips & Tricks for Apache Kafka®Tips & Tricks for Apache Kafka®
Tips & Tricks for Apache Kafka®confluent
 
LCA14: LCA14-412: GPGPU on ARM SoC session
LCA14: LCA14-412: GPGPU on ARM SoC sessionLCA14: LCA14-412: GPGPU on ARM SoC session
LCA14: LCA14-412: GPGPU on ARM SoC sessionLinaro
 
Kubernetes @ Squarespace: Kubernetes in the Datacenter
Kubernetes @ Squarespace: Kubernetes in the DatacenterKubernetes @ Squarespace: Kubernetes in the Datacenter
Kubernetes @ Squarespace: Kubernetes in the DatacenterKevin Lynch
 
Plny12 galera-cluster-best-practices
Plny12 galera-cluster-best-practicesPlny12 galera-cluster-best-practices
Plny12 galera-cluster-best-practicesDimas Prasetyo
 
Citi TechTalk Session 2: Kafka Deep Dive
Citi TechTalk Session 2: Kafka Deep DiveCiti TechTalk Session 2: Kafka Deep Dive
Citi TechTalk Session 2: Kafka Deep Diveconfluent
 
Introduction to apache kafka
Introduction to apache kafkaIntroduction to apache kafka
Introduction to apache kafkaSamuel Kerrien
 
Testing Persistent Storage Performance in Kubernetes with Sherlock
Testing Persistent Storage Performance in Kubernetes with SherlockTesting Persistent Storage Performance in Kubernetes with Sherlock
Testing Persistent Storage Performance in Kubernetes with SherlockScyllaDB
 
WKSctl: Gitops Management of Kubernetes Clusters
WKSctl: Gitops Management of Kubernetes ClustersWKSctl: Gitops Management of Kubernetes Clusters
WKSctl: Gitops Management of Kubernetes ClustersWeaveworks
 
Container Orchestration from Theory to Practice
Container Orchestration from Theory to PracticeContainer Orchestration from Theory to Practice
Container Orchestration from Theory to PracticeDocker, Inc.
 
Kubecon seattle 2018 workshop slides
Kubecon seattle 2018 workshop slidesKubecon seattle 2018 workshop slides
Kubecon seattle 2018 workshop slidesWeaveworks
 
Kernel Recipes 2015 - So you want to write a Linux driver framework
Kernel Recipes 2015 - So you want to write a Linux driver frameworkKernel Recipes 2015 - So you want to write a Linux driver framework
Kernel Recipes 2015 - So you want to write a Linux driver frameworkAnne Nicolas
 
A Hitchhiker's Guide to Apache Kafka Geo-Replication with Sanjana Kaundinya ...
 A Hitchhiker's Guide to Apache Kafka Geo-Replication with Sanjana Kaundinya ... A Hitchhiker's Guide to Apache Kafka Geo-Replication with Sanjana Kaundinya ...
A Hitchhiker's Guide to Apache Kafka Geo-Replication with Sanjana Kaundinya ...HostedbyConfluent
 

Similaire à Bringing Kafka Without Zookeeper Into Production with Colin McCabe | Kafka Summit London 2022 (20)

Kafka’s New Control Plane: The Quorum Controller | Colin McCabe, Confluent
Kafka’s New Control Plane: The Quorum Controller | Colin McCabe, ConfluentKafka’s New Control Plane: The Quorum Controller | Colin McCabe, Confluent
Kafka’s New Control Plane: The Quorum Controller | Colin McCabe, Confluent
 
Disaster Recovery Options Running Apache Kafka in Kubernetes with Rema Subra...
 Disaster Recovery Options Running Apache Kafka in Kubernetes with Rema Subra... Disaster Recovery Options Running Apache Kafka in Kubernetes with Rema Subra...
Disaster Recovery Options Running Apache Kafka in Kubernetes with Rema Subra...
 
Kafka Needs no Keeper( Jason Gustafson & Colin McCabe, Confluent) Kafka Summi...
Kafka Needs no Keeper( Jason Gustafson & Colin McCabe, Confluent) Kafka Summi...Kafka Needs no Keeper( Jason Gustafson & Colin McCabe, Confluent) Kafka Summi...
Kafka Needs no Keeper( Jason Gustafson & Colin McCabe, Confluent) Kafka Summi...
 
Scaling Up Logging and Metrics
Scaling Up Logging and MetricsScaling Up Logging and Metrics
Scaling Up Logging and Metrics
 
Kafka Needs No Keeper
Kafka Needs No KeeperKafka Needs No Keeper
Kafka Needs No Keeper
 
OpenTelemetry For Operators
OpenTelemetry For OperatorsOpenTelemetry For Operators
OpenTelemetry For Operators
 
Kubernetes @ Squarespace (SRE Portland Meetup October 2017)
Kubernetes @ Squarespace (SRE Portland Meetup October 2017)Kubernetes @ Squarespace (SRE Portland Meetup October 2017)
Kubernetes @ Squarespace (SRE Portland Meetup October 2017)
 
Clusternaut: Orchestrating  Percona XtraDB Cluster with Kubernetes
Clusternaut:  Orchestrating  Percona XtraDB Cluster with KubernetesClusternaut:  Orchestrating  Percona XtraDB Cluster with Kubernetes
Clusternaut: Orchestrating  Percona XtraDB Cluster with Kubernetes
 
Tips & Tricks for Apache Kafka®
Tips & Tricks for Apache Kafka®Tips & Tricks for Apache Kafka®
Tips & Tricks for Apache Kafka®
 
LCA14: LCA14-412: GPGPU on ARM SoC session
LCA14: LCA14-412: GPGPU on ARM SoC sessionLCA14: LCA14-412: GPGPU on ARM SoC session
LCA14: LCA14-412: GPGPU on ARM SoC session
 
Kubernetes @ Squarespace: Kubernetes in the Datacenter
Kubernetes @ Squarespace: Kubernetes in the DatacenterKubernetes @ Squarespace: Kubernetes in the Datacenter
Kubernetes @ Squarespace: Kubernetes in the Datacenter
 
Plny12 galera-cluster-best-practices
Plny12 galera-cluster-best-practicesPlny12 galera-cluster-best-practices
Plny12 galera-cluster-best-practices
 
Citi TechTalk Session 2: Kafka Deep Dive
Citi TechTalk Session 2: Kafka Deep DiveCiti TechTalk Session 2: Kafka Deep Dive
Citi TechTalk Session 2: Kafka Deep Dive
 
Introduction to apache kafka
Introduction to apache kafkaIntroduction to apache kafka
Introduction to apache kafka
 
Testing Persistent Storage Performance in Kubernetes with Sherlock
Testing Persistent Storage Performance in Kubernetes with SherlockTesting Persistent Storage Performance in Kubernetes with Sherlock
Testing Persistent Storage Performance in Kubernetes with Sherlock
 
WKSctl: Gitops Management of Kubernetes Clusters
WKSctl: Gitops Management of Kubernetes ClustersWKSctl: Gitops Management of Kubernetes Clusters
WKSctl: Gitops Management of Kubernetes Clusters
 
Container Orchestration from Theory to Practice
Container Orchestration from Theory to PracticeContainer Orchestration from Theory to Practice
Container Orchestration from Theory to Practice
 
Kubecon seattle 2018 workshop slides
Kubecon seattle 2018 workshop slidesKubecon seattle 2018 workshop slides
Kubecon seattle 2018 workshop slides
 
Kernel Recipes 2015 - So you want to write a Linux driver framework
Kernel Recipes 2015 - So you want to write a Linux driver frameworkKernel Recipes 2015 - So you want to write a Linux driver framework
Kernel Recipes 2015 - So you want to write a Linux driver framework
 
A Hitchhiker's Guide to Apache Kafka Geo-Replication with Sanjana Kaundinya ...
 A Hitchhiker's Guide to Apache Kafka Geo-Replication with Sanjana Kaundinya ... A Hitchhiker's Guide to Apache Kafka Geo-Replication with Sanjana Kaundinya ...
A Hitchhiker's Guide to Apache Kafka Geo-Replication with Sanjana Kaundinya ...
 

Plus de HostedbyConfluent

Build Real-time Machine Learning Apps on Generative AI with Kafka Streams
Build Real-time Machine Learning Apps on Generative AI with Kafka StreamsBuild Real-time Machine Learning Apps on Generative AI with Kafka Streams
Build Real-time Machine Learning Apps on Generative AI with Kafka StreamsHostedbyConfluent
 
When Only the Last Writer Wins We All Lose: Active-Active Geo-Replication in ...
When Only the Last Writer Wins We All Lose: Active-Active Geo-Replication in ...When Only the Last Writer Wins We All Lose: Active-Active Geo-Replication in ...
When Only the Last Writer Wins We All Lose: Active-Active Geo-Replication in ...HostedbyConfluent
 
Apache Kafka's Next-Gen Rebalance Protocol: Towards More Stable and Scalable ...
Apache Kafka's Next-Gen Rebalance Protocol: Towards More Stable and Scalable ...Apache Kafka's Next-Gen Rebalance Protocol: Towards More Stable and Scalable ...
Apache Kafka's Next-Gen Rebalance Protocol: Towards More Stable and Scalable ...HostedbyConfluent
 
Using Kafka at Scale - A Case Study of Micro Services Data Pipelines at Evern...
Using Kafka at Scale - A Case Study of Micro Services Data Pipelines at Evern...Using Kafka at Scale - A Case Study of Micro Services Data Pipelines at Evern...
Using Kafka at Scale - A Case Study of Micro Services Data Pipelines at Evern...HostedbyConfluent
 
Rule Based Asset Management Workflow Automation at Netflix
Rule Based Asset Management Workflow Automation at NetflixRule Based Asset Management Workflow Automation at Netflix
Rule Based Asset Management Workflow Automation at NetflixHostedbyConfluent
 
Scalable E-Commerce Data Pipelines with Kafka: Real-Time Analytics, Batch, ML...
Scalable E-Commerce Data Pipelines with Kafka: Real-Time Analytics, Batch, ML...Scalable E-Commerce Data Pipelines with Kafka: Real-Time Analytics, Batch, ML...
Scalable E-Commerce Data Pipelines with Kafka: Real-Time Analytics, Batch, ML...HostedbyConfluent
 
Indeed Flex: The Story of a Revolutionary Recruitment Platform
Indeed Flex: The Story of a Revolutionary Recruitment PlatformIndeed Flex: The Story of a Revolutionary Recruitment Platform
Indeed Flex: The Story of a Revolutionary Recruitment PlatformHostedbyConfluent
 
Forecasting Kafka Lag Issues with Machine Learning
Forecasting Kafka Lag Issues with Machine LearningForecasting Kafka Lag Issues with Machine Learning
Forecasting Kafka Lag Issues with Machine LearningHostedbyConfluent
 
Getting Under the Hood of Kafka Streams: Optimizing Storage Engines to Tune U...
Getting Under the Hood of Kafka Streams: Optimizing Storage Engines to Tune U...Getting Under the Hood of Kafka Streams: Optimizing Storage Engines to Tune U...
Getting Under the Hood of Kafka Streams: Optimizing Storage Engines to Tune U...HostedbyConfluent
 
Maximizing Real-Time Data Processing with Apache Kafka and InfluxDB: A Compre...
Maximizing Real-Time Data Processing with Apache Kafka and InfluxDB: A Compre...Maximizing Real-Time Data Processing with Apache Kafka and InfluxDB: A Compre...
Maximizing Real-Time Data Processing with Apache Kafka and InfluxDB: A Compre...HostedbyConfluent
 
Accelerating Path to Production for Generative AI-powered Applications
Accelerating Path to Production for Generative AI-powered ApplicationsAccelerating Path to Production for Generative AI-powered Applications
Accelerating Path to Production for Generative AI-powered ApplicationsHostedbyConfluent
 
Optimize Costs and Scale Your Streaming Applications with Virtually Unlimited...
Optimize Costs and Scale Your Streaming Applications with Virtually Unlimited...Optimize Costs and Scale Your Streaming Applications with Virtually Unlimited...
Optimize Costs and Scale Your Streaming Applications with Virtually Unlimited...HostedbyConfluent
 
Don’t Let Degradation Bring You Down: Automatically Detect & Remediate Degrad...
Don’t Let Degradation Bring You Down: Automatically Detect & Remediate Degrad...Don’t Let Degradation Bring You Down: Automatically Detect & Remediate Degrad...
Don’t Let Degradation Bring You Down: Automatically Detect & Remediate Degrad...HostedbyConfluent
 
Go Big or Go Home: Approaching Kafka Replication at Scale
Go Big or Go Home: Approaching Kafka Replication at ScaleGo Big or Go Home: Approaching Kafka Replication at Scale
Go Big or Go Home: Approaching Kafka Replication at ScaleHostedbyConfluent
 
What's in store? Part Deux; Creating Custom Queries with Kafka Streams IQv2
What's in store? Part Deux; Creating Custom Queries with Kafka Streams IQv2What's in store? Part Deux; Creating Custom Queries with Kafka Streams IQv2
What's in store? Part Deux; Creating Custom Queries with Kafka Streams IQv2HostedbyConfluent
 
A Trifecta of Real-Time Applications: Apache Kafka, Flink, and Druid
A Trifecta of Real-Time Applications: Apache Kafka, Flink, and DruidA Trifecta of Real-Time Applications: Apache Kafka, Flink, and Druid
A Trifecta of Real-Time Applications: Apache Kafka, Flink, and DruidHostedbyConfluent
 
From Raw Data to an Interactive Data App in an Hour: Powered by Snowpark Python
From Raw Data to an Interactive Data App in an Hour: Powered by Snowpark PythonFrom Raw Data to an Interactive Data App in an Hour: Powered by Snowpark Python
From Raw Data to an Interactive Data App in an Hour: Powered by Snowpark PythonHostedbyConfluent
 
Beyond Monoliths: Thrivent’s Lessons in Building a Modern Integration Archite...
Beyond Monoliths: Thrivent’s Lessons in Building a Modern Integration Archite...Beyond Monoliths: Thrivent’s Lessons in Building a Modern Integration Archite...
Beyond Monoliths: Thrivent’s Lessons in Building a Modern Integration Archite...HostedbyConfluent
 
Exactly-Once Semantics Revisited: Distributed Transactions across Flink and K...
Exactly-Once Semantics Revisited: Distributed Transactions across Flink and K...Exactly-Once Semantics Revisited: Distributed Transactions across Flink and K...
Exactly-Once Semantics Revisited: Distributed Transactions across Flink and K...HostedbyConfluent
 

Plus de HostedbyConfluent (20)

Build Real-time Machine Learning Apps on Generative AI with Kafka Streams
Build Real-time Machine Learning Apps on Generative AI with Kafka StreamsBuild Real-time Machine Learning Apps on Generative AI with Kafka Streams
Build Real-time Machine Learning Apps on Generative AI with Kafka Streams
 
When Only the Last Writer Wins We All Lose: Active-Active Geo-Replication in ...
When Only the Last Writer Wins We All Lose: Active-Active Geo-Replication in ...When Only the Last Writer Wins We All Lose: Active-Active Geo-Replication in ...
When Only the Last Writer Wins We All Lose: Active-Active Geo-Replication in ...
 
Apache Kafka's Next-Gen Rebalance Protocol: Towards More Stable and Scalable ...
Apache Kafka's Next-Gen Rebalance Protocol: Towards More Stable and Scalable ...Apache Kafka's Next-Gen Rebalance Protocol: Towards More Stable and Scalable ...
Apache Kafka's Next-Gen Rebalance Protocol: Towards More Stable and Scalable ...
 
Using Kafka at Scale - A Case Study of Micro Services Data Pipelines at Evern...
Using Kafka at Scale - A Case Study of Micro Services Data Pipelines at Evern...Using Kafka at Scale - A Case Study of Micro Services Data Pipelines at Evern...
Using Kafka at Scale - A Case Study of Micro Services Data Pipelines at Evern...
 
Rule Based Asset Management Workflow Automation at Netflix
Rule Based Asset Management Workflow Automation at NetflixRule Based Asset Management Workflow Automation at Netflix
Rule Based Asset Management Workflow Automation at Netflix
 
Scalable E-Commerce Data Pipelines with Kafka: Real-Time Analytics, Batch, ML...
Scalable E-Commerce Data Pipelines with Kafka: Real-Time Analytics, Batch, ML...Scalable E-Commerce Data Pipelines with Kafka: Real-Time Analytics, Batch, ML...
Scalable E-Commerce Data Pipelines with Kafka: Real-Time Analytics, Batch, ML...
 
Indeed Flex: The Story of a Revolutionary Recruitment Platform
Indeed Flex: The Story of a Revolutionary Recruitment PlatformIndeed Flex: The Story of a Revolutionary Recruitment Platform
Indeed Flex: The Story of a Revolutionary Recruitment Platform
 
Forecasting Kafka Lag Issues with Machine Learning
Forecasting Kafka Lag Issues with Machine LearningForecasting Kafka Lag Issues with Machine Learning
Forecasting Kafka Lag Issues with Machine Learning
 
Getting Under the Hood of Kafka Streams: Optimizing Storage Engines to Tune U...
Getting Under the Hood of Kafka Streams: Optimizing Storage Engines to Tune U...Getting Under the Hood of Kafka Streams: Optimizing Storage Engines to Tune U...
Getting Under the Hood of Kafka Streams: Optimizing Storage Engines to Tune U...
 
Maximizing Real-Time Data Processing with Apache Kafka and InfluxDB: A Compre...
Maximizing Real-Time Data Processing with Apache Kafka and InfluxDB: A Compre...Maximizing Real-Time Data Processing with Apache Kafka and InfluxDB: A Compre...
Maximizing Real-Time Data Processing with Apache Kafka and InfluxDB: A Compre...
 
Accelerating Path to Production for Generative AI-powered Applications
Accelerating Path to Production for Generative AI-powered ApplicationsAccelerating Path to Production for Generative AI-powered Applications
Accelerating Path to Production for Generative AI-powered Applications
 
Optimize Costs and Scale Your Streaming Applications with Virtually Unlimited...
Optimize Costs and Scale Your Streaming Applications with Virtually Unlimited...Optimize Costs and Scale Your Streaming Applications with Virtually Unlimited...
Optimize Costs and Scale Your Streaming Applications with Virtually Unlimited...
 
Don’t Let Degradation Bring You Down: Automatically Detect & Remediate Degrad...
Don’t Let Degradation Bring You Down: Automatically Detect & Remediate Degrad...Don’t Let Degradation Bring You Down: Automatically Detect & Remediate Degrad...
Don’t Let Degradation Bring You Down: Automatically Detect & Remediate Degrad...
 
Streaming is a Detail
Streaming is a DetailStreaming is a Detail
Streaming is a Detail
 
Go Big or Go Home: Approaching Kafka Replication at Scale
Go Big or Go Home: Approaching Kafka Replication at ScaleGo Big or Go Home: Approaching Kafka Replication at Scale
Go Big or Go Home: Approaching Kafka Replication at Scale
 
What's in store? Part Deux; Creating Custom Queries with Kafka Streams IQv2
What's in store? Part Deux; Creating Custom Queries with Kafka Streams IQv2What's in store? Part Deux; Creating Custom Queries with Kafka Streams IQv2
What's in store? Part Deux; Creating Custom Queries with Kafka Streams IQv2
 
A Trifecta of Real-Time Applications: Apache Kafka, Flink, and Druid
A Trifecta of Real-Time Applications: Apache Kafka, Flink, and DruidA Trifecta of Real-Time Applications: Apache Kafka, Flink, and Druid
A Trifecta of Real-Time Applications: Apache Kafka, Flink, and Druid
 
From Raw Data to an Interactive Data App in an Hour: Powered by Snowpark Python
From Raw Data to an Interactive Data App in an Hour: Powered by Snowpark PythonFrom Raw Data to an Interactive Data App in an Hour: Powered by Snowpark Python
From Raw Data to an Interactive Data App in an Hour: Powered by Snowpark Python
 
Beyond Monoliths: Thrivent’s Lessons in Building a Modern Integration Archite...
Beyond Monoliths: Thrivent’s Lessons in Building a Modern Integration Archite...Beyond Monoliths: Thrivent’s Lessons in Building a Modern Integration Archite...
Beyond Monoliths: Thrivent’s Lessons in Building a Modern Integration Archite...
 
Exactly-Once Semantics Revisited: Distributed Transactions across Flink and K...
Exactly-Once Semantics Revisited: Distributed Transactions across Flink and K...Exactly-Once Semantics Revisited: Distributed Transactions across Flink and K...
Exactly-Once Semantics Revisited: Distributed Transactions across Flink and K...
 

Dernier

Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXTarek Kalaji
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Brian Pichman
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?IES VE
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDELiveplex
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 

Dernier (20)

Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBX
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 
201610817 - edge part1
201610817 - edge part1201610817 - edge part1
201610817 - edge part1
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 

Bringing Kafka Without Zookeeper Into Production with Colin McCabe | Kafka Summit London 2022

  • 1. Bringing Kafka Without ZooKeeper Into Production Colin McCabe Principal Engineer Confluent
  • 2. About Me 2 I work on Apache Kafka at Confluent. Kafka Committer & PMC Member
  • 3. 3 ● KRaft Architecture ● Deploying KRaft ● Rolling KRaft Clusters ● Troubleshooting KRaft Clusters ● Upgrading from ZooKeeper ● Roadmap ● Q&A Table of Contents
  • 6. How Kafka Uses ZooKeeper 6 Stores most persistent metadata in ZooKeeper ● Topics ● Partitions ● Configurations ● Quotas ● ACLs Also uses ZooKeeper for coordinating cluster membership. /brokers/ids/0 /brokers/ids/1 /brokers/topics/foo /brokers/topics/foo/partitions/0/state /brokers/topics/foo/partitions/1/state /brokers/topics/foo/partitions/2/state ...
  • 8. Controller Startup with ZK 8 ● One broker wins the election in ZooKeeper to be the controller
  • 10. Controller Startup with ZK 10 ● UpdateMetadataRequest ● LeaderAndIsrRequest All partitions
  • 11. Problems with ZK-based Controller Startup 11 ● Have to load all metadata synchronously on startup: ○ O(num_partitions), O(num_brokers) ○ Controller is unavailable during this time ■ Cold start: cluster unavailable. ■ Controller restart: admin ops and ISR changes unavailable ● Have to send all metadata to all brokers on startup 3 minutes
  • 12. How KRaft Replaces ZooKeeper 12 ● Instead of ZooKeeper, we have an internal topic named __cluster_metadata ○ Replicated with KRaft ○ Single partition ○ The leader is the active controller ● KRaft: Raft for Kafka ○ The Raft protocol implemented in Kafka ■ Records committed by a majority of nodes ○ Self-managed quorum ■ Doesn’t rely on an external system for leader election
  • 13. Metadata Records 13 ● Binary records containing metadata ○ KRPC format ○ Auto-generated from protocol schemas ○ Can also be translated into JSON for human readability ● Two ways to evolve format ○ New record versions ○ Tagged fields ● Some records are deltas that apply changes to existing state { "type": "REGISTER_BROKER_RECORD", "version": 0, "data": { "brokerId": 1, "incarnationId": "P3UFsWoNR-erL9PK98YLsA", "brokerEpoch": 0, "endPoints": [ { "name": "PLAINTEXT", "host": "localhost", "port": 9092, "securityProtocol": 0 } ], "features": [], "rack": null } }
  • 14. Metadata as An Ordered Log 14 TopicRecord(name=foo, id=rtkInsMkQPiEBj6uz67rrQ) PartitionRecord(id=rtkInsMkQPiEBj6uz67rrQ, index=0, …) PartitionRecord(id=rtkInsMkQPiEBj6uz67rrQ, index=1, …) PartitionRecord(id=rtkInsMkQPiEBj6uz67rrQ, index=2, …) ConfigRecord(name=num.io.threads, value=...) RegisterBrokerRecord(id=4, endpoints=…, …) 10434 10435 10436 10437 10438 10439 2 1
  • 15. Controller Startup with KRaft 15 ● The blue nodes are designated KRaft controllers
  • 16. Controller Startup with KRaft 16 ● The KRaft controllers will elect a single leader ● The leader will have all previously committed records
  • 17. Controller Startup with KRaft 17 ● The newly elected KRaft controller is ready immediately ● Brokers fetch only the metadata they need ● New brokers and brokers that are behind fetch snapshots
  • 18. Controller Failover 18 ZK Mode KRaft Mode • Win controller election • Load all topics and partitions • Send LeaderAndIsr + UpdateMetadata to all brokers • Win KRaft election • Start handling requests from brokers
  • 19. Rolling Nodes 19 ZK Mode KRaft Mode Restarted node begins with no metadata. Must wait for full metadata update RPCs from controller. Restarted node consults its local metadata cache, transfers only what it doesn’t have, based on metadata offset.
  • 20. Calling an Admin API with ZK 20 ● External client connects to a random broker
  • 21. Calling an Admin API with ZK 21 ● Broker alters ZooKeeper ● Example: create znodes for new topic
  • 22. Calling an Admin API with ZK 22 ● Znode watch triggers ● Controller loads changes
  • 23. Calling an Admin API with ZK 23 ● Controller pushes out changes to brokers ● Incremental LeaderAndIsr ● Incremental UpdateMetadata
  • 24. ZK-based Programming Model 24 Controller Thread ZooKeeper Create /brokers/topics/foo Request Handler /brokers/topics Changed List /brokers/topics Compute changes List /brokers/topics/foo
  • 25. Problems 25 ● What if the controller fails to send an update to a specific broker? ○ Metadata divergence ○ No easy way to know what we have and what we don’t have ● Difficult programming model ○ Multiple writers to ZooKeeper ○ Can’t assume your cache is up-to-date! ● ZK is the bottleneck for all admin operations ○ Often 1 admin operation leads to many ZK operations ○ Blocking
  • 26. Calling an Admin API with KRaft 26 ● External client connects to a random broker
  • 27. Calling an Admin API with KRaft 27 ● Broker forwards the request to the active controller for processing
  • 28. Calling an Admin API with KRaft 28 ● Controller creates metadata records and persists them to __cluster_metadata
  • 29. Calling an Admin API with KRaft 29 ● Once the records have been committed to the metadata log, the active controller returns the result to the forwarding broker, which returns it to the external client
  • 30. Calling an Admin API with KRaft 30 ● Brokers are continuously fetching metadata from the active controller. ● They become aware of the admin changes by reading the new metadata records.
  • 31. KRaft-based Programming Model 31 Raft Quorum Controller CreateTopicsRequest Request Handler Write new records Last stable offset advances CreateTopicsResponse
  • 32. KRaft-based Programming Model: Pipelining 32 Raft Quorum Controller CreateTopicsRequest Request Handler Write new records Last stable offset advances CreateTopicsResponse
  • 33. Admin Operations in KRaft 33 ● Pull-based metadata propagation model can recover from RPC send failures easily ● Simpler programming model: single-writer ● Pipelining means that we can have multiple metadata operations in flight at once
  • 35. Getting KRaft 35 ● The latest Apache (and Confluent) releases support both ZK mode and KRaft mode ○ KRaft mode is supported from the same code base – not a branch ● KRaft mode is not yet recommended for production ○ In Apache 2.8, KRaft was in early access ○ As of Apache 3.0, KRaft is in preview ○ Definitely use the latest version if you are trying it out – many bugs have been fixed ● ZooKeeper mode will be supported for a while ○ I’ll discuss the release plans in more detail in the Roadmap section of this talk
  • 36. Creating a New KRaft Cluster ● First two steps are new! 36 $ kafka-storage.sh random-uuid MX1HbXq8TFymY2SYR1hGYg $ kafka-storage.sh format -c ./config/kraft/controller.properties --cluster-id MX1HbXq8TFymY2SYR1hGYg Formatting /tmp/kraft-controller-logs ● Generate a new random uuid with kafka-storage tool ● On each node, use the kafka-storage tool to format storage directories ● Start all nodes
  • 37. The Role of Formatting 37 ● In ZK mode, brokers auto-format storage directories if they are blank on startup. ● In KRaft mode, we must format each broker and controller before starting the nodes. ● Why not auto-format? ○ Avoid admin mistakes or filesystem issues that could lose data ■ If a volume is not mounted, the directory may appear empty ■ The same reasons why databases and filesystems require a formatting step. ○ We need a way to bootstrap security-related dynamic configurations that the quorum and the brokers need to function ■ SCRAM configurations ■ Dynamic SSL configurations ■ Metadata version
  • 38. Bootstrapping 38 ZK Mode KRaft Mode • Start up ZooKeeper cluster • Set SCRAM users, dynamic security configs, etc. in ZK • Start brokers • Generate cluster ID • Run format tool on all nodes, specifying cluster ID, SCRAM, security configs, metadata version, etc. • Start nodes
  • 39. KRaft Controllers ● A small pool of nodes that we designate ahead of time ○ Hot standbys for the active controller ○ Single log directory for __cluster_metadata ● Sizing is very similar to ZooKeeper ○ Typically 3 controller nodes ○ May use more to provide more resilience ○ Just like with ZK, need to keep a majority of nodes alive ● Can be configured by copying over broker configuration file ○ Set node.roles to controller ○ Set node.id 39
  • 40. 4 0 Combined Mode Controller processes in the same JVM as broker processes Shared IDs, shared JVMs, shared nodes Can have single JVM Kafka cluster Deploying KRaft Controllers Separate Mode Controller processes on separate nodes Different IDs, different JVMs, different nodes Better isolation, easier rolls Separate Mode with Kubernetes Bin-Packing Controllers in separate k8s pods Different IDs, different JVMs, maybe shared nodes
  • 41. Some Considerations for Deploying Controllers 41 ● Isolation ● Avoid co-location in busy clusters ● Try to avoid situations where traffic from outside disrupts internals ● High Availability ● Try to not to have multiple controllers in a single failure domain ● In clusters which span 3 availability zones, put a controller in each zone Controller
  • 42. New Controller Endpoint 42 Broker Controller EXTERNAL INTERBROKER CONTROLLER ● Only exposed on controllers ● Not an advertised listener or broker listener ● Should not be accessible externally
  • 43. Configuring KRaft Clusters ● node.id ○ Replaces broker.id on both brokers and controllers. ● process.roles ○ broker ○ controller ○ broker,controller ● controller.quorum.voters ○ 1@controller1.9093,2@controller2:9093,3@controller3:9093 ○ Statically configured ● controller.listener.names ○ Tell brokers and other controllers how to connect to the controller ○ These are not advertised listeners and cannot appear in “advertised.listeners” on the broker ○ On the controller they appear in “listeners” 43
  • 45. Rolling KRaft Controllers 45 ● Much lighter roll process ○ No cluster-destabilizing full metadata updates to send ● KRaft controllers can be rolled very quickly compared to brokers ○ They do not manage lots of data directories ○ Make sure that monitoring software can keep up ● KRaft controller software can be upgraded on a separate schedule ○ Forwards and backwards compatibility with alternate broker versions
  • 46. Monitoring Rolls 46 ● ZK Mode ○ Under-replicated partitions ○ ActiveControllerCount ○ LeaderAndIsr request time ○ UpdateMetadata request time ○ ZooKeeperExpiresPerSec ● KRaft Mode ○ Under-replicated partitions ■ Shows broker health ○ ActiveControllerCount ○ MetadataOffset ○ SnapshotLag ○ SnapshotSizeBytes ○ MetadataCommitRate ○ MetadataCommitLatency ○ Current Raft state (follower, leader, candidate, observer) ■ Shows controller health ○ FencedBrokerCount ○ ActiveBrokerCount
  • 47. Creating New Partitions During Cluster Rolls in ZK 47 ● In ZK mode, controller “forgets about” brokers that are temporary down during a roll ● In a 3-node cluster, if one node is rolling, we don’t have enough nodes to create a partition with 3x replication. ● Must run with 4 nodes, even if we don’t need all 4. Rolling! Replica 1 Replica 2 Replica 3 New Partition
  • 48. 48 ● In KRaft mode, the controller remembers brokers that are temporary down during a roll ● In KRaft mode, those brokers enter “fenced state” ○ Can place new replicas on these nodes if needed ○ Brokers will be unfenced once they come up ● Many small clusters can now use 3 nodes rather than 4. Rolling! Replica 1 Replica 2 Replica 3 New Partition Creating New Partitions During Cluster Rolls in KRaft
  • 49. Handling Version Skew 49 ● During the roll, old and new software versions must co-exist ● Older brokers can’t understand newer APIs and protocols ● inter.broker.protocol solves this in the ZK world ○ Controls what RPC protocols brokers use when communicating with each other ○ Controls what features brokers support ● Examples ○ Whether to use AlterPartitions for ISRs ○ Whether to use topic IDs ○ Whether to use the new version of some API (ListOffsets, FetchRequest, etc.) KAFKA_2_4_IV0, KAFKA_2_4_IV1, KAFKA_2_5_IV0, KAFKA_2_6_IV0, KAFKA_2_7_IV0, KAFKA_2_7_IV1, KAFKA_2_7_IV2, KAFKA_2_8_IV0, KAFKA_2_8_IV1, KAFKA_3_0_IV0, KAFKA_3_0_IV1, KAFKA_3_1_IV0, KAFKA_3_2_IV0
  • 50. Problems with Inter Broker Protocol in ZK 50 ● inter.broker.protocol version must be manually configured ○ If it is left out of the configuration file, it defaults to the latest version, which is probably not what the user wants ● Because inter.broker.protocol is statically configured, it can’t be changed without restarting all the nodes ○ To upgrade to a new version AND get the new features requires a “double roll” in ZK mode ● No downgrade support ○ It is safe to downgrade between some pairs of versions, but not others.
  • 51. Introducing metadata.version 51 ● In KRaft mode, inter.broker.protocol is replaced by metadata.version ● Each new inter.broker.protocol version has a corresponding metadata.version ● metadata.version is dynamically configured ○ Does not require a roll to change ○ It is changed by invoking a controller API. ○ “Guard rails” ■ The controller will refuse to do the upgrade if some brokers have not been rolled ● Supports downgrade!
  • 52. Downgrading metadata.version 52 ● Like upgrade, downgrade can be done dynamically from the command line ● Two kinds of downgrades ○ Safe: no metadata will be lost by the downgrade ○ Unsafe: some metadata may be cleared during the downgrade ● Unsafe downgrades require the operator to provide an override flag
  • 54. Troubleshooting KRaft Clusters 54 ● In KRaft, __cluster_metadata replaces ZooKeeper as the store of record ● It’s often helpful to examine the metadata log to see what happened ● All batches come with timestamps ● Offsets are uniform across the cluster ○ A specific record will have the same offset on each controller and broker ● Several tools for looking at metadata logs ○ kafka-dump-log ○ kafka-metadata-shell
  • 55. kafka-dump-log 55 $ ./bin/kafka-dump-log.sh --cluster-metadata-decoder --files /tmp/logs/__cluster_metadata-0/00000000000000000000.log Dumping /tmp/logs/__cluster_metadata-0/00000000000000000000.log Starting offset: 0 baseOffset: 0 lastOffset: 0 count: 1 baseSequence: -1 [...] | offset: 0 CreateTime: 1650857270775 keySize: 4 valueSize: 19 sequence: -1 headerKeys: [] controlType: LEADER_CHANGE(2) baseOffset: 1 lastOffset: 1 count: 1 baseSequence: -1 [...]{"type":"REGISTER_BROKER_RECORD","version":0,"data":{"...
  • 56. Interactive Metadata Shell 56 ● Replaces zookeeper-shell in KRaft clusters ● Data sources ○ Snapshot ○ Running controller cluster ● Reads __cluster_metadata log entries into memory ● Constructs a “virtual filesystem” with the cluster’s information ● Commands available: ls, pwd, cd, find, etc.
  • 57. kafka-metadata-shell 57 $ ./bin/kafka-metadata-shell.sh --snapshot /tmp/logs/__cluster_metadata-0/00000000000000000000.log Loading... Starting... [ Kafka Metadata Shell ] >> ls brokers configs local metadataQuorum topicIds topics >> cat /brokers/1/registration RegisterBrokerRecord(brokerId=1, incarnationId=tHo3Z8dYSuONV5hA82BVug, brokerEpoch=0, endPoints=[BrokerEndpoint(name='PLAINTEXT', host='localhost', port=9092, securityProtocol=0)], features=[], rack=null, fenced=true)
  • 58. Monitoring the Quorum 58 ● Metrics ○ MetadataOffset ○ SnapshotLag ○ SnapshotSizeBytes ○ MetadataCommitRate ○ MetadataCommitLatency ○ Current Raft state (follower, leader, candidate, observer) ■ Important for controller health ● DescribeQuorum RPC ○ Leader ID ○ Leader epoch ○ High water mark ○ Current voters ■ Controllers ○ Current observers ■ Brokers ■ Possible metadata shell instances, etc. ○ Log end offset of all followers
  • 60. Upgrading Clusters from ZK Mode 60 ZK mode ZK mode ZK mode ● Initially: all metadata in ZooKeeper. ● All brokers in ZK mode.
  • 61. Upgrading Clusters from ZK Mode 61 61 ZK mode ZK mode ZK mode KRaft controller ● We add a quorum of new KRaft controllers to the cluster KRaft controller KRaft controller
  • 62. Upgrading Clusters from ZK Mode 62 62 ZK mode ZK mode ZK mode ● The KRaft controllers elect a leader from their ranks and force that leader to be the cluster leader in ZK. KRaft controller KRaft controller KRaft controller
  • 63. Upgrading Clusters from ZK Mode 63 63 ZK mode ZK mode ZK mode ● The KRaft controller loads all metadata from ZK, and sends out the appropriate LeaderAndIsr, UMRs. ● Future metadata changes go into the KRaft quorum. KRaft controller KRaft controller KRaft controller metadata
  • 64. Upgrading Clusters from ZK Mode 64 64 ZK mode ZK mode KRaft mode ● Now brokers can be rolled one by one. ● KRaft controller will still send LeaderAndIsr, etc. to un-upgraded brokers. KRaft controller KRaft controller KRaft controller
  • 65. Upgrading Clusters from ZK Mode 65 65 ZK mode KRaft mode KRaft mode ● This relies on brokers never directly communicating with ZK (except to register themselves) ● KIP-591 Forwarding KRaft controller KRaft controller KRaft controller
  • 66. Upgrading Clusters from ZK Mode 66 66 KRaft mode KRaft mode KRaft mode ● Once all brokers have been rolled, ZK can be removed. KRaft controller KRaft controller KRaft controller
  • 68. KRaft Project Phases 1. Design 2. Initial Implementation 3. Available for testing 4. Available for production 5. Bridge release 6. ZK removed 68
  • 69. KRaft Project Phases 69 1. Design 2. Initial Implementation 3. Available for testing 4. Available for production 5. Bridge release 6. ZK removed
  • 70. Design Phase 70 ● During the design phase, we had many upstream discussions about the goals of the KRaft project (called the KIP-500 project at the time) and how we would evolve the project to meet them. ● September 2019: vote for “KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum” passes ● June 2020: vote for “KIP-591: Redirect Zookeeper Mutation Protocols to The Controller” passes (some revisions will be made later) ● August 2020: vote for “KIP-595: A Raft Protocol for the Quorum” passes ● September 2020: vote for “KIP-631: The Quorum-Based Kafka Controller” passes ● October 2020: vote for “KIP-630: Kafka Raft Snapshots” passes ● Several other minor KIPs filled gaps in our admin APIs which were being filled by direct ZK access
  • 71. Initial Implementation Phase 71 ● During this phase, we put the ideas from the design phase into practice. ● The implementation was pretty unstable during this time period – many big changes and fixes were happening all the time. ● There were many minor changes to remove ZK dependencies as well. ● Late 2020: KIP-500 work started in a branch. ● Early 2021: all code merged to trunk ○ New controller code ○ New self-managed Raft quorum implementation ● April 2021: Apache 2.8 released ○ First Apache release with KRaft ○ KRaft in Early Access ● September 2021: Apache 3.0 release ○ First release with KRaft snapshot support ○ Added reassignment and EOS support as well ○ Preview
  • 72. Testing Phase 72 ● This is where we are now. ● KRaft is in preview mode ○ Not recommended for production ○ Available for testing ● January 2022: Apache Kafka 3.1 released ○ Many bug fixes, new tests ○ Some new KRaft metrics ● May 2022: Apache Kafka 3.2 released (soon) ○ New KRaft-based authorizer
  • 73. Production Phase 73 ● Coming soon! ● During this phase, we will recommend KRaft for production use. ○ Also known as “going GA” ● There may still be some feature gaps ○ SCRAM ○ JBOD support ○ Delegation token support ● Most importantly, we will not have support for upgrading from ZK yet in this phase. ○ Therefore, in this phase, KRaft will be useful for new clusters, but not for existing ones
  • 74. Bridge Release 74 ● A bridge release is a release that supports upgrading from ZK mode to KRaft mode. ○ As described in the section on upgrading from ZK, this requires brokers to channel their admin changes through the controller, rather than mutating ZK directly. ● We will make multiple bridge releases ● We will work hard to close all the remaining feature gaps during this phase, so that there are no longer any users stuck on ZK.
  • 75. Last Phase: Removing ZK Support 75 ● Eventually, we will want to remove ZK support so that we no longer have to maintain two controller implementations ● Before we can remove ZK support, we have to deprecate it, and then make a new major release ● Timeline for full removal is TBD