"Using just the basic tools that come with Apache Kafka makes operating your own cluster hard work and time consuming. Tasks like moving partitions across brokers, scaling up clusters, or decommissioning nodes are harder than they need to be. Cruise Control is an open source tool to make managing and operating Kafka much easier. It helps you perform these common tasks, but it’s also capable of automatically healing your cluster if brokers crash. With its UI you can easily do basic monitoring too. Getting the most out of Cruise Control will enable you to manage Kafka clusters effectively across a range of Kafka infrastructures. We will see demos to showcase how to apply the powers of Cruise Control: rebalance load on the cluster by a simple command, or make your brokers ready for maintenance. We will also learn about the self-healing feature that can repair your cluster if nodes crash and restore offline replicas to ensure your Kafka cluster is fully functioning. Join this session if you want to learn how to use Cruise Control to automate Kafka cluster management and make your team’s life easier."

1. Introduction to Kafka Cruise Control
Viktor Somogyi-Vass

2. Viktor Somogyi-Vass
viktor.somogyi@cloudera.com

3. 3
© 2023 Cloudera, Inc. All rights reserved.
Operating Kafka clusters are hard.

4. 4
© 2023 Cloudera, Inc. All rights reserved.

5. 5
© 2023 Cloudera, Inc. All rights reserved.
1
Surprises
There can be data center
outages, maintenance can
go wrong and upgrades can
go wrong.

6. 6
© 2023 Cloudera, Inc. All rights reserved.
1
Surprises
There can be data center
outages, maintenance can
go wrong and upgrades can
go wrong.
2
Scaling
New brokers won’t get
populated. What to put on
them? Where to put
partitions when
decommissioning brokers?

7. 7
© 2023 Cloudera, Inc. All rights reserved.
1
Surprises
There can be data center
outages, maintenance can
go wrong and upgrades can
go wrong.
3
Performance
Brokers can be overloaded
or even underloaded.
Rebalances can ruin your
cluster.
2
Scaling
New brokers won’t get
populated. What to put on
them? Where to put
partitions when
decommissioning brokers?

8. 8
© 2023 Cloudera, Inc. All rights reserved.
How do we solve these problems?

9. 9
© 2023 Cloudera, Inc. All rights reserved.

10. 10
© 2023 Cloudera, Inc. All rights reserved.
SOME HIGHLIGHTS

11. 11
© 2023 Cloudera, Inc. All rights reserved.
SOME HIGHLIGHTS
SELF-HEALING
By rebalances
Detect—failures and anomalies are
detected automatically
Rebalance—healing will be done via
rebalancing partitions to healthy
brokers

12. 12
© 2023 Cloudera, Inc. All rights reserved.
SOME HIGHLIGHTS
SELF-HEALING
By rebalances
ADMIN
Scaling and maintenance
Detect—failures and anomalies are
detected automatically
Rebalance—healing will be done via
rebalancing partitions to healthy
brokers
Upscaling—brokers can be added to
Cruise Control and populated with
partitions
Downscaling—brokers can be
removed from the cluster and
partitions will be reassigned
optimally

13. 13
© 2023 Cloudera, Inc. All rights reserved.
SOME HIGHLIGHTS
SELF-HEALING
By rebalances
ADMIN
Scaling and maintenance
MONITORING
Even load everywhere
Detect—failures and anomalies are
detected automatically
Rebalance—healing will be done via
rebalancing partitions to healthy
brokers
Upscaling—brokers can be added to
Cruise Control and populated with
partitions
Downscaling—brokers can be
removed from the cluster and
partitions will be reassigned
optimally
Metrics—are collected periodically
to ensure an up-to-date view of the
cluster
Estimation—partitions level
utilization is estimated with a linear
model

14. 14
© 2023 Cloudera, Inc. All rights reserved.
How does it work?

15. 15
© 2023 Cloudera, Inc. All rights reserved.
ARCHITECTURE

16. 16
© 2023 Cloudera, Inc. All rights reserved.
ARCHITECTURE

17. 17
© 2023 Cloudera, Inc. All rights reserved.
ARCHITECTURE

18. 18
© 2023 Cloudera, Inc. All rights reserved.
ARCHITECTURE

19. 19
© 2023 Cloudera, Inc. All rights reserved.
ARCHITECTURE

20. 20
© 2023 Cloudera, Inc. All rights reserved.
ARCHITECTURE

21. 21
© 2023 Cloudera, Inc. All rights reserved.
THE LOAD MONITOR

22. 22
© 2023 Cloudera, Inc. All rights reserved.
THE LOAD MONITOR

23. 23
© 2023 Cloudera, Inc. All rights reserved.
THE LOAD MONITOR

24. 24
© 2023 Cloudera, Inc. All rights reserved.
THE LOAD MONITOR

25. 25
© 2023 Cloudera, Inc. All rights reserved.
THE LOAD MONITOR

26. 26
© 2023 Cloudera, Inc. All rights reserved.
THE LOAD MONITOR

27. 27
© 2023 Cloudera, Inc. All rights reserved.
THE ANALYZER
The brain

28. 28
© 2023 Cloudera, Inc. All rights reserved.
THE ANALYZER
The brain
Goals
Define the
characteristics of an
optimal aspect

29. 29
© 2023 Cloudera, Inc. All rights reserved.
THE ANALYZER
The brain
Goals
Define the
characteristics of an
optimal aspect
Heuristic model
An iterative model
defines how it
reaches the optimal
load

30. 30
© 2023 Cloudera, Inc. All rights reserved.
THE ANALYZER
The brain
For each goal g in the goal list ordered by priority {
For each broker b {
while b does not meet g’s requirement {
For each replica r on b sorted by the resource utilization density {
Move r (or the leadership of r) to another eligible broker b’ so b’
still satisfies g and all the satisfied goals
Finish the optimization for b once g is satisfied.
}
Fail the optimization if g is a hard goal and is not satisfied for b
}
}
Add g to the satisfied goals
}
Goals
Define the
characteristics of an
optimal aspect
Heuristic model
An iterative model
defines how it
reaches the optimal
load

31. 31
© 2023 Cloudera, Inc. All rights reserved.
THE ANOMALY DETECTOR

32. 32
© 2023 Cloudera, Inc. All rights reserved.
THE ANOMALY DETECTOR
Broker failure
A non-empty broker
crashes. Results in
under-replication.

33. 33
© 2023 Cloudera, Inc. All rights reserved.
THE ANOMALY DETECTOR
Disk failure
A non-empty disk
dies. Some replicas
might get lost.
Broker failure
A non-empty broker
crashes. Results in
under-replication.

34. 34
© 2023 Cloudera, Inc. All rights reserved.
THE ANOMALY DETECTOR
Goal violation
An optimization goal
is violated.
Unbalanced cluster.
Disk failure
A non-empty disk
dies. Some replicas
might get lost.
Broker failure
A non-empty broker
crashes. Results in
under-replication.

35. 35
© 2023 Cloudera, Inc. All rights reserved.
THE ANOMALY DETECTOR
Goal violation
An optimization goal
is violated.
Unbalanced cluster.
Disk failure
A non-empty disk
dies. Some replicas
might get lost.
Broker failure
A non-empty broker
crashes. Results in
under-replication.
Fixable

36. 36
© 2023 Cloudera, Inc. All rights reserved.
THE ANOMALY DETECTOR
Goal violation
An optimization goal
is violated.
Unbalanced cluster.
Disk failure
A non-empty disk
dies. Some replicas
might get lost.
Metric anomaly
One of the collected
metric observes an
unexpected value.
Broker failure
A non-empty broker
crashes. Results in
under-replication.
Fixable

37. 37
© 2023 Cloudera, Inc. All rights reserved.
THE ANOMALY DETECTOR
Goal violation
An optimization goal
is violated.
Unbalanced cluster.
Disk failure
A non-empty disk
dies. Some replicas
might get lost.
Metric anomaly
One of the collected
metric observes an
unexpected value.
Topic anomaly
One or more topics
violate user-defined
properties.
Broker failure
A non-empty broker
crashes. Results in
under-replication.
Fixable

38. 38
© 2023 Cloudera, Inc. All rights reserved.
THE ANOMALY DETECTOR
Goal violation
An optimization goal
is violated.
Unbalanced cluster.
Disk failure
A non-empty disk
dies. Some replicas
might get lost.
Metric anomaly
One of the collected
metric observes an
unexpected value.
Topic anomaly
One or more topics
violate user-defined
properties.
Broker failure
A non-empty broker
crashes. Results in
under-replication.
Fixable Not
fixable

39. 39
© 2023 Cloudera, Inc. All rights reserved.
EXECUTOR

40. 40
© 2023 Cloudera, Inc. All rights reserved.
EXECUTOR
Long running
Depending on the size
of the executed task.

41. 41
© 2023 Cloudera, Inc. All rights reserved.
EXECUTOR
Interruptible
At any point it can be
stopped safely. No
“undo” functionality.
Long running
Depending on the size
of the executed task.

42. 42
© 2023 Cloudera, Inc. All rights reserved.
EXECUTOR
Reassignment
Optimization is done
by reassigning
partitions between
disks, brokers and
adjusting leadership
Interruptible
At any point it can be
stopped safely. No
“undo” functionality.
Long running
Depending on the size
of the executed task.

43. 43
© 2023 Cloudera, Inc. All rights reserved.
EXECUTOR
Reassignment
Optimization is done
by reassigning
partitions between
disks, brokers and
adjusting leadership
Concurrent
Reassignment is
concurrent, limits are
set either manually or
by the concurrency
adjuster
Interruptible
At any point it can be
stopped safely. No
“undo” functionality.
Long running
Depending on the size
of the executed task.

44. 44
© 2023 Cloudera, Inc. All rights reserved.
So how do we use it?

46. 46
© 2023 Cloudera, Inc. All rights reserved.
What else can it do?

47. 47
© 2023 Cloudera, Inc. All rights reserved.
SLOW BROKER FINDER

48. 48
© 2023 Cloudera, Inc. All rights reserved.
SLOW BROKER FINDER
DETECTION
With metrics
Log flush—broker log flush 99.9th
percentile is used, both absolute
and relative to incoming traffic
Scoring—brokers have score, every
anomaly increases it and every
normal behavior decreases it

49. 49
© 2023 Cloudera, Inc. All rights reserved.
SLOW BROKER FINDER
DETECTION
With metrics
REMEDIATION
Demoting and rebalance
Log flush—broker log flush 99.9th
percentile is used, both absolute
and relative to incoming traffic
Scoring—brokers have score, every
anomaly increases it and every
normal behavior decreases it
Demoting—brokers will be demoted
as a first attempt
Removal—slow brokers can be
removed from the cluster if the
problem persists after demotion

50. 50
© 2023 Cloudera, Inc. All rights reserved.
PROVISIONER

51. 51
© 2023 Cloudera, Inc. All rights reserved.
PROVISIONER
RIGHTSIZING
With API
Underprovisioned—the cluster lacks
certain kind of resources
Overprovisioned—by removing
resources and rebalancing it can
achieve some cost saving

52. 52
© 2023 Cloudera, Inc. All rights reserved.
PROVISIONER
RIGHTSIZING
With API
Underprovisioned—the cluster lacks
certain kind of resources
Overprovisioned—by removing
resources and rebalancing it can
achieve some cost saving
RECOMMENDATION
With goals
Resource—a goal can request a
resource, like broker, disk to be
added or removed
Constraints—every goal can give
constraints, e.g. racks for which
brokers should not be added

53. 53
© 2023 Cloudera, Inc. All rights reserved.
PROVISIONER
IMPLEMENTATION
Provider dependent
API—there is an API that can be
used as a basis for your
implementation
Providers—for all providers like AWS
or Azure you need to add glue code
to acquire or release resources
RIGHTSIZING
With API
Underprovisioned—the cluster lacks
certain kind of resources
Overprovisioned—by removing
resources and rebalancing it can
achieve some cost saving
RECOMMENDATION
With goals
Resource—a goal can request a
resource, like broker, disk to be
added or removed
Constraints—every goal can give
constraints, e.g. racks for which
brokers should not be added

54. 54
© 2023 Cloudera, Inc. All rights reserved.
CONCURRENCY ADJUSTER
Inter-broker replica reassignment limits aren’t easy

55. 55
© 2023 Cloudera, Inc. All rights reserved.
CONCURRENCY ADJUSTER
MANUAL LIMITS
Challenging
Destructive—if the manually set
limit is too high, then it can be fast
but also overwhelm the cluster
Slow—If the limit is too small then
rebalances can last for a long time
Inter-broker replica reassignment limits aren’t easy

56. 56
© 2023 Cloudera, Inc. All rights reserved.
CONCURRENCY ADJUSTER
MANUAL LIMITS
Challenging
Destructive—if the manually set
limit is too high, then it can be fast
but also overwhelm the cluster
Slow—If the limit is too small then
rebalances can last for a long time
FEEDBACK LOOP
Adaptive limit
Metrics—metrics are collected to
assess whether rebalance
concurrency needs adjustments
Limits—if metrics are over a
threshold then concurrency will
adjust with AIMD algorithm
Inter-broker replica reassignment limits aren’t easy

57. 57
© 2023 Cloudera, Inc. All rights reserved.
CONCURRENCY ADJUSTER
RESULTS
AIMD
Automated—no need for constant
oversight for setting concurrency in
a fluctuating traffic environment
Fast—rebalances will complete
faster due to the optimal limit
Stable—clients won’t experience any
side effects caused by rebalances
MANUAL LIMITS
Challenging
Destructive—if the manually set
limit is too high, then it can be fast
but also overwhelm the cluster
Slow—If the limit is too small then
rebalances can last for a long time
FEEDBACK LOOP
Adaptive limit
Metrics—metrics are collected to
assess whether rebalance
concurrency needs adjustments
Limits—if metrics are over a
threshold then concurrency will
adjust with AIMD algorithm
Inter-broker replica reassignment limits aren’t easy

58. 58
© 2023 Cloudera, Inc. All rights reserved.
Security matters

59. 59
© 2023 Cloudera, Inc. All rights reserved.
SECURITY IN CRUISE CONTROL

60. 60
© 2023 Cloudera, Inc. All rights reserved.
SECURITY IN CRUISE CONTROL
Basic Auth
The simple basic
authentication that
sends username and
password

61. 61
© 2023 Cloudera, Inc. All rights reserved.
SECURITY IN CRUISE CONTROL
SPNEGO
Kerberos over HTTP.
Tokens are
negotiated via the
SPNEGO protocol.
Basic Auth
The simple basic
authentication that
sends username and
password

62. 62
© 2023 Cloudera, Inc. All rights reserved.
SECURITY IN CRUISE CONTROL
SPNEGO
Kerberos over HTTP.
Tokens are
negotiated via the
SPNEGO protocol.
Basic Auth
The simple basic
authentication that
sends username and
password
Trusted Proxy
If a process acts on
behalf of a user, the
doAs header will
forward the
username.

63. 63
© 2023 Cloudera, Inc. All rights reserved.
SECURITY IN CRUISE CONTROL
SPNEGO
Kerberos over HTTP.
Tokens are
negotiated via the
SPNEGO protocol.
Basic Auth
The simple basic
authentication that
sends username and
password
Trusted Proxy
If a process acts on
behalf of a user, the
doAs header will
forward the
username.
TLS/HTTPS
Communication is
secured via HTTPS to
the server and TLS to
Zookeeper or Kafka

64. 64
© 2023 Cloudera, Inc. All rights reserved.
SECURITY IN CRUISE CONTROL
SPNEGO
Kerberos over HTTP.
Tokens are
negotiated via the
SPNEGO protocol.
Basic Auth
The simple basic
authentication that
sends username and
password
Trusted Proxy
If a process acts on
behalf of a user, the
doAs query param
will forward the
username.
TLS/HTTPS
Communication is
secured via HTTPS to
the server and TLS to
Zookeeper or Kafka
Authorization
Simple model with
viewer, user and
admin roles.

65. THANK YOU