Confluent Platform 6.0 and Project Metamorphosis complete the event streaming platform by providing elastic scalability, infinite storage, global access, and transforming Kafka. Key features include self-balancing clusters and dynamic scaling on Confluent Cloud, tiered storage and infinite retention on the platform, and cluster linking to simplify hybrid and multi-cloud deployments. These new capabilities help remove limitations on scale, storage, and deployment that traditionally challenged Kafka applications.

1. How Confluent Completes the
Event Streaming Platform
With Confluent Platform 6.0 and Project Metamorphosis
@DanRosanova
@AddisonHuddy

2. As our roadmap may change in the future, the features
referred to herein may change, may not be delivered
on time or may not be delivered at all.
This information is not a commitment to deliver any
functionality and customers should make their
purchasing decisions based upon features that are
currently available.

3. Hadoop ... Device
Logs ... App ...MicroserviceMainframes
Data
Warehouse Splunk ...
Data Stores Logs 3rd Party Apps Custom Apps / Microservices
Real-time
Inventory
Real-time Fraud
Detection
Real-time
Customer 360
Machine
Learning
Models
Real-time Data
Transformation
...
Contextual Event-Driven Applications
Universal Event Pipeline
CONFLUENT

4. Elastic
Confluent Cloud
● Self-serve clusters
● GBps+ scale
Confluent Platform 6.0
● Self-Balancing
Clusters
Cost-effective
Confluent Cloud
● 99.5% Basic SLA
● New connectors
Confluent Platform 6.0
● Admin REST API
Infinite
Confluent Cloud
● Infinite retention
● Schema registry
Confluent Platform 6.0
● Tiered Storage
● ksqlDB
Transforming Kafka with Confluent Platform 6.0
and Project Metamorphosis
Global
Confluent Cloud
● Cluster linking
● New cloud regions
Confluent Platform 6.0
● Cluster Linking
(preview)

5. Elastic Scalability

6. Confluent Cloud
Milliseconds Minutes
Basic, Standard [0-100Mbps]
Do Nothing
Elastic Scaling w/ fully-managed Confluent Cloud
*Even in public clouds provider quotas for VMs, disks, security groups can be encountered causing delays. Confluent has these limits raised already.
Dedicated [Mbps - Gbps]
1 Click—Select CKU from drop
down in cluster management UI and
click Apply Changes
Other Kafka Services
Days - Weeks
Determine how much capacity is needed
Procure capacity*
Configure new brokers
a. Disks b. OS c. Network d. Kafka (application)
Identify partitions on specific brokers to
rebalance & topics they are part of
For each Topic: migrate partitions
a. Increase ISR +1 b. Wait for new replica to sync
c. Failover master d. Reduce ISR -1 e. Delete old replica

7. Dynamic Scaling with Confluent Platform
T3, P0
T1, P0
T2, P1
T3, P0
T1, P1
T2, P0
T3, P1
Broker 1 Broker 2 New: Broker 3
T1, P0 T2, P1T1, P1
T2, P0T3, P1
Broker 1 Broker 2 Broker 3
Scaling with OSS Kafka
T3, P0T1, P0 T2, P1
Broker 1 Broker 2 New: Broker 3
Object Store
Broker 1 Broker 2 Broker 3
Object Store
T3, P1T1, P1 T2, P0
T3, P0T1, P0 T2, P1T3, P1 T1, P1 T2, P0
Scaling with Tiered Storage + Self-
balancing Clusters on Kubernetes
Rebalance in Hours... Rebalance in Seconds...

8. Infinite Storage

9. 9
Historically, the
way to scale
storage in Kafka is
to scale brokers.
Broker storage can be expensive and is
tightly bound to compute:
Brokers

10. Confluent Cloud
Infinite Retention
Scale up storage without
adding brokers by
removing storage limits
on topics to enable
infinite data retention in
Kafka
Cluster
Broker 1 Broker 2 Broker 2
Topic 1
Topic 2
Topic 3
Topic 2
Topic 3
Topic 2
Topic 3
Topic 1Topic 1

11. Confluent Platform
Tiered Storage
Offload old events to
cheaper, 3rd party object
storage to reduce costs
and enable infinite data
retention in Kafka
11
Tiered Storage enables Kafka to
recognize two layers of storage:
Brokers
Object Storage
Offload old data
to object store

12. Global Access

13. 13
Cluster Linking
(preview)
Cluster Linking
simplifies hybrid-cloud
and multi-cloud
deployments for Kafka
Connecting independent clusters or
migrating clusters presents major
challenges:
1. Requires deploying a separate Connect
cluster, adding architectural complexity
and operational burden
1. Offsets are NOT preserved, so
consumers are at risk of skipping or
rereading messages
1
2

14. 14
Cluster Linking
(preview)
Cluster Linking
simplifies hybrid-cloud
and multi-cloud
deployments for Kafka
Cluster Linking requires no additional
infrastructure and preserves offsets:

16. Confluent Developer
developer.confluent.io
Learn Kafka.
Start building with
Apache Kafka at
Confluent Developer.

17. Thank you!
@DanRosanova
drosanova@confluent.io
@AddisonHuddy
addison@confluent.io
cnfl.io/meetups cnfl.io/slackcnfl.io/blog