Leo Yuen, Confluent, Senior Partner Solutions Engineer Confluent Platform 是一个企业级的平台，创新的功能与技术让Kafka变得更完善， 不仅加速应用程序开发和连通性，通过流处理实现转换也简化了大规模的企业运营并满足严格的架构要求。本次分享主要介绍两个新技术，包括分层存储与集群链接。 https://www.meetup.com/Beijing-Kafka-Meetup/events/281812695/

1. Confluent流处理平台之Kafka新技术分享
Leo Yuen
Partner Solutions Engineer

2. 动态性能和弹性
Dynamic Performance & Elasticity
Self-Balancing Clusters | Tiered Storage
Flexible DevOps Automation
Operator | Ansible
GUI-driven Mgmt & Monitoring
Control Center | Proactive Support
Event Streaming Database
ksqlDB
Rich Pre-built Ecosystem
Connectors | Hub | Schema Registry
Multi-language Development
Non-Java Clients | REST Proxy
Admin REST APIs
Global Resilience
Multi-region Clusters | Replicator |
Cluster Linking
Data Compatibility
Schema Registry | Schema Validation
Enterprise-grade Security
RBAC | Secrets | Audit Logs
TCO / ROI
Revenue / Cost / Risk Impact
Complete Engagement Model
Efficient
Operations at Scale
Unrestricted
Developer Productivity
Production-stage
Prerequisites
Partnership for
Business Success
Freedom of Choice
Committer-driven Expertise
Open Source | Community licensed
Fully Managed Cloud Service
Self-managed Software
Training Partners
Enterprise
Support
Professional
Services
ARCHITECT
OPERATOR
DEVELOPER BUYER
Apache Kafka

3. 3
分层存储
Tiered Storage
分层存储通过解耦
Kafka 中的计算层和存
储层，实现无限数据保留
和弹性可扩展性
事件流是存储密集型的，尤其是
随着客户的成熟：
...
Micro-
service
...
SFDC App
Splunk
...
Device
Logs
Object
Storage
Main-
frame
...
Hadoop
Data Stores
3rd Party Apps Custom Apps /
Microservices
Logs

4. 4
代理存储可能很昂贵并且与计算
紧密相关：
Brokers
分层存储
Tiered Storage
分层存储通过解耦
Kafka 中的计算层和存
储层，实现无限数据保留
和弹性可扩展性

5. 5
分层存储允许 Kafka 识别两层存
储：
Brokers
Cost-effective
Object Storage
Offload old data
to object store
分层存储
Tiered Storage
分层存储通过解耦
Kafka 中的计算层和存
储层，实现无限数据保留
和弹性可扩展性

6. 6
分层存储提供了三个主要优势，
彻底改变了我们客户体验 Kafka
的方式：
无限数据保留
Infinite data retention
Reimagine what event streaming apps can do
降低基础设施成本
Reduced infrastructure costs
Offload data to cost-effective object storage
平台弹性
Platform elasticity
Scale compute and storage independently
分层存储
Tiered Storage
分层存储通过解耦
Kafka 中的计算层和存
储层，实现无限数据保留
和弹性可扩展性

7. 全球韧性
Dynamic Performance & Elasticity
Self-Balancing Clusters | Tiered Storage
Flexible DevOps Automation
Operator | Ansible
GUI-driven Mgmt & Monitoring
Control Center | Proactive Support
Event Streaming Database
ksqlDB
Rich Pre-built Ecosystem
Connectors | Hub | Schema Registry
Multi-language Development
Non-Java Clients | REST Proxy
Admin REST APIs
Global Resilience
Multi-region Clusters | Replicator |
Cluster Linking
Data Compatibility
Schema Registry | Schema Validation
Enterprise-grade Security
RBAC | Secrets | Audit Logs
TCO / ROI
Revenue / Cost / Risk Impact
Complete Engagement Model
Efficient
Operations at Scale
Unrestricted
Developer Productivity
Production-stage
Prerequisites
Partnership for
Business Success
Freedom of Choice
Committer-driven Expertise
Open Source | Community licensed
Fully Managed Cloud Service
Self-managed Software
Training Partners
Enterprise
Support
Professional
Services
ARCHITECT
OPERATOR
DEVELOPER BUYER
Apache Kafka

8. 在独立集群或迁移集群之间共享数
据带来了主要挑战：
1. Requires deploying a separate Connect
cluster, adding architectural complexity
2. Offsets are not preserved, so messages
are at risk of being skipped or reread
8
1
2
0 1 2 3 4 ...
4 5 6 7 8 ...
Topic 1, DC 1:
Topic 1, DC 2:
DC 1: DC 2:
集群链接
Cluster Linking
集群链接简化了 Kafka
的混合云和多云部署

9. 9
Cluster Linking 不需要额外的基础
设施并保留偏移量：
Migrate
clusters to
Confluent
Cloud
集群链接
Cluster Linking
集群链接简化了 Kafka
的混合云和多云部署

10. Cluster Linking 还在公共云之间提供了一
个经济高效、安全且高性能的传输层：
App
App
App
App
App
App
App
App
App
App
App
App
高网络成本
复杂的管理
低网络成本
一次移动，多次阅读
Cluster Linking
10
集群链接
Cluster Linking
集群链接简化了 Kafka
的混合云和多云部署

11. 混合云和多云战略为企业提供了
显着的好处：
消除数据孤岛 Remove data silos and ensure
data exists wherever your business needs it
Leverage 同类最佳解决方案 best of breed
solutions across different public cloud
providers
Offload data infrastructure to 全托管服务
fully managed services, like Confluent
Cloud
避免供应商锁定 Avoid vendor lock-in and
utilize the most cost-effective vendors
11
集群链接
Cluster Linking
集群链接简化了 Kafka
的混合云和多云部署

12. 12
Demo - Clustering Linking