Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
1
Deploying Confluent
Platform in Production
Gwen Shapira, Product Manager
2
Welcome to
Best Practices for Apache
Kafka in Production
Online Talk Series
3
About me
• Product Manager
• Apache Kafka Committer
• Tweets a lot @gwenshap
4
About you
• Know Linux
• Installed Confluent Platform
• Ran an example application
• Need to productionize Kafka
• Ready...
5
Components Resources
Planning Scaling
Deployment
6
Components of
Confluent Platform
7
ZooKeeper ZooKeeper ZooKeeper
8
ZooKeeper ZooKeeper ZooKeeper
Kafka Kafka Kafka
9
ZooKeeper ZooKeeper ZooKeeper
Kafka Kafka Kafka
Java app C# app
Go app Python app
REST Proxy
10
ZooKeeper ZooKeeper ZooKeeper
Kafka Kafka Kafka
Java app C# app
Go app Python app
REST Proxy
Connect
Connectors
Streams...
11
ZooKeeper ZooKeeper ZooKeeper
Kafka Kafka Kafka
Java app C# app
Go app Python app
REST Proxy
Connect
Connectors
Streams...
12
ZooKeeper ZooKeeper ZooKeeper
Kafka Kafka Kafka
Java app C# app
Go app Python app
REST Proxy
Connect
Connectors
Streams...
13
State in Kafka Stateless – No Storage
Planning for storage and containers
State on local disk
ZooKeeperKafka
Streams
ap...
14
Required Resources
15
The Storage Questions
How Much?
• Kafka:
Throughput * Retention
• Zookeeper:
Very little
• Control Center:
Lots
• Strea...
16
Memory
Page Cache JVM Heap Off Heap Memory
ZooKeeper 1-4 GB
Kafka The more the
merrier
# partitions * max fetch
+ compa...
17
• CPU
• Keep an eye on Zookeeper and Kafka
• High CPU usually caused by misconfig
• Or by… Compression, encryption, hig...
18
Special Considerations for Clouds
• Virtual cores are relatively weak
• Network is typically weak
• Shared storage is t...
19
Planning a Deployment
of Confluent Platform
20
1. Which components do I need?
2. Small or Large cluster?
Kafka just for one team and one app?
Centralized cluster for ...
21
Small Cluster:
Place many related
components
on each node
22
Large Cluster:
Separate nodes
Scale each component as
needed
23
Scaling Confluent Platform
as you grow
24
Some Stuff you can change easily:
• Additional producers
• Additional schema registries
• Additional REST Proxies
• Add...
25
Some Stuff requires “enough partitions”:
• Additional consumers
• Additional streams instances
• Additional sink connec...
26
Some Stuff takes time/resources to change:
• Additional partitions
• Additional brokers
• Additional Zookeeper instance...
27
Monitor your cluster closely to know when to add resources
28
1. Key components in Confluent Platform and their requirements
2. Key resources and how to use them effectively
3. Plan...
29
Thank You!
Prochain SlideShare
Chargement dans…5
×

Deploying Confluent Platform for Production

4 294 vues

Publié le

Presented by Gwen Shapira as part 1 of 5 in the Best Practices for Apache Kafka in Production, Confluent Online Talk Series

Publié dans : Logiciels

Deploying Confluent Platform for Production

  1. 1. 1 Deploying Confluent Platform in Production Gwen Shapira, Product Manager
  2. 2. 2 Welcome to Best Practices for Apache Kafka in Production Online Talk Series
  3. 3. 3 About me • Product Manager • Apache Kafka Committer • Tweets a lot @gwenshap
  4. 4. 4 About you • Know Linux • Installed Confluent Platform • Ran an example application • Need to productionize Kafka • Ready to Learn
  5. 5. 5 Components Resources Planning Scaling Deployment
  6. 6. 6 Components of Confluent Platform
  7. 7. 7 ZooKeeper ZooKeeper ZooKeeper
  8. 8. 8 ZooKeeper ZooKeeper ZooKeeper Kafka Kafka Kafka
  9. 9. 9 ZooKeeper ZooKeeper ZooKeeper Kafka Kafka Kafka Java app C# app Go app Python app REST Proxy
  10. 10. 10 ZooKeeper ZooKeeper ZooKeeper Kafka Kafka Kafka Java app C# app Go app Python app REST Proxy Connect Connectors Streams apps
  11. 11. 11 ZooKeeper ZooKeeper ZooKeeper Kafka Kafka Kafka Java app C# app Go app Python app REST Proxy Connect Connectors Streams apps Schema Registry
  12. 12. 12 ZooKeeper ZooKeeper ZooKeeper Kafka Kafka Kafka Java app C# app Go app Python app REST Proxy Connect Connectors Streams apps Schema Registry Replicator ADB Control Center
  13. 13. 13 State in Kafka Stateless – No Storage Planning for storage and containers State on local disk ZooKeeperKafka Streams apps Control Center Java app C# app Go app Python app REST ProxyConnect Connectors Schema Registry Replicator ADB
  14. 14. 14 Required Resources
  15. 15. 15 The Storage Questions How Much? • Kafka: Throughput * Retention • Zookeeper: Very little • Control Center: Lots • Streams: It is complicated Hardware • Are SSDs worth it? • Is shared storage ok? Configuration • RAID vs JBOD • XFS or EXT4 • Zookeeper log Partitions • Per topic • Per broker • Total
  16. 16. 16 Memory Page Cache JVM Heap Off Heap Memory ZooKeeper 1-4 GB Kafka The more the merrier # partitions * max fetch + compaction buffer + ~10% Kafka Connect # tasks * # partitions * memory buffer per partition Kafka Streams 10GB Buffer Cache + 1MB per partition or 50MB per broker =~ 32GB J RocksDB Rest Proxy ~ 1 GB Schema Registry ~ 1 GB Clients Java client – batching and retries Other clients – batching and retries Control Center It is a streams app… 32GB It is a streams app…
  17. 17. 17 • CPU • Keep an eye on Zookeeper and Kafka • High CPU usually caused by misconfig • Or by… Compression, encryption, high request rate • Network • 1GbE = 100MB/s rate (including replication!) • Leave room for catching up • Compress!
  18. 18. 18 Special Considerations for Clouds • Virtual cores are relatively weak • Network is typically weak • Shared storage is typically awesome
  19. 19. 19 Planning a Deployment of Confluent Platform
  20. 20. 20 1. Which components do I need? 2. Small or Large cluster? Kafka just for one team and one app? Centralized cluster for larger organization? 3. Other requirements? Availability, retention, latency, throughput. 4. How many clusters? Ask yourself:
  21. 21. 21 Small Cluster: Place many related components on each node
  22. 22. 22 Large Cluster: Separate nodes Scale each component as needed
  23. 23. 23 Scaling Confluent Platform as you grow
  24. 24. 24 Some Stuff you can change easily: • Additional producers • Additional schema registries • Additional REST Proxies • Additional connectors and tasks • Additional replicas
  25. 25. 25 Some Stuff requires “enough partitions”: • Additional consumers • Additional streams instances • Additional sink connectors
  26. 26. 26 Some Stuff takes time/resources to change: • Additional partitions • Additional brokers • Additional Zookeeper instances • Additional Control Centers (impossible right now)
  27. 27. 27 Monitor your cluster closely to know when to add resources
  28. 28. 28 1. Key components in Confluent Platform and their requirements 2. Key resources and how to use them effectively 3. Planning your deployment 4. Monitoring in order to scale 5. Want to learn more? There is a paper! https://www.confluent.io/whitepaper/confluent-enterprise-reference- architecture/ What did we learn?
  29. 29. 29 Thank You!

×