Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
@gamussa | #kafkasummit | @ConfluentINc
Kafka on Kubernetes:
Does it really have to be
«The Hard Way»?
April, 2019 / New Y...
Raffle, yeah 🚀
Follow @gamussa @confluentinc
📸 🖼 👬
Tag @gamussa
With #kafkasummit
@gamussa | #kafkasummit | @ConfluentINc
3
Evolution of #devkafkaops
Shell scripts
ansible/chef
Docker
Kubernetes
@gamussa | #kafkasummit | @ConfluentINc
4
@gamussa | #kafkasummit | @ConfluentINc
5
🙋
@gamussa | #kafkasummit | @ConfluentINc
6
Who run stateless
workloads in Kubernetes?
Who thinks it’s a good
idea?
Who run ...
@gamussa | #kafkasummit | @ConfluentINc
7
kafkaesque world of Kafka on
Kubernetes
@gamussa | #kafkasummit | @ConfluentINc
8
Well, it’s tricky ©
Translating an existing architecture to Kubernetes
External ...
@gamussa | #kafkasummit | @ConfluentINc
9
We just need to
deploy Kafka on
Kubernetes
@gamussa | #kafkasummit | @ConfluentINc
10
We will use
confluentinc/cp-helm-
charts
@gamussa | #kafkasummit | @ConfluentINc
11
Helm Charts is just a GO
Templates.
How Charts help with
rolling restart?
@gamussa | #kafkasummit | @ConfluentINc
12
We will use
StatefulSets
with OrderedReady
@gamussa | #kafkasummit | @ConfluentINc
13
We need SRE /
Operator knowledge to
manage the platform.
You need Operator!
@gamussa | #kafkasummit | @ConfluentINc
14
@gamussa | #kafkasummit | @ConfluentINc
15
Show me your
Operator
@gamussa | #kafkasummit | @ConfluentINc
16
Demo
@gamussa | #kafkasummit | @ConfluentINc
17
DO KAFKA ON KUBERNETES DEMO
AND EVERYONE LOOSES THEIR MIND
@gamussa | #kafkasummit | @ConfluentINc
18
What just happened?
ZK and Kafka deployed
Security with TLS is configured
Exter...
@gamussa | #kafkasummit | @ConfluentINc
19
Confluent Operator - Automated
Security Configuration
SASL PLAIN and Mutual TLS...
@gamussa | #kafkasummit | @ConfluentINc
20
Confluent Operator - Scale
Automate Scaling:
Spin up new brokers, connect worke...
@gamussa | #kafkasummit | @ConfluentINc
21
Be like Justin!
@gamussa | #kafkasummit | @ConfluentINc
22
Rolling Upgrade
Kafka Broker Upgrades:
1. Stop the broker, upgrade
Kafka
2. Wai...
@gamussa | #kafkasummit | @ConfluentINc
23
Will it fly?
vs.
@gamussa | #kafkasummit | @ConfluentINc
24
GA Plans● We are in private Preview
Release now
● 24 customers testing the
Oper...
@@gamussa | #kafkasummit | @ConfluentINc
Thanks!
@gamussa
viktor@confluent.io
michael.ng@confluent.io
https://slackpass.io...
26
Prochain SlideShare
Chargement dans…5
×

Kafka on Kubernetes: Does it really have to be "The Hard Way"? (Viktor Gamov and Michael Ng, Confluent) Kafka Summit NYC 2019

597 vues

Publié le

When it comes to choosing a distributed streaming platform for real-time data pipelines, everyone knows the answer: Apache Kafka! And when it comes to deploying applications at scale without needing to integrate different pieces of infrastructure yourself, the answer nowadays is increasingly Kubernetes. However, with all great things, the devil is truly in the details. While Kubernetes does provide all the building blocks that are needed, a lot of thought is required to truly create an enterprise-grade Kafka platform that can be used in production. In this technical deep dive, Michael and Viktor will go through challenges and pitfalls of managing Kafka on Kubernetes as well as the goals and lessons learned from the development of the Confluent Operator for Kubernetes. NOTE: This talk will be delivered with Michael Ng, product manager, Confluent

Publié dans : Technologie
  • Soyez le premier à commenter

Kafka on Kubernetes: Does it really have to be "The Hard Way"? (Viktor Gamov and Michael Ng, Confluent) Kafka Summit NYC 2019

  1. 1. @gamussa | #kafkasummit | @ConfluentINc Kafka on Kubernetes: Does it really have to be «The Hard Way»? April, 2019 / New York, 2019 @gamussa | #kafkasummit | @ConfluentINc
  2. 2. Raffle, yeah 🚀 Follow @gamussa @confluentinc 📸 🖼 👬 Tag @gamussa With #kafkasummit
  3. 3. @gamussa | #kafkasummit | @ConfluentINc 3 Evolution of #devkafkaops Shell scripts ansible/chef Docker Kubernetes
  4. 4. @gamussa | #kafkasummit | @ConfluentINc 4
  5. 5. @gamussa | #kafkasummit | @ConfluentINc 5 🙋
  6. 6. @gamussa | #kafkasummit | @ConfluentINc 6 Who run stateless workloads in Kubernetes? Who thinks it’s a good idea? Who run stateful workloads in Kubernetes? Who thinks it’s a good idea? 🙋
  7. 7. @gamussa | #kafkasummit | @ConfluentINc 7 kafkaesque world of Kafka on Kubernetes
  8. 8. @gamussa | #kafkasummit | @ConfluentINc 8 Well, it’s tricky © Translating an existing architecture to Kubernetes External access to brokers and other components Persistent Storage options on prem and clouds Security Configuration and Upgrades #devkafkaops
  9. 9. @gamussa | #kafkasummit | @ConfluentINc 9 We just need to deploy Kafka on Kubernetes
  10. 10. @gamussa | #kafkasummit | @ConfluentINc 10 We will use confluentinc/cp-helm- charts
  11. 11. @gamussa | #kafkasummit | @ConfluentINc 11 Helm Charts is just a GO Templates. How Charts help with rolling restart?
  12. 12. @gamussa | #kafkasummit | @ConfluentINc 12 We will use StatefulSets with OrderedReady
  13. 13. @gamussa | #kafkasummit | @ConfluentINc 13 We need SRE / Operator knowledge to manage the platform. You need Operator!
  14. 14. @gamussa | #kafkasummit | @ConfluentINc 14
  15. 15. @gamussa | #kafkasummit | @ConfluentINc 15 Show me your Operator
  16. 16. @gamussa | #kafkasummit | @ConfluentINc 16 Demo
  17. 17. @gamussa | #kafkasummit | @ConfluentINc 17 DO KAFKA ON KUBERNETES DEMO AND EVERYONE LOOSES THEIR MIND
  18. 18. @gamussa | #kafkasummit | @ConfluentINc 18 What just happened? ZK and Kafka deployed Security with TLS is configured External access is configured Monitoring is enabled
  19. 19. @gamussa | #kafkasummit | @ConfluentINc 19 Confluent Operator - Automated Security Configuration SASL PLAIN and Mutual TLS Authentication Automate configuration of truststores and keystores with secret objects Automate configuration of Kafka and all Confluent Platform Components
  20. 20. @gamussa | #kafkasummit | @ConfluentINc 20 Confluent Operator - Scale Automate Scaling: Spin up new brokers, connect workers easily Distribute partitions to new brokers: Determine balancing plan Execute balancing plan Monitor Resources
  21. 21. @gamussa | #kafkasummit | @ConfluentINc 21 Be like Justin!
  22. 22. @gamussa | #kafkasummit | @ConfluentINc 22 Rolling Upgrade Kafka Broker Upgrades: 1. Stop the broker, upgrade Kafka 2. Wait for Partition Leader reassignment 3. Start the upgraded broker 4. Wait for zero under- replicated partitions 5. Upgrade the next broker
  23. 23. @gamussa | #kafkasummit | @ConfluentINc 23 Will it fly? vs.
  24. 24. @gamussa | #kafkasummit | @ConfluentINc 24 GA Plans● We are in private Preview Release now ● 24 customers testing the Operator in Preview: ● Global customers ● Banks, Fin Tech, Retailers, Consumer Tech ● We are in the final stages of Preview and about to launch soon
  25. 25. @@gamussa | #kafkasummit | @ConfluentINc Thanks! @gamussa viktor@confluent.io michael.ng@confluent.io https://slackpass.io/confluentcommunity #kubernetes
  26. 26. 26

×