Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with Kafka on Kubernetes ( Kamala Dasika, Pivotal and Michael Ng, Confluent) Kafka Summit London 2019
Microservices, events, containers, and orchestrators are dominating our vernacular today. As operations teams adapt to support these technologies in production, cloud-native platforms like Cloud Foundry and Kubernetes have quickly risen to serve as force multipliers of automation, productivity and value. Kafka is providing developers a critically important component as they build and modernize applications to cloud-native architecture. This talk will explore:
• Why cloud-native platforms and why run Kafka on Kubernetes?
• What kind of workloads are best suited for this combination?
• Tips to determine the path forward for legacy monoliths in your application portfolio
• Running Kafka as a Streaming Platform on Container Orchestration
Similaire à Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with Kafka on Kubernetes ( Kamala Dasika, Pivotal and Michael Ng, Confluent) Kafka Summit London 2019
Similaire à Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with Kafka on Kubernetes ( Kamala Dasika, Pivotal and Michael Ng, Confluent) Kafka Summit London 2019 (20)
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Modern Cloud-Native Streaming Platforms: Event Streaming Microservices with Kafka on Kubernetes ( Kamala Dasika, Pivotal and Michael Ng, Confluent) Kafka Summit London 2019
2. Changing Model for Application Delivery
Development Deployment
Sparingly at
designated times
Ready for prod at
any time
Architecture
Abstraction
Layer
“Day 2” Ops
App Server on Machine
App on “disposable”
infrastructure
App and Data
Monolith
Microservices /
Composite app
Linear / Sequential
Short cycles, test
driven, iterative
Many tools, ad hoc
automation
Manage services,
not servers
3. Driven by Modern Cloud Native Platform
Cont
inuous Del
ivery
DEV OPS
Customer
Feedback
Customer
Delivery
An idea in the morning can ship by evening
Microservices
Release #1
Microservices
Release #2
7. Hardware
IaaS
CaaS
PaaS
FaaS
Strategic goal: Push as many workloads as technically
feasible to the top of the platform hierarchy
Higher flexibility and
less enforcement of
standards
Lower development
complexity and higher
operational efficiency
9. Goal: Operational consistency AND multiple abstractions
FREEDOM: Run Your App on Any Cloud
AUTOMATION: Lowers Ops Cost and Increases Velocity
RESILIENCY: Auto-Scale, Blue-green Deploys, Health Mgmt.
GOVERNANCE: Common Developer-Self-Serviced Frameworks
VISIBILITY: Full Stack Logging, Monitoring & Auditability
SPEED & SCALE: Cut Cycle Time from Months to Days/Hours
Built for Cloud
Native
New Initiatives
Existing Workloads
Modernize to
Cloud-Native?
10. Outcomes trump technology bingo
Needs High Dev Efficiency
Needs Low Dev Efficiency
App Rarely changes App Frequently changes
• Traditional
apps
• Leave as-is
• Re-architect
• Replatform
• CI/CD
• Evaluate
• Evaluate
11. Evaluating the Enterprise App Portfolio
Business
Technical Economic
Score
• Business criticality
• Risk tolerance
• Change frequency
• Lifecycle stage
• Domain expert availability
• Licensing costs (HW / SW)
• Time-to-market implications
• Revenue impact-fullness
• Codebase
• Suitable framework / runtime
• Relatively lightweight footprint
• No “hard wired” file system
dependencies
• Usage / workload
For each app
12. Application Transformation Strategies
Re-Host
Containerize aka
“lift and shift” to
CaaS and gain
infrastructure +
platform ops benefits
Re-Platform
Minimal readiness
effort to run on PaaS
(4 -7 factor apps +
data service) that
can run on PaaS
Re-Factor
Convert to cloud-
native/microservice
(12-15 factor app +
data service) runs
well on PaaS/FaaS
Re-Build
Decompose and
rewrite from the
ground up
Application Portfolio
Funneling Process
Learn More: Tools and Recipes to Replatform Monolithic Apps to Modern Cloud Environments
13. Evolution of the enterprise application portfolio
Learn More: Microservices, Events, and Breaking the Data Monolith with Kafka
15. “The world’s most valuable resource is no longer oil, but data” - The
Economist, May 6 2017
16. Apache Kafka is a Distributed Event Streaming
Platform
Process streams of events In real time, as they occur
110101
010111
001101
100010
Publish and subscribe to
streams of events
Similar to a message queue or
enterprise messaging system
110101
010111
001101
100010
Store streams of events In a fault tolerant way
110101
010111
001101
100010
20. Events enable Microservice Architectures - Let’s
refactor!
Submit
Order
Orders
Service
Shipping
Service
Customer
Service
Webserver
E-Commerce
Microservices (with RPC)
- Orders Service calls
Shipping Service to tell it
to ship item.
- Shipping service looks
up address to ship to
(from Customer Service)
- No Kafka !shipOrder() getCustomer()
21. Refactoring Orders and Shipping
Message Broker (Kafka)
Submit
Order
Order
Created
getCustomer()
RPC
Orders
Service
Shipping
Service
Customer
Service
Webserver
- Orders Service no longer
knows about the Shipping
service (or any other service).
- Events are fire and forget.
25. Why run Kafka on
Kubernetes?
33% of Confluent
Customers plan to run
Production Kafka on
Kubernetes
26. Microservices - driving container and container
orchestration adoption
Breaking down the monolith - enables dev teams to work and scale
independently
µ
Containers allow services to be independently deployable services
● Each service has its own purpose & implementation language
● Each service has its own dependencies
Orchestration - running containers at scale
● Microservices = container deployments at scale
● Service discovery
● Scale services independently
● High Availability, Self Healing of services
● Hardware Abstraction
28. There are some challenges
Translating an existing architecture to Kubernetes: K8s pods,
nodes, stateful sets, custom resources
External access: What do we do with advertised listeners on
Kubernetes?
Managing configuration of Kafka, Zookeeper and all Confluent
Components
Persistent Storage options on prem and clouds
Security Configuration and Upgrades
29. Confluent Operator enables you to:
Automate provisioning of Kafka pods
in minutes
Monitor SLAs through Confluent
Control Center or Prometheus
Scale Kafka elastically & Automate
rolling updates
Built on our first hand knowledge of
running Confluent at scale
Confluent Operator is
our Kubernetes
Operator and our first
step towards becoming
cloud native
CONFIDENTIAL
31. Confluent Operator - Scale Horizontally
Automate Scaling:
Spin up new brokers, connect workers
easily
Distribute partitions to new brokers:
Determine balancing plan
Execute balancing plan
Monitor Resources
32. Confluent Operator - Rolling Upgrade of all
components
Automated Rolling Upgrades of all
components - Kafka Brokers, Zookeeper,
Connect, Control Center
Kafka Broker Upgrades:
1. Stop the broker, upgrade Kafka
2. Wait for Partition Leader reassignment
3. Start the upgraded broker
4. Wait for zero under-replicated partitions
5. Upgrade the next broker
33. Confluent Operator - Automated Security
Configuration
SASL PLAIN and Mutual TLS
Authentication
Automate configuration of truststores
and keystores with secret objects
Automate configuration of Kafka and all
Confluent Platform Components
34. Demo – Ride Sharing Location Tracking
2 Web Apps powered by a
microservice each
Apps are a ride sharing driver and
rider location tracking
Rider hails a ride and the driver
drives towards the rider
Real time location events powered
by Kafka on Kubernetes
Rider Driver
35. Confluent Operator GA Plans
We are in private Preview Release now
24 customers testing the Operator in Preview:
● Global customers
● Banks, Fin Tech, Retailers, Consumer Tech
We are in the final stages of Preview and about to launch soon.