SlideShare une entreprise Scribd logo
1  sur  59
The Top 5 Event Streaming
Use Cases & Architectures in 2021
Hybrid Architectures, Edge Computing, Machine Learning, Cybersecurity, Service Mesh
Kai Waehner
Field CTO
contact@kai-waehner.de
LinkedIn
@KaiWaehner
www.confluent.io
www.kai-waehner.de
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Gartner Top
Strategic Technology
Trends for 2021
https://www.gartner.com/smarterwithgartner/gartner-top-strategic-technology-trends-for-2021/
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
STREAM
PROCESSING
Create and store
materialized views
Filter
Analyze in-flight
Time
C CC
Event Streaming
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Global Scale
Real-time
Persistent Storage
Stream Processing
Data Integration
Apache Kafka
The De-facto Standard for Real-Time Event Streaming
Edge
Cloud
Data LakeDatabases
Datacenter
IoT
SaaS AppsMobile
Microservices Machine
Learning
Apache
Kafka
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Improve
Customer
Experience
(CX)
Increase
Revenue
(make money)
Business
Value
Decrease
Costs
(save money)
Core Business
Platform
Increase
Operational
Efficiency
Migrate
to Cloud
Mitigate
Risk (protect
money)
Key Drivers
Strategic
Objectives
(sample)
Fraud
Detection
IoT sensor
ingestion
Digital
replatforming/
Mainframe Offload
Connected Car: Navigation &
improved in-car experience: Audi
Customer 360
Simplifying Omni-channel Retail at
Scale: Target
Faster transactional
processing / analysis
incl. Machine Learning / AI
Mainframe Offload: RBC
Microservices
Architecture
Online Fraud Detection
Online Security
(syslog, log aggregation,
Splunk replacement)
Middleware
replacement
Regulatory
Digital
Transformation
Application Modernization: Multiple
Examples
Website / Core
Operations
(Central Nervous
System)
The [Silicon Valley] Digital Natives;
LinkedIn, Netflix, Uber, Yelp...
Predictive Maintenance: Audi
Streaming Platform in a regulated
environment (e.g. Electronic Medical
Records): Celmatix
Real-time app
updates
Real Time Streaming Platform for
Communications and Beyond: Capital
One
Developer Velocity - Building
Stateful Financial Applications with
Kafka Streams: Funding Circle
Detect Fraud & Prevent Fraud in
Real Time: PayPal
Kafka as a Service - A Tale of
Security and Multi-Tenancy: Apple
Example Use Cases
$↑
$↓
$↔
Example Case Studies
(of many)
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
The Top 5 Event Streaming
Use Cases & Architectures in 2021
1) Hybrid Architectures
2) Edge Deployments Outside the Data Center
3) Service Mesh based Microservice Architectures
4) Streaming Machine Learning
5) Next-Generation Cybersecurity
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
The Top 5 Event Streaming
Use Cases & Architectures in 2021
1) Hybrid Architectures
2) Edge Deployments Outside the Data Center
3) Service Mesh based Microservice Architectures
4) Streaming Machine Learning
5) Next-Generation Cybersecurity
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Why Kafka in Multiple Data Centers?
* Not a representative survey J
** ‘Many DCs’ does NOT necessarily mean more than one Kafka Cluster
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Disaster Recovery – RPO and RTO
RPO = Recovery Point Objective
RTO = Recovery Time Objective
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Disaster Recovery @ JPMorgan
https://www.confluent.io/kafka-summit-san-francisco-2019/secure-kafka-at-scale-in-true-multi-tenant-environment
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Cluster Linking
• Hybrid-cloud and multi-cloud
• No additional infrastructure (such as Kafka Connect or MirrorMaker)
• Just configuration
• Regional or global
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
What is the right Hybrid Kafka Architecture for you?
(Hint: This is hard à Let’s guide you by our experts)
12
Latency
> 50ms
Latency
< 50ms
RTO = 0 RTO > 0 RPO = 0 RPO > 0 Single Region Multi-Region Global
Stretched Cluster
x x x x
Replicator
x x x x* x
Cluster-Linking
x x x x* x
MRC Sync
x x x x**
MRC Observer
x x x x**
* With a stretched cluster in a single region, you still have RTO & RPO = 0
** Requires 3 regions minimum
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Kafka as a Service – Fully Managed?
Infrastructure
management
(commodity)
Scaling
● Upgrades (latest stable version of Kafka)
● Patching
● Maintenance
● Sizing (retention, latency, throughput, storage, etc.)
● Data balancing for optimal performance
● Performance tuning for real-time and latency requirements
● Fixing Kafka bugs
● Uptime monitoring and proactive remediation of issues
● Recovery support from data corruption
● Scaling the cluster as needed
● Data balancing the cluster as nodes are added
● Support for any Kafka issue with less than X minutes response time
Infra-as-a-Service
Harness full power of Kafka
Kafka-specific
management
Platform-as-a-Service
Evolve as you need
Future-proof
Mission-critical reliability
Most Kafka-as-a-Service offerings are partially-managed
Kafka as a Service should be a serverless experience with consumption-based pricing!
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
The Top 5 Event Streaming
Use Cases & Architectures in 2021
1) Hybrid Architectures
2) Edge Deployments Outside the Data Center
3) Service Mesh based Microservice Architectures
4) Streaming Machine Learning
5) Next-Generation Cybersecurity
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
What is the “Edge” for Kafka?
• Edge is NOT a data center
• Kafka clients AND the Kafka broker(s)
• Offline business continuity
• Often 100+ locations
• Low-footprint and low-touch
• Hybrid integration
Example:
Single broker, 1 GB Ram
100 MB/sec
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
CRM
3rd party
payment
provider
Context-specific
real-time upsell
Customer data
Payment processing and
fraud detection as a service
Manager
Get report
API
Customer Customer
Customer
data
Train
schedule
Payment
data
Loyalty
information
Streams of real time events
Customer
data
Train
schedule
Payment
data
Loyalty
information
Streams of real time events
Customer
data
Train
schedule
Payment
data
Loyalty
information
Streams of real time events
Hybrid Architecture
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Food
Inventory Loyalty
System
Traveler
Information
Orders Upsell to
first class
Customer
data
Train
schedule
Payment
data
Loyalty
information
Streams of real time events
Updated
SchedulesEvent Streaming
at the Edge
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Data Processing
at the Edge
Time
P
C1
C2
C3
Know-your-customer
Loyalty app, predictive behavior, …
Estimated
time of arrival
Connect to the
gaming server
for kids
Play games, earn rewards, communicate
with other kids in the train, …
Always on (even “offline”)
Replayability
Reduced traffic cost
Better latency
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Edge Kafka @ Royal Caribbean
https://www.confluent.io/kafka-summit-lon19/seamless-guest-experience-with-kafka-streams/
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Edge Integration and Analytics @ WPX Energy
Edge processing and
replication to the cloud
in real-time at scale
in the oil&gas industry
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
The Top 5 Event Streaming
Use Cases & Architectures in 2021
1) Hybrid Architectures
2) Edge Deployments Outside the Data Center
3) Service Mesh based Microservice Architectures
4) Streaming Machine Learning
5) Next-Generation Cybersecurity
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Microservices to the rescue?
• Significant Operations Overhead
• Substantial DevOps Skills Required
• Implicit Interfaces
• Duplication Of Effort
http://highscalability.com/blog/2014/4/8/microservices-not-a-free-lunch.html
• Distributed System Complexity
• Asynchronicity Is Difficult
• Testability Challenges
22
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Microservices can lead to Death-star Architectures
Netflix: https://www.slideshare.net/brucewong3/the-case-for-chaos
Twitter: https://twitter.com/adrianco/status/441883572618948608
Hail-o: http://www.sudo.hailoapp.com/services/2015/03/09/journey-into-a-microservice-world-part-3/
450+ microservices 500+ microservices 500+ microservices
23
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Key Requirements for Microservices
•Decoupled
•Flexible
•Operationally Transparent
•Data Aware
•Elastic
24
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Kafka Connect
Kafka Cluster
CRM
Integration
Domain-Driven Design (DDD) for your Microservice Architecture
Legacy
Integration
Custom
Application
ESB Connector
Java / KSQL /
Kafka Streams
Schema
Registry
Event Streaming Platform
CRM Domain Legacy Domain Payment Domain
è Independent and loosely coupled, but scalable, highly available and reliable!
25
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Key Requirements for Microservices
•Decoupled
•Flexible
•Operationally Transparent
•Data Aware
•Elastic
26
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Cloud-Native Deployment leveraging Kubernetes
27
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Service Mesh
A microservice pattern to move visibility, reliability, and
security primitives for service-to-service communication into
the infrastructure layer, out of the application layer.
https://www.infoq.com/articles/linkerd-v2-production-adoption/
28
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Service Proxy Features
• Metrics without instrumenting apps
• Trace flow of requests across services
• One stable URI for each service
• Service discovery
• Monitor request latency
• Routing - A/B testing, green/blue deployments
• Circuit breaking
• Protocol translation (HTTP, gRPC, Kafka Protocol, etc.)
• Mutual TLS (mTLS)
• SSL Termination
• Integrate with 3rd party tools like Prometheus, Grafana,
Zipkin, etc.
• Much more…
Observability
“is by far the most important thing that a Proxy and the Service
Mesh provide in a distributed Microservice architecture!” Matt Klein
29
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Example: Lyft today with “envoy” Proxy
• 100% (!!!) communication coverage - Everything talks through Envoy Proxies
• à Make monitoring, debugging, firefighting as consistent as possible
https://www.youtube.com/watch?v=55yi4MMVBi4
Matt Klein at QCon NY 2018
30
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Kafka Connect
Kafka Cluster
CRM
Integration
Clients and Servers are Independent (including their Ops Teams)
Legacy
Integration
Custom
Application
ESB Connector
Java / KSQL /
Kafka Streams
Schema
Registry
Event Streaming Platform
CRM Domain Legacy Domain Payment Domain
Proxy
Proxy
Proxy
Proxy
Proxy
Proxy
Control
Plane
31
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Example: Kafka + Confluent REST Proxy
Envoy
Proxy
I am using REST too!
Kafka? Never heard
of her.
I’m using REST
to talk to a
service
I’m proxying
REST.
And also
logging stuff
to Kafka
Confluent
REST Proxy
I support only
TCP!
HTTP
HTTP
32
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Example: Kafka + Envoy Kafka Protocol Filter
Envoy
Proxy
I am using REST too!
Kafka? Never heard
of her.
I’m using REST
to talk to a
service
I’m proxying
REST.
And also
logging stuff
to Kafka
HTTP
TCP
(Kafka Protocol)
33
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Implementation: Kafka + Istio @ Banzai Cloud
34
https://banzaicloud.com/blog/kafka-on-istio-performance/
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Implementation: Kafka + Istio
35
https://banzaicloud.com/blog/kafka-on-istio-performance/
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Implementation: Kafka + ksqlDB + Istio
36
https://banzaicloud.com/blog/supertubes-ksql/
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
(Potential) Features for
Kafka + Service Mesh Implementation
Protocol conversion from HTTP / gRPC
to Kafka
• Tap feature to dump to a Kafka stream
• Protocol parsing for observability
(stats, logging, and trace linking with
HTTP RPCs)
• Shadow requests to a Kafka stream
instead of HTTP / gRPC shadow
• Integrate with Kafka Connect and its
whole ecosystem of connectors
Validation of Events
• Serialization format
(JSON, Avro, Protobuf, etc.)
• Message schema
• Headers, attributes, etc.
Security
• SSL Termination
• Mutual TLS (mTLS)
• Authorization
Proxy features
• Dynamic Routing
• Rate limiting at both the L4 connection
and L7 message level
• Filter, add compression, …
• Automatic topic name conversion (e.g. for
canary release or blue/green deployment)
Monitoring and Tracing
• Request logs and stats
• Data lineage / audit log
• Audit log by taking request logs and
enriching them with the user info.
• Client specific metrics (Byte rate per
client id / per consumer groups,
versions of the client libraries,
consumer lag monitoring for the entire
data center)
37
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Key Requirements for Microservices
•Decoupled
•Flexible
•Operationally Transparent
•Data Aware
•Elastic
38
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
The Top 5 Event Streaming
Use Cases & Architectures in 2021
1) Hybrid Architectures
2) Edge Deployments Outside the Data Center
3) Service Mesh based Microservice Architectures
4) Streaming Machine Learning
5) Next-Generation Cybersecurity
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Apache Kafka as Infrastructure for ML
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Apache Kafka’s Open Ecosystem as Infrastructure for ML
Kafka
Streams/
ksqlDB
Kafka Connect
Confluent REST Proxy
Confluent Schema Registry
Go/.NET/Python
Kafka Producer
ksqlDB
Python
Client
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Streaming Analytics for
Predictive Maintenance at Scale
42
IoT
Integration
Layer
Batch
Analytics
Platform
BI
Dashboard
Streaming
Platform
Big Data
Integration
Layer
Car Sensors
Streaming Platform
Other Components
Real Time
Monitoring
System
All
Data
Critical
Data
Ingest
Data
Potential
Detect
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Streaming Analytics for
Predictive Maintenance at Scale
43
IoT
Integration
Layer
Batch
Analytics
Platform
BI
Dashboard
Streaming
Platform
Big Data
Integration
Layer
Car Sensors
Streaming Platform
Analytics Platform
Other Components
Real Time
Monitoring
System
All
Data
Critical
Data
Ingest
Data
Potential
DetectAnalytics
Platform
Train
Analytic
Model
Data
Processing
Analytic
Model
Preprocess
Data
Consume
Data
Deploy
Analytic Model
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Preprocessing
with ksqlDB
44
SELECT car_id, event_id, car_model_id, sensor_input
FROM car_sensor c
LEFT JOIN car_models m ON c.car_model_id = m.car_model_id
WHERE m.car_model_type ='Audi_A8';
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Direct streaming ingestion
for model training
with TensorFlow I/O + Kafka Plugin
(no additional data storage
like S3 or HDFS required!)
Time
Model BModel A
Producer
Distributed
Commit Log
Streaming Ingestion and Model Training
with TensorFlow IO
https://github.com/tensorflow/io
45
Model X
(at a later time)
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Confluent Tiered Storage for Kafka
46
(Only available in Confluent Platform)
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Local Predictions
Model Training
in the Cloud
Model Deployment
at the Edge
Analytic Model
Separation of
Model Training and Model Inference
47
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
“CREATE STREAM AnomalyDetection AS
SELECT sensor_id,
detectAnomaly(sensor_values)
FROM car_engine;“
User Defined Function (UDF)
Model Deployment with
Apache Kafka, ksqlDB
and TensorFlow
48
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
The Top 5 Event Streaming
Use Cases & Architectures in 2021
1) Hybrid Architectures
2) Edge Deployments Outside the Data Center
3) Service Mesh based Microservice Architectures
4) Streaming Machine Learning
5) Next-Generation Cybersecurity
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Cybersecurity
The threat is real!
Challenges
Stealing IP
DDoS
Ransomware / wiperware
WannaCry, NotPetya, …
Damage: Billions of dollars
”Supply chain attack”
Digital
Transformation
Networking
Communication
Connectivity
Open standards
”Always-on”
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Legacy SIEM needs to evolve
ForwarderNetwork traffic
Firewall logs
RDBMS
Application logs
Adaptors
Beats
Sensor Data
Challenges:
● Proprietary forwarders that can only
send data to single source
● Data locked from being shared
● Difficult to scale with growing data
volumes
● Prohibitively high indexing costs
● Unable to filter out noisy data
● Slow batch processing
HTTP proxy logs
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
AI/ML
Modernized security information and event management (SIEM)
Filter, transform
aggregate
APP SIEM Index
Search
Curated streams
Forensic
Archive
HDFS
S3
Big Query
CDC
Syslog
Network traffic
Firewall logs
RDBMS
Application logs
Payment Data
HTTP proxy logs
QRadar
Arcsight
Splunk
Elastic
APP
Stateful
real-time analytics
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Cyber Intelligence Platform
leveraging Kafka Connect, Kafka Streams, Multi-Region Clusters (MRC), and more…
https://www.intel.com/content/www/us/en/it-management/intel-it-best-practices/modern-scalable-cyber-intelligence-platform-kafka.html
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Fraud Detection
at Scale in Real-Time for Billions of Messages
https://www.infoq.com/presentations/paypal-data-service-fraud
https://conferences.oreilly.com/strata/strata-ny-2018/public/schedule/detail/69459.html
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
How does
Confluent
help?
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
The Rise of Event Streaming
2010
Apache Kafka
created at LinkedIn by
Confluent founders
2014
2020
80%
Fortune 100
Companies
trust and use
Apache Kafka
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
I N V E S T M E N T & T I M E
VALUE
3
4
5
1
2
Event Streaming Maturity Model
Initial Awareness /
Pilot (1 Kafka Cluster)
Start to Build Pipeline /
Deliver 1 New Outcome
(1 Kafka Cluster)
Mission-Critical
Deployment
(Stretched, Hybrid, Multi-
Region)
Build Contextual Event-
Driven Apps
(Stretched, Hybrid,
Multi-Region)
Central Nervous System
(Global Kafka)
Product, Support, Training, Partners, Technical Account Management...
Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent
Confluent Platform
Fully Managed Cloud ServiceSelf Managed Software FREEDOM OF
CHOICE
COMMITTER-DRIVEN
EXPERTISE
PartnersTrainingProfessional
Services
Enterprise
Support
Apache Kafka
EFFICIENT
OPERATIONS AT SCALE
PRODUCTION-
STAGE PREREQUISITES
UNRESTRICTED
DEVELOPER PRODUCTIVITY
SQL-based
Stream Processing
KSQL (ksqlDB)
Rich Pre-built Ecosystem
Connectors | Hub | Schema Registry
Multi-language Development
non-Java clients | REST Proxy
GUI-driven Mgmt & Monitoring
Control Center
Flexible DevOps Automation
Operator | Ansible
Dynamic Performance &
Elasticity
Auto Data Balancer | Tiered Storage
Enterprise-grade Security
RBAC | Secrets | Audit logs
Data Compatibility
Schema Registry | Schema Validation
Global Resilience
Multi-Region Clusters | Replicator
Developer Operator Architect
Open Source | Community licensed
PARTNERSHIP
FOR BUSINESS SUCCESS
Complete
Engagement Model
Revenue / Cost / Risk
Impact
TCO / ROI
Executive Buyer
Kai Waehner
Field CTO
contact@kai-waehner.de
@KaiWaehner
www.kai-waehner.de
www.confluent.io
LinkedIn
Questions? Feedback?
Let’s connect!

Contenu connexe

Tendances

Apache Kafka in Financial Services - Use Cases and Architectures
Apache Kafka in Financial Services - Use Cases and ArchitecturesApache Kafka in Financial Services - Use Cases and Architectures
Apache Kafka in Financial Services - Use Cases and ArchitecturesKai Wähner
 
When NOT to use Apache Kafka?
When NOT to use Apache Kafka?When NOT to use Apache Kafka?
When NOT to use Apache Kafka?Kai Wähner
 
Streaming all over the world Real life use cases with Kafka Streams
Streaming all over the world  Real life use cases with Kafka StreamsStreaming all over the world  Real life use cases with Kafka Streams
Streaming all over the world Real life use cases with Kafka Streamsconfluent
 
Top use cases for 2022 with Data in Motion and Apache Kafka
Top use cases for 2022 with Data in Motion and Apache KafkaTop use cases for 2022 with Data in Motion and Apache Kafka
Top use cases for 2022 with Data in Motion and Apache Kafkaconfluent
 
The Rise Of Event Streaming – Why Apache Kafka Changes Everything
The Rise Of Event Streaming – Why Apache Kafka Changes EverythingThe Rise Of Event Streaming – Why Apache Kafka Changes Everything
The Rise Of Event Streaming – Why Apache Kafka Changes EverythingKai Wähner
 
Microservices in the Apache Kafka Ecosystem
Microservices in the Apache Kafka EcosystemMicroservices in the Apache Kafka Ecosystem
Microservices in the Apache Kafka Ecosystemconfluent
 
Kafka Tutorial - Introduction to Apache Kafka (Part 1)
Kafka Tutorial - Introduction to Apache Kafka (Part 1)Kafka Tutorial - Introduction to Apache Kafka (Part 1)
Kafka Tutorial - Introduction to Apache Kafka (Part 1)Jean-Paul Azar
 
Scaling a Core Banking Engine Using Apache Kafka | Peter Dudbridge, Thought M...
Scaling a Core Banking Engine Using Apache Kafka | Peter Dudbridge, Thought M...Scaling a Core Banking Engine Using Apache Kafka | Peter Dudbridge, Thought M...
Scaling a Core Banking Engine Using Apache Kafka | Peter Dudbridge, Thought M...HostedbyConfluent
 
Can Apache Kafka Replace a Database?
Can Apache Kafka Replace a Database?Can Apache Kafka Replace a Database?
Can Apache Kafka Replace a Database?Kai Wähner
 
Kafka 101 and Developer Best Practices
Kafka 101 and Developer Best PracticesKafka 101 and Developer Best Practices
Kafka 101 and Developer Best Practicesconfluent
 
Apache Flink and what it is used for
Apache Flink and what it is used forApache Flink and what it is used for
Apache Flink and what it is used forAljoscha Krettek
 
What is Apache Kafka and What is an Event Streaming Platform?
What is Apache Kafka and What is an Event Streaming Platform?What is Apache Kafka and What is an Event Streaming Platform?
What is Apache Kafka and What is an Event Streaming Platform?confluent
 
CDC patterns in Apache Kafka®
CDC patterns in Apache Kafka®CDC patterns in Apache Kafka®
CDC patterns in Apache Kafka®confluent
 
ksqlDB - Stream Processing simplified!
ksqlDB - Stream Processing simplified!ksqlDB - Stream Processing simplified!
ksqlDB - Stream Processing simplified!Guido Schmutz
 
Streaming Data and Stream Processing with Apache Kafka
Streaming Data and Stream Processing with Apache KafkaStreaming Data and Stream Processing with Apache Kafka
Streaming Data and Stream Processing with Apache Kafkaconfluent
 
Building Data Pipelines in Python
Building Data Pipelines in PythonBuilding Data Pipelines in Python
Building Data Pipelines in PythonC4Media
 
Introduction to Apache Kafka
Introduction to Apache KafkaIntroduction to Apache Kafka
Introduction to Apache KafkaShiao-An Yuan
 
Apache Kafka - Martin Podval
Apache Kafka - Martin PodvalApache Kafka - Martin Podval
Apache Kafka - Martin PodvalMartin Podval
 
Apache Kafka 0.8 basic training - Verisign
Apache Kafka 0.8 basic training - VerisignApache Kafka 0.8 basic training - Verisign
Apache Kafka 0.8 basic training - VerisignMichael Noll
 
Securing Kafka
Securing Kafka Securing Kafka
Securing Kafka confluent
 

Tendances (20)

Apache Kafka in Financial Services - Use Cases and Architectures
Apache Kafka in Financial Services - Use Cases and ArchitecturesApache Kafka in Financial Services - Use Cases and Architectures
Apache Kafka in Financial Services - Use Cases and Architectures
 
When NOT to use Apache Kafka?
When NOT to use Apache Kafka?When NOT to use Apache Kafka?
When NOT to use Apache Kafka?
 
Streaming all over the world Real life use cases with Kafka Streams
Streaming all over the world  Real life use cases with Kafka StreamsStreaming all over the world  Real life use cases with Kafka Streams
Streaming all over the world Real life use cases with Kafka Streams
 
Top use cases for 2022 with Data in Motion and Apache Kafka
Top use cases for 2022 with Data in Motion and Apache KafkaTop use cases for 2022 with Data in Motion and Apache Kafka
Top use cases for 2022 with Data in Motion and Apache Kafka
 
The Rise Of Event Streaming – Why Apache Kafka Changes Everything
The Rise Of Event Streaming – Why Apache Kafka Changes EverythingThe Rise Of Event Streaming – Why Apache Kafka Changes Everything
The Rise Of Event Streaming – Why Apache Kafka Changes Everything
 
Microservices in the Apache Kafka Ecosystem
Microservices in the Apache Kafka EcosystemMicroservices in the Apache Kafka Ecosystem
Microservices in the Apache Kafka Ecosystem
 
Kafka Tutorial - Introduction to Apache Kafka (Part 1)
Kafka Tutorial - Introduction to Apache Kafka (Part 1)Kafka Tutorial - Introduction to Apache Kafka (Part 1)
Kafka Tutorial - Introduction to Apache Kafka (Part 1)
 
Scaling a Core Banking Engine Using Apache Kafka | Peter Dudbridge, Thought M...
Scaling a Core Banking Engine Using Apache Kafka | Peter Dudbridge, Thought M...Scaling a Core Banking Engine Using Apache Kafka | Peter Dudbridge, Thought M...
Scaling a Core Banking Engine Using Apache Kafka | Peter Dudbridge, Thought M...
 
Can Apache Kafka Replace a Database?
Can Apache Kafka Replace a Database?Can Apache Kafka Replace a Database?
Can Apache Kafka Replace a Database?
 
Kafka 101 and Developer Best Practices
Kafka 101 and Developer Best PracticesKafka 101 and Developer Best Practices
Kafka 101 and Developer Best Practices
 
Apache Flink and what it is used for
Apache Flink and what it is used forApache Flink and what it is used for
Apache Flink and what it is used for
 
What is Apache Kafka and What is an Event Streaming Platform?
What is Apache Kafka and What is an Event Streaming Platform?What is Apache Kafka and What is an Event Streaming Platform?
What is Apache Kafka and What is an Event Streaming Platform?
 
CDC patterns in Apache Kafka®
CDC patterns in Apache Kafka®CDC patterns in Apache Kafka®
CDC patterns in Apache Kafka®
 
ksqlDB - Stream Processing simplified!
ksqlDB - Stream Processing simplified!ksqlDB - Stream Processing simplified!
ksqlDB - Stream Processing simplified!
 
Streaming Data and Stream Processing with Apache Kafka
Streaming Data and Stream Processing with Apache KafkaStreaming Data and Stream Processing with Apache Kafka
Streaming Data and Stream Processing with Apache Kafka
 
Building Data Pipelines in Python
Building Data Pipelines in PythonBuilding Data Pipelines in Python
Building Data Pipelines in Python
 
Introduction to Apache Kafka
Introduction to Apache KafkaIntroduction to Apache Kafka
Introduction to Apache Kafka
 
Apache Kafka - Martin Podval
Apache Kafka - Martin PodvalApache Kafka - Martin Podval
Apache Kafka - Martin Podval
 
Apache Kafka 0.8 basic training - Verisign
Apache Kafka 0.8 basic training - VerisignApache Kafka 0.8 basic training - Verisign
Apache Kafka 0.8 basic training - Verisign
 
Securing Kafka
Securing Kafka Securing Kafka
Securing Kafka
 

Similaire à Top 5 Event Streaming Use Cases for 2021 with Apache Kafka

App modernization on AWS with Apache Kafka and Confluent Cloud
App modernization on AWS with Apache Kafka and Confluent CloudApp modernization on AWS with Apache Kafka and Confluent Cloud
App modernization on AWS with Apache Kafka and Confluent CloudKai Wähner
 
Apache Kafka as Event Streaming Platform for Microservice Architectures
Apache Kafka as Event Streaming Platform for Microservice ArchitecturesApache Kafka as Event Streaming Platform for Microservice Architectures
Apache Kafka as Event Streaming Platform for Microservice ArchitecturesKai Wähner
 
Data In Motion Paris 2023
Data In Motion Paris 2023Data In Motion Paris 2023
Data In Motion Paris 2023confluent
 
Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...
Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...
Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...HostedbyConfluent
 
Kappa vs Lambda Architectures and Technology Comparison
Kappa vs Lambda Architectures and Technology ComparisonKappa vs Lambda Architectures and Technology Comparison
Kappa vs Lambda Architectures and Technology ComparisonKai Wähner
 
Architecture patterns for distributed, hybrid, edge and global Apache Kafka d...
Architecture patterns for distributed, hybrid, edge and global Apache Kafka d...Architecture patterns for distributed, hybrid, edge and global Apache Kafka d...
Architecture patterns for distributed, hybrid, edge and global Apache Kafka d...Kai Wähner
 
Apache Kafka vs. Traditional Middleware (Kai Waehner, Confluent) Frankfurt 20...
Apache Kafka vs. Traditional Middleware (Kai Waehner, Confluent) Frankfurt 20...Apache Kafka vs. Traditional Middleware (Kai Waehner, Confluent) Frankfurt 20...
Apache Kafka vs. Traditional Middleware (Kai Waehner, Confluent) Frankfurt 20...confluent
 
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...confluent
 
Supply Chain Optimization with Apache Kafka
Supply Chain Optimization with Apache KafkaSupply Chain Optimization with Apache Kafka
Supply Chain Optimization with Apache KafkaKai Wähner
 
Beyond the Brokers: A Tour of the Kafka Ecosystem
Beyond the Brokers: A Tour of the Kafka EcosystemBeyond the Brokers: A Tour of the Kafka Ecosystem
Beyond the Brokers: A Tour of the Kafka Ecosystemconfluent
 
Beyond the brokers - A tour of the Kafka ecosystem
Beyond the brokers - A tour of the Kafka ecosystemBeyond the brokers - A tour of the Kafka ecosystem
Beyond the brokers - A tour of the Kafka ecosystemDamien Gasparina
 
Beyond the brokers - Un tour de l'écosystème Kafka
Beyond the brokers - Un tour de l'écosystème KafkaBeyond the brokers - Un tour de l'écosystème Kafka
Beyond the brokers - Un tour de l'écosystème KafkaFlorent Ramiere
 
Au delà des brokers, un tour de l’environnement Kafka | Florent Ramière
Au delà des brokers, un tour de l’environnement Kafka | Florent RamièreAu delà des brokers, un tour de l’environnement Kafka | Florent Ramière
Au delà des brokers, un tour de l’environnement Kafka | Florent Ramièreconfluent
 
IoT Architectures for Apache Kafka and Event Streaming - Industry 4.0, Digita...
IoT Architectures for Apache Kafka and Event Streaming - Industry 4.0, Digita...IoT Architectures for Apache Kafka and Event Streaming - Industry 4.0, Digita...
IoT Architectures for Apache Kafka and Event Streaming - Industry 4.0, Digita...Kai Wähner
 
Building event-driven (Micro)Services with Apache Kafka Ecosystem
Building event-driven (Micro)Services with Apache Kafka EcosystemBuilding event-driven (Micro)Services with Apache Kafka Ecosystem
Building event-driven (Micro)Services with Apache Kafka EcosystemGuido Schmutz
 
IoT and Event Streaming at Scale with Apache Kafka
IoT and Event Streaming at Scale with Apache KafkaIoT and Event Streaming at Scale with Apache Kafka
IoT and Event Streaming at Scale with Apache Kafkaconfluent
 
Apache Kafka + Apache Mesos + Kafka Streams - Highly Scalable Streaming Micro...
Apache Kafka + Apache Mesos + Kafka Streams - Highly Scalable Streaming Micro...Apache Kafka + Apache Mesos + Kafka Streams - Highly Scalable Streaming Micro...
Apache Kafka + Apache Mesos + Kafka Streams - Highly Scalable Streaming Micro...Kai Wähner
 
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB)
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB)Apache Kafka vs. Integration Middleware (MQ, ETL, ESB)
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB)Kai Wähner
 
Azure Containers & Serverless Technology Options (After-Tech-Summit-2018 Edit...
Azure Containers & Serverless Technology Options (After-Tech-Summit-2018 Edit...Azure Containers & Serverless Technology Options (After-Tech-Summit-2018 Edit...
Azure Containers & Serverless Technology Options (After-Tech-Summit-2018 Edit...Yoichi Kawasaki
 
Introduction to Apache Kafka and why it matters - Madrid
Introduction to Apache Kafka and why it matters - MadridIntroduction to Apache Kafka and why it matters - Madrid
Introduction to Apache Kafka and why it matters - MadridPaolo Castagna
 

Similaire à Top 5 Event Streaming Use Cases for 2021 with Apache Kafka (20)

App modernization on AWS with Apache Kafka and Confluent Cloud
App modernization on AWS with Apache Kafka and Confluent CloudApp modernization on AWS with Apache Kafka and Confluent Cloud
App modernization on AWS with Apache Kafka and Confluent Cloud
 
Apache Kafka as Event Streaming Platform for Microservice Architectures
Apache Kafka as Event Streaming Platform for Microservice ArchitecturesApache Kafka as Event Streaming Platform for Microservice Architectures
Apache Kafka as Event Streaming Platform for Microservice Architectures
 
Data In Motion Paris 2023
Data In Motion Paris 2023Data In Motion Paris 2023
Data In Motion Paris 2023
 
Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...
Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...
Applying ML on your Data in Motion with AWS and Confluent | Joseph Morais, Co...
 
Kappa vs Lambda Architectures and Technology Comparison
Kappa vs Lambda Architectures and Technology ComparisonKappa vs Lambda Architectures and Technology Comparison
Kappa vs Lambda Architectures and Technology Comparison
 
Architecture patterns for distributed, hybrid, edge and global Apache Kafka d...
Architecture patterns for distributed, hybrid, edge and global Apache Kafka d...Architecture patterns for distributed, hybrid, edge and global Apache Kafka d...
Architecture patterns for distributed, hybrid, edge and global Apache Kafka d...
 
Apache Kafka vs. Traditional Middleware (Kai Waehner, Confluent) Frankfurt 20...
Apache Kafka vs. Traditional Middleware (Kai Waehner, Confluent) Frankfurt 20...Apache Kafka vs. Traditional Middleware (Kai Waehner, Confluent) Frankfurt 20...
Apache Kafka vs. Traditional Middleware (Kai Waehner, Confluent) Frankfurt 20...
 
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or ...
 
Supply Chain Optimization with Apache Kafka
Supply Chain Optimization with Apache KafkaSupply Chain Optimization with Apache Kafka
Supply Chain Optimization with Apache Kafka
 
Beyond the Brokers: A Tour of the Kafka Ecosystem
Beyond the Brokers: A Tour of the Kafka EcosystemBeyond the Brokers: A Tour of the Kafka Ecosystem
Beyond the Brokers: A Tour of the Kafka Ecosystem
 
Beyond the brokers - A tour of the Kafka ecosystem
Beyond the brokers - A tour of the Kafka ecosystemBeyond the brokers - A tour of the Kafka ecosystem
Beyond the brokers - A tour of the Kafka ecosystem
 
Beyond the brokers - Un tour de l'écosystème Kafka
Beyond the brokers - Un tour de l'écosystème KafkaBeyond the brokers - Un tour de l'écosystème Kafka
Beyond the brokers - Un tour de l'écosystème Kafka
 
Au delà des brokers, un tour de l’environnement Kafka | Florent Ramière
Au delà des brokers, un tour de l’environnement Kafka | Florent RamièreAu delà des brokers, un tour de l’environnement Kafka | Florent Ramière
Au delà des brokers, un tour de l’environnement Kafka | Florent Ramière
 
IoT Architectures for Apache Kafka and Event Streaming - Industry 4.0, Digita...
IoT Architectures for Apache Kafka and Event Streaming - Industry 4.0, Digita...IoT Architectures for Apache Kafka and Event Streaming - Industry 4.0, Digita...
IoT Architectures for Apache Kafka and Event Streaming - Industry 4.0, Digita...
 
Building event-driven (Micro)Services with Apache Kafka Ecosystem
Building event-driven (Micro)Services with Apache Kafka EcosystemBuilding event-driven (Micro)Services with Apache Kafka Ecosystem
Building event-driven (Micro)Services with Apache Kafka Ecosystem
 
IoT and Event Streaming at Scale with Apache Kafka
IoT and Event Streaming at Scale with Apache KafkaIoT and Event Streaming at Scale with Apache Kafka
IoT and Event Streaming at Scale with Apache Kafka
 
Apache Kafka + Apache Mesos + Kafka Streams - Highly Scalable Streaming Micro...
Apache Kafka + Apache Mesos + Kafka Streams - Highly Scalable Streaming Micro...Apache Kafka + Apache Mesos + Kafka Streams - Highly Scalable Streaming Micro...
Apache Kafka + Apache Mesos + Kafka Streams - Highly Scalable Streaming Micro...
 
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB)
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB)Apache Kafka vs. Integration Middleware (MQ, ETL, ESB)
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB)
 
Azure Containers & Serverless Technology Options (After-Tech-Summit-2018 Edit...
Azure Containers & Serverless Technology Options (After-Tech-Summit-2018 Edit...Azure Containers & Serverless Technology Options (After-Tech-Summit-2018 Edit...
Azure Containers & Serverless Technology Options (After-Tech-Summit-2018 Edit...
 
Introduction to Apache Kafka and why it matters - Madrid
Introduction to Apache Kafka and why it matters - MadridIntroduction to Apache Kafka and why it matters - Madrid
Introduction to Apache Kafka and why it matters - Madrid
 

Plus de Kai Wähner

Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)
Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)
Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)Kai Wähner
 
Kafka for Live Commerce to Transform the Retail and Shopping Metaverse
Kafka for Live Commerce to Transform the Retail and Shopping MetaverseKafka for Live Commerce to Transform the Retail and Shopping Metaverse
Kafka for Live Commerce to Transform the Retail and Shopping MetaverseKai Wähner
 
The Heart of the Data Mesh Beats in Real-Time with Apache Kafka
The Heart of the Data Mesh Beats in Real-Time with Apache KafkaThe Heart of the Data Mesh Beats in Real-Time with Apache Kafka
The Heart of the Data Mesh Beats in Real-Time with Apache KafkaKai Wähner
 
Apache Kafka vs. Cloud-native iPaaS Integration Platform Middleware
Apache Kafka vs. Cloud-native iPaaS Integration Platform MiddlewareApache Kafka vs. Cloud-native iPaaS Integration Platform Middleware
Apache Kafka vs. Cloud-native iPaaS Integration Platform MiddlewareKai Wähner
 
Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?
Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?
Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?Kai Wähner
 
Serverless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
Serverless Kafka and Spark in a Multi-Cloud Lakehouse ArchitectureServerless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
Serverless Kafka and Spark in a Multi-Cloud Lakehouse ArchitectureKai Wähner
 
Data Streaming with Apache Kafka in the Defence and Cybersecurity Industry
Data Streaming with Apache Kafka in the Defence and Cybersecurity IndustryData Streaming with Apache Kafka in the Defence and Cybersecurity Industry
Data Streaming with Apache Kafka in the Defence and Cybersecurity IndustryKai Wähner
 
Apache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare IndustryApache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare IndustryKai Wähner
 
Apache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare IndustryApache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare IndustryKai Wähner
 
Apache Kafka for Real-time Supply Chain in the Food and Retail Industry
Apache Kafka for Real-time Supply Chainin the Food and Retail IndustryApache Kafka for Real-time Supply Chainin the Food and Retail Industry
Apache Kafka for Real-time Supply Chain in the Food and Retail IndustryKai Wähner
 
Apache Kafka Landscape for Automotive and Manufacturing
Apache Kafka Landscape for Automotive and ManufacturingApache Kafka Landscape for Automotive and Manufacturing
Apache Kafka Landscape for Automotive and ManufacturingKai Wähner
 
Event Streaming CTO Roundtable for Cloud-native Kafka Architectures
Event Streaming CTO Roundtable for Cloud-native Kafka ArchitecturesEvent Streaming CTO Roundtable for Cloud-native Kafka Architectures
Event Streaming CTO Roundtable for Cloud-native Kafka ArchitecturesKai Wähner
 
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...Kai Wähner
 
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...Kai Wähner
 
Apache Kafka in the Transportation and Logistics
Apache Kafka in the Transportation and LogisticsApache Kafka in the Transportation and Logistics
Apache Kafka in the Transportation and LogisticsKai Wähner
 
Apache Kafka for Cybersecurity and SIEM / SOAR Modernization
Apache Kafka for Cybersecurity and SIEM / SOAR ModernizationApache Kafka for Cybersecurity and SIEM / SOAR Modernization
Apache Kafka for Cybersecurity and SIEM / SOAR ModernizationKai Wähner
 
Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4....
Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4....Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4....
Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4....Kai Wähner
 
Serverless Kafka on AWS as Part of a Cloud-native Data Lake Architecture
Serverless Kafka on AWS as Part of a Cloud-native Data Lake ArchitectureServerless Kafka on AWS as Part of a Cloud-native Data Lake Architecture
Serverless Kafka on AWS as Part of a Cloud-native Data Lake ArchitectureKai Wähner
 
IBM Cloud Pak for Integration with Confluent Platform powered by Apache Kafka
IBM Cloud Pak for Integration with Confluent Platform powered by Apache KafkaIBM Cloud Pak for Integration with Confluent Platform powered by Apache Kafka
IBM Cloud Pak for Integration with Confluent Platform powered by Apache KafkaKai Wähner
 
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?Kai Wähner
 

Plus de Kai Wähner (20)

Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)
Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)
Apache Kafka as Data Hub for Crypto, NFT, Metaverse (Beyond the Buzz!)
 
Kafka for Live Commerce to Transform the Retail and Shopping Metaverse
Kafka for Live Commerce to Transform the Retail and Shopping MetaverseKafka for Live Commerce to Transform the Retail and Shopping Metaverse
Kafka for Live Commerce to Transform the Retail and Shopping Metaverse
 
The Heart of the Data Mesh Beats in Real-Time with Apache Kafka
The Heart of the Data Mesh Beats in Real-Time with Apache KafkaThe Heart of the Data Mesh Beats in Real-Time with Apache Kafka
The Heart of the Data Mesh Beats in Real-Time with Apache Kafka
 
Apache Kafka vs. Cloud-native iPaaS Integration Platform Middleware
Apache Kafka vs. Cloud-native iPaaS Integration Platform MiddlewareApache Kafka vs. Cloud-native iPaaS Integration Platform Middleware
Apache Kafka vs. Cloud-native iPaaS Integration Platform Middleware
 
Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?
Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?
Data Warehouse vs. Data Lake vs. Data Streaming – Friends, Enemies, Frenemies?
 
Serverless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
Serverless Kafka and Spark in a Multi-Cloud Lakehouse ArchitectureServerless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
Serverless Kafka and Spark in a Multi-Cloud Lakehouse Architecture
 
Data Streaming with Apache Kafka in the Defence and Cybersecurity Industry
Data Streaming with Apache Kafka in the Defence and Cybersecurity IndustryData Streaming with Apache Kafka in the Defence and Cybersecurity Industry
Data Streaming with Apache Kafka in the Defence and Cybersecurity Industry
 
Apache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare IndustryApache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare Industry
 
Apache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare IndustryApache Kafka in the Healthcare Industry
Apache Kafka in the Healthcare Industry
 
Apache Kafka for Real-time Supply Chain in the Food and Retail Industry
Apache Kafka for Real-time Supply Chainin the Food and Retail IndustryApache Kafka for Real-time Supply Chainin the Food and Retail Industry
Apache Kafka for Real-time Supply Chain in the Food and Retail Industry
 
Apache Kafka Landscape for Automotive and Manufacturing
Apache Kafka Landscape for Automotive and ManufacturingApache Kafka Landscape for Automotive and Manufacturing
Apache Kafka Landscape for Automotive and Manufacturing
 
Event Streaming CTO Roundtable for Cloud-native Kafka Architectures
Event Streaming CTO Roundtable for Cloud-native Kafka ArchitecturesEvent Streaming CTO Roundtable for Cloud-native Kafka Architectures
Event Streaming CTO Roundtable for Cloud-native Kafka Architectures
 
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...
Apache Kafka in the Public Sector (Government, National Security, Citizen Ser...
 
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...
Telco 4.0 - Payment and FinServ Integration for Data in Motion with 5G and Ap...
 
Apache Kafka in the Transportation and Logistics
Apache Kafka in the Transportation and LogisticsApache Kafka in the Transportation and Logistics
Apache Kafka in the Transportation and Logistics
 
Apache Kafka for Cybersecurity and SIEM / SOAR Modernization
Apache Kafka for Cybersecurity and SIEM / SOAR ModernizationApache Kafka for Cybersecurity and SIEM / SOAR Modernization
Apache Kafka for Cybersecurity and SIEM / SOAR Modernization
 
Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4....
Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4....Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4....
Apache Kafka in the Automotive Industry (Connected Vehicles, Manufacturing 4....
 
Serverless Kafka on AWS as Part of a Cloud-native Data Lake Architecture
Serverless Kafka on AWS as Part of a Cloud-native Data Lake ArchitectureServerless Kafka on AWS as Part of a Cloud-native Data Lake Architecture
Serverless Kafka on AWS as Part of a Cloud-native Data Lake Architecture
 
IBM Cloud Pak for Integration with Confluent Platform powered by Apache Kafka
IBM Cloud Pak for Integration with Confluent Platform powered by Apache KafkaIBM Cloud Pak for Integration with Confluent Platform powered by Apache Kafka
IBM Cloud Pak for Integration with Confluent Platform powered by Apache Kafka
 
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?
Apache Kafka and API Management / API Gateway – Friends, Enemies or Frenemies?
 

Dernier

MinionLabs_Mr. Gokul Srinivas_Young Entrepreneur
MinionLabs_Mr. Gokul Srinivas_Young EntrepreneurMinionLabs_Mr. Gokul Srinivas_Young Entrepreneur
MinionLabs_Mr. Gokul Srinivas_Young EntrepreneurPriyadarshini T
 
Building Generative AI-infused apps: what's possible and how to start
Building Generative AI-infused apps: what's possible and how to startBuilding Generative AI-infused apps: what's possible and how to start
Building Generative AI-infused apps: what's possible and how to startMaxim Salnikov
 
User Experience Designer | Kaylee Miller Resume
User Experience Designer | Kaylee Miller ResumeUser Experience Designer | Kaylee Miller Resume
User Experience Designer | Kaylee Miller ResumeKaylee Miller
 
VuNet software organisation powerpoint deck
VuNet software organisation powerpoint deckVuNet software organisation powerpoint deck
VuNet software organisation powerpoint deckNaval Singh
 
Telebu Social -Whatsapp Business API : Mastering Omnichannel Business Communi...
Telebu Social -Whatsapp Business API : Mastering Omnichannel Business Communi...Telebu Social -Whatsapp Business API : Mastering Omnichannel Business Communi...
Telebu Social -Whatsapp Business API : Mastering Omnichannel Business Communi...telebusocialmarketin
 
openEuler Community Overview - a presentation showing the current scale
openEuler Community Overview - a presentation showing the current scaleopenEuler Community Overview - a presentation showing the current scale
openEuler Community Overview - a presentation showing the current scaleShane Coughlan
 
Flutter the Future of Mobile App Development - 5 Crucial Reasons.pdf
Flutter the Future of Mobile App Development - 5 Crucial Reasons.pdfFlutter the Future of Mobile App Development - 5 Crucial Reasons.pdf
Flutter the Future of Mobile App Development - 5 Crucial Reasons.pdfMind IT Systems
 
Large Scale Architecture -- The Unreasonable Effectiveness of Simplicity
Large Scale Architecture -- The Unreasonable Effectiveness of SimplicityLarge Scale Architecture -- The Unreasonable Effectiveness of Simplicity
Large Scale Architecture -- The Unreasonable Effectiveness of SimplicityRandy Shoup
 
BATbern52 Swisscom's Journey into Data Mesh
BATbern52 Swisscom's Journey into Data MeshBATbern52 Swisscom's Journey into Data Mesh
BATbern52 Swisscom's Journey into Data MeshBATbern
 
MUT4SLX: Extensions for Mutation Testing of Stateflow Models
MUT4SLX: Extensions for Mutation Testing of Stateflow ModelsMUT4SLX: Extensions for Mutation Testing of Stateflow Models
MUT4SLX: Extensions for Mutation Testing of Stateflow ModelsUniversity of Antwerp
 
Technical improvements. Reasons. Methods. Estimations. CJ
Technical improvements.  Reasons. Methods. Estimations. CJTechnical improvements.  Reasons. Methods. Estimations. CJ
Technical improvements. Reasons. Methods. Estimations. CJpolinaucc
 
BusinessGPT - SECURITY AND GOVERNANCE FOR GENERATIVE AI.pptx
BusinessGPT  - SECURITY AND GOVERNANCE  FOR GENERATIVE AI.pptxBusinessGPT  - SECURITY AND GOVERNANCE  FOR GENERATIVE AI.pptx
BusinessGPT - SECURITY AND GOVERNANCE FOR GENERATIVE AI.pptxAGATSoftware
 
03.2024_North America VMUG Optimizing RevOps using the power of ChatGPT in Ma...
03.2024_North America VMUG Optimizing RevOps using the power of ChatGPT in Ma...03.2024_North America VMUG Optimizing RevOps using the power of ChatGPT in Ma...
03.2024_North America VMUG Optimizing RevOps using the power of ChatGPT in Ma...jackiepotts6
 
Steps to Successfully Hire Ionic Developers
Steps to Successfully Hire Ionic DevelopersSteps to Successfully Hire Ionic Developers
Steps to Successfully Hire Ionic Developersmichealwillson701
 
CYBER SECURITY AND CYBER CRIME COMPLETE GUIDE.pLptx
CYBER SECURITY AND CYBER CRIME COMPLETE GUIDE.pLptxCYBER SECURITY AND CYBER CRIME COMPLETE GUIDE.pLptx
CYBER SECURITY AND CYBER CRIME COMPLETE GUIDE.pLptxBarakaMuyengi
 
Leveling Up your Branding and Mastering MERN: Fullstack WebDev
Leveling Up your Branding and Mastering MERN: Fullstack WebDevLeveling Up your Branding and Mastering MERN: Fullstack WebDev
Leveling Up your Branding and Mastering MERN: Fullstack WebDevpmgdscunsri
 
Unlocking the Power of IoT: A comprehensive approach to real-time insights
Unlocking the Power of IoT: A comprehensive approach to real-time insightsUnlocking the Power of IoT: A comprehensive approach to real-time insights
Unlocking the Power of IoT: A comprehensive approach to real-time insightsconfluent
 
Einstein Copilot Conversational AI for your CRM.pdf
Einstein Copilot Conversational AI for your CRM.pdfEinstein Copilot Conversational AI for your CRM.pdf
Einstein Copilot Conversational AI for your CRM.pdfCloudMetic
 
Mobile App Development company Houston
Mobile  App  Development  company HoustonMobile  App  Development  company Houston
Mobile App Development company Houstonjennysmithusa549
 
Mobile App Development process | Expert Tips
Mobile App Development process | Expert TipsMobile App Development process | Expert Tips
Mobile App Development process | Expert Tipsmichealwillson701
 

Dernier (20)

MinionLabs_Mr. Gokul Srinivas_Young Entrepreneur
MinionLabs_Mr. Gokul Srinivas_Young EntrepreneurMinionLabs_Mr. Gokul Srinivas_Young Entrepreneur
MinionLabs_Mr. Gokul Srinivas_Young Entrepreneur
 
Building Generative AI-infused apps: what's possible and how to start
Building Generative AI-infused apps: what's possible and how to startBuilding Generative AI-infused apps: what's possible and how to start
Building Generative AI-infused apps: what's possible and how to start
 
User Experience Designer | Kaylee Miller Resume
User Experience Designer | Kaylee Miller ResumeUser Experience Designer | Kaylee Miller Resume
User Experience Designer | Kaylee Miller Resume
 
VuNet software organisation powerpoint deck
VuNet software organisation powerpoint deckVuNet software organisation powerpoint deck
VuNet software organisation powerpoint deck
 
Telebu Social -Whatsapp Business API : Mastering Omnichannel Business Communi...
Telebu Social -Whatsapp Business API : Mastering Omnichannel Business Communi...Telebu Social -Whatsapp Business API : Mastering Omnichannel Business Communi...
Telebu Social -Whatsapp Business API : Mastering Omnichannel Business Communi...
 
openEuler Community Overview - a presentation showing the current scale
openEuler Community Overview - a presentation showing the current scaleopenEuler Community Overview - a presentation showing the current scale
openEuler Community Overview - a presentation showing the current scale
 
Flutter the Future of Mobile App Development - 5 Crucial Reasons.pdf
Flutter the Future of Mobile App Development - 5 Crucial Reasons.pdfFlutter the Future of Mobile App Development - 5 Crucial Reasons.pdf
Flutter the Future of Mobile App Development - 5 Crucial Reasons.pdf
 
Large Scale Architecture -- The Unreasonable Effectiveness of Simplicity
Large Scale Architecture -- The Unreasonable Effectiveness of SimplicityLarge Scale Architecture -- The Unreasonable Effectiveness of Simplicity
Large Scale Architecture -- The Unreasonable Effectiveness of Simplicity
 
BATbern52 Swisscom's Journey into Data Mesh
BATbern52 Swisscom's Journey into Data MeshBATbern52 Swisscom's Journey into Data Mesh
BATbern52 Swisscom's Journey into Data Mesh
 
MUT4SLX: Extensions for Mutation Testing of Stateflow Models
MUT4SLX: Extensions for Mutation Testing of Stateflow ModelsMUT4SLX: Extensions for Mutation Testing of Stateflow Models
MUT4SLX: Extensions for Mutation Testing of Stateflow Models
 
Technical improvements. Reasons. Methods. Estimations. CJ
Technical improvements.  Reasons. Methods. Estimations. CJTechnical improvements.  Reasons. Methods. Estimations. CJ
Technical improvements. Reasons. Methods. Estimations. CJ
 
BusinessGPT - SECURITY AND GOVERNANCE FOR GENERATIVE AI.pptx
BusinessGPT  - SECURITY AND GOVERNANCE  FOR GENERATIVE AI.pptxBusinessGPT  - SECURITY AND GOVERNANCE  FOR GENERATIVE AI.pptx
BusinessGPT - SECURITY AND GOVERNANCE FOR GENERATIVE AI.pptx
 
03.2024_North America VMUG Optimizing RevOps using the power of ChatGPT in Ma...
03.2024_North America VMUG Optimizing RevOps using the power of ChatGPT in Ma...03.2024_North America VMUG Optimizing RevOps using the power of ChatGPT in Ma...
03.2024_North America VMUG Optimizing RevOps using the power of ChatGPT in Ma...
 
Steps to Successfully Hire Ionic Developers
Steps to Successfully Hire Ionic DevelopersSteps to Successfully Hire Ionic Developers
Steps to Successfully Hire Ionic Developers
 
CYBER SECURITY AND CYBER CRIME COMPLETE GUIDE.pLptx
CYBER SECURITY AND CYBER CRIME COMPLETE GUIDE.pLptxCYBER SECURITY AND CYBER CRIME COMPLETE GUIDE.pLptx
CYBER SECURITY AND CYBER CRIME COMPLETE GUIDE.pLptx
 
Leveling Up your Branding and Mastering MERN: Fullstack WebDev
Leveling Up your Branding and Mastering MERN: Fullstack WebDevLeveling Up your Branding and Mastering MERN: Fullstack WebDev
Leveling Up your Branding and Mastering MERN: Fullstack WebDev
 
Unlocking the Power of IoT: A comprehensive approach to real-time insights
Unlocking the Power of IoT: A comprehensive approach to real-time insightsUnlocking the Power of IoT: A comprehensive approach to real-time insights
Unlocking the Power of IoT: A comprehensive approach to real-time insights
 
Einstein Copilot Conversational AI for your CRM.pdf
Einstein Copilot Conversational AI for your CRM.pdfEinstein Copilot Conversational AI for your CRM.pdf
Einstein Copilot Conversational AI for your CRM.pdf
 
Mobile App Development company Houston
Mobile  App  Development  company HoustonMobile  App  Development  company Houston
Mobile App Development company Houston
 
Mobile App Development process | Expert Tips
Mobile App Development process | Expert TipsMobile App Development process | Expert Tips
Mobile App Development process | Expert Tips
 

Top 5 Event Streaming Use Cases for 2021 with Apache Kafka

  • 1. The Top 5 Event Streaming Use Cases & Architectures in 2021 Hybrid Architectures, Edge Computing, Machine Learning, Cybersecurity, Service Mesh Kai Waehner Field CTO contact@kai-waehner.de LinkedIn @KaiWaehner www.confluent.io www.kai-waehner.de
  • 2. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Gartner Top Strategic Technology Trends for 2021 https://www.gartner.com/smarterwithgartner/gartner-top-strategic-technology-trends-for-2021/
  • 3. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent STREAM PROCESSING Create and store materialized views Filter Analyze in-flight Time C CC Event Streaming
  • 4. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Global Scale Real-time Persistent Storage Stream Processing Data Integration Apache Kafka The De-facto Standard for Real-Time Event Streaming Edge Cloud Data LakeDatabases Datacenter IoT SaaS AppsMobile Microservices Machine Learning Apache Kafka
  • 5. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Improve Customer Experience (CX) Increase Revenue (make money) Business Value Decrease Costs (save money) Core Business Platform Increase Operational Efficiency Migrate to Cloud Mitigate Risk (protect money) Key Drivers Strategic Objectives (sample) Fraud Detection IoT sensor ingestion Digital replatforming/ Mainframe Offload Connected Car: Navigation & improved in-car experience: Audi Customer 360 Simplifying Omni-channel Retail at Scale: Target Faster transactional processing / analysis incl. Machine Learning / AI Mainframe Offload: RBC Microservices Architecture Online Fraud Detection Online Security (syslog, log aggregation, Splunk replacement) Middleware replacement Regulatory Digital Transformation Application Modernization: Multiple Examples Website / Core Operations (Central Nervous System) The [Silicon Valley] Digital Natives; LinkedIn, Netflix, Uber, Yelp... Predictive Maintenance: Audi Streaming Platform in a regulated environment (e.g. Electronic Medical Records): Celmatix Real-time app updates Real Time Streaming Platform for Communications and Beyond: Capital One Developer Velocity - Building Stateful Financial Applications with Kafka Streams: Funding Circle Detect Fraud & Prevent Fraud in Real Time: PayPal Kafka as a Service - A Tale of Security and Multi-Tenancy: Apple Example Use Cases $↑ $↓ $↔ Example Case Studies (of many)
  • 6. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent The Top 5 Event Streaming Use Cases & Architectures in 2021 1) Hybrid Architectures 2) Edge Deployments Outside the Data Center 3) Service Mesh based Microservice Architectures 4) Streaming Machine Learning 5) Next-Generation Cybersecurity
  • 7. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent The Top 5 Event Streaming Use Cases & Architectures in 2021 1) Hybrid Architectures 2) Edge Deployments Outside the Data Center 3) Service Mesh based Microservice Architectures 4) Streaming Machine Learning 5) Next-Generation Cybersecurity
  • 8. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Why Kafka in Multiple Data Centers? * Not a representative survey J ** ‘Many DCs’ does NOT necessarily mean more than one Kafka Cluster
  • 9. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Disaster Recovery – RPO and RTO RPO = Recovery Point Objective RTO = Recovery Time Objective
  • 10. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Disaster Recovery @ JPMorgan https://www.confluent.io/kafka-summit-san-francisco-2019/secure-kafka-at-scale-in-true-multi-tenant-environment
  • 11. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Cluster Linking • Hybrid-cloud and multi-cloud • No additional infrastructure (such as Kafka Connect or MirrorMaker) • Just configuration • Regional or global
  • 12. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent What is the right Hybrid Kafka Architecture for you? (Hint: This is hard à Let’s guide you by our experts) 12 Latency > 50ms Latency < 50ms RTO = 0 RTO > 0 RPO = 0 RPO > 0 Single Region Multi-Region Global Stretched Cluster x x x x Replicator x x x x* x Cluster-Linking x x x x* x MRC Sync x x x x** MRC Observer x x x x** * With a stretched cluster in a single region, you still have RTO & RPO = 0 ** Requires 3 regions minimum
  • 13. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Kafka as a Service – Fully Managed? Infrastructure management (commodity) Scaling ● Upgrades (latest stable version of Kafka) ● Patching ● Maintenance ● Sizing (retention, latency, throughput, storage, etc.) ● Data balancing for optimal performance ● Performance tuning for real-time and latency requirements ● Fixing Kafka bugs ● Uptime monitoring and proactive remediation of issues ● Recovery support from data corruption ● Scaling the cluster as needed ● Data balancing the cluster as nodes are added ● Support for any Kafka issue with less than X minutes response time Infra-as-a-Service Harness full power of Kafka Kafka-specific management Platform-as-a-Service Evolve as you need Future-proof Mission-critical reliability Most Kafka-as-a-Service offerings are partially-managed Kafka as a Service should be a serverless experience with consumption-based pricing!
  • 14. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent The Top 5 Event Streaming Use Cases & Architectures in 2021 1) Hybrid Architectures 2) Edge Deployments Outside the Data Center 3) Service Mesh based Microservice Architectures 4) Streaming Machine Learning 5) Next-Generation Cybersecurity
  • 15. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent What is the “Edge” for Kafka? • Edge is NOT a data center • Kafka clients AND the Kafka broker(s) • Offline business continuity • Often 100+ locations • Low-footprint and low-touch • Hybrid integration Example: Single broker, 1 GB Ram 100 MB/sec
  • 16. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent CRM 3rd party payment provider Context-specific real-time upsell Customer data Payment processing and fraud detection as a service Manager Get report API Customer Customer Customer data Train schedule Payment data Loyalty information Streams of real time events Customer data Train schedule Payment data Loyalty information Streams of real time events Customer data Train schedule Payment data Loyalty information Streams of real time events Hybrid Architecture
  • 17. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Food Inventory Loyalty System Traveler Information Orders Upsell to first class Customer data Train schedule Payment data Loyalty information Streams of real time events Updated SchedulesEvent Streaming at the Edge
  • 18. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Data Processing at the Edge Time P C1 C2 C3 Know-your-customer Loyalty app, predictive behavior, … Estimated time of arrival Connect to the gaming server for kids Play games, earn rewards, communicate with other kids in the train, … Always on (even “offline”) Replayability Reduced traffic cost Better latency
  • 19. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Edge Kafka @ Royal Caribbean https://www.confluent.io/kafka-summit-lon19/seamless-guest-experience-with-kafka-streams/
  • 20. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Edge Integration and Analytics @ WPX Energy Edge processing and replication to the cloud in real-time at scale in the oil&gas industry
  • 21. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent The Top 5 Event Streaming Use Cases & Architectures in 2021 1) Hybrid Architectures 2) Edge Deployments Outside the Data Center 3) Service Mesh based Microservice Architectures 4) Streaming Machine Learning 5) Next-Generation Cybersecurity
  • 22. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Microservices to the rescue? • Significant Operations Overhead • Substantial DevOps Skills Required • Implicit Interfaces • Duplication Of Effort http://highscalability.com/blog/2014/4/8/microservices-not-a-free-lunch.html • Distributed System Complexity • Asynchronicity Is Difficult • Testability Challenges 22
  • 23. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Microservices can lead to Death-star Architectures Netflix: https://www.slideshare.net/brucewong3/the-case-for-chaos Twitter: https://twitter.com/adrianco/status/441883572618948608 Hail-o: http://www.sudo.hailoapp.com/services/2015/03/09/journey-into-a-microservice-world-part-3/ 450+ microservices 500+ microservices 500+ microservices 23
  • 24. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Key Requirements for Microservices •Decoupled •Flexible •Operationally Transparent •Data Aware •Elastic 24
  • 25. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Kafka Connect Kafka Cluster CRM Integration Domain-Driven Design (DDD) for your Microservice Architecture Legacy Integration Custom Application ESB Connector Java / KSQL / Kafka Streams Schema Registry Event Streaming Platform CRM Domain Legacy Domain Payment Domain è Independent and loosely coupled, but scalable, highly available and reliable! 25
  • 26. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Key Requirements for Microservices •Decoupled •Flexible •Operationally Transparent •Data Aware •Elastic 26
  • 27. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Cloud-Native Deployment leveraging Kubernetes 27
  • 28. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Service Mesh A microservice pattern to move visibility, reliability, and security primitives for service-to-service communication into the infrastructure layer, out of the application layer. https://www.infoq.com/articles/linkerd-v2-production-adoption/ 28
  • 29. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Service Proxy Features • Metrics without instrumenting apps • Trace flow of requests across services • One stable URI for each service • Service discovery • Monitor request latency • Routing - A/B testing, green/blue deployments • Circuit breaking • Protocol translation (HTTP, gRPC, Kafka Protocol, etc.) • Mutual TLS (mTLS) • SSL Termination • Integrate with 3rd party tools like Prometheus, Grafana, Zipkin, etc. • Much more… Observability “is by far the most important thing that a Proxy and the Service Mesh provide in a distributed Microservice architecture!” Matt Klein 29
  • 30. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Example: Lyft today with “envoy” Proxy • 100% (!!!) communication coverage - Everything talks through Envoy Proxies • à Make monitoring, debugging, firefighting as consistent as possible https://www.youtube.com/watch?v=55yi4MMVBi4 Matt Klein at QCon NY 2018 30
  • 31. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Kafka Connect Kafka Cluster CRM Integration Clients and Servers are Independent (including their Ops Teams) Legacy Integration Custom Application ESB Connector Java / KSQL / Kafka Streams Schema Registry Event Streaming Platform CRM Domain Legacy Domain Payment Domain Proxy Proxy Proxy Proxy Proxy Proxy Control Plane 31
  • 32. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Example: Kafka + Confluent REST Proxy Envoy Proxy I am using REST too! Kafka? Never heard of her. I’m using REST to talk to a service I’m proxying REST. And also logging stuff to Kafka Confluent REST Proxy I support only TCP! HTTP HTTP 32
  • 33. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Example: Kafka + Envoy Kafka Protocol Filter Envoy Proxy I am using REST too! Kafka? Never heard of her. I’m using REST to talk to a service I’m proxying REST. And also logging stuff to Kafka HTTP TCP (Kafka Protocol) 33
  • 34. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Implementation: Kafka + Istio @ Banzai Cloud 34 https://banzaicloud.com/blog/kafka-on-istio-performance/
  • 35. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Implementation: Kafka + Istio 35 https://banzaicloud.com/blog/kafka-on-istio-performance/
  • 36. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Implementation: Kafka + ksqlDB + Istio 36 https://banzaicloud.com/blog/supertubes-ksql/
  • 37. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent (Potential) Features for Kafka + Service Mesh Implementation Protocol conversion from HTTP / gRPC to Kafka • Tap feature to dump to a Kafka stream • Protocol parsing for observability (stats, logging, and trace linking with HTTP RPCs) • Shadow requests to a Kafka stream instead of HTTP / gRPC shadow • Integrate with Kafka Connect and its whole ecosystem of connectors Validation of Events • Serialization format (JSON, Avro, Protobuf, etc.) • Message schema • Headers, attributes, etc. Security • SSL Termination • Mutual TLS (mTLS) • Authorization Proxy features • Dynamic Routing • Rate limiting at both the L4 connection and L7 message level • Filter, add compression, … • Automatic topic name conversion (e.g. for canary release or blue/green deployment) Monitoring and Tracing • Request logs and stats • Data lineage / audit log • Audit log by taking request logs and enriching them with the user info. • Client specific metrics (Byte rate per client id / per consumer groups, versions of the client libraries, consumer lag monitoring for the entire data center) 37
  • 38. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Key Requirements for Microservices •Decoupled •Flexible •Operationally Transparent •Data Aware •Elastic 38
  • 39. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent The Top 5 Event Streaming Use Cases & Architectures in 2021 1) Hybrid Architectures 2) Edge Deployments Outside the Data Center 3) Service Mesh based Microservice Architectures 4) Streaming Machine Learning 5) Next-Generation Cybersecurity
  • 40. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Apache Kafka as Infrastructure for ML
  • 41. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Apache Kafka’s Open Ecosystem as Infrastructure for ML Kafka Streams/ ksqlDB Kafka Connect Confluent REST Proxy Confluent Schema Registry Go/.NET/Python Kafka Producer ksqlDB Python Client
  • 42. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Streaming Analytics for Predictive Maintenance at Scale 42 IoT Integration Layer Batch Analytics Platform BI Dashboard Streaming Platform Big Data Integration Layer Car Sensors Streaming Platform Other Components Real Time Monitoring System All Data Critical Data Ingest Data Potential Detect
  • 43. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Streaming Analytics for Predictive Maintenance at Scale 43 IoT Integration Layer Batch Analytics Platform BI Dashboard Streaming Platform Big Data Integration Layer Car Sensors Streaming Platform Analytics Platform Other Components Real Time Monitoring System All Data Critical Data Ingest Data Potential DetectAnalytics Platform Train Analytic Model Data Processing Analytic Model Preprocess Data Consume Data Deploy Analytic Model
  • 44. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Preprocessing with ksqlDB 44 SELECT car_id, event_id, car_model_id, sensor_input FROM car_sensor c LEFT JOIN car_models m ON c.car_model_id = m.car_model_id WHERE m.car_model_type ='Audi_A8';
  • 45. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Direct streaming ingestion for model training with TensorFlow I/O + Kafka Plugin (no additional data storage like S3 or HDFS required!) Time Model BModel A Producer Distributed Commit Log Streaming Ingestion and Model Training with TensorFlow IO https://github.com/tensorflow/io 45 Model X (at a later time)
  • 46. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Confluent Tiered Storage for Kafka 46 (Only available in Confluent Platform)
  • 47. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Local Predictions Model Training in the Cloud Model Deployment at the Edge Analytic Model Separation of Model Training and Model Inference 47
  • 48. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent “CREATE STREAM AnomalyDetection AS SELECT sensor_id, detectAnomaly(sensor_values) FROM car_engine;“ User Defined Function (UDF) Model Deployment with Apache Kafka, ksqlDB and TensorFlow 48
  • 49. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent The Top 5 Event Streaming Use Cases & Architectures in 2021 1) Hybrid Architectures 2) Edge Deployments Outside the Data Center 3) Service Mesh based Microservice Architectures 4) Streaming Machine Learning 5) Next-Generation Cybersecurity
  • 50. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Cybersecurity The threat is real! Challenges Stealing IP DDoS Ransomware / wiperware WannaCry, NotPetya, … Damage: Billions of dollars ”Supply chain attack” Digital Transformation Networking Communication Connectivity Open standards ”Always-on”
  • 51. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Legacy SIEM needs to evolve ForwarderNetwork traffic Firewall logs RDBMS Application logs Adaptors Beats Sensor Data Challenges: ● Proprietary forwarders that can only send data to single source ● Data locked from being shared ● Difficult to scale with growing data volumes ● Prohibitively high indexing costs ● Unable to filter out noisy data ● Slow batch processing HTTP proxy logs
  • 52. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent AI/ML Modernized security information and event management (SIEM) Filter, transform aggregate APP SIEM Index Search Curated streams Forensic Archive HDFS S3 Big Query CDC Syslog Network traffic Firewall logs RDBMS Application logs Payment Data HTTP proxy logs QRadar Arcsight Splunk Elastic APP Stateful real-time analytics
  • 53. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Cyber Intelligence Platform leveraging Kafka Connect, Kafka Streams, Multi-Region Clusters (MRC), and more… https://www.intel.com/content/www/us/en/it-management/intel-it-best-practices/modern-scalable-cyber-intelligence-platform-kafka.html
  • 54. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Fraud Detection at Scale in Real-Time for Billions of Messages https://www.infoq.com/presentations/paypal-data-service-fraud https://conferences.oreilly.com/strata/strata-ny-2018/public/schedule/detail/69459.html
  • 55. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent How does Confluent help?
  • 56. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent The Rise of Event Streaming 2010 Apache Kafka created at LinkedIn by Confluent founders 2014 2020 80% Fortune 100 Companies trust and use Apache Kafka
  • 57. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent I N V E S T M E N T & T I M E VALUE 3 4 5 1 2 Event Streaming Maturity Model Initial Awareness / Pilot (1 Kafka Cluster) Start to Build Pipeline / Deliver 1 New Outcome (1 Kafka Cluster) Mission-Critical Deployment (Stretched, Hybrid, Multi- Region) Build Contextual Event- Driven Apps (Stretched, Hybrid, Multi-Region) Central Nervous System (Global Kafka) Product, Support, Training, Partners, Technical Account Management...
  • 58. Real-Life Use Cases & Architectures for Event Streaming with Apache Kafka and Confluent Confluent Platform Fully Managed Cloud ServiceSelf Managed Software FREEDOM OF CHOICE COMMITTER-DRIVEN EXPERTISE PartnersTrainingProfessional Services Enterprise Support Apache Kafka EFFICIENT OPERATIONS AT SCALE PRODUCTION- STAGE PREREQUISITES UNRESTRICTED DEVELOPER PRODUCTIVITY SQL-based Stream Processing KSQL (ksqlDB) Rich Pre-built Ecosystem Connectors | Hub | Schema Registry Multi-language Development non-Java clients | REST Proxy GUI-driven Mgmt & Monitoring Control Center Flexible DevOps Automation Operator | Ansible Dynamic Performance & Elasticity Auto Data Balancer | Tiered Storage Enterprise-grade Security RBAC | Secrets | Audit logs Data Compatibility Schema Registry | Schema Validation Global Resilience Multi-Region Clusters | Replicator Developer Operator Architect Open Source | Community licensed PARTNERSHIP FOR BUSINESS SUCCESS Complete Engagement Model Revenue / Cost / Risk Impact TCO / ROI Executive Buyer