Gunnar Morling presents on change data streaming patterns for microservices using Debezium. Debezium is an open source platform for change data capture that retrieves change events from transaction logs of different databases. It streams these events to Apache Kafka in a unified format. This allows microservices to stay synchronized by consuming the change events and keeping their local data stores in sync without direct database access. Various patterns are demonstrated including microservice data synchronization, leveraging single message transformations, and ensuring data quality.

1. Change Data Streaming Patterns forChange Data Streaming Patterns for
Microservices With DebeziumMicroservices With Debezium
Gunnar MorlingGunnar Morling
Software Engineer

2. Gunnar MorlingGunnar Morling
Open source software engineer at Red Hat
Debezium
Hibernate
Spec Lead for Bean Validation 2.0
Other projects: Deptective, MapStruct
Java Champion
gunnar@hibernate.org
@gunnarmorling
http://in.relation.to/gunnar-morling/
#KafkaSummit #Debezium @gunnarmorling

3. A Common ProblemA Common Problem
Updating Multiple ResourcesUpdating Multiple Resources
@gunnarmorling
Database
Order
Service
#KafkaSummit #Debezium

4. A Common ProblemA Common Problem
Updating Multiple ResourcesUpdating Multiple Resources
@gunnarmorling
Cache
Database
Order
Service
#KafkaSummit #Debezium

5. A Common ProblemA Common Problem
Updating Multiple ResourcesUpdating Multiple Resources
@gunnarmorling
Cache
Database
Order
Service
Search
Index
#KafkaSummit #Debezium

6. A Common ProblemA Common Problem
Updating Multiple ResourcesUpdating Multiple Resources
@gunnarmorling
Order
Service
Cache
Database
Search
Index
“ Friends Don't Let Friends Do Dual Writes
#KafkaSummit #Debezium

7. A Better SolutionA Better Solution
Streaming Change Events From the DatabaseStreaming Change Events From the Database
@gunnarmorling
Order
Service
#KafkaSummit #Debezium

8. A Better SolutionA Better Solution
Streaming Change Events From the DatabaseStreaming Change Events From the Database
@gunnarmorling
Order
Service
C C U C U U D C
C - Create
U - Update
D - Delete
Change Data
Capture
#KafkaSummit #Debezium

9. A Better SolutionA Better Solution
Streaming Change Events From the DatabaseStreaming Change Events From the Database
@gunnarmorling
Order
Service
C C U C U U D C
C - Create
U - Update
D - Delete
Change Data
Capture
#KafkaSummit #Debezium

10. Change Data CaptureChange Data Capture
With DebeziumWith Debezium

11. DebeziumDebezium
Change Data Capture PlatformChange Data Capture Platform
Retrieves change events from TX logs from different DBs
Transparent to writing apps
Comprehensive type support (PostGIS etc.)
Snapshotting, Filtering etc.
Fully open-source, very active community
Latest version: 0.9 (based on Kafka 2.2)
Production deployments at multiple companies (e.g.
WePay, Trivago, BlaBlaCar etc.)
@gunnarmorling#KafkaSummit #Debezium

12. Advantages of Log-based CDCAdvantages of Log-based CDC
Tailing the transaction logTailing the transaction log
All data changes are captured
No polling delay or overhead
Transparent to writing applications and models
Can capture deletes
Can capture old record state and further meta data
Different formats/APIs, but Debezium deals with this
@gunnarmorling#KafkaSummit #Debezium

13. DebeziumDebezium
CDC Use CasesCDC Use Cases
Update or invalidate caches
Enable full-text search via Elasticsearch, Solr etc.
Data replication
Microservices data exchange
Auditing/historization
Update CQRS read models
Enable streaming queries
@gunnarmorling#KafkaSummit #Debezium

14. Change Event StructureChange Event Structure
Key: PK of table
Value: Describing the change event
Before state,
After state,
Source info
Serialization formats:
JSON
Avro
{
"before": null,
"after": {
"id": 1004,
"first_name": "Anne",
"last_name": "Kretchmar",
"email": "annek@noanswer.org"
},
"source": {
"name": "dbserver1",
"server_id": 0,
"ts_sec": 0,
"file": "mysql­bin.000003",
"pos": 154,
"row": 0,
"snapshot": true,
"db": "inventory",
"table": "customers"
},
"op": "c",
"ts_ms": 1486500577691
}
@gunnarmorling#KafkaSummit #Debezium

15. Debezium ConnectorsDebezium Connectors
MySQL
Postgres
MongoDB
SQL Server
Oracle (Tech Preview, based on XStream)
Possible future additions
Cassandra?
MariaDB?
@gunnarmorling#KafkaSummit #Debezium

16. @gunnarmorling
Postgres
MySQL
Apache Kafka
CDC with Debezium and Kafka ConnectCDC with Debezium and Kafka Connect
#KafkaSummit #Debezium

17. CDC with Debezium and Kafka ConnectCDC with Debezium and Kafka Connect
@gunnarmorling
Postgres
MySQL
Apache KafkaKafka Connect Kafka Connect
#KafkaSummit #Debezium

18. @gunnarmorling
Postgres
MySQL
Apache KafkaKafka Connect Kafka Connect
DBZ PG
DBZ
MySQL
CDC with Debezium and Kafka ConnectCDC with Debezium and Kafka Connect
#KafkaSummit #Debezium

19. @gunnarmorling
Postgres
MySQL
Kafka Connect Kafka ConnectApache Kafka
DBZ PG
DBZ
MySQL
Elasticsearch
ES
Connector
CDC with Debezium and Kafka ConnectCDC with Debezium and Kafka Connect
#KafkaSummit #Debezium

20. MicroserviceMicroservice
CDC PatternsCDC Patterns

21. @gunnarmorling
Order Item Stock
App
Local DB Local DB Local DB
App App
Item ChangesStock Changes
Pattern: Microservice DataPattern: Microservice Data
SynchronizationSynchronization
Microservice ArchitecturesMicroservice Architectures
Propagate data between different
services without coupling
Each service keeps
optimised views locally
#KafkaSummit #Debezium

22. Source DB
(with "Events" table)
Kafka Connect Apache Kafka
DBZ Order Events
Credit Worthiness Check Events
Pattern: OutboxPattern: Outbox
Separate Events TableSeparate Events Table
@gunnarmorling
Order
Service
Shipment
Service
Customer
Service
ID Category Type Payload
123 Order OrderCreated { "id" : 123, ... }
456 Order OrderDetail-
Canceled
{ "id" : 456, ... }
789 ... ... ...
Id AggregateType AggregateId Type Payload
ec6e Order 123 OrderCreated { "id" : 123, ... }
8af8 Order 456 OrderDetailCanceled { "id" : 456, ... }
890b Customer 789 InvoiceCreated { "id" : 789, ... }
"Outbox" table
#KafkaSummit #Debezium

23. Pattern: Microservice ExtractionPattern: Microservice Extraction
Migrating from Monoliths to MicroservicesMigrating from Monoliths to Microservices
Extract microservice for single component(s)
Keep write requests against running monolith
Stream changes to extracted microservice
Test new functionality
Switch over, evolve schema only afterwards
@gunnarmorling#KafkaSummit #Debezium

24. Pattern: Leverage the Powers of SMTsPattern: Leverage the Powers of SMTs
Single Message TransformationsSingle Message Transformations
Aggregate sharded tables to single topic
Keep compatibility with existing consumers
Format conversions, e.g. for dates
Ensure compatibility with sink connectors
Extracting "after" state only
Expand MongoDB's JSON structures
@gunnarmorling#KafkaSummit #Debezium

25. Pattern: Ensuring Data QualityPattern: Ensuring Data Quality
Detecting Missing or Wrong DataDetecting Missing or Wrong Data
Constantly compare record counts on source and sink side
Raise alert if threshold is reached
Compare every n-th record field by field
E.g. have all records compared within one week
@gunnarmorling#KafkaSummit #Debezium

26. DemoDemo

27. Running Debezium on KubernetesRunning Debezium on Kubernetes
AMQ Streams: Enterprise Distribution of Apache KafkaAMQ Streams: Enterprise Distribution of Apache Kafka
Provides
Container images for Apache Kafka, Connect, Zookeeper and MirrorMaker
Operators for managing/configuring Apache Kafka clusters, topics and users
Kafka Consumer, Producer and Admin clients, Kafka Streams
Supported by Red Hat
Upstream Community: Strimzi
@gunnarmorling#KafkaSummit #Debezium

28. Support for DebeziumSupport for Debezium
Red Hat IntegrationRed Hat Integration
Debezium is being productized as part of the Red Hat
Integration product
Initially Microsoft SQL Server, MySQL, PostgreSQL,
and MongoDB connectors
Integrated with AMQ Streams
Developer Preview to be released soon; general
availability (GA) planned for later this year
@gunnarmorling#KafkaSummit #Debezium

29. SummarySummary
CDC enables use cases such as replication, microservices
data exchange and much more
Debezium: CDC for a growing number of databases
Contributions welcome!
Tell us about your feature requests and ideas!
@gunnarmorling
“ Friends Don't Let Friends Do Dual Writes
#KafkaSummit #Debezium

30. ResourcesResources
Website:
Source code, examples, Compose files etc.
Discussion group
Strimzi (Kafka on Kubernetes/OpenShift)
Latest news: @debezium
http://debezium.io/
https://github.com/debezium
https://groups.google.com/forum/
#!forum/debezium
http://strimzi.io/
@gunnarmorling#KafkaSummit #Debezium