Two way data sync between legacy and your brand new micro-service architecture

1. Two way data sync between legacy
and your brand new micro-service
architecture

2. Brice LEPORINI
43 YO
CTO PALO IT France
Senior Developer
@blep

3. SMALL ENOUGH TO CARE, BIG ENOUGH TO DELIVER
Key figures & offices
350+
Experts Nationalities
30+
Turnover
30M
Organic Growth
+15%
Debt
0
Held by Managers
100%
Hong Kong
Singapore
Thailand
Bangkok
Mexico
Mexico City
Australia
Sydney
France
Paris, Nancy, Nantes, Toulouse
@blep

4. Replace a teenager monolith (15 YO)
Micro-service oriented architecture
Progressive migration
The mission
@blep

5. @blep

6. Microservices ? Why?
Distinct life cycle for each service
Ability to scale out specific service
@blep

7. Gradual migration
No big bang
Strangler pattern: features / bounded
contexts gradually replaced
Canary releases
@blep

8. The legacy and the micro-services
have to run side by side.
They can modify concurrently same
data
@blep

9. Schéma sync ms <-> legacy
@blep

10. Conflictless !
@blep

11. Periods in the migration of each context
● The new context is not fully effective for everyone → the legacy system is the master of
data
● The new context is battle tested → the corresponding features are no longer available in
the legacy but it may need to be notified of changes for other features
@blep

12. Remember how work databases
@blep

14. You can’t change what
happened.
But you can add an event
that cancels a previous one.
Events are immutable
What happens if a
cancelation event is
consumed before the event
it cancels!
Events are ordered
Strong coupling with the
snapshot will imply complex
migration during
application’s life.
Beware of tight coupling
your event model
@blep

15. Entity state in MS environment
=
Last entity found in the snapshot
+
Events without validation
@blep

16. Yes it is a kind of
CQRS + ES
usage
CQRS = Command Query Responsability Segregation
ES = Event Sourcing
@blep

17. Legacy —> Microservices snapshot synchronization
No interrogation allowed
Minimal latency
@blep

18. Legacy —> Microservices snapshot synchronization
Business events , however :
● Huge legacy code base, missing events may occur
● High costs
Database triggers :
● Run with each DML, high impact on performance and scalability
@blep

19. CDC : Change Data Capture
Logs every change that occurs in the database
Nothing to change in the legacy application
Technical options: Debezium, built in in MSSQL
Technical events
@blep

20. Consideration for CDC Usage in SQL Server
One update on one row ⇒ two CDC rows with previous state and new state
1000 on one row ⇒ 2000 CDC rows…
One bulk update …
Note:
● Beware on the tables you activate CDC: high traffic tables need your attention …
○ Performance
○ File system filling
● CDC tables should be stored in other disks than business tables
@blep

21. Retail example: order article count
select count(*) from tbl_item
where fkid_order=?
Item count may be inferred from the sum of
article prices compared to total order price
@blep

22. What if one or more items come with
0 as price?
@blep

23. This design is based on the
eventual consistency
principle which means that you may
be inconsistent for a short moment
(a few milliseconds?) but you will
finally be consistent
@blep

24. Keystone : no message can be lost
Apache Kafka:
● Data streaming platform
● Messages are persisted and replicated
● Delivery order is kept by partition
● Distributed, fault tolerant and horizontally scalable
@blep

25. @blep

26. Data path: changes coming from legacy
@blep

27. Data path: changes coming from microservices
@blep

28. Rollout procedure
@blep