@crichardson
Handling Eventual Consistency in
JVM Microservices with Event
Sourcing
Chris Richardson
Founder of Eventuate....
@crichardson
Presentation goal
Show how event sourcing is a great
foundation for a microservice
architecture
@crichardson
About Chris
http://learnmicroservices.io
@crichardson
About Kenny
@kennybastani
@crichardson
Agenda
Overview of event sourcing
The problem with microservices, transactions and queries
Using events to ma...
@crichardson
Traditional persistence
Order
id
state
….
101 ACCEPTED
Order table
…
ID STATE …
102 … …
@crichardson
But how did we get here?
Who did what and when?
What was the state of the
Order last Monday at 3:01pm?
@crichardson
Order example
History
@crichardson
Task example State
History
Audit
@crichardson
Usually
auditing, history and temporal
queries
is additional code and/or
an after-thought
@crichardson
Event sourcing
For each domain object (i.e. DDD aggregate):
Identify (state changing) domain events, e.g. use...
@crichardson
Persists events NOT current state
Event table
Entity type
Event
id
Entity
id
Event
data
Order 902101 …OrderAp...
@crichardson
Replay events to recreate
state
Order
state
OrderCreated(…)
OrderAccepted(…)
OrderShipped(…)
Events
Periodica...
@crichardson
Benefits of event sourcing
Reifies state changes:
Built in, reliable audit log
temporal queries
Preserved histo...
@crichardson
Drawbacks of event sourcing
Requires application rewrite
Weird and unfamiliar style of programming
Events liv...
@crichardson
Demo
@crichardson
Agenda
Overview of event sourcing
The problem with microservices, transactions and queries
Using events to ma...
@crichardson
The Microservice architecture
tackles complexity through
modularization
@crichardson
Microservice architecture
Browser
Mobile
Device
Store
Front UI
API
Gateway
Catalog
Service
Review
Service
Ord...
@crichardson
But there are challenges
implementing
transactions and queries
@crichardson
ACID transactions cannot be
used
BEGIN TRANSACTION
…
SELECT ORDER_TOTAL
FROM ORDERS WHERE CUSTOMER_ID = ?
…
S...
@crichardson
2PC is not a viable option
Guarantees consistency
BUT
2PC is best avoided
Not supported by many NoSQL databas...
@crichardson
Queries can’t use joins
SELECT *
FROM CUSTOMER c, ORDER o
WHERE
c.id = o.ID
AND o.ORDER_TOTAL > 100000
AND o....
@crichardson
Agenda
Overview of event sourcing
The problem with microservices, transactions and queries
Using events to ma...
@crichardson
Event-driven architecture
@crichardson
Order Management
Order
id : 4567
total: 343
state = CREATED
Customer Management
Customer
creditLimit : 12000
...
@crichardson
How atomically update
database and publish an event
Order Service
Order
Database
Message Broker
insert Order
...
@crichardson
Failure = inconsistent system
Order Service
Order
Database
Message Broker
insert Order
publish
OrderCreatedEv...
@crichardson
2PC is not an option
@crichardson
How to reliably publish
events when state changes?
@crichardson
Agenda
Overview of event sourcing
The problem with microservices, transactions and queries
Using events to ma...
@crichardson
Event sourcing = event-centric
persistence
Application
Database
Event store
update
publish
X
@crichardson
Event Store
Application architecture
Order 123 Customer 456
OrderCreated
OrderApproved
…
CustomerCreated
Cust...
@crichardson
Request handling in an event sourced application
HTTP
Handler
Event
Store
pastEvents = findEvents(entityId)
Or...
@crichardson
Event Store publishes events
consumed by other services
Event
Store
Event
Subscriber
subscribe(EventTypes)
pu...
@crichardson
Event Store publishes events
consumed by other services
Event
Store
Event
Subscriber
subscribe(EventTypes)
pu...
Event store = database + message
broker
Hybrid database and
message broker
Implementations:
Home grown/DIY
geteventstore.c...
@crichardson
Agenda
Overview of event sourcing
The problem with microservices, transactions and queries
Using events to ma...
@crichardson
Find recent, valuable
customers
SELECT *
FROM CUSTOMER c, ORDER o
WHERE
c.id = o.ID
AND o.ORDER_TOTAL > 10000...
@crichardson
Use Command Query
Responsibility Segregation
(CQRS)
@crichardson
Command Query Responsibility
Segregation (CQRS)
Application logic
Commands Queries
X
POST
PUT
DELETE
GET
@crichardson
Command Query Responsibility
Segregation (CQRS)
Command side
Commands
Event Sourcing
domain
objects
Event Sto...
@crichardson
Query side design
Event Store
Updater
View Updater
Service
Events
Reader
HTTP GET
Request
View Query
Service
...
@crichardson
Persisting a customer and
order history in MongoDB
{
"_id" : "0000014f9a45004b 0a00270000000000",
"_class" : ...
@crichardson
Persisting customers and order info
using Spring Data for MongoDB...
@crichardson
Persisting customers and order
using Spring Data for MongoDB...
Benefits and drawbacks of
CQRS
Benefits
Necessary in an event sourced
architecture
Separation of concerns =
simpler command ...
@crichardson
Demo
@crichardson
Summary
Microservice architecture functionally decomposes an
application into services
Transactions and queri...
@crichardson
chris@chrisrichardson.net kbastani@pivotal.io
http://learnmicroservices.io http://pivotal.io
Questions?
Prochain SlideShare
Chargement dans…5
×

Handling Eventual Consistency in JVM Microservices with Event Sourcing (javaone 2016)

7 518 vues

Publié le

This is the talk that the Kenny Bastani and I gave at JavaOne 2016.

When you’re building JVM applications in a microservice architecture, managing state becomes a distributed systems problem. Instead of being able to manage state as transactions inside the boundaries of a single monolithic application, a microservice must be able to manage consistency by using transactions that are distributed across a network of many different applications and databases. This session explores the problems of data consistency and high availability in JVM-based microservices and how to use event sourcing to solve these problems.

Publié dans : Logiciels

Handling Eventual Consistency in JVM Microservices with Event Sourcing (javaone 2016)

  1. 1. @crichardson Handling Eventual Consistency in JVM Microservices with Event Sourcing Chris Richardson Founder of Eventuate.io @crichardson http://eventuate.io Copyright © 2015. Chris Richardson Consulting, Inc. All rights reserved Kenny Bastani Spring Developer Advocate @kennybastani http://pivotal.io
  2. 2. @crichardson Presentation goal Show how event sourcing is a great foundation for a microservice architecture
  3. 3. @crichardson About Chris http://learnmicroservices.io
  4. 4. @crichardson About Kenny @kennybastani
  5. 5. @crichardson Agenda Overview of event sourcing The problem with microservices, transactions and queries Using events to maintain consistency Event sourcing in a microservice architecture Implementing queries with CQRS
  6. 6. @crichardson Traditional persistence Order id state …. 101 ACCEPTED Order table … ID STATE … 102 … …
  7. 7. @crichardson But how did we get here? Who did what and when? What was the state of the Order last Monday at 3:01pm?
  8. 8. @crichardson Order example History
  9. 9. @crichardson Task example State History Audit
  10. 10. @crichardson Usually auditing, history and temporal queries is additional code and/or an after-thought
  11. 11. @crichardson Event sourcing For each domain object (i.e. DDD aggregate): Identify (state changing) domain events, e.g. use Event Storming Define Event classes For example, Order events: OrderCreated, OrderCancelled, OrderApproved, OrderRejected, OrderShipped
  12. 12. @crichardson Persists events NOT current state Event table Entity type Event id Entity id Event data Order 902101 …OrderApproved Order 903101 …OrderShipped Event type Order 901101 …OrderCreated
  13. 13. @crichardson Replay events to recreate state Order state OrderCreated(…) OrderAccepted(…) OrderShipped(…) Events Periodically snapshot to avoid loading all events
  14. 14. @crichardson Benefits of event sourcing Reifies state changes: Built in, reliable audit log temporal queries Preserved history More easily implement future requirements Eliminates O/R mapping problem (mostly) Reliable event publishing: publishes events needed by predictive analytics etc, user notifications,…
  15. 15. @crichardson Drawbacks of event sourcing Requires application rewrite Weird and unfamiliar style of programming Events live forever carefully evolve schema Querying the event store can be challenging Current state is no longer directly available Often need to maintain views for efficiency
  16. 16. @crichardson Demo
  17. 17. @crichardson Agenda Overview of event sourcing The problem with microservices, transactions and queries Using events to maintain consistency Event sourcing in a microservice architecture Implementing queries with CQRS
  18. 18. @crichardson The Microservice architecture tackles complexity through modularization
  19. 19. @crichardson Microservice architecture Browser Mobile Device Store Front UI API Gateway Catalog Service Review Service Order Service … Service Catalog Database Review Database Order Database … Database HTML REST REST Apply X-axis and Z-axis scaling to each service independently
  20. 20. @crichardson But there are challenges implementing transactions and queries
  21. 21. @crichardson ACID transactions cannot be used BEGIN TRANSACTION … SELECT ORDER_TOTAL FROM ORDERS WHERE CUSTOMER_ID = ? … SELECT CREDIT_LIMIT FROM CUSTOMERS WHERE CUSTOMER_ID = ? … INSERT INTO ORDERS … … COMMIT TRANSACTION Private to the Order Service Private to the Customer Service Requires 2PC
  22. 22. @crichardson 2PC is not a viable option Guarantees consistency BUT 2PC is best avoided Not supported by many NoSQL databases etc. CAP theorem 2PC impacts availability ….
  23. 23. @crichardson Queries can’t use joins SELECT * FROM CUSTOMER c, ORDER o WHERE c.id = o.ID AND o.ORDER_TOTAL > 100000 AND o.STATE = 'SHIPPED' AND c.CREATION_DATE > ? Customer Service Order Service
  24. 24. @crichardson Agenda Overview of event sourcing The problem with microservices, transactions and queries Using events to maintain consistency Event sourcing in a microservice architecture Implementing queries with CQRS
  25. 25. @crichardson Event-driven architecture
  26. 26. @crichardson Order Management Order id : 4567 total: 343 state = CREATED Customer Management Customer creditLimit : 12000 creditReservations: {} Customer creditLimit : 12000 creditReservations: { 4567 -> 343} Order id : 4567 total: 343 state = APPROVED Eventually consistent credit checking Message Bus createOrder() Publishes: Subscribes to: Subscribes to: publishes: OrderCreatedEvent CreditReservedEvent OrderCreatedEvent CreditReservedEvent
  27. 27. @crichardson How atomically update database and publish an event Order Service Order Database Message Broker insert Order publish OrderCreatedEvent dual write problem ?
  28. 28. @crichardson Failure = inconsistent system Order Service Order Database Message Broker insert Order publish OrderCreatedEvent X
  29. 29. @crichardson 2PC is not an option
  30. 30. @crichardson How to reliably publish events when state changes?
  31. 31. @crichardson Agenda Overview of event sourcing The problem with microservices, transactions and queries Using events to maintain consistency Event sourcing in a microservice architecture Implementing queries with CQRS
  32. 32. @crichardson Event sourcing = event-centric persistence Application Database Event store update publish X
  33. 33. @crichardson Event Store Application architecture Order 123 Customer 456 OrderCreated OrderApproved … CustomerCreated CustomerCreditReserved … CreateOrder UpdateOrder GetOrder Subscribe Order Service CreateCustomer UpdateCustomer GetCustomer Subscribe Customer Service
  34. 34. @crichardson Request handling in an event sourced application HTTP Handler Event Store pastEvents = findEvents(entityId) Order new() applyEvents(pastEvents) newEvents = processCmd(someCmd) saveEvents(newEvents) (optimistic locking) Order Service applyEvents(newEvents)
  35. 35. @crichardson Event Store publishes events consumed by other services Event Store Event Subscriber subscribe(EventTypes) publish(event) publish(event) Customer update() Customer Service
  36. 36. @crichardson Event Store publishes events consumed by other services Event Store Event Subscriber subscribe(EventTypes) publish(event) publish(event) CQRS View update() Service Xyz send notifications …
  37. 37. Event store = database + message broker Hybrid database and message broker Implementations: Home grown/DIY geteventstore.com by Greg Young http://eventuate.io (mine) Event Store Save aggregate events Get aggregate events Subscribe to events
  38. 38. @crichardson Agenda Overview of event sourcing The problem with microservices, transactions and queries Using events to maintain consistency Event sourcing in a microservice architecture Implementing queries with CQRS
  39. 39. @crichardson Find recent, valuable customers SELECT * FROM CUSTOMER c, ORDER o WHERE c.id = o.ID AND o.ORDER_TOTAL > 100000 AND o.STATE = 'SHIPPED' AND c.CREATION_DATE > ? Customer Service Order Service What if event sourcing is used?…. is no longer easy
  40. 40. @crichardson Use Command Query Responsibility Segregation (CQRS)
  41. 41. @crichardson Command Query Responsibility Segregation (CQRS) Application logic Commands Queries X POST PUT DELETE GET
  42. 42. @crichardson Command Query Responsibility Segregation (CQRS) Command side Commands Event Sourcing domain objects Event Store Events Query side Queries (Materialized) View Events POST PUT DELETE GET
  43. 43. @crichardson Query side design Event Store Updater View Updater Service Events Reader HTTP GET Request View Query Service View Store e.g. MongoDB ElasticSearch Neo4J update query
  44. 44. @crichardson Persisting a customer and order history in MongoDB { "_id" : "0000014f9a45004b 0a00270000000000", "_class" : "net.chrisrichardson…..views.orderhistory.CustomerView", "version" : NumberLong(5), "orders" : { "0000014f9a450063 0a00270000000000" : { "state" : "APPROVED", "orderId" : "0000014f9a450063 0a00270000000000", "orderTotal" : { "amount" : "1234" } }, "0000014f9a450063 0a00270000000001" : { "state" : "REJECTED", "orderId" : "0000014f9a450063 0a00270000000001", "orderTotal" : { "amount" : "3000" } } }, "name" : "Fred", "creditLimit" : { "amount" : "2000" } } Denormalized = efficient lookup
  45. 45. @crichardson Persisting customers and order info using Spring Data for MongoDB...
  46. 46. @crichardson Persisting customers and order using Spring Data for MongoDB...
  47. 47. Benefits and drawbacks of CQRS Benefits Necessary in an event sourced architecture Separation of concerns = simpler command and query models Supports multiple denormalized views Improved scalability and performance Drawbacks Complexity Potential code duplication Replication lag/eventually consistent views
  48. 48. @crichardson Demo
  49. 49. @crichardson Summary Microservice architecture functionally decomposes an application into services Transactions and queries resist decomposition Use an event-driven architecture based on event sourcing to maintain data consistency Implement queries using CQRS
  50. 50. @crichardson chris@chrisrichardson.net kbastani@pivotal.io http://learnmicroservices.io http://pivotal.io Questions?

×