Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
SIMPLE MACHINES
Sagas in Kafka
SAGAS IN KAFKA !1
Stephen Zoio — Principal Consultant
SIMPLE MACHINES SAGAS IN KAFKA
!2
Journey to Sagas
Coordinating microservices Sagas
Kafka as a database
Events
A protocol ...
SIMPLE MACHINES SAGAS IN KAFKA
!3
Microservice Transactions
www.
Flight Booking
Hotel Booking
Car Rental Payment
Trips
?
SIMPLE MACHINES SAGAS IN KAFKA
!4
Two Phase Commit
www.
Flight Booking
Hotel Booking
Car Rental Payment
Trips
Prepare
Resp...
SIMPLE MACHINES SAGAS IN KAFKA
!5
Saga Transaction
www.
Flight Booking
Hotel Booking
Car Rental Payment
Trips
Action
Succe...
SIMPLE MACHINES SAGAS IN KAFKA
!6
Saga Failure
www.
Flight Booking
Hotel Booking
Car Rental Payment
Trips
Action
Success R...
SIMPLE MACHINES SAGAS IN KAFKA
!7
Saga Data Consistency
SQL Database
A Atomicity
C Consistency
I Isolation
D Durability
Sa...
SIMPLE MACHINES SAGAS IN KAFKA
!8
Kafka and Event Sourcing
Kafka as a database?
Event sourcing
Event processingKafka
Event...
SIMPLE MACHINES SAGAS IN KAFKA
!9
Event Log Consistency
e.g. Bank Account
Events
Simple Sourcing
• Balance always positive...
An API to support
these consistency
requirements
SIMPLE MACHINES SAGAS IN KAFKA
!11
Simple Sourcing Concepts
Type Represents Example
C Command Intent debit Bob’s account $...
SIMPLE MACHINES SAGAS IN KAFKA
!12
Simple Sourcing Data Flow
Kafka
Command request
Aggregate state
Command response
Event
...
SIMPLE MACHINES SAGAS IN KAFKA
!13
Simple Sourcing Programming Model
In Scala types
(Command, Aggregate) => Either[Error, ...
SIMPLE MACHINES SAGAS IN KAFKA
!14
Event Sourced Apps
Coordinating between aggregates: Bike auction scenario
Account Aggre...
SIMPLE MACHINES SAGAS IN KAFKA
!15
Auction App
Coordinating between aggregates
Charlie
$250
Account Aggregate
Key Total Re...
SIMPLE MACHINES SAGAS IN KAFKA
!16
Representing Sagas
A stateful directed acyclic graph (DAG)
a undo
A
b c undo
C
d undo
D...
SIMPLE MACHINES SAGAS IN KAFKA
!17
Representing Sagas
Auction bid saga
reserve
account
cancel
reservation
bid on auction
c...
SIMPLE MACHINES
(Event, Aggregate) => Aggregate
(Command, Aggregate) => Either[Error, List[Event]]
Key => Aggregate
COMMAN...
SIMPLE MACHINES SAGAS IN KAFKA
!19
Useful Patterns
Pessimistic lock
Claim lock
Perform action on resource
Release lock
Cla...
SIMPLE MACHINES SAGAS IN KAFKA
!20
Saga State a undo
A
b c undo
C
d undo
D
e
Pending
In progress
Completed
Failed
N/A
• Sa...
SIMPLE MACHINES SAGAS IN KAFKA
!21
Saga State a undo
A
b c undo
C
d undo
D
e
Pending
In progress
Completed
Failed
N/A
Fail...
SIMPLE MACHINES SAGAS IN KAFKA
!22
Saga DSL
Scala DSL loosely based on Akka Streams
a
b
c
a ~> b ~> c
SIMPLE MACHINES SAGAS IN KAFKA
!23
Saga DSL
with undo (compensation) actions
a ~> b.withUndo(undoB) ~> c
a
b
c
undo
B
SIMPLE MACHINES SAGAS IN KAFKA
!24
Saga DSL
parallel execution
a undo
A
b c undo
C
d undo
D
e
a.withUndo(undoA) ~> paralle...
SIMPLE MACHINES SAGAS IN KAFKA
!25
Saga DSL
These are all equivalent:
(a ~> b ~> c) ~> d ~> e
a ~> (b ~> c ~> d) ~> e
a ~>...
SIMPLE MACHINES SAGAS IN KAFKA
!26
Simple Sagas
A simple Kafka-based Sagas framework
Saga coordinator
Action processors
Cl...
SIMPLE MACHINES SAGAS IN KAFKA
!27
Saga Coordinator
Saga request Saga response
Action request Action response
Saga coordin...
SIMPLE MACHINES SAGAS IN KAFKA
!28
Saga Coordinator Saga request
Saga(A, B, C, D, E)
Saga state transition
Start(saga=Saga...
SIMPLE MACHINES SAGAS IN KAFKA
!29
Saga Coordinator Apps
Application main method
SagaApp<SpecificRecord> sagaApp = SagaApp...
SIMPLE MACHINES SAGAS IN KAFKA
!30
Action Processor
{
"sagaId": 1,
"commandId": 34,
"command": "BookFlight(...)",
"actionT...
SIMPLE MACHINES SAGAS IN KAFKA
!31
Event Sourcing Action Processor
Action request
Key Value
Saga ID DebitAccount(amount=10...
SIMPLE MACHINES SAGAS IN KAFKA
!32
Async Action Processor
Action request
Key Value
Saga id ThirdPartyPayment(amount=50)
Sa...
SIMPLE MACHINES
streamApp
.withActionProcessor(EventSourcingBuilder.apply(
accountCommandSpec,
topicBuilder -> topicBuilde...
SIMPLE MACHINES SAGAS IN KAFKA
!34
Saga Client
public interface SagaAPI<A> {
FutureResult<SagaError, SagaId> submitSaga(Sa...
SIMPLE MACHINES SAGAS IN KAFKA
!35
Deployment
Single node
Application Instance
Saga coordinator
Action processor 1
Action ...
SIMPLE MACHINES SAGAS IN KAFKA
!36
Deployment
Application Instance 1
Separate sagas and actions
Application Instance 2
Kaf...
SIMPLE MACHINES
Action processors
SAGAS IN KAFKA
!37
Deployment
Saga coordinators
Kafka brokers
Resilient cluster setup
ap...
SIMPLE MACHINES SAGAS IN KAFKA
!38
Saga coordinators
Kafka brokers
Event Sourcing
Hybrid Architectures
Async / Http
Simple...
SIMPLE MACHINES
Questions…
SAGAS IN KAFKA !39
https://simplesource.io/
Prochain SlideShare
Chargement dans…5
×

Simplifying Distributed Transactions with Sagas in Kafka (Stephen Zoio, Simple Machines) Kafka Summit London 2019

Microservices are seen as the way to simplify complex systems, until you need to coordinate a transaction across services, and in that instant, the dream ends. Transactions involving multiple services can lead to a spaghetti web of interactions. Protocols such as two-phase commit come with complexity and performance bottlenecks. The Saga pattern involves a simplified transactional model. In sagas, a sequence of actions are executed, and if any action fails, a compensating action is executed for each of the actions that have already succeeded. This is particularly well suited to long-running and cross-microservice transactions. In this talk we introduce the new Simple Sagas library (https://github.com/simplesourcing/simplesagas). Built using Kafka streams, it provides a scalable fault tolerance event-based transaction processing engine. We walk through a use case of coordinating a sequence of complex financial transactions. We demonstrate the easy to use DSL, show how the system copes with failure, and discuss this overall approach to building scalable transactional systems in an event-driven streaming context.

Simplifying Distributed Transactions with Sagas in Kafka (Stephen Zoio, Simple Machines) Kafka Summit London 2019

  1. 1. SIMPLE MACHINES Sagas in Kafka SAGAS IN KAFKA !1 Stephen Zoio — Principal Consultant
  2. 2. SIMPLE MACHINES SAGAS IN KAFKA !2 Journey to Sagas Coordinating microservices Sagas Kafka as a database Events A protocol for distributed and long running transactions
  3. 3. SIMPLE MACHINES SAGAS IN KAFKA !3 Microservice Transactions www. Flight Booking Hotel Booking Car Rental Payment Trips ?
  4. 4. SIMPLE MACHINES SAGAS IN KAFKA !4 Two Phase Commit www. Flight Booking Hotel Booking Car Rental Payment Trips Prepare Response Confirm Acknowledgement • Too strict • Too slow • Too complex Not that popular in practice…
  5. 5. SIMPLE MACHINES SAGAS IN KAFKA !5 Saga Transaction www. Flight Booking Hotel Booking Car Rental Payment Trips Action Success Response • Much simpler protocol • Minimises latency and comms • Services don’t need to care about sagas
  6. 6. SIMPLE MACHINES SAGAS IN KAFKA !6 Saga Failure www. Flight Booking Hotel Booking Car Rental Payment Trips Action Success Response Compensation Failure work backwards, executing compensating actions Compensation Response
  7. 7. SIMPLE MACHINES SAGAS IN KAFKA !7 Saga Data Consistency SQL Database A Atomicity C Consistency I Isolation D Durability Sagas A Atomicity C Consistency I Isolation D Durability We give up isolation, but we can live without it…
  8. 8. SIMPLE MACHINES SAGAS IN KAFKA !8 Kafka and Event Sourcing Kafka as a database? Event sourcing Event processingKafka Events WHY NOT STORE THE DATA LIKE WE USE IT? WE’RE ALREADY USING KAFKA FOR MESSAGING
  9. 9. SIMPLE MACHINES SAGAS IN KAFKA !9 Event Log Consistency e.g. Bank Account Events Simple Sourcing • Balance always positive • Account never debited twice Taking advantage of • Kafka exactly once processing • Per topic-partition ordering guarantee
  10. 10. An API to support these consistency requirements
  11. 11. SIMPLE MACHINES SAGAS IN KAFKA !11 Simple Sourcing Concepts Type Represents Example C Command Intent debit Bob’s account $100 E Event Outcome Bob’s account debited $100 A Aggregate State Bob’s account balance=$700, available=$650 K Key Identity Bob’s account number Based on CQRS (Command query responsibility segregation)
  12. 12. SIMPLE MACHINES SAGAS IN KAFKA !12 Simple Sourcing Data Flow Kafka Command request Aggregate state Command response Event Bob’s balance >$100? Debit Bob’s account $100 Return Success Publish event AccountDebited(bob, 100) Return error response Debit Bob’s account $100 Yes No Everything happens in a single atomic transaction
  13. 13. SIMPLE MACHINES SAGAS IN KAFKA !13 Simple Sourcing Programming Model In Scala types (Command, Aggregate) => Either[Error, List[Event]] (Event, Aggregate) => Aggregate Key => Aggregate COMMAND HANDLER EVENT AGGREGATOR INITIAL VALUE
  14. 14. SIMPLE MACHINES SAGAS IN KAFKA !14 Event Sourced Apps Coordinating between aggregates: Bike auction scenario Account Aggregate Key Total Reserved Bob’s a/c 400 200 Charlie’s a/c 500 – Auction Aggregate Key Bidder Amount Bicycle Bob 200 Reserve Charlie’s account $250 Bid $250 for Charlie Release $200 for Bob $200 Current bid: Bob $200 Bob $250 Next bid: Charlie $250 Charlie 250 Auction Aggregate Key Bidder Amount Bicycle Charlie 250 –
  15. 15. SIMPLE MACHINES SAGAS IN KAFKA !15 Auction App Coordinating between aggregates Charlie $250 Account Aggregate Key Total Reserved Bob’s a/c 400 – Charlie’s a/c 500 Alice’s a/c 500 Auction Aggregate Key Bidder Amount Bicycle Charlie 250 Reserve Bob’s account $300 Bid $300 for Bob Undo Bob’s reservation Reserve Alice’s account $350 Bid $350 for Alice Release $250 for Charlie Charlie has the current best bid of $250 Bob $300 Bob bid’s again, this time $300 Alice $350 Meanwhile Alice arrives and bids $350 300 350 300 350Alice Sagas are suitable for synchronising between multiple aggregates in an event sourced application - -250
  16. 16. SIMPLE MACHINES SAGAS IN KAFKA !16 Representing Sagas A stateful directed acyclic graph (DAG) a undo A b c undo C d undo D e a undo A Saga actions Undo (compensation) actions
  17. 17. SIMPLE MACHINES SAGAS IN KAFKA !17 Representing Sagas Auction bid saga reserve account cancel reservation bid on auction cancel previous reservation
  18. 18. SIMPLE MACHINES (Event, Aggregate) => Aggregate (Command, Aggregate) => Either[Error, List[Event]] Key => Aggregate COMMAND HANDLER EVENT AGGREGATOR INITIAL VALUE (_, _) => true (_, aggregate) => if (aggregate) Error("email taken") else List(EventClaimed) _ => false SAGAS IN KAFKA !18 Useful Patterns Enforcing uniqueness Claim Email Type Example C Command claim email address, rejects command if aggregate is set A Aggregate either true or false K Key email address *Each user must have a unique email Claim email address: bob@xyz.com Create user with email address: bob@xyz.com Claim email Create user release email
  19. 19. SIMPLE MACHINES SAGAS IN KAFKA !19 Useful Patterns Pessimistic lock Claim lock Perform action on resource Release lock Claim lock Create user release lock Create userDo actions Release lock
  20. 20. SIMPLE MACHINES SAGAS IN KAFKA !20 Saga State a undo A b c undo C d undo D e Pending In progress Completed Failed N/A • Saga state (in progress, in failure) • Action state (pending, in progress) • The actions themselves (for dynamic actions) a undo A b c undo C d undo D e a undo A b c undo C d undo D e a undo A b c undo C d undo D e a undo A b c undo C d undo D e a undo A b c undo C d undo D e a undo A b c undo C d undo D e a undo A b c undo C d undo D e a undo A b c undo C d undo D e a undo A b c undo C d undo D e
  21. 21. SIMPLE MACHINES SAGAS IN KAFKA !21 Saga State a undo A b c undo C d undo D e Pending In progress Completed Failed N/A Failure mode saga into switches into undo mode a undo A b c undo C d undo D e undo A b undo C undo D undo E undo A b undo C undo D undo E undo A b undo C undo D undo E undo A b undo C undo D undo E undo A b undo C undo D undo E
  22. 22. SIMPLE MACHINES SAGAS IN KAFKA !22 Saga DSL Scala DSL loosely based on Akka Streams a b c a ~> b ~> c
  23. 23. SIMPLE MACHINES SAGAS IN KAFKA !23 Saga DSL with undo (compensation) actions a ~> b.withUndo(undoB) ~> c a b c undo B
  24. 24. SIMPLE MACHINES SAGAS IN KAFKA !24 Saga DSL parallel execution a undo A b c undo C d undo D e a.withUndo(undoA) ~> parallel(b, c.withUndo(undoC)) ~> d.withUndo(undoD) ~> e
  25. 25. SIMPLE MACHINES SAGAS IN KAFKA !25 Saga DSL These are all equivalent: (a ~> b ~> c) ~> d ~> e a ~> (b ~> c ~> d) ~> e a ~> List(b, c, d).inSeries ~> e (a ~> b) ~> c ~> (d ~> e) ~> is associative This enables a building block approach to constructing sagas
  26. 26. SIMPLE MACHINES SAGAS IN KAFKA !26 Simple Sagas A simple Kafka-based Sagas framework Saga coordinator Action processors Client
  27. 27. SIMPLE MACHINES SAGAS IN KAFKA !27 Saga Coordinator Saga request Saga response Action request Action response Saga coordinator { "sagaId": 12, "actions": [ {"actionId": 1, "command": "BookFlight(...)"}, {"actionId": 2, "command": "BookCar(...)"} ], "dependencies": {"2": [1], "4": [2, 3]} } { "sagaId": 12, "success": true } { "actionId": 1, "command": "BookFlight(...)"} } { "actionId": 1, "success": true }
  28. 28. SIMPLE MACHINES SAGAS IN KAFKA !28 Saga Coordinator Saga request Saga(A, B, C, D, E) Saga state transition Start(saga=Saga(A, B, C, D, E)) ActionStarted(action=A) ActionFinished(action=A) ActionStarted(action=B,C) ActionFinished(action=C) Saga state Saga(A, B, C, D, E) Saga(A=started, B, C, D, E) Saga(A=finished, B, C, D, E) Saga(A=finished, B=started, C=started,..) Saga(A=finished, B=finished, C=started,..) Saga Saga Client Saga request Saga response Saga state transition Aggregate state Event processor Saga state machine Action request Action response Action processor Saga coordinator
  29. 29. SIMPLE MACHINES SAGAS IN KAFKA !29 Saga Coordinator Apps Application main method SagaApp<SpecificRecord> sagaApp = SagaApp.of( sagaSpec, actionSpec, topicBuilder -> topicBuilder.withTopicPrefix("saga-")); sagaApp.withActions( "account_action", "auction_action", "payment_action"); sagaApp.run(properties);
  30. 30. SIMPLE MACHINES SAGAS IN KAFKA !30 Action Processor { "sagaId": 1, "commandId": 34, "command": "BookFlight(...)", "actionType": "book_flight" } { "sagaId": 1, "commandId": 34, "success": true, "undo": { "command": "CancelFlight(...)", "actionType": "cancel_flight" } } Action request Action response Action processor Idempotent effect Use the built in ones, or roll your own ?
  31. 31. SIMPLE MACHINES SAGAS IN KAFKA !31 Event Sourcing Action Processor Action request Key Value Saga ID DebitAccount(amount=100) Saga Coordinator Command processor Event Aggregator Action request Simple Sourcing Command request Aggregate state Event Action response Command request Key Value Bob’s a/c DebitAccount(amount=100) Event Key Value Bob’s a/c DebitAccount(amount=100) Aggregate state Key Value Bob’s a/c Account(balance=500) Bob’s a/c DebitAccount(amount=100) Command response Key Value Bob’s a/c DebitAccount(amount=100) Action response Key Value Bob’s a/c DebitAccount(amount=100) Command response Event sourcing action processor saga-long per-aggregate consistency guarantee
  32. 32. SIMPLE MACHINES SAGAS IN KAFKA !32 Async Action Processor Action request Key Value Saga id ThirdPartyPayment(amount=50) Saga Coordinator Action request Action response Async output Key Value ? Payment(token=XXX) Action response Key Value Saga id Response(success=true, undoCommand=CancelPayment(X XX)) Action output Async action processor Microservice { "success": true, “booking_reference": “XXX" } Callback Async / http invoke undo actions can also be defined dynamically from web service result update saga with new undo action
  33. 33. SIMPLE MACHINES streamApp .withActionProcessor(EventSourcingBuilder.apply( accountCommandSpec, topicBuilder -> topicBuilder.withTopicPrefix("action-"), topicBuilder -> topicBuilder.withTopicPrefix("command-") .withActionProcessor(EventSourcingBuilder.apply( auctionCommandSpec, topicBuilder -> topicBuilder.withTopicPrefix("action-"), topicBuilder -> topicBuilder.withTopicPrefix("command-") streamApp.run(configProperties); SAGAS IN KAFKA !33 Action Processor Apps EventSourcingSpec<SpecificRecord, AccountCommand, AccountId, AccountCommand> accountCommandSpec = EventSourcingSpec.builder() .actionType("account_action") .aggregateName("account") .decode(a -> Result.success((AccountCommand) a)) .commandMapper(c -> c) .keyMapper(AccountCommand::getId) .sequenceMapper(c -> Sequence.position(c.getSequence())) .timeout(Duration.ofSeconds(30)) .build(); Application main method EventSourcingSpec<SpecificRecord, AuctionCommand, AuctionId, AuctionCommand> auctionCommandSpec = ...;
  34. 34. SIMPLE MACHINES SAGAS IN KAFKA !34 Saga Client public interface SagaAPI<A> { FutureResult<SagaError, SagaId> submitSaga(SagaRequest<A> request); FutureResult<SagaError, SagaResponse> getSagaResponse( SagaId requestId, Duration timeout); } submitSaga Saga request Saga response Saga Coordinator getSagaResponse Consumer Private response topic COMPLETES FUTURE client is typically another service, such as a web application
  35. 35. SIMPLE MACHINES SAGAS IN KAFKA !35 Deployment Single node Application Instance Saga coordinator Action processor 1 Action processor 2 Kafka broker Kafka broker
  36. 36. SIMPLE MACHINES SAGAS IN KAFKA !36 Deployment Application Instance 1 Separate sagas and actions Application Instance 2 Kafka broker Kafka broker Microservice Saga coordinator Action processor 1 (event sourcing) Action processor 2 (event sourcing) Action processor 3 (async)
  37. 37. SIMPLE MACHINES Action processors SAGAS IN KAFKA !37 Deployment Saga coordinators Kafka brokers Resilient cluster setup application instance failureapplication instance recovery Kafka broker failureKafka broker recovery
  38. 38. SIMPLE MACHINES SAGAS IN KAFKA !38 Saga coordinators Kafka brokers Event Sourcing Hybrid Architectures Async / Http Simple Sourcing Other microservices
  39. 39. SIMPLE MACHINES Questions… SAGAS IN KAFKA !39 https://simplesource.io/

×