Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

Opportunities and Pitfalls of Event-Driven Utopia

56 vues

Publié le

Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2B24UoY.

Bernd Rücker goes over the concepts, the advantages, and the pitfalls of event-driven utopia. He shares real-life stories or points to source code examples. Filmed at qconnewyork.com.

Bernd Rücker is co-founder and developer advocate at Camunda. Previously, he has helped automating highly scalable core workflows at global companies including T-Mobile, Lufthansa, Zalando. He is currently focused on new workflow automation paradigms that fit into modern architectures around distributed systems, microservices, domain-driven design, event-driven architecture and reactive systems.

Publié dans : Technologie
  • Soyez le premier à commenter

  • Soyez le premier à aimer ceci

Opportunities and Pitfalls of Event-Driven Utopia

  1. 1. Opportunities & Pitfalls of Event-Driven Utopia @berndruecker
  2. 2. InfoQ.com: News & Community Site • 750,000 unique visitors/month • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • News 15-20 / week • Articles 3-4 / week • Presentations (videos) 12-15 / week • Interviews 2-3 / week • Books 1 / month Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ event-driven-benefits-pitfalls/
  3. 3. Presented at QCon New York www.qconnewyork.com Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide
  4. 4. Why this talk
  5. 5. Why this talk
  6. 6. Service 1 Agenda Service 2 Events on the inside Events on the outside 1 3
  7. 7. Service 1 Agenda Service 2 Events on the inside Events on the outside 1 3 Events inside out2
  8. 8. Service 1 Agenda Service 2 Events on the inside1 Events on the outside3 Events inside out2
  9. 9. Once upon a time… RDMS Application Client BBC architecture (box - arrow – box – arrow - cylinder) Every architecture diagram you'll ever need
  10. 10. The great thing about this architecture RDMS Application DB gurantees (e.g. ACID)
  11. 11. The problem not webscale resiliency is expensive RDMS Application RDMS
  12. 12. Pat Helland https://vimeo.com/52831373
  13. 13. Append-only Log … bank account created +2,500 $ transfered -14.99$ paid by credit card … Persistent change RDMS Account # Balance 12345 2,500$ Persistent state Current Balance = 2,485.01 $
  14. 14. Append-only Log … bank account created +2,500 $ transfered -14.99$ paid by credit card … Persistent change Event Bank Account Created 2019/04/16 11:00 # 12345 Event Money Transfer Received 2,500$ 2019/04/16 11:00 # 12345
  15. 15. Event Sourcing in a nutshell Customer Business Logic Save event 2. Read events Replay … Internal State 3. Build internal state 5. Add Customer Created Event 1. Create Customer 4. Validation and Invariant Checks e.g. Customer Created, Customer Credit Limit Approved, …Customer Event Store 6. Async publish Customer Created Domain Event
  16. 16. Working without distributed transactions Customer Business Logic Save event 2. Read events Replay … Internal State 3. Build internal state 5. Add Customer Created Event 1. Create Customer 4. Validation and Invariant Checks e.g. Customer Created, Customer Credit Limit Approved, …Customer Event Store 6. Async publish Customer Created Domain Event This is the only atomic operation required
  17. 17. Traditional Architecture Customer 1. Create Customer Account RDMS Business Logic Open Account 2. Persist state changes 3. Remote Communication
  18. 18. Pat Helland “ Distributed Systems Guru Worked at Amazon, Microsoft & Salesforce @berndruecker
  19. 19. Pat Helland Grown-Ups Don’t Use Distributed Transactions “ Distributed Systems Guru Worked at Amazon, Microsoft & Salesforce
  20. 20. Open Account Outbox pattern in traditional architectures Customer 1. Create Customer Account RDMS Business Logic Job: „Open Account“ 2. Persist state changes 3. Remote CommunicationExecute: „Open Account“ TX 1 TX 2 Async after first transaction!
  21. 21. Outbox pattern – Implementation Approaches Scheduler Database Transaction Log Workflow Automation
  22. 22. Idempotency Customer 1. Create Customer Account RDMS Business Logic Job: „Open Account“ 2. Persist state changes 3. Remote Communication Async after transaction! Execute: „Open Account“ Capture Request TX 1 TX 2
  23. 23. Idempotency Customer 1. Create Customer Account RDMS Business Logic Job: „Open Account“ 2. Persist state changes 3. Remote Communication Async after transaction! Execute: „Open Account“ Capture Request TX 1 TX 3TX 2
  24. 24. Working without distributed transactions Customer Business Logic Save event 2. Read events Replay … Internal State 3. Build internal state 5. Add Customer Created Event 1. Create Customer 4. Validation and Invariant Checks e.g. Customer Created, Customer Credit Limit Approved, …Customer Event Store 6. Async publish Customer Created Domain Event This is the only atomic operation required
  25. 25. Events on the inside. An example from my world
  26. 26. mail@berndruecker.io @berndruecker Bernd Ruecker Co-founder and Chief Technologist of Camunda
  27. 27. We offer two different workflow engines. Why? Camunda ZeebePersistent State Persistent change
  28. 28. Workflow Instance Id Current Activity State 2 RetrievePayment running Workflow Instance Id Current Activity State 2 ShipGoods running Workflow Instance Id Current Activity State 2 OrderDelivered ended 2. 3. UPDATE 2. UPDATE 1. INSERT 3.1. RDMS Workflow Engine
  29. 29. Append-only Log1. create workflow instance workflow instance created start event occured sequence flow taken activity activated task created lock created task locked complete task task completed activity completed sequence flow taken … … 2. 2. 1. Workflow Engine
  30. 30. Event Handling, Replication & Single Writer Follower Follower complete task command task completed event 1 send 2 append command Broker (Leader) Stream Processor 4 process 7 store & replicate event 6 append event 3 store & replicate command 5 respond Single Writer (single thread)
  31. 31. What we do different Follower Follower complete task command task completed event 1 send 2 append command Leader Stream Processor 4 process 7 store & replicate event 6 append event 3 store & replicate command 5 respond Single Writer (single thread) Store and replay commands Delete records that are fully processed Persist & replicate internal state
  32. 32. Consistency Availability Partition
  33. 33. Zeebe is CP Follower Follower complete task command task completed event 1 send 2 append command Leader Stream Processor 4 process 7 store & replicate event 6 append event 3 store & replicate command 5 respond Single Writer (single thread)
  34. 34. Horizontal scalability by partitioning Partition 1 Partition 2 Partition 3 Partition 4 Every workflow instance is exactly handled by one partition instance id: 2-42 instance id: 3-66 Stream Processor Single Writer (single thread)
  35. 35. Queries and read models Zeebe Broker Zeebe Broker Streaming Exporter ask ask
  36. 36. Recap 1 – Events on the inside # Natural mechanism to build scalable services in distributed systems (with Outbox & co included) But # You have to think about reads, queries & eventual consistency # Much less industry experience available @berndruecker
  37. 37. Service 1 Agenda Service 2 Events on the inside Events on the outside 1 3 Events inside out2
  38. 38. Event Store and Messaging Customer … 1. Create Customer Customer Event Store
  39. 39. Merge Messaging and Event Store Customer … 1. Create Customer Customer Event Store
  40. 40. Merge messaging and event store Customer … 1. Create Customer Shared Event Store
  41. 41. Enter the world of Kafka…
  42. 42. Merge messaging and event store Customer … 1. Create Customer Shared Event Store
  43. 43. Kafka as transport Customer … 1. Create Customer Used as queue (but persistent!)
  44. 44. Service 1 Agenda Service 2 Events on the inside Events on the outside 1 3 Events inside out2
  45. 45. Once upon a time Billing Customer Change Address
  46. 46. Event Notification Address changed Billing Customer
  47. 47. Event Notification Address changed Billing Customer Billing Customer Reverse direction of dependency direction of dependency Change Address
  48. 48. Event Notification Address changed Billing Customer AdressChanged { customerId: 42 } Ask for details
  49. 49. Event-carried State Transfer Address changed Billing Customer AddressChanged { customerId: 42, address: ... } CustomerChanged { customerId: 42, status: A, address: ..., } AddressChanged { customerId: 42, oldAddress: ... newAddress: ... } CustomerMoved { ...,
  50. 50. This decision is complex Address changed Billing Customer Billing Customer Reverse direction of dependency direction of dependency Change Address
  51. 51. Example Change Address Address Submit From bla@company.com Date 2019-04-23 09.05 To confirm your address change please click on this link: http://company.com/confirm?id=82e97d49-166c-4862- 9973-4db348e6225d Incoming Email
  52. 52. Example Customer Notification Address change confirmed Change Adress Address change requested http://company.com/confirm?id=82e97 d49-166c-4862-9973-4db348e6225d direction of dependency
  53. 53. Example Customer Notification ‚Confirmation‘ approved Change Adress http://company.com/confirm?id=82e97 d49-166c-4862-9973-4db348e6225d Send mail ‚Confirmation‘ Address changed direction of dependency
  54. 54. Challenge: Command vs. Event Event Command vs
  55. 55. It is NOTabout communication protocols Address changed Billing Customer Billing Customer Change Address It can be messaging, REST, whatever, ….
  56. 56. Manifold ways of transport …
  57. 57. Manifold ways of transport …
  58. 58. ? Event Command Query Message Record Event Fact, happened in the past, immutable Intend, Want s.th. to happen, The intention itself is a fact
  59. 59. ? Event Command Query Message Record Event
  60. 60. Commands in disguise The Customer Needs To Be Sent A Message To Confirm Address Change Event Send Message Wording of Sender Wording of recipient
  61. 61. Examples PaymentOrder Subscription Retrieve Payment More general, does not need to know who is retrieving payments Customer Order Address Changed Billing More general, does not need to care about who is interessted in address changes Notification Send Mail Global service Order Notification Order Placed Payment Received Goods Shipped Service that can handle notifications for orders autonomously
  62. 62. Distributed Monoliths Authorization Service Document Context Page Context Document attached Page created Document moved Page moved …
  63. 63. Define stable contract/API instead Authorization Service Document Context Page Context Add auth …
  64. 64. Next challenge: Event chains
  65. 65. Event Chains Adress Check Credit Check Registration @berndruecker Customer Event Bus Registration requested Credit checked Address checked Customer registered
  66. 66. Event Chains Adress Check Credit Check Registration @berndruecker Customer Event Bus Registration requested Credit checked Address checked Customer registered How does customer registration work?
  67. 67. The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years. https://martinfowler.com/articles/201701-event-driven.html @berndruecker
  68. 68. The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years. https://martinfowler.com/articles/201701-event-driven.html @berndruecker
  69. 69. The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years. https://martinfowler.com/articles/201701-event-driven.html @berndruecker
  70. 70. Monitoring Workflows Across Microservices https://www.infoq.com/articles/monitor-workflow-collaborating-microservices @berndruecker
  71. 71. Typical approachesDistributed Tracing Data Lake / Event Monitoring Process Mining Process Tracking @berndruecker
  72. 72. What we currently build with customers… Camunda Optimize Elastic Registration requested Credit checked Address checked Customer registered @berndruecker
  73. 73. All great – until you have to move…
  74. 74. Changes required for an additional check Adress Check Credit Check Registration Criminal Check @berndruecker Customer Event Bus Registration requested Credit checked Customer registered Address checked
  75. 75. Changes required for an additional check Adress Check Credit Check Registration Criminal Check @berndruecker Customer Event Bus Registration requested Credit checked Customer registered Address checked Criminal checked
  76. 76. Alternative flow Adress Check Credit Check Registration Criminal Check @berndruecker Customer Kafka Customer registered Registration requested Address checked Credit checked Criminal checked
  77. 77. Alternative flow Adress Check Credit Check Registration Criminal Check @berndruecker Customer Kafka Customer registered Registration requested Address checked Credit checked Criminal checked „Credit checks got more expensive, do that only if all other checks succeed“
  78. 78. Keep it stable, just move sticks with yellow color to the top. How hard can it be?
  79. 79. What we wanted Photo by Lijian Zhang, available under Creative Commons SA 2.0 License and Pedobear19 / CC BY-SA 4.0 @berndruecker
  80. 80. Orchestration Adress CheckCredit Check Registration Customer Kafka Registration requested Credit checked Address checked Customer registeredCheck credit Check address Customer checked Customer On-boarding Of course these two services could be merged
  81. 81. Changes Adress CheckCredit Check Registration Customer Kafka Registration requested Credit checked Address checked Customer registeredCheck credit Check address Customer checked Customer On-boarding Criminal Check Crimes checked Check crimes
  82. 82. Comparison 2 changes criminal check can be deployed first 2 changes, criminal check can be deployed first See also https://www.infoworld.com/article/3391592/ how-to-tame-event-driven-microservices.html
  83. 83. In my world… Customer On-boarding Leverage Workflow Engine & BPMN within Service
  84. 84. Customer On-boarding Local Orchestration Central Orchestration Service
  85. 85. Recap 2 # Commands vs. Events: Decide about the direction of dependencies # Beware of event-chains and avoid losing sight # Balance choreography and orchestration @berndruecker
  86. 86. Service 2Service 1 Recap Events on the inside Events on the outside 1 3 Persistent state vs persistent change Event sourcing & Event Store Consistency & CAP Read Models & CQRS Events as API Event vs Command Event chains & visibility Orchestration vs Choreography Shared Event Store Events inside out2
  87. 87. Want to see code?
  88. 88. Nothing for the faint of heart… Events on the inside Events on the outside
  89. 89. Nothing for the faint of heart… …but doable… …and worth it
  90. 90. Thank you! @berndruecker
  91. 91. mail@berndruecker.io @berndruecker https://berndruecker.io https://medium.com/berndruecker https://github.com/berndruecker https://www.infoq.com/articles/events- workflow-automation Contact: Slides: Blog: Code: https://www.infoworld.com/article/3254777/ application-development/ 3-common-pitfalls-of-microservices- integrationand-how-to-avoid-them.html https://thenewstack.io/5-workflow-automation- use-cases-you-might-not-have-considered/
  92. 92. Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ event-driven-benefits-pitfalls/

×