Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
@berndruecker
Monitoring and Orchestration
of Your Microservices
Landscape with
Kafka and Zeebe
Berlin, Germany
mail@berndruecker.io
@berndruecker
Bernd Ruecker
Co-founder and
Chief Technologist of
Camunda
@berndruecker
Simplified example:
dash button
Photo by 0xF2, available under Creative Commons BY-ND 2.0
license. https://www.flickr.com/...
Three steps…
@berndruecker
(Micro-)services
Checkout
Payment
Inventory
Shipment
@berndruecker
Autonomous (micro-)services
Checkout
Payment
Inventory
Shipment
Dedicated Application Processes
Dedicated infrastructure
D...
Data, data, data, …
Source: Microservices orchestration survey,
July 2018, 354 responses
@berndruecker
92 %
@berndruecker
Example
Checkout
Payment
Inventory
Shipment
The button blinks if we can
ship within 24 hours
@berndruecker
Request/response: temporal coupling
Checkout
Payment
Inventory
Shipment
Request
Response
The button blinks if we can
ship ...
Temporal decoupling with events and read models
Checkout
Payment
Inventory
Shipment
Good
Stored
Read
Model
Good
Fetched
Th...
Order
Placed
Payment
Received
Goods
Fetched
Notification
Checkout
Payment
Inventory
Shipment
Event-driven architecture
@be...
Peer-to-peer event chains
Checkout
Payment
Inventory
Shipment
Order
placed
Payment
received
Goods
shipped
Goods
fetched
@b...
Peer-to-peer event chains
Checkout
Payment
Inventory
Shipment
Order
placed
Payment
received
Goods
shipped
Goods
fetched
@b...
The danger is that it's very easy to make
nicely decoupled systems with event
notification, without realizing that you're
...
The danger is that it's very easy to make
nicely decoupled systems with event
notification, without realizing that you're
...
The danger is that it's very easy to make
nicely decoupled systems with event
notification, without realizing that you're
...
Monitoring Workflows Across Microservices
https://www.infoq.com/articles/monitor-workflow-collaborating-microservices
@ber...
Typical approachesDistributed Tracing
Data Lake / Event Monitoring
Process Mining
Process Tracking
@berndruecker
Tracking
Checkout Inventory
Payment Shipment
Kafka
Zeebe
@berndruecker
Tracking + BI
Checkout Inventory
Payment Shipment
Kafka
Zeebe
Camunda
Optimize
Other reporting
/ BIO solutions
@berndrueck...
Tracking + BI
Checkout Inventory
Payment Shipment
Kafka
Zeebe
Camunda
Optimize
Other reporting
/ BIO solutions
@berndrueck...
Tracking + BI
Checkout Inventory
Payment Shipment
Kafka
Zeebe
Camunda
Optimize
Other reporting
/ BIO solutions
This is tot...
Live hacking
https://github.com/berndruecker/flowing-retail/
Zeebe.io
Zeebe Architecture
Operate
We do two workflow engines
Camunda ZeebePersistent
State
Persistent
change
RDMS
Demo architecture
Checkout Inventory
Payment Shipment
Kafka
Zeebe
Spring
Boot
Services
Spring Cloud Streams
Kafka Connect
...
But you can start acting upon certain events
@berndruecker
Vodafone, Liongate & WDW
Presented at CamundaCOn Berlin 2018
@berndruecker
What we currently build with some customers…
Camunda
Optimize
Elastic
17
Registration
requested
Credit
checked
Address
che...
Stefan Tilkov: Microservice Patterns & Antipatterns - MicroXchg 2018
@berndruecker
Peer-to-peer event chains
Checkout
Payment
Inventory
Shipment
Order
placed
Payment
received
Goods
shipped
Goods
fetched
Fe...
Peer-to-peer event chains
Checkout
Payment
Inventory
Shipment
Fetch the goods
before the
payment
Goods
fetched
Order
place...
What we wanted
Photo by Lijian Zhang, available under Creative Commons SA 2.0 License and Pedobear19 / CC BY-SA 4.0
@bernd...
„Challenges?“
Source:
Microservices orchestration survey,
July 2018, 354 responses
https://camunda.com/microservices-orche...
Order
Extract the end-to-end responsibility
Checkout
Payment
Inventory
Shipment
*Commands have an intent about
what needs ...
Order
It is about where to decide about the coupling!
Checkout
Payment
Inventory
Shipment
Order
placed
Retrieve
payment
Or...
Order
It is about where to decide about the coupling!
Checkout
Payment
Inventory
Shipment
Order
placed
Retrieve
payment
It...
Order
Checkout
Payment
Inventory
Shipment
Stateful orchestration
This orchestration
requires state
@berndruecker
Order
Checkout
Payment
Inventory
Shipment
@berndruecker
Now it is easy to change the flow
@berndruecker
Live hacking
https://github.com/berndruecker/flowing-retail/
Demo architecture
Checkout Inventory
Payment Shipment
Kafka
Zeebe
Spring
Boot
Services
Spring Cloud Streams
Order
Alternat...
BPMN
Business Process
Model and Notation
ISO Standard
@berndruecker
BPMN supports integration use cases
Aggregator
Re-Sequencer
Combined with timeouts:
Zeebe supports Message TTL
@berndrueck...
BPMN supports integration use cases
Compensation
@berndruecker
Biz Dev Ops
@berndruecker
Hybrid architectures
Order
Credit
Card
Customer
Credit
Payment
Pub/Sub
e.g. Kafka
Pub/Sub
Work Distribution
e.g. External ...
Order
Checkout
Payment
Inventory
Shipment
Every service owns its workflow!
@berndruecker
Finding the right balance
61
Observe Microservice Paradigms
● Loose Coupling
● Autonomy
● Event Collaboration
Ensure Flow ...
@berndruecker
Kafka
Zeebe
Produce
Record
Consume
Records
Event-
Stream
@berndruecker
Kafka
Zeebe
low latency,
high-throughput
low frequency,
latency doesn‘t matter
What people think workflow
au...
How?
@berndruecker
@berndruecker
Recap
• In event-driven-systems it is easy to loose sight of the flow
• Visibility is essential to survive
• You need to b...
Thank you!
@berndruecker
mail@berndruecker.io
@berndruecker
https://berndruecker.io
https://medium.com/berndruecker
https://github.com/berndruecker...
Prochain SlideShare
Chargement dans…5
×

Monitoring and Orchestration of your Microservices Landscape with Kafka and Zeebe (Meetup)

468 vues

Publié le

Slides from a joined meetup of Confluent/Kafka and Camunda/zeebe

Publié dans : Technologie
  • Soyez le premier à commenter

Monitoring and Orchestration of your Microservices Landscape with Kafka and Zeebe (Meetup)

  1. 1. @berndruecker Monitoring and Orchestration of Your Microservices Landscape with Kafka and Zeebe
  2. 2. Berlin, Germany mail@berndruecker.io @berndruecker Bernd Ruecker Co-founder and Chief Technologist of Camunda
  3. 3. @berndruecker
  4. 4. Simplified example: dash button Photo by 0xF2, available under Creative Commons BY-ND 2.0 license. https://www.flickr.com/photos/0xf2/29873149904/ @berndruecker
  5. 5. Three steps… @berndruecker
  6. 6. (Micro-)services Checkout Payment Inventory Shipment @berndruecker
  7. 7. Autonomous (micro-)services Checkout Payment Inventory Shipment Dedicated Application Processes Dedicated infrastructure Dedicated Development Teams @berndruecker
  8. 8. Data, data, data, … Source: Microservices orchestration survey, July 2018, 354 responses @berndruecker
  9. 9. 92 % @berndruecker
  10. 10. Example Checkout Payment Inventory Shipment The button blinks if we can ship within 24 hours @berndruecker
  11. 11. Request/response: temporal coupling Checkout Payment Inventory Shipment Request Response The button blinks if we can ship within 24 hours @berndruecker
  12. 12. Temporal decoupling with events and read models Checkout Payment Inventory Shipment Good Stored Read Model Good Fetched The button blinks if we can ship within 24 hours *Events are facts about what happened (in the past) @berndruecker
  13. 13. Order Placed Payment Received Goods Fetched Notification Checkout Payment Inventory Shipment Event-driven architecture @berndruecker
  14. 14. Peer-to-peer event chains Checkout Payment Inventory Shipment Order placed Payment received Goods shipped Goods fetched @berndruecker
  15. 15. Peer-to-peer event chains Checkout Payment Inventory Shipment Order placed Payment received Goods shipped Goods fetched @berndruecker
  16. 16. The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years. https://martinfowler.com/articles/201701-event-driven.html @berndruecker
  17. 17. The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years. https://martinfowler.com/articles/201701-event-driven.html @berndruecker
  18. 18. The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years. https://martinfowler.com/articles/201701-event-driven.html @berndruecker
  19. 19. Monitoring Workflows Across Microservices https://www.infoq.com/articles/monitor-workflow-collaborating-microservices @berndruecker
  20. 20. Typical approachesDistributed Tracing Data Lake / Event Monitoring Process Mining Process Tracking @berndruecker
  21. 21. Tracking Checkout Inventory Payment Shipment Kafka Zeebe @berndruecker
  22. 22. Tracking + BI Checkout Inventory Payment Shipment Kafka Zeebe Camunda Optimize Other reporting / BIO solutions @berndruecker
  23. 23. Tracking + BI Checkout Inventory Payment Shipment Kafka Zeebe Camunda Optimize Other reporting / BIO solutions @berndruecker
  24. 24. Tracking + BI Checkout Inventory Payment Shipment Kafka Zeebe Camunda Optimize Other reporting / BIO solutions This is totally non-invasive But also limited in power @berndruecker
  25. 25. Live hacking https://github.com/berndruecker/flowing-retail/
  26. 26. Zeebe.io
  27. 27. Zeebe Architecture Operate
  28. 28. We do two workflow engines Camunda ZeebePersistent State Persistent change RDMS
  29. 29. Demo architecture Checkout Inventory Payment Shipment Kafka Zeebe Spring Boot Services Spring Cloud Streams Kafka Connect Operate @berndruecker
  30. 30. But you can start acting upon certain events @berndruecker
  31. 31. Vodafone, Liongate & WDW Presented at CamundaCOn Berlin 2018 @berndruecker
  32. 32. What we currently build with some customers… Camunda Optimize Elastic 17 Registration requested Credit checked Address checked Customer registered @berndruecker
  33. 33. Stefan Tilkov: Microservice Patterns & Antipatterns - MicroXchg 2018 @berndruecker
  34. 34. Peer-to-peer event chains Checkout Payment Inventory Shipment Order placed Payment received Goods shipped Goods fetched Fetch the goods before the payment @berndruecker
  35. 35. Peer-to-peer event chains Checkout Payment Inventory Shipment Fetch the goods before the payment Goods fetched Order placed Payment received Goods shipped @berndruecker
  36. 36. What we wanted Photo by Lijian Zhang, available under Creative Commons SA 2.0 License and Pedobear19 / CC BY-SA 4.0 @berndruecker
  37. 37. „Challenges?“ Source: Microservices orchestration survey, July 2018, 354 responses https://camunda.com/microservices-orchestration-survey-results-2018/ @berndruecker
  38. 38. Order Extract the end-to-end responsibility Checkout Payment Inventory Shipment *Commands have an intent about what needs to happen in the future Payment received Order placed Retrieve payment @berndruecker
  39. 39. Order It is about where to decide about the coupling! Checkout Payment Inventory Shipment Order placed Retrieve payment Order decides . to listen to the event . to issue the command @berndruecker
  40. 40. Order It is about where to decide about the coupling! Checkout Payment Inventory Shipment Order placed Retrieve payment It can still be messaging! @berndruecker
  41. 41. Order Checkout Payment Inventory Shipment Stateful orchestration This orchestration requires state @berndruecker
  42. 42. Order Checkout Payment Inventory Shipment @berndruecker
  43. 43. Now it is easy to change the flow @berndruecker
  44. 44. Live hacking https://github.com/berndruecker/flowing-retail/
  45. 45. Demo architecture Checkout Inventory Payment Shipment Kafka Zeebe Spring Boot Services Spring Cloud Streams Order Alternative: Kafka Connect @berndruecker
  46. 46. BPMN Business Process Model and Notation ISO Standard @berndruecker
  47. 47. BPMN supports integration use cases Aggregator Re-Sequencer Combined with timeouts: Zeebe supports Message TTL @berndruecker
  48. 48. BPMN supports integration use cases Compensation @berndruecker
  49. 49. Biz Dev Ops @berndruecker
  50. 50. Hybrid architectures Order Credit Card Customer Credit Payment Pub/Sub e.g. Kafka Pub/Sub Work Distribution e.g. External Task Request/Response e.g. REST @berndruecker
  51. 51. Order Checkout Payment Inventory Shipment Every service owns its workflow! @berndruecker
  52. 52. Finding the right balance 61 Observe Microservice Paradigms ● Loose Coupling ● Autonomy ● Event Collaboration Ensure Flow Manageability • End-To-End Visibility • Meet Business Process SLA • Support Continuous Process Improvement @berndruecker
  53. 53. @berndruecker Kafka Zeebe Produce Record Consume Records Event- Stream
  54. 54. @berndruecker Kafka Zeebe low latency, high-throughput low frequency, latency doesn‘t matter What people think workflow automation can do What we currently teach workflow automation to be able to do What workflow automation can already do today Workflow Automation at scale
  55. 55. How? @berndruecker
  56. 56. @berndruecker
  57. 57. Recap • In event-driven-systems it is easy to loose sight of the flow • Visibility is essential to survive • You need to balance orchestration and chorography • Distributed systems need stateful orchestration • Workflow automation is an essential building block • BPMN is very powerful and provides BizDevOps • Zeebe rocks :-) @berndruecker
  58. 58. Thank you! @berndruecker
  59. 59. mail@berndruecker.io @berndruecker https://berndruecker.io https://medium.com/berndruecker https://github.com/berndruecker https://www.infoq.com/articles/events- workflow-automation Contact: Slides: Blog: Code: https://www.infoworld.com/article/3254777/ application-development/ 3-common-pitfalls-of-microservices- integrationand-how-to-avoid-them.html https://thenewstack.io/5-workflow-automation- use-cases-you-might-not-have-considered/

×