Slides from my talk at the Kafka Summit San Francisco 2018 about orchestration vs. choreography as well as track vs. manage event flows. Source code for live demos available on https://github.com/berndruecker/flowing-retail

1. @berndruecker
The Big Picture
Monitoring and Orchestration
of Your Microservices
Landscape with
Kafka and Zeebe

2. Some
Service
Some
Service
Some
Service
Some
Service
Some
Service
Some
Service
Some
Service
Microservices…

3. Data, data, data, …
Source: Microservices orchestration survey,
July 2018, 354 responses

4. 92 %

5. The microservice community favours an
alternative approach: smart endpoints and
dumb pipes.
[… Microservices …] are choreographed using
simple RESTish protocols rather than complex
protocols such as WS-Choreography or BPEL or
orchestration by a central tool.
https://www.martinfowler.com/articles/microservices.html

7. De-coupled services with event backbone
Checkout Inventory
Payment Shipment
Kafka

8. Event-driven
Checkout
Payment
Inventory
Shipment
Order
Placed
Payment
Received
Goods
Fetched

9. Peer-to-peer event chain
Checkout
Payment
Inventory
Shipment

10. The danger is that it's very easy to make
nicely decoupled systems with event
notification, without realizing that you're
losing sight of that larger-scale flow, and
thus set yourself up for trouble in future
years.
https://martinfowler.com/articles/201701-event-driven.html

11. The danger is that it's very easy to make
nicely decoupled systems with event
notification, without realizing that you're
losing sight of that larger-scale flow, and
thus set yourself up for trouble in future
years.
https://martinfowler.com/articles/201701-event-driven.html

12. The danger is that it's very easy to make
nicely decoupled systems with event
notification, without realizing that you're
losing sight of that larger-scale flow, and
thus set yourself up for trouble in future
years.
https://martinfowler.com/articles/201701-event-driven.html

13. Challenges?

14. Microservice pioneers
have become aware

15. What we wanted
Photo by Lijian Zhang, available under Creative Commons SA 2.0 License and Pedobear19 / CC BY-SA 4.0

16. Finding the right balance
16
Observe Microservice Paradigms
● Loose Coupling
● Autonomy
● Event Collaboration
Ensure Flow Manageability
• End-To-End Visibility
• Meet Business Process SLA
• Support Continuous Process
Improvement

17. Workflow automation | orchestration
is an essential building block
in microservice architectures
Of course I am opinioated – but I truly believe this!

18. Workflow automation | orchestration
is an essential building block
in microservice architectures
Of course I am opinioated – but I truly believe this!
Orchestration is not a central thing!
„It is not BPEL“

19. Warning:
Contains Opinion

20. HQ Germany (Berlin) + Offices in US
bernd.ruecker@camunda.com
@berndruecker
Bernd Ruecker
Co-founder and
Developer Advocate of
Camunda

22. Different levels of leveraging workflow automation
Manage the flowTrack the flow
Non-invasive, read-only
Additionally to choreography
Limited in power
Needs to be incorporated
Mixture of choreography and orchestration
Leverage full potential

23. A possible journey from pogo to beauty…
Manage the flowTrack the flow
Non-invasive, read-only
Additionally to choreography
Limited in power
Needs to be incorporated
Mixture of choreography and orchestration
Leverage full potential

24. Tracking
Checkout Inventory
Payment Shipment
Kafka
Zeebe

25. Tracking + BI
Checkout Inventory
Payment Shipment
Kafka
Zeebe
Camunda
Optimize
Other reporting
/ BIO solutions

26. Tracking + BI
Checkout Inventory
Payment Shipment
Kafka
Workflow
Engine
Camunda
Optimize
Other reporting
/ BIO solutions

27. Tracking + BI
Checkout Inventory
Payment Shipment
Kafka
Zeebe
Camunda
Optimize
Other reporting
/ BIO solutions
This is totally non-invasive
But also limited in power

28. But you can start acting upon certain events

29. Live hacking
https://github.com/berndruecker/flowing-retail/

30. Demo architecture
Checkout Inventory
Payment Shipment
Kafka
Zeebe
Spring
Boot
Services
Spring Cloud Streams
Kafka Connect
Zeebe Simple
Monitor

31. Why so shy?
Photo by LuidmilaKot, available under Creative Commons CC0 1.0 License.

32. A possible journey away from pogo…
Manage the flowTrack the flow
Non-invasive, read-only
Additionally to choreography
Limited in power
Needs to be incorporated
Mixture of choreography and orchestration
Leverage full potential

33. Changing the flow
Checkout
Payment
Inventory
Shipment
Fetch the goods
before the
payment

34. Changing the flow
Checkout
Payment
Inventory
Shipment
Fetch the goods
before the
payment

35. Photo by born1945, available under Creative Commons BY 2.0 license.

36. Order
Checkout
Payment
Inventory
Shipment
Important responsibilities need a home

37. Distributed systems

38. Distributed systems

39. Distributed systems

40. Order
Checkout
Payment
Inventory
Shipment
Stateful orchestration
This orchestration
requires state

41. Order
Checkout
Payment
Inventory
Shipment

42. Now it is easy to change the flow

43. Manage the flow
Checkout Inventory
Payment Shipment
Kafka
Zeebe
Order
Owned heredeployed here

44. Live hacking
https://github.com/berndruecker/flowing-retail/

45. Demo architecture
Checkout Inventory
Payment Shipment
Kafka
Zeebe
Spring
Boot
Services
Spring Cloud Streams
Order
Alternative:
Kafka Connect

46. BPMN
Business Process
Model and Notation
ISO Standard

47. BPMN supports integration use cases
Aggregator
Re-Sequencer
Combined with timeouts:
Zeebe supports Message TTL

48. BPMN supports integration use cases
Compensation

49. Biz Dev Ops

50. Hybrid architectures
Order
Credit
Card
Customer
Credit
Payment
Pub/Sub
e.g. Kafka
Pub/Sub
Work Distribution
e.g. External Task
Request/Response
e.g. REST

51. Order
Checkout
Payment
Inventory
Shipment
Every service owns its workflow!

52. Central Zeebe or …
Checkout Inventory
Payment Shipment
Kafka
Zeebe
Order
Owned heredeployed here
Owned here

53. …decentral Zeebe
Checkout Inventory
Payment Shipment
Kafka
ZeebeOrderZeebe

54. Workflow automation at scale!
low latency,
high-throughput
low frequency,
latency doesn‘t matter
What people think workflow
automation can do
What we currently teach
workflow automation
to be able to do
What workflow automation
can already do today

56. Vodafone, Liongate & WDW
Presented at CamundaCOn Berlin 2018

57. Recap
• In event-driven-systems it is easy to loose sight of the flow
• Visibility is essential to survive
• You need to balance orchestration and chorography
• Distributed systems need stateful orchestration
• Track, monitor or manage the flow – it might be a journey
• Workflow automation is an essential building block
• BPMN is very powerful and provides BizDevOps

58. Thank you!

59. mail@berndruecker.io
@berndruecker
https://berndruecker.io
https://medium.com/berndruecker
https://github.com/berndruecker
https://www.infoq.com/articles/events-
workflow-automation
Contact:
Slides:
Blog:
Code:
https://www.infoworld.com/article/3254777/
application-development/
3-common-pitfalls-of-microservices-
integrationand-how-to-avoid-them.html
https://thenewstack.io/5-workflow-automation-
use-cases-you-might-not-have-considered/