Slides from my talk at JAX 2019

1. Workflow Automation
Reinvented
@berndruecker

2. What people think, when I say „Workflow Automation“…

3. What I think, when I say „Workflow Automation“
…

4. What people think, when I say „BPM“…
Death by properties panel
Low-code is great!
(You can get rid
of your developers!)
@berndruecker

5. What I think, when I say „BPM“
+

7. public static void main(String[] args) {
ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration()
.buildProcessEngine();
engine.getRepositoryService().createDeployment() //
.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") //
.startEvent()
.serviceTask("Step1").camundaClass(SysoutDelegate.class)
.serviceTask("Step2").camundaClass(SysoutDelegate.class)
.endEvent()
.done()
).deploy();
engine.getRuntimeService().startProcessInstanceByKey(
"flow", Variables.putValue("city", "New York"));
}
public class SysoutDelegate implements JavaDelegate {
public void execute(DelegateExecution execution) throws Exception {
System.out.println("Hello " + execution.getVariable("city"));
}
}
What do I mean by
„lightweight?“
@berndruecker

8. public static void main(String[] args) {
ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration()
.buildProcessEngine();
engine.getRepositoryService().createDeployment() //
.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") //
.startEvent()
.serviceTask("Step1").camundaClass(SysoutDelegate.class)
.serviceTask("Step2").camundaClass(SysoutDelegate.class)
.endEvent()
.done()
).deploy();
engine.getRuntimeService().startProcessInstanceByKey(
"flow", Variables.putValue("city", "New York"));
}
public class SysoutDelegate implements JavaDelegate {
public void execute(DelegateExecution execution) throws Exception {
System.out.println("Hello " + execution.getVariable("city"));
}
}
Build engine
in one line of
code
(using in-
memory H2)
@berndruecker

9. public static void main(String[] args) {
ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration()
.buildProcessEngine();
engine.getRepositoryService().createDeployment() //
.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow")
.startEvent()
.serviceTask("Step1").camundaClass(SysoutDelegate.class)
.serviceTask("Step2").camundaClass(SysoutDelegate.class)
.endEvent()
.done()
).deploy();
engine.getRuntimeService().startProcessInstanceByKey(
"flow", Variables.putValue("city", "New York"));
}
public class SysoutDelegate implements JavaDelegate {
public void execute(DelegateExecution execution) throws Exception {
System.out.println("Hello " + execution.getVariable("city"));
}
}
Define flow
e.g. in Java
DSL
@berndruecker

10. public static void main(String[] args) {
ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration()
.buildProcessEngine();
engine.getRepositoryService().createDeployment() //
.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow")
.startEvent()
.serviceTask("Step1").camundaClass(SysoutDelegate.class)
.serviceTask("Step2").camundaClass(SysoutDelegate.class)
.endEvent()
.done()
).deploy();
engine.getRuntimeService().startProcessInstanceByKey(
"flow", Variables.putValue("city", "New York"));
}
public class SysoutDelegate implements JavaDelegate {
public void execute(DelegateExecution execution) throws Exception {
System.out.println("Hello " + execution.getVariable("city"));
}
}
Define flow
e.g. in Java
DSL
@berndruecker

11. BPMN
Business Process
Model and Notation
ISO Standard
@berndruecker

12. public static void main(String[] args) {
ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration()
.buildProcessEngine();
engine.getRepositoryService().createDeployment() //
.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow")
.startEvent()
.serviceTask("Step1").camundaClass(SysoutDelegate.class)
.serviceTask("Step2").camundaClass(SysoutDelegate.class)
.endEvent()
.done()
).deploy();
engine.getRuntimeService().startProcessInstanceByKey(
"flow", Variables.putValue("city", "New York"));
}
public class SysoutDelegate implements JavaDelegate {
public void execute(DelegateExecution execution) throws Exception {
System.out.println("Hello " + execution.getVariable("city"));
}
}
We can attach
code…
@berndruecker

13. public static void main(String[] args) {
ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration()
.buildProcessEngine();
engine.getRepositoryService().createDeployment() //
.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow")
.startEvent()
.serviceTask("Step1").camundaClass(SysoutDelegate.class)
.serviceTask("Step2").camundaClass(SysoutDelegate.class)
.endEvent()
.done()
).deploy();
engine.getRuntimeService().startProcessInstanceByKey(
"flow", Variables.putValue("city", "New York"));
}
public class SysoutDelegate implements JavaDelegate {
public void execute(DelegateExecution execution) throws Exception {
System.out.println("Hello " + execution.getVariable("city"));
}
}
…that is
called when
workflow
instances pass
through
@berndruecker

14. public static void main(String[] args) {
ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration()
.buildProcessEngine();
engine.getRepositoryService().createDeployment() //
.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow")
.startEvent()
.serviceTask("Step1").camundaClass(SysoutDelegate.class)
.serviceTask("Step2").camundaClass(SysoutDelegate.class)
.endEvent()
.done()
).deploy();
engine.getRuntimeService().startProcessInstanceByKey(
"flow", Variables.putValue("city", "New York"));
}
public class SysoutDelegate implements JavaDelegate {
public void execute(DelegateExecution execution) throws Exception {
System.out.println("Hello " + execution.getVariable("city"));
}
}
Start
instances
@berndruecker

15. Architecture choices
Your app or service
WF Engine
Your app or service
WF
Engine
WF Engine
logic or
service
WF Engine
logic or
service
Call,
send message,
connector, …
(Long-)Polling
Pub/Sub
PushEmbedded
Server Pull
vs vs
See also https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91

16. Why workflow automation?

17. https://thenewstack.io/5-workflow-automation-use-cases-you-might-not-have-considered/

18. <= milliseconds seconds minutes, weeks, …
Business
IT
long runningalways short running short running, but
potentially long running
Use cases for workflow automation

19. <= milliseconds seconds minutes, weeks, …
Business
IT
Business processes
automation
long runningalways short running short running, but
potentially long running
Use cases for workflow automation

20. Real-life examples

21. <= milliseconds seconds minutes, weeks, …
Business
IT
Business processes
automation
long runningalways short running short running, but
potentially long running
Use cases for workflow automation

24. <= milliseconds seconds minutes, weeks, …
Business
IT
Business processes
automation
long runningalways short running short running, but
potentially long running
Use cases for workflow automation
Communication in
distributed systems

25. Payment
Ever called a RESTAPI?
Credit
Card
REST

26. Distributed systems
“A distributed system is one in which the failure of a computer
you didn’t even know existed can render your own computer unusable.”
Lamport

27. Payment
Stateful retry
Credit
Card
REST

28. Distributed systems introduce complexity you have to tackle!
Credit
Card
Payment
REST

29. It is impossible to
differentiate certain
failure scenarios.
Independant of
communication style!
Service
Provider
Client

30. Distributed systems introduce complexity you have to tackle!
Credit
Card
Payment
REST

31. Distributed systems introduce complexity you have to tackle!
Credit
Card
Payment
REST
Cancel
charge

32. Live hacking

33. Warning:
Contains Opinion

34. Berlin, Germany
mail@berndruecker.io
@berndruecker
Bernd Ruecker
Co-founder and
Chief Technologist of
Camunda

35. Live hacking
https://github.com/berndruecker/flowing-retail/tree/master/rest/java/payment-camunda
https://github.com/berndruecker/flowing-retail-payment-rest-spring-statemachine

36. Embedded, Push
Your app or service
WF Engine
logic or
service
Call,
send message,
connector, …
Push
Embedded
Raises questions around
Threads,
Transactions, …

37. Spring StateMachine docs

38. Aspects to consider
Camunda Spring State Machine
Workflow Definition BPMN 2.0
(graphical, XML, Model API)
Java API, UML-Generator
Visual? YES No
Tooling Modeler, Cockpit, Optimize -
Storage Runtime RDMS up to you
Storage History RDMS -
Scalability Stateless Engine,
RDMS is limit,
Sharding possible
up to you
Fault tolerance If RDMS is HA up to you
Supported programming
languages
Java, REST,
Language Clients (JS, C#)
Java
HOMEWORK

39. See more on http://berndruecker.io/

40. Long running services*
provide a better API!
@berndruecker
* Services that have the capability to keep state and thus can be potentially long running

41. Example
Order Payment
Retrieve
Payment
@berndruecker

42. Example
Order Payment
Credit
Card
Retrieve
Payment
@berndruecker

43. Example
Order Payment
Credit
Card
Retrieve
Payment
Rejected
@berndruecker

44. Example
Order Payment
If the credit
card was
rejected, the
customer can
provide new
details
Credit
Card
Retrieve
Payment
Rejected
Rejected
@berndruecker

45. Example
Order Payment
Client of dumb endpoints easily become a god services.
If the credit
card was
rejected, the
customer can
provide new
details
Credit
Card
Retrieve
Payment
Rejected
Rejected
@berndruecker

46. Payment
failed
Who is responsible to deal with problems?
Order Payment
If the credit
card was
rejected, the
customer can
provide new
details
Credit
Card
Retrieve
Payment
Rejected
Payment
received
@berndruecker

47. Payment
failed
Long running services
Order Payment
Credit
Card
Retrieve
Payment
Rejected
Payment
received
Smart endpoints are
potentially long-running
@berndruecker

48. <= milliseconds seconds minutes, weeks, …
Business
IT
Business processes
automation
long runningalways short running short running, but
potentially long running
Use cases for workflow automation
Communication in
distributed systems

49. <= milliseconds seconds minutes, weeks, …
Business
IT
Business processes
automation
Communication in
distributed systems
long runningalways short running short running, but
potentially long running
Use cases for workflow automation
Orchestration

50. Microservices…
Service
B
Some
Service
Some
Service Some
Service
Service
C
Service
A
Service
D
Monolith
Functionality
A
Functionality
B
Functionality
C
Functionality
D

52. Simplified version…
@berndruecker

53. (Micro-)services
Checkout
Payment
Inventory
Shipment
@berndruecker

54. Order
Placed
Payment
Received
Goods
Fetched
Notification
Event-driven architecture
@berndruecker
Checkout
Payment
Inventory
Shipment

55. Peer-to-peer event chains
Checkout
Payment
Inventory
Shipment
Order
placed
Payment
received
Goods
shipped
Goods
fetched
@berndruecker

56. Peer-to-peer event chains
Checkout
Payment
Inventory
Shipment
Order
placed
Payment
received
Goods
shipped
Goods
fetched
@berndruecker

57. The danger is that it's very easy to make
nicely decoupled systems with event
notification, without realizing that you're
losing sight of that larger-scale flow, and
thus set yourself up for trouble in future
years.
https://martinfowler.com/articles/201701-event-driven.html
@berndruecker

58. The danger is that it's very easy to make
nicely decoupled systems with event
notification, without realizing that you're
losing sight of that larger-scale flow, and
thus set yourself up for trouble in future
years.
https://martinfowler.com/articles/201701-event-driven.html
@berndruecker

59. The danger is that it's very easy to make
nicely decoupled systems with event
notification, without realizing that you're
losing sight of that larger-scale flow, and
thus set yourself up for trouble in future
years.
https://martinfowler.com/articles/201701-event-driven.html
@berndruecker

60. Peer-to-peer event chains
Checkout
Payment
Inventory
Shipment
Order
placed
Payment
received
Goods
shipped
Goods
fetched
Fetch the goods
before the
payment
@berndruecker

61. Peer-to-peer event chains
Checkout
Payment
Inventory
Shipment
Fetch the goods
before the
payment
Goods
fetched
Order
placed
Payment
received
Goods
shipped
@berndruecker

62. What we wanted
Photo by Lijian Zhang, available under Creative Commons SA 2.0 License and Pedobear19 / CC BY-SA 4.0
@berndruecker

63. Order
Extract the end-to-end responsibility
Checkout
Payment
Inventory
Shipment
*Commands have an intent about
what needs to happen in the future
Payment
received
Order
placed
Retrieve
payment
@berndruecker

64. Order
It still can be messaging!
Checkout
Payment
Inventory
Shipment
Order
placed
Retrieve
payment
@berndruecker

65. Workflows implement stateful orchestration logic
@berndruecker

66. No BPM(N) monoliths
https://blog.bernd-ruecker.com/avoiding-the-bpm-monolith-when-using-bounded-contexts-d86be6308d8
@berndruecker

67. Microservice pioneers
have become aware

68. https://netflix.github.io/conductor/metadata/kitchensink/

69. Live hacking
https://github.com/berndruecker/flowing-retail/

70. Architecture choices
WF
Engine
Service
Server
Pull
Service
WF
Engine
Service
Server
Queue /
Topic Name
Topic Name
Service
Pull

71. Aspects to consider
Zeebe Conductor
Workflow Definition BPMN 2.0 (graphical, XML,
Model API)
JSON
Visual? YES No
Tooling Modeler, Operate Simple UI
Storage Runtime File System (via RocksDB) Dynomite + Elasticsearch
Storage History Elasticsearch -
Supported programming
languages
grpc,
Language clients (Java, C#,
JS, Python, C#)
REST, Language Clients
(Java, Python), grpc
Licensing ASL ASL
HOMEWORK

72. Camunda vs. Zeebe
Camunda ZeebePersistent
State
Persistent
change
RDMS

73. Workflow
Instance
Id
Current
Activity
State
2 RetrievePayment running
Workflow
Instance
Id
Current
Activity
State
2 ShipGoods running
Workflow
Instance
Id
Current
Activity
State
2 OrderDelivered ended
2.
3. UPDATE
2. UPDATE
1. INSERT
3.1.
RDMS
Workflow
Engine

74. Append-only
Log1.
create
workflow
instance
workflow
instance
created
start
event
occured
sequence
flow taken
activity
activated
task
created
lock
created
task
locked
complete
task
task
completed
activity
completed
sequence
flow taken
…
…
2.
2.
3.1.
Workflow
Engine

75. Event Handling, Replication & Single Writer
Follower
Follower
complete task
command
task completed
event
1 send
2 append command
Broker
(Leader)
Stream
Processor
4 process
7
store & replicate
event
6 append event
3
store & replicate
command
5 respond
Single Writer
(single thread)

76. Horizontal scalability by partitioning
Partition 1
Partition 2
Partition 3
Partition 4
Every workflow instance is
exactly handled by one
partition
instance id: 2-42
instance id: 3-66
Stream
Processor Single Writer
(single thread)

77. See more on http://berndruecker.io/

78. <= milliseconds seconds minutes, weeks, …
Business
IT
Business processes
automation
Communication in
distributed systems
long runningalways short running short running, but
potentially long running
Use cases for workflow automation
Orchestration

79. <= milliseconds seconds minutes, weeks, …
Business
IT
Business processes
automation
Communication in
distributed systems
long runningalways short running short running, but
potentially long running
Use cases for workflow automation
Orchestration
Consistency

80. The classical example
book
hotel
book
car
book
flight
1. 2. 3.
Failure!
book
trip

81. Pat Helland
“
Distributed Systems Guru
Worked at Amazon,
Microsoft & Salesforce
@berndruecker

82. Pat Helland
Grown-Ups Don’t Use
Distributed Transactions
“
Distributed Systems Guru
Worked at Amazon,
Microsoft & Salesforce
@berndruecker

83. The classical example
Saga
book
hotel
book
car
book
flight
cancel
hotel
cancel
car
1. 2. 3.
5.6.
Failure
book
trip
Trigger compensations

84. Live hacking

85. BPMN
Saga Pattern
(implemented by BPMN compensation)
https://github.com/berndruecker/trip-booking-saga-java

86. Step Functions
https://read.acloud.guru/how-the-saga-pattern-manages-failures-with-aws-lambda-and-
step-functions-bc8f7129f900

87. Flow language is important!
Think of more complicated scenarios…

88. Architecture for Serverless
WF Engine
(Long-)Polling Push
Pull
function
Server
glue layer
logic or
service
Easy to hop
over firewalls

89. Aspects to consider
Camunda AWS Step Functions
Workflow
Definition
BPMN 2.0 (graphical, XML,
Model API)
JSON
Visual? YES Simple, only runtime
Tooling Modeler, Cockpit,
Optimize
-
Scalability Stateless Engine
With RDMS
Up to ~ 1k state
transitions / sec*
Deployment Embedded or Server Cloud (Server)
Licensing ASL Proprietary
HOMEWORK

90. See more on http://berndruecker.io/

91. <= milliseconds seconds minutes, weeks, …
Business
IT
Business processes
automation
Distributed
Transactions
Orchestration
Communication in
distributed systems
long runningalways short running short running, but
potentially long running
Use cases for workflow automation

92. <= milliseconds seconds minutes, weeks, …
Business
IT
Business processes
automation
Distributed
Transactions
Decision
Automation
Orchestration
Communication in
distributed systems
long runningalways short running short running, but
potentially long running
Use cases for workflow automation

93. Photo by ErikaWittlieb, available under Creative Commons CC0 1.0 license.

94. Decisions with DMN

95. Workflow Automation is important in modern architectures!

96. Thoughts on the state machine | workflow engine market

97. Thoughts on the state machine | workflow engine market
OSS Workflow or
Orchestration Engines
Stack Vendors,
Pure Play BPMS
Low Code Platforms
Homegrown frameworks
to scratch an itch
Integration Frameworks
Cloud Offerings
Uber, Netflix, AirBnb, ING, … AWS Step Functions,
Azure Durable Functions, …
Camunda, Zeebe, jBPM,
Activiti, Flowable, Mistral, …
PEGA, IBM, SAG, …
Apache Airflow,
Spring Data Flow, …
Apache Camel,
Balerina, …
Data
Pipelines

98. Does it support stateful operations?
Does it support the necessary flow logic?
Does it support BizDevOps?
Does it scale?

99. https://zeebe.io/blog/2019/03/zeebe-comparison-evaluation-criteria/

100. My personal pro-tip for a shortlist ;-)
OSS Workflow or
Orchestration Engines
Stack Vendors,
Pure Play BPMS
Low Code Platforms
Homegrown frameworks
to scratch an itch
Integration Frameworks
Cloud Offerings
Data
Pipelines
Camunda & Zeebe

101. Thank you!
@berndruecker

102. mail@berndruecker.io
@berndruecker
https://berndruecker.io
https://medium.com/berndruecker
https://github.com/berndruecker
https://www.infoq.com/articles/events-
workflow-automation
Contact:
Slides:
Blog:
Code:
https://www.infoworld.com/article/3254777/
application-development/
3-common-pitfalls-of-microservices-
integrationand-how-to-avoid-them.html
https://thenewstack.io/5-workflow-automation-
use-cases-you-might-not-have-considered/