The document presents an overview of the challenges in developing a benchmark for Business Process Model and Notation (BPMN) workflow management systems (WfMS). Key challenges include defining a realistic workload model based on real-world BPMN processes, handling asynchronous workflow execution, and determining relevant performance metrics and key performance indicators. The goal is to design the first benchmark that can assess and compare the performance of WfMSs compliant with the BPMN 2.0 standard.
[2024]Digital Global Overview Report 2024 Meltwater.pdf
Towards a Benchmark for BPMN Engines
1. Vincenzo Ferme, Cesare Pautasso | University of Lugano (USI)
Towards a
Benchmark for
BPMN Engines
2. Towards a Benchmark for BPMN Engines
2
University of Stuttgart (IAAS)
Germany
University of Lugano (USI)
Switzerland
Prof. Cesare Pautasso
Vincenzo Ferme
Ana Ivanchikj
Marco Argenti
Gabriele Cerfoglio
Simone D’Avico
Abdil Cakal,Andreia Faria Carvalho
Nicolò Linder, Sonny Monti
Prof. Frank Leymann
Dr. Dieter H. Roller
Marigianna Skouradaki
Abdul Wahab
Nagarjuna Siddam
Tayyaba Azad
BaluVenu Thayil
Credits
3. Towards a Benchmark for BPMN Engines
3
The Impact of TPC Benchmark on DBMS Performance
Source: http://www.tpc.org/tpcc/results/tpcc_results.asp
4. Towards a Benchmark for BPMN Engines
4
What is a Workflow?
Register
Reserv.
Send
Doc.
Confirm
Request
5. Towards a Benchmark for BPMN Engines
4
What is a Workflow?
Register
Reserv.
Send
Doc.
Confirm
Request
Bob
1
6. Towards a Benchmark for BPMN Engines
4
What is a Workflow?
Register
Reserv.
Send
Doc.
Confirm
Request
Bob
1
Reservation
Service
2
Bob
2
7. Towards a Benchmark for BPMN Engines
4
What is a Workflow?
Register
Reserv.
Send
Doc.
Confirm
Request
Bob
1
Reservation
Service
2
Bob
2
Alice
3
8. Towards a Benchmark for BPMN Engines
4
What is a Workflow?
Register
Reserv.
Send
Doc.
Confirm
Request
Bob
1
Reservation
Service
2
Bob
2
Alice
3
4
Confirmation
Handler
9. Towards a Benchmark for BPMN Engines
Workflow Management System
4
What is a Workflow?
Register
Reserv.
Send
Doc.
Confirm
Request
Bob
1
Reservation
Service
2
Bob
2
Alice
3
4
Confirmation
Handler
10. Towards a Benchmark for BPMN Engines
Applications
What is a Workflow Management System (WfMS)?
5
WfMS Users
Instance
Database
Application Server
DBMS
Web
Service
D
A
B
C
11. Towards a Benchmark for BPMN Engines
6Application Server
Workflow Management System’s Main Components
WES
Job
Executor
Core
Engine
Application Server
12. Towards a Benchmark for BPMN Engines
6Application Server
Workflow Management System’s Main Components
WES
Job
Executor
Core
Engine
Process Navigator
D
A
B
C
Application Server
13. Towards a Benchmark for BPMN Engines
6Application Server
Workflow Management System’s Main Components
WES
Job
Executor
Core
Engine
Process Navigator
D
A
B
C
Task Dispatcher Users
Application
Application Server
14. Towards a Benchmark for BPMN Engines
6Application Server
Workflow Management System’s Main Components
WES
Job
Executor
Core
Engine
Process Navigator
D
A
B
C
Task Dispatcher Users
Application
Application Server
Event Handler
Service Invoker
Web
Service
Application
15. Towards a Benchmark for BPMN Engines
6Application Server
Workflow Management System’s Main Components
WES
Job
Executor
Core
Engine
Process Navigator
D
A
B
C
Instance
Database
DBMS
Persistent Manager
Transaction Manager
Task Dispatcher Users
Application
Application Server
Event Handler
Service Invoker
Web
Service
Application
16. Towards a Benchmark for BPMN Engines
System’s Architecture
• Distributed workflow support
• Migrating workflow objects support
• Transactional workflow support
Deployment Infrastructure
• Standalone
• Cluster deployment
• Cloud deployment
• Mobile deployment
Supported Languages
• BPMN, BPEL, Petri-Nets, …
7
Functionality
• Dynamic handling of Workflows
• Integration capabilities
Workflow Management Systems’ Diversity
BPELEPC XPDL BPMN
1992 1998 2002 2004
PNML
2008
BPMN 2.0
2011
17. Towards a Benchmark for BPMN Engines
System’s Architecture
• Distributed workflow support
• Migrating workflow objects support
• Transactional workflow support
Deployment Infrastructure
• Standalone
• Cluster deployment
• Cloud deployment
• Mobile deployment
Supported Languages
• BPMN, BPEL, Petri-Nets, …
8
Functionality
• Dynamic handling of Workflows
• Integration capabilities
Workflow Management Systems’ Diversity
BPELEPC XPDL BPMN
1992 1998 2002 2004
PNML
2008
BPMN 2.0
2011
18. Towards a Benchmark for BPMN Engines
System’s Architecture
• Distributed workflow support
• Migrating workflow objects support
• Transactional workflow support
Deployment Infrastructure
• Standalone
• Cluster deployment
• Cloud deployment
• Mobile deployment
Supported Languages
• BPMN, BPEL, Petri-Nets, …
9
Workflow Management Systems’ Diversity
Functionality
• Dynamic handling of Workflows
• Integration capabilities
BPELEPC XPDL BPMN
1992 1998 2002 2004
PNML
2008
BPMN 2.0
2011
19. Towards a Benchmark for BPMN Engines
System’s Architecture
• Distributed workflow support
• Migrating workflow objects support
• Transactional workflow support
Deployment Infrastructure
• Standalone
• Cluster deployment
• Cloud deployment
• Mobile deployment
Supported Languages
• BPMN, BPEL, Petri-Nets, …
10
Workflow Management Systems’ Diversity
Functionality
• Dynamic handling of Workflows
• Integration capabilities
BPELEPC XPDL BPMN
1992 1998 2002 2004
PNML
2008
BPMN 2.0
2011
20. Towards a Benchmark for BPMN Engines
11
https://en.wikipedia.org/wiki/List_of_BPMN_2.0_engines
Jan 2011
BPMN 2.0
Jan 2014
BPMN 2.0.2
ISO/IEC 19510
Aug 2009
BETA
BPMN 2.0
NumberofBPMN2.0WfMSs
0
5
10
15
20
25
Year of the First Version Supporting BPMN 2.0
2009 2010 2011 2012 2013 2014 2015
1
BPMN 2.0: A Widely Adopted Standard (?)
23. Towards a Benchmark for BPMN Engines
12
Data Flow
A B
C
B
A
Control Flow
BPMN 2.0
24. Towards a Benchmark for BPMN Engines
12
Data Flow
A B
C
B
A
Control Flow
Events
BPMN 2.0
25. Towards a Benchmark for BPMN Engines
12
Data Flow
A B
C
B
A
Control Flow
Events Activities Task Types Execution Behaviour
A B
C D
E F
G
H
I
BPMN 2.0
26. Towards a Benchmark for BPMN Engines
13
Design and implement the first benchmark to
assess and compare the performance of WfMSs
that are compliant with Business Process Model
and Notation 2.0 standard.
”
“
BenchFlow Project
the goal
http://benchflow.inf.usi.ch
27. Towards a Benchmark for BPMN Engines
14
end-users, vendors, developers
Why do we Need a Benchmark?
28. Towards a Benchmark for BPMN Engines
14
end-users, vendors, developers
Why do we Need a Benchmark?
1. How to choose the best WfMS in accordance
with the company’s technical requirements?
2. How to choose the best WfMS
in accordance with the company’s
business process models (workflows)?
B
C
D
29. Towards a Benchmark for BPMN Engines
14
end-users, vendors, developers
Why do we Need a Benchmark?
1. How to choose the best WfMS in accordance
with the company’s technical requirements?
2. How to choose the best WfMS
in accordance with the company’s
business process models (workflows)?
B
C
D
4. How to identify WfMS’s bottlenecks?
3. How to evaluate performance improvements
during WfMS’s development?
30. Towards a Benchmark for BPMN Engines
15
Workload
Mix
Load Functions
Test Data
Workload
Model
WfMS
Configurations
Test
Types
Workload
Metrics
KPIs
Derive
Performance
Data
Benchmark Elements for BenchFlow
31. Towards a Benchmark for BPMN Engines
16
Test Types Load Test, Stress Test
Benchmark Element Limitations
State of the Art Limitations (BPEL WfMS)
[SEKE ’14]
Röck et al.
32. Towards a Benchmark for BPMN Engines
16
Test Types Load Test, Stress TestWorkloadModel
Workload Mix
Load Functions
Test Data
Unclear Definition
Simplistic
Not Realistic
Benchmark Element Limitations
State of the Art Limitations (BPEL WfMS)
[SEKE ’14]
Röck et al.
33. Towards a Benchmark for BPMN Engines
16
Test Types Load Test, Stress TestWorkloadModel
Workload Mix
Load Functions
Test Data
Unclear Definition
Simplistic
Not Realistic
Systems
Number of WfMS
WfMS Configurations
2–5
Not Extensive
Benchmark Element Limitations
State of the Art Limitations (BPEL WfMS)
[SEKE ’14]
Röck et al.
34. Towards a Benchmark for BPMN Engines
16
Test Types Load Test, Stress TestWorkloadModel
Workload Mix
Load Functions
Test Data
Unclear Definition
Simplistic
Not Realistic
Systems
Number of WfMS
WfMS Configurations
2–5
Not Extensive
Analyses
Metrics
KPIs
Too General
Too General
Benchmark Element Limitations
State of the Art Limitations (BPEL WfMS)
[SEKE ’14]
Röck et al.
35. Towards a Benchmark for BPMN Engines
• Relevant
• Representative
• Portable
• Scalable
• Simple
Benchmarking Requirements
• Repeatable
• Vendor-neutral
• Accessible
• Efficient
• Affordable
• K. Huppler, The art of building a good benchmark, 2009
• J. Gray, The Benchmark Handbook for Database and Transaction Systems, 1993
• S. E. Sim, S. Easterbrook et al., Using benchmarking to advance research: A
challenge to software engineering, 2003
References:
36. Towards a Benchmark for BPMN Engines
18
Main Challenges in Benchmarking BPMN 2.0 WfMSs
37. Towards a Benchmark for BPMN Engines
18
Main Challenges in Benchmarking BPMN 2.0 WfMSs
WORKLOAD MODEL
38. Towards a Benchmark for BPMN Engines
18
Main Challenges in Benchmarking BPMN 2.0 WfMSs
Workload Mix
20%
80%
C
A
B
D
WORKLOAD MODEL
39. Towards a Benchmark for BPMN Engines
18
Main Challenges in Benchmarking BPMN 2.0 WfMSs
Workload Mix
20%
80%
C
A
B
D
WORKLOAD MODEL
Test Data
40%
60%
C
A
B
U S
40. Towards a Benchmark for BPMN Engines
18
Main Challenges in Benchmarking BPMN 2.0 WfMSs
Load FunctionsWorkload Mix
20%
80%
C
A
B
D
WORKLOAD MODEL
Test Data
40%
60%
C
A
B
U S
41. Towards a Benchmark for BPMN Engines
18
Main Challenges in Benchmarking BPMN 2.0 WfMSs
Load FunctionsWorkload Mix
20%
80%
C
A
B
D
BENCHMARK EXECUTION
WORKLOAD MODEL
Test Data
40%
60%
C
A
B
U S
42. Towards a Benchmark for BPMN Engines
18
Main Challenges in Benchmarking BPMN 2.0 WfMSs
Load FunctionsWorkload Mix
20%
80%
C
A
B
D
WfMS-specific APIs
and BPMN 2.0 Customisations
x
WfMSUsers
Client→WfMS
WfMSWebServices
ApplicationWfMS
BENCHMARK EXECUTION
WORKLOAD MODEL
Test Data
40%
60%
C
A
B
U S
43. Towards a Benchmark for BPMN Engines
18
Main Challenges in Benchmarking BPMN 2.0 WfMSs
Load FunctionsWorkload Mix
20%
80%
C
A
B
D
WfMS-specific APIs
and BPMN 2.0 Customisations
x
WfMSUsers
Client→WfMS
WfMSWebServices
ApplicationWfMS
BENCHMARK EXECUTION
WORKLOAD MODEL
Test Data
40%
60%
C
A
B
U S
Asynchronous Execution
of Workflows
WfMS
?
start
44. Towards a Benchmark for BPMN Engines
18
Main Challenges in Benchmarking BPMN 2.0 WfMSs
Load FunctionsWorkload Mix
20%
80%
C
A
B
D
WfMS-specific APIs
and BPMN 2.0 Customisations
x
WfMSUsers
Client→WfMS
WfMSWebServices
ApplicationWfMS
BENCHMARK EXECUTION
WORKLOAD MODEL
ANALYSES
Test Data
40%
60%
C
A
B
U S
Asynchronous Execution
of Workflows
WfMS
?
start
45. Towards a Benchmark for BPMN Engines
18
Main Challenges in Benchmarking BPMN 2.0 WfMSs
Performance
Metrics and KPIs
Load FunctionsWorkload Mix
20%
80%
C
A
B
D
WfMS-specific APIs
and BPMN 2.0 Customisations
x
WfMSUsers
Client→WfMS
WfMSWebServices
ApplicationWfMS
BENCHMARK EXECUTION
WORKLOAD MODEL
ANALYSES
Test Data
40%
60%
C
A
B
U S
Asynchronous Execution
of Workflows
WfMS
?
start
46. Towards a Benchmark for BPMN Engines
19
Define the Workload Mix
based on real-world BPMN 2.0 process models
gram
Register
Reserv.
Send
Doc.
Confirm
Request
47. Towards a Benchmark for BPMN Engines
19
Define the Workload Mix
based on real-world BPMN 2.0 process models
gram
Register
Reserv.
Send
Doc.
Confirm
Request
Obtain Real-World BPMN 2.0 Process Models
MAIN CHALLENGE:
48. Towards a Benchmark for BPMN Engines
19
Define the Workload Mix
based on real-world BPMN 2.0 process models
gram
Register
Reserv.
Send
Doc.
Confirm
Request
Obtain Real-World BPMN 2.0 Process Models
MAIN CHALLENGE:
gram
Register
Reserv.
Send
Doc.
Confirm
Request
B
A
C
49. Towards a Benchmark for BPMN Engines
> 100 METRICS:
Size, Structure, External
Interaction, Data Handling,
Complexity
20
Define the Workload Mix
characterise real-world BPMN 2.0 process models collections
http://benchflow.inf.usi.ch/bpmeter
BPMeter:
Web Service and application
for static analysis of BPMN
2.0 collections
50. Towards a Benchmark for BPMN Engines
21
NUMBER OF WfMSs SUPPORTING THE FEATURE
NUMBEROFREAL-WORLDMODELS
USINGTHEFEATURE
M
O
CK
DATA
Define the Workload Mix
heterogeneous usage and support of BPMN 2.0
51. Towards a Benchmark for BPMN Engines
21
NUMBER OF WfMSs SUPPORTING THE FEATURE
NUMBEROFREAL-WORLDMODELS
USINGTHEFEATURE
0
200
4
F400
5
600
2
700
8
800
12
950
10
M
O
CK
DATA
Define the Workload Mix
heterogeneous usage and support of BPMN 2.0
52. Towards a Benchmark for BPMN Engines
22
Different Types of Benchmarks
BPMN 2.0 SUPPORTED ELEMENTS
BPMN2.0APPLICATIONDOMAIN
CUSTOMCORE ADVANCED FULL
core, advanced, full, custom
53. Towards a Benchmark for BPMN Engines
22
Different Types of Benchmarks
BPMN 2.0 SUPPORTED ELEMENTS
BPMN2.0APPLICATIONDOMAIN
CUSTOMCORE ADVANCED FULL
core, advanced, full, custom
YOUR
COMPANY
54. Towards a Benchmark for BPMN Engines
22
Different Types of Benchmarks
BPMN 2.0 SUPPORTED ELEMENTS
BPMN2.0APPLICATIONDOMAIN
CUSTOMCORE ADVANCED FULL
core, advanced, full, custom
YOUR
COMPANY
DOMAIN
SPECIFIC
55. Towards a Benchmark for BPMN Engines
22
Different Types of Benchmarks
BPMN 2.0 SUPPORTED ELEMENTS
BPMN2.0APPLICATIONDOMAIN
CUSTOMCORE ADVANCED FULL
core, advanced, full, custom
YOUR
COMPANY
DOMAIN
SPECIFIC
GENERAL
56. Towards a Benchmark for BPMN Engines
23
Workload Mix Characterisation Overview
Models
Collection
Characterisation
Characterisation
Criteria
Benchmark
Workflows
Characterise
Selection
Criteria
57. Towards a Benchmark for BPMN Engines
23
Workload Mix Characterisation Overview
Reoccurring Structures
A
H
Models
Collection
Characterisation
Characterisation
Criteria
Benchmark
Workflows
Characterise
Selection
Criteria
58. Towards a Benchmark for BPMN Engines
23
Workload Mix Characterisation Overview
Clusters of ModelsReoccurring Structures
A
H
Models
Collection
Characterisation
Characterisation
Criteria
Benchmark
Workflows
Characterise
Selection
Criteria
59. Towards a Benchmark for BPMN Engines
24
A
B
C
D
H
F
E
G
REAL-WORLD
PROCESSES
Define the Workload Mix
discover reoccurring structural patterns
60. Towards a Benchmark for BPMN Engines
24
A
B
C
D
H
F
E
G
REAL-WORLD
PROCESSES
[SOSE ’15]
a3
a2
a1
a6
a5
a4
Graph
Matching
REOCCURRING STRUCTURES
Skouradaki et al.
Define the Workload Mix
discover reoccurring structural patterns
61. Towards a Benchmark for BPMN Engines
24
User Defined
Selection Criteria
A
B
C
D
H
F
E
G
REAL-WORLD
PROCESSES
[SOSE ’15]
a3
a2
a1
a6
a5
a4
Graph
Matching
REOCCURRING STRUCTURES
Skouradaki et al.
Define the Workload Mix
discover reoccurring structural patterns
62. Towards a Benchmark for BPMN Engines
24
User Defined
Selection Criteria
A
B
C
D
H
F
E
G
REAL-WORLD
PROCESSES
WORKLOAD MIX
a1
a2
a1
a3 a5
50%
50%
Composition
Criteria
[SOSE ’15]
a3
a2
a1
a6
a5
a4
Graph
Matching
REOCCURRING STRUCTURES
Skouradaki et al.
Define the Workload Mix
discover reoccurring structural patterns
63. Towards a Benchmark for BPMN Engines
25
Define the Workload Mix
discover clusters of models
A
B
C
D
H
F
E
G
REAL-WORLD
PROCESSES
64. Towards a Benchmark for BPMN Engines
25
Define the Workload Mix
discover clusters of models
A
B
C
D
H
F
E
G
REAL-WORLD
PROCESSES
> 100 METRICS: Size, Structure, External Interaction, Data Handling, Complexity
CLUSTERS OF MODELS
F
E
G
B
C
D
A
H
PCA
+
Clustering
65. Towards a Benchmark for BPMN Engines
25
Define the Workload Mix
discover clusters of models
A
B
C
D
H
F
E
G
REAL-WORLD
PROCESSES
> 100 METRICS: Size, Structure, External Interaction, Data Handling, Complexity
CLUSTERS OF MODELS
F
E
G
B
C
D
A
H
PCA
+
Clustering
WORKLOAD MIX
50%
50%
Clustroid
Models
B
C
D
A
66. Towards a Benchmark for BPMN Engines
Define the Test Data
26
gram
Register
Reserv.
Send
Doc.
Confirm
Request
different types of test data
67. Towards a Benchmark for BPMN Engines
Define the Test Data
26
gram
Register
Reserv.
Send
Doc.
Confirm
Request
Test Data:
different types of test data
68. Towards a Benchmark for BPMN Engines
Define the Test Data
26
gram
Register
Reserv.
Send
Doc.
Confirm
Request
Test Data:
40%
60%
Condition Data
different types of test data
69. Towards a Benchmark for BPMN Engines
Define the Test Data
26
gram
Register
Reserv.
Send
Doc.
Confirm
Request
Test Data:
40%
60%
Condition Data
different types of test data
70. Towards a Benchmark for BPMN Engines
Define the Test Data
26
gram
Register
Reserv.
Send
Doc.
Confirm
Request
Test Data:
40%
60%
Condition Data
different types of test data
71. Towards a Benchmark for BPMN Engines
Define the Test Data
26
gram
Register
Reserv.
Send
Doc.
Confirm
Request
Test Data:
…
40%
60%
Condition Data
different types of test data
72. Towards a Benchmark for BPMN Engines
27
am
Register
Reserv.
Send
Doc.
Confirm
Request
Define the Load Functions
different types of load functions
73. Towards a Benchmark for BPMN Engines
27
am
Register
Reserv.
Send
Doc.
Confirm
Request
Define the Load Functions
P(branch)
different types of load functions
74. Towards a Benchmark for BPMN Engines
27
am
Register
Reserv.
Send
Doc.
Confirm
Request
Define the Load Functions
P(branch)
different types of load functions
75. Towards a Benchmark for BPMN Engines
27
am
Register
Reserv.
Send
Doc.
Confirm
Request
Define the Load Functions
P(branch)
different types of load functions
76. Towards a Benchmark for BPMN Engines
27
am
Register
Reserv.
Send
Doc.
Confirm
Request
Define the Load Functions
P(branch)
different types of load functions
77. Towards a Benchmark for BPMN Engines
28
Define the Load Functions
m
Register
Reserv.
Send
Doc.
Confirm
Request
Process Logs
process logs, experts’ opinion
78. Towards a Benchmark for BPMN Engines
28
Define the Load Functions
m
Register
Reserv.
Send
Doc.
Confirm
Request
Process Logs
process logs, experts’ opinion
79. Towards a Benchmark for BPMN Engines
28
Define the Load Functions
m
Register
Reserv.
Send
Doc.
Confirm
Request
Process Logs
process logs, experts’ opinion
80. Towards a Benchmark for BPMN Engines
28
Define the Load Functions
m
Register
Reserv.
Send
Doc.
Confirm
Request
Process Logs
40%
60%
P(branch)
process logs, experts’ opinion
81. Towards a Benchmark for BPMN Engines
28
Define the Load Functions
m
Register
Reserv.
Send
Doc.
Confirm
Request
Process Logs
Completion Time
Completion Time
40%
60%
P(branch)
process logs, experts’ opinion
82. Towards a Benchmark for BPMN Engines
28
Define the Load Functions
m
Register
Reserv.
Send
Doc.
Confirm
Request
Process Logs
Response Time
Completion Time
Completion Time
40%
60%
P(branch)
process logs, experts’ opinion
83. Towards a Benchmark for BPMN Engines
28
Define the Load Functions
m
Register
Reserv.
Send
Doc.
Confirm
Request
Process Logs
Response Time
Requests Rate
Completion Time
Completion Time
40%
60%
P(branch)
process logs, experts’ opinion
84. Towards a Benchmark for BPMN Engines
28
Define the Load Functions
m
Register
Reserv.
Send
Doc.
Confirm
Request
Process Logs
Response Time
Requests Rate
Completion Time
Completion Time
40%
60%
P(branch)
Experts
process logs, experts’ opinion
85. Towards a Benchmark for BPMN Engines
29
Test Types and the Workload Model
Performance Test Types
• Load testing
• Stress testing
• Soak testing
• Spike testing
• Scalability testing
• Capacity testing
• Configuration testing
• Isolation testing
• …
Workload Model
86. Towards a Benchmark for BPMN Engines
29
Test Types and the Workload Model
Performance Test Types
• Load testing
• Stress testing
• Soak testing
• Spike testing
• Scalability testing
• Capacity testing
• Configuration testing
• Isolation testing
• …
Workload Model
0
40
80
10
20
30
40
50
LoadFunction
0
200
400
10
20
30
40
50
89. Towards a Benchmark for BPMN Engines
30
Main Challenges in Benchmarking BPMN 2.0 WfMSs
Performance
Metrics and KPIs
WfMS-specific APIs
and BPMN 2.0 Customisations
x
WfMSUsers
Client→WfMS
WfMSWebServices
ApplicationWfMS
BENCHMARK EXECUTION ANALYSES
Load FunctionsWorkload Mix
20%
80%
C
A
B
D
WORKLOAD MODEL
Test Data
40%
60%
C
A
B
U S
Asynchronous Execution
of Workflows
WfMS
?
start
90. Towards a Benchmark for BPMN Engines
30
Main Challenges in Benchmarking BPMN 2.0 WfMSs
Performance
Metrics and KPIs
WfMS-specific APIs
and BPMN 2.0 Customisations
x
WfMSUsers
Client→WfMS
WfMSWebServices
ApplicationWfMS
BENCHMARK EXECUTION ANALYSES
Load FunctionsWorkload Mix
20%
80%
C
A
B
D
WORKLOAD MODEL
Test Data
40%
60%
C
A
B
U S
Asynchronous Execution
of Workflows
WfMS
?
start
91. Towards a Benchmark for BPMN Engines
31
Instance
Database
DBMSFaban Drivers
ContainersServers
DATA
TRANSFORMERS
ANALYSERS
Performance
Metrics
Performance
KPIs
harness
WfMS
TestExecutionAnalyses
Faban
+
Web
Service
Enabling the Benchmark Execution and Analyses
BenchFlow framework
Minio
92. Towards a Benchmark for BPMN Engines
Performance Benchmark
BenchFlow Framework
32
System Under Test (SUT)
core requirements & functionalities
93. Towards a Benchmark for BPMN Engines
Performance Benchmark
BenchFlow Framework
32
• Automate the SUT deployment
• Simplify the SUT’s deployment
configuration
System Under Test (SUT)
core requirements & functionalities
94. Towards a Benchmark for BPMN Engines
• Manage the benchmark lifecycle
• Simulate the entities interacting
with the WfMS
• Accommodate and automate
different kinds of performance tests
• Ensure reliable execution
• Ensure repeatability
Performance Benchmark
BenchFlow Framework
32
• Automate the SUT deployment
• Simplify the SUT’s deployment
configuration
System Under Test (SUT)
core requirements & functionalities
95. Towards a Benchmark for BPMN Engines
33
BenchFlow Framework
system under test
Docker Engine
96. Towards a Benchmark for BPMN Engines
33
BenchFlow Framework
system under test
Docker Engine
Containers
97. Towards a Benchmark for BPMN Engines
33
BenchFlow Framework
system under test
Docker Engine
Docker Machine
provides
Containers
98. Towards a Benchmark for BPMN Engines
33
BenchFlow Framework
system under test
Docker Swarm
Docker Engine
Docker Machine
provides
Containers
99. Towards a Benchmark for BPMN Engines
33
BenchFlow Framework
system under test
Docker Swarm
Docker Engine
Docker Machine
provides manages
Containers
Servers
100. Towards a Benchmark for BPMN Engines
33
BenchFlow Framework
system under test
Docker Swarm
Docker Engine
Docker Compose
Docker Machine
provides manages
Containers
Servers
101. Towards a Benchmark for BPMN Engines
33
BenchFlow Framework
system under test
Docker Swarm
Docker Engine
Docker Compose
Docker Machine
provides manages
Containers
Servers
SUT’s Deployment Conf.
deploys
102. Towards a Benchmark for BPMN Engines
Manages the Benchmark Lifecycle
BenchFlow Framework
performance benchmark
34
Faban
103. Towards a Benchmark for BPMN Engines
Manages the Benchmark Lifecycle
BenchFlow Framework
performance benchmark
34
Faban
Simulates interacting entities
104. Towards a Benchmark for BPMN Engines
Manages the Benchmark Lifecycle
BenchFlow Framework
performance benchmark
34
Faban
Simulates interacting entities
Ensures repeatability
• the behaviour of a load driver
is described using code
• load drivers can be released
105. Towards a Benchmark for BPMN Engines
Manages the Benchmark Lifecycle
BenchFlow Framework
performance benchmark
34
Faban
Simulates interacting entities
Ensures repeatability
• the behaviour of a load driver
is described using code
• load drivers can be released
paired with Docker:
SUT’s initial state is
frozen in the container
106. Towards a Benchmark for BPMN Engines
Manages the Benchmark Lifecycle
BenchFlow Framework
performance benchmark
34
Faban
Simulates interacting entities
Ensures reliable execution
• checks the environment condition
• validates the benchmark execution
• ensures as few interferences as
possible on the SUT during
measurements
Ensures repeatability
• the behaviour of a load driver
is described using code
• load drivers can be released
paired with Docker:
SUT’s initial state is
frozen in the container
107. Towards a Benchmark for BPMN Engines
Manages the Benchmark Lifecycle
BenchFlow Framework
performance benchmark
34
Faban
Simulates interacting entities
Ensures reliable execution
• checks the environment condition
• validates the benchmark execution
• ensures as few interferences as
possible on the SUT during
measurements
Ensures repeatability
• the behaviour of a load driver
is described using code
• load drivers can be released
paired with Docker:
SUT’s initial state is
frozen in the container
Automates performance tests
108. Towards a Benchmark for BPMN Engines
35
software adapters
WfMS-specific APIs
DBMSFaban Drivers
ContainersServers
harness
WfMS
TestExecution
Web
Service
Adapters
109. Towards a Benchmark for BPMN Engines
WfMS-specific APIs
36
WfMS Users
D
A
B
C
Instance
Database
Application Server
DBMS
Web
Service
Load Driver
custom APIs
• Load Driver APIs
• Web Service APIs
• User APIs
• Event APIs
APIs:
110. Towards a Benchmark for BPMN Engines
37
WfMS-specific APIs
requirements from the WfMS
ADVANCED FULLCORE
111. Towards a Benchmark for BPMN Engines
37
WfMS-specific APIs
requirements from the WfMS
ADVANCED FULLCORE
WfMS
Load Driver APIs
Deploy Workflow
Start Workflow
112. Towards a Benchmark for BPMN Engines
37
WfMS-specific APIs
requirements from the WfMS
ADVANCED FULLCORE
WfMS
Load Driver APIs
Deploy Workflow
Start Workflow
WfMS
Claim Task Complete Task
User and Web Service APIs
+
Create User Create Group
Pending User Tasks
Invoke WS
113. Towards a Benchmark for BPMN Engines
37
WfMS-specific APIs
requirements from the WfMS
ADVANCED FULLCORE
WfMS
Load Driver APIs
Deploy Workflow
Start Workflow
+
Event APIs
Issue Event
WfMS
Pending Event Tasks
WfMS
Claim Task Complete Task
User and Web Service APIs
+
Create User Create Group
Pending User Tasks
Invoke WS
114. Towards a Benchmark for BPMN Engines
38
BPMN 2.0 Customisations
models deployment version
WfMS
Deploy Workflow
115. Towards a Benchmark for BPMN Engines
38
BPMN 2.0 Customisations
models deployment version
WfMS
Deploy Workflow
BPMN 2.0 Standard
XML Serialisation
WfMS A
BPMN 2.0
Serialisation
WfMS B
BPMN 2.0
Serialisation
116. Towards a Benchmark for BPMN Engines
38
BPMN 2.0 Customisations
models deployment version
WfMS
Deploy Workflow
BPMN 2.0 Standard
XML Serialisation
WfMS A
BPMN 2.0
Serialisation
WfMS B
BPMN 2.0
Serialisation
XML
Processor
117. Towards a Benchmark for BPMN Engines
38
BPMN 2.0 Customisations
models deployment version
WfMS
Deploy Workflow
BPMN 2.0 Standard
XML Serialisation
WfMS A
BPMN 2.0
Serialisation
WfMS B
BPMN 2.0
Serialisation
XML
Processor
WfMS A WfMS B
|| ||
118. Towards a Benchmark for BPMN Engines
38
BPMN 2.0 Customisations
models deployment version
WfMS
Deploy Workflow
BPMN 2.0 Standard
XML Serialisation
WfMS A
BPMN 2.0
Serialisation
WfMS B
BPMN 2.0
Serialisation
XML
Processor
WfMS A WfMS B
|| ||
119. Towards a Benchmark for BPMN Engines
38
BPMN 2.0 Customisations
models deployment version
WfMS
Deploy Workflow
BPMN 2.0 Standard
XML Serialisation
WfMS A
BPMN 2.0
Serialisation
WfMS B
BPMN 2.0
Serialisation
XML
Processor
Customised
Test Data
Formatting
+WfMS A WfMS B
|| ||
120. Towards a Benchmark for BPMN Engines
39
Asynchronous Execution of Workflows
WfMS
Start Workflow
overview
121. Towards a Benchmark for BPMN Engines
Users
D
A
B
C
Start
Application Server
Web
Service
Load Driver
DBMS
WfMS
Instance
Database
39
Asynchronous Execution of Workflows
WfMS
Start Workflow
overview
122. Towards a Benchmark for BPMN Engines
Users
D
A
B
C
Start
Application Server
Web
Service
Load Driver
DBMS
WfMS
Instance
Database
39
Asynchronous Execution of Workflows
WfMS
Start Workflow
End
overview
123. Towards a Benchmark for BPMN Engines
40
DBMSFaban Drivers
ContainersServers
harness
WfMS
TestExecution
Web
Service
Asynchronous Execution of Workflows
monitors
124. Towards a Benchmark for BPMN Engines
40
DBMSFaban Drivers
ContainersServers
harness
WfMS
TestExecution
Web
Service
MONITOR
Asynchronous Execution of Workflows
monitors
125. Towards a Benchmark for BPMN Engines
41
Asynchronous Execution of Workflows
DBMSFaban Drivers
ContainersServers
harness
WfMS
TestExecution
Web
Service
• CPU usage
• Database state
Examples of Monitors:Monitors’ Characteristics:
• Microservices
• Lightweight (written in Go)
• As less invasive on the SUT as possible
• Expose REST APIs towards the Drivers
monitors
126. Towards a Benchmark for BPMN Engines
41
Asynchronous Execution of Workflows
DBMSFaban Drivers
ContainersServers
harness
WfMS
TestExecution
Web
Service
CPU
Monitor
DB
Monitor
• CPU usage
• Database state
Examples of Monitors:Monitors’ Characteristics:
• Microservices
• Lightweight (written in Go)
• As less invasive on the SUT as possible
• Expose REST APIs towards the Drivers
monitors
127. Towards a Benchmark for BPMN Engines
42
Main Challenges in Benchmarking BPMN 2.0 WfMSs
Performance
Metrics and KPIs
Load FunctionsWorkload Mix
20%
80%
C
A
B
D
WfMS-specific APIs
and BPMN 2.0 Customisations
x
WfMSUsers
Client→WfMS
WfMSWebServices
ApplicationWfMS
BENCHMARK EXECUTION
WORKLOAD MODEL
ANALYSES
Test Data
40%
60%
C
A
B
U S
Asynchronous Execution
of Workflows
WfMS
?
start
128. Towards a Benchmark for BPMN Engines
42
Main Challenges in Benchmarking BPMN 2.0 WfMSs
Performance
Metrics and KPIs
Load FunctionsWorkload Mix
20%
80%
C
A
B
D
WfMS-specific APIs
and BPMN 2.0 Customisations
x
WfMSUsers
Client→WfMS
WfMSWebServices
ApplicationWfMS
BENCHMARK EXECUTION
WORKLOAD MODEL
ANALYSES
Test Data
40%
60%
C
A
B
U S
Asynchronous Execution
of Workflows
WfMS
?
start
129. Towards a Benchmark for BPMN Engines
Performance Metrics and KPIs
43
WfMS Users
Load Driver
Instance
Database
Application Server
DBMS
Web
Service
D
A
B
C
levels
130. Towards a Benchmark for BPMN Engines
Performance Metrics and KPIs
43
WfMS Users
Load Driver
Instance
Database
Application Server
DBMS
Web
Service
D
A
B
C
Metrics and KPIs
• Engine Level
levels
131. Towards a Benchmark for BPMN Engines
Performance Metrics and KPIs
43
WfMS Users
Load Driver
Instance
Database
Application Server
DBMS
Web
Service
D
A
B
C
Metrics and KPIs
• Engine Level
• Process Level
levels
132. Towards a Benchmark for BPMN Engines
Performance Metrics and KPIs
43
WfMS Users
Load Driver
Instance
Database
Application Server
DBMS
Web
Service
D
A
B
C
Metrics and KPIs
• Engine Level
• Process Level
• Feature Level
levels
133. Towards a Benchmark for BPMN Engines
Performance Metrics and KPIs
43
WfMS Users
Load Driver
Instance
Database
Application Server
DBMS
Web
Service
D
A
B
C
Metrics and KPIs
• Engine Level
• Process Level
• Feature Level
• Interactions
levels
134. Towards a Benchmark for BPMN Engines
44
Performance Metrics and KPIs
examples and relevant users
135. Towards a Benchmark for BPMN Engines
44
ENGINE LEVEL
PROCESS LEVEL
FEATURE LEVEL
Performance Metrics and KPIs
examples and relevant users
136. Towards a Benchmark for BPMN Engines
44
Throughput
Response Time
Capacity
Latency
Flexibility to Spike
ENGINE LEVEL
PROCESS LEVEL
FEATURE LEVEL
Performance Metrics and KPIs
examples and relevant users
137. Towards a Benchmark for BPMN Engines
44
Throughput
Response Time
Capacity
Latency
Flexibility to Spike
CPU Usage
RAM Usage
Duration of Process
ENGINE LEVEL
PROCESS LEVEL
FEATURE LEVEL
Performance Metrics and KPIs
examples and relevant users
138. Towards a Benchmark for BPMN Engines
44
Delay of Timer Event
Space Used by a Data Input
Duration of Message Event
Throughput
Response Time
Capacity
Latency
Flexibility to Spike
CPU Usage
RAM Usage
Duration of Process
ENGINE LEVEL
PROCESS LEVEL
FEATURE LEVEL
Performance Metrics and KPIs
examples and relevant users
139. Towards a Benchmark for BPMN Engines
44
Delay of Timer Event
Space Used by a Data Input
Duration of Message Event
Throughput
Response Time
Capacity
Latency
Flexibility to Spike
CPU Usage
RAM Usage
Duration of Process
ENGINE LEVEL
PROCESS LEVEL
FEATURE LEVEL
END-USERS
VENDORS
DEVELOPERS
Performance Metrics and KPIs
examples and relevant users
140. Towards a Benchmark for BPMN Engines
45
Performance Metrics and KPIs
minimal data requirements
Accessibility of the Data
DBMS
WfMS
141. Towards a Benchmark for BPMN Engines
45
Performance Metrics and KPIs
minimal data requirements
• Workflow & Construct:
• Start Time
• End Time
• [Duration]
Availability of Timing DataAccessibility of the Data
DBMS
WfMS
142. Towards a Benchmark for BPMN Engines
45
Performance Metrics and KPIs
minimal data requirements
• Workflow & Construct:
• Start Time
• End Time
• [Duration]
Availability of Timing DataAccessibility of the Data
DBMS
WfMS
Availability of Execution State
State of the workflow execution. E.g., running, completed, error
143. Towards a Benchmark for BPMN Engines
Performance Metrics and KPIs
46
collect data
DBMSFaban Drivers
ContainersServers
harness
WfMS
TestExecution
Web
Service
144. Towards a Benchmark for BPMN Engines
Performance Metrics and KPIs
46
collect data
DBMSFaban Drivers
ContainersServers
harness
WfMS
TestExecution
Web
Service
Minio
Instance
Database
Analyses
COLLECTORS
145. Towards a Benchmark for BPMN Engines
47
DBMSFaban Drivers
ContainersServers
harness
WfMS
TestExecution
Web
Service
• Container’s Stats (e.g., CPU usage)
• Database dump
• Applications Logs
Examples of Collectors:Collectors’ Characteristics:
• Microservices
• Lightweight (written in Go)
• Two types: online and offline
• Buffer data locally
• Expose REST APIs towards Drivers
Performance Metrics and KPIs
collect data
146. Towards a Benchmark for BPMN Engines
47
DBMSFaban Drivers
ContainersServers
harness
WfMS
TestExecution
Web
Service
Stats
Collector
DB
Collector
• Container’s Stats (e.g., CPU usage)
• Database dump
• Applications Logs
Examples of Collectors:Collectors’ Characteristics:
• Microservices
• Lightweight (written in Go)
• Two types: online and offline
• Buffer data locally
• Expose REST APIs towards Drivers
Performance Metrics and KPIs
collect data
147. Towards a Benchmark for BPMN Engines
48
Performance Metrics and KPIs
amount of data
REALISTIC
DATA
Number of
Repetitions
1
10
5
3
148. Towards a Benchmark for BPMN Engines
48
Performance Metrics and KPIs
amount of data
REALISTIC
DATA
Number of
Repetitions
1
10
5
3
Number of
WfMSs
1
149. Towards a Benchmark for BPMN Engines
49
Performance Metrics and KPIs
transform data
DBMSFaban Drivers
ContainersServers
harness
WfMS
Web
Service
TestExecution
Instance
Database
DATA
TRANSFORMERS
Analyses
Data Mappers
150. Towards a Benchmark for BPMN Engines
49
Performance Metrics and KPIs
transform data
DBMSFaban Drivers
ContainersServers
harness
WfMS
Web
Service
TestExecution
Instance
Database
DATA
TRANSFORMERS
Analyses
• custom DB format
• unstructured logs
• …
Data Mappers
151. Towards a Benchmark for BPMN Engines
49
Performance Metrics and KPIs
transform data
DBMSFaban Drivers
ContainersServers
harness
WfMS
Web
Service
TestExecution
Instance
Database
DATA
TRANSFORMERS
Analyses
• custom DB format
• unstructured logs
• …
ETL Process
Data Mappers
152. Towards a Benchmark for BPMN Engines
50
Performance Metrics and KPIs
transform data
DBMSFaban Drivers
ContainersServers
harness
WfMS
Web
Service
TestExecution
DATA
TRANSFORMERS
Analyses
fast and general
engine for large-scale
data processing
Data Mappers
Instance
Database
153. Towards a Benchmark for BPMN Engines
DBMSFaban Drivers
ContainersServers
harness
WfMS
Web
Service
TestExecution
51
Analyses
DATA
TRANSFORMERS
ANALYSERS
Performance
Metrics
Performance
KPIs
Analyses
Performance Metrics and KPIs
analyse data
Data Mappers
Instance
Database
155. Towards a Benchmark for BPMN Engines
52
Highlights
Why a Benchmark for WfMSs
156. Towards a Benchmark for BPMN Engines
52
Highlights
Why a Benchmark for WfMSs Benchmark’s Main Challenges
157. Towards a Benchmark for BPMN Engines
52
Highlights
Why a Benchmark for WfMSs Benchmark’s Main Challenges
BenchFlow Framework
158. Towards a Benchmark for BPMN Engines
53
Future Work
BenchFlow framework
1. Finalise the release of the BenchFlow framework as
an open-source project on GitHub and DockerHub;
benchflowhttps://github.com/benchflow
159. Towards a Benchmark for BPMN Engines
53
Future Work
BenchFlow framework
1. Finalise the release of the BenchFlow framework as
an open-source project on GitHub and DockerHub;
benchflowhttps://github.com/benchflow
2. Simplify and automate the execution of common
performance tests: load test, spike test, scalability
test, …
160. Towards a Benchmark for BPMN Engines
54
Future Work
experiments
1. Perform the first real-world experiments
161. Towards a Benchmark for BPMN Engines
54
Future Work
experiments
1. Perform the first real-world experiments
Execute Different Test Types
• Stress testing
• Spike testing
• Scalability testing
• Configuration testing
162. Towards a Benchmark for BPMN Engines
54
Future Work
experiments
1. Perform the first real-world experiments
Execute Different Test Types
• Stress testing
• Spike testing
• Scalability testing
• Configuration testing
Use Real-World Workflows
163. Towards a Benchmark for BPMN Engines
54
Future Work
experiments
1. Perform the first real-world experiments
Execute Different Test Types
• Stress testing
• Spike testing
• Scalability testing
• Configuration testing
Production-like Configurations
Use Real-World Workflows
164. Towards a Benchmark for BPMN Engines
54
Future Work
experiments
1. Perform the first real-world experiments
Execute Different Test Types
• Stress testing
• Spike testing
• Scalability testing
• Configuration testing
Realistic:
• Load Functions
• Test Data
Production-like Configurations
Use Real-World Workflows
165. Towards a Benchmark for BPMN Engines
55
Future Work
experiments
2. Increase the number of supported WfMSs
APIs:
CORE: Load Driver APIs
+
ADVANCED: User and Web
Service APIs
+ FULL: Event APIs
166. Towards a Benchmark for BPMN Engines
55
Future Work
experiments
2. Increase the number of supported WfMSs
APIs:
CORE: Load Driver APIs
+
ADVANCED: User and Web
Service APIs
+ FULL: Event APIs
WfMS N
BPMN 2.0
Serialisation
customised test
data formatting+WfMS N
||
Deployable BPMN 2.0 Models:
167. Towards a Benchmark for BPMN Engines
55
Future Work
experiments
2. Increase the number of supported WfMSs
APIs:
CORE: Load Driver APIs
+
ADVANCED: User and Web
Service APIs
+ FULL: Event APIs
WfMS N
BPMN 2.0
Serialisation
customised test
data formatting+WfMS N
||
Deployable BPMN 2.0 Models:
Data:
• Accessibility of the data
• Availability of timing data
• Availability of execution
state
168. Towards a Benchmark for BPMN Engines
56
•We want to characterise the Workload Mix using Real-World process models
• Share your executable BPMN 2.0 process models, even anonymised
Process Models
•We want to characterise the Load Functions using Real-World behaviours
• Share your execution logs, even anonymised
Execution Logs
Call for Collaboration
WfMSs, process models, process logs
•We want to add more and more WfMSs to the benchmark
• Contact us for collaboration, and BenchFlow framework support
WfMSs
169. Towards a Benchmark for BPMN Engines
Join Us @ ICWE 2016 in Lugano!
http://icwe2016.inf.usi.ch
170. Towards a Benchmark for BPMN Engines
Vincenzo Ferme, Cesare Pautasso | University of Lugano (USI)
vincenzo.ferme@usi.ch
@VincenzoFerme
http://benchflow.inf.usi.ch
c.pautasso@ieee.org
@pautasso
benchflowbenchflow
172. Towards a Benchmark for BPMN Engines
[SEKE ’14]
C. Röck, S. Harrer, and G. Wirtz. Performance benchmarking of BPEL engines: A
comparison framework, status quo evaluation and challenges. In Proc. of the 26th
International Conference on Software Engineering and Knowledge Engineering, SEKE
2014, pages 31–34, 2014.
60
[SOSE ’15]
M. Skouradaki, K. Goerlach, M. Hahn, and F. Leymann. Application of Sub-Graph
Isomorphism to Extract Reoccurring Structures from BPMN 2.0 Process Models.
In Proc. of the 9th International IEEE Symposium on Service-Oriented System
Engineering, SOSE 2015, 2015.
Cited Work
173. Towards a Benchmark for BPMN Engines
61
Published Work
[BTW ’15]
C. Pautasso, V. Ferme, D. Roller, F. Leymann, and M. Skouradaki. Towards workflow
benchmarking: Open research challenges. In Proc. of the 16th conference on
Database Systems for Business,Technology, and Web, BTW 2015, pages 331–350, 2015.
[SSP ’14]
M. Skouradaki, D. H. Roller, F. Leymann, V. Ferme, and C. Pautasso. Technical open
challenges on benchmarking workflow management systems. In Proc. of the 2014
Symposium on Software Performance, SSP 2014, pages 105–112, 2014.
[ICPE ’15]
M. Skouradaki, D. H. Roller, L. Frank, V. Ferme, and C. Pautasso. On the Road to
Benchmarking BPMN 2.0 Workflow Engines. In Proc. of the 6th ACM/SPEC
International Conference on Performance Engineering, ICPE ’15, pages 301–304, 2015.
174. Towards a Benchmark for BPMN Engines
62
[CLOSER ’15]
M. Skouradaki,V. Ferme, F. Leymann, C. Pautasso, and D. H. Roller. “BPELanon”: Protect
business processes on the cloud. In Proc. of the 5th International Conference on
Cloud Computing and Service Science, CLOSER 2015. SciTePress, 2015.
Published Work
[SOSE ’15]
M. Skouradaki, K. Goerlach, M. Hahn, and F. Leymann. Application of Sub-Graph
Isomorphism to Extract Reoccurring Structures from BPMN 2.0 Process Models.
In Proc. of the 9th International IEEE Symposium on Service-Oriented System
Engineering, SOSE 2015, 2015.
[BPM ’15]
V. Ferme, A. Ivanchikj, C. Pautasso. A Framework for Benchmarking BPMN 2.0
Workflow Management Systems. In Proc. of the 13th International Conference on
Business Process Management, BPM ’15, pages 251-259, 2015.
175. Towards a Benchmark for BPMN Engines
63
[BPMD ’15]
A. Ivanchikj,V. Ferme, C. Pautasso. BPMeter: Web Service and Application for Static
Analysis of BPMN 2.0 Collections. In Proc. of the 13th International Conference on
Business Process Management [Demo], BPM ’15, pages 30-34, 2015.
Published Work