Executing hundreds or thousands of process instances per second? Yes, it's possible. This webinar is about best practices for high-load situations, and how to scale Camunda BPM horizontally.
2. Daniel Meyer
Process Engine Expert
Technical Project Lead
@meyerdan | daniel.meyer@camunda.com
Bernd Rücker
10+ years experience with workflow and Java
Co-Founder of Camunda
Evangelist & Head of Consulting
@berndruecker | bernd.ruecker@camunda.com
Your speakers today
4. It always depends
−On hardware
−On software environment (OS, Java, App Server, Database, …)
−On Service Tasks in the process
−On network topology (e.g. remote database, web services, …)
−On concurrent requests, database load, …
There is no simple answer to performance
But we always succeed – in each and every real-life situation
−Handling millions of process instances / day
−Handling more than 1.000 process instances / second
−Handling thousands of parallel users
Performance is a difficult topic
5. We are much faster than competition
see http://camunda.com/landing/whitepaper-camunda-jbpm/
In our tests, Camunda‘s
throughput was 10x –
30x higher than with
JBoss jBPM.
6. 1. Understand basic engine architecture
2. Understand influence parameters on performance
3. Discuss performance improvement approaches
4. See example figures / measurements
5. Discuss future scenarios (e.g. sharding, NoSQL, …)
What we do today
16. Process/Task Variables
−Show in list
−Use in Search/Filter
Support for Pagination
Big number of users accessing the tasklist very often
Implementation challenge
Provide a generic database schema
Complex data types are serialized – no SQL-JOIN possible
Variables are stored in one row per variable – multiple SQL-JOINs
might be required
Some customers use 10-30 variables
Tasklist Requirements
17. Add Process Variables optimized (and only used) for Queries
−Extract attributes
−Combine variables to work with LIKE
Use own queries
−Native – if you want to improve the WHERE
−Custom – if you want to SELECT multiple information at once
Own TaskInfo or ProcessInstanceInfo entities
−Persisted as MyBatis or JPA entities
−Combine all attributes – allow to query tasks without (or with one)
JOIN only
−Synchronisation via Listener – or use ProcessInstanceInfo as single
source
Solution Approaches: Tasklist
18. Example
Customer
- customerId
- company
- …
Your DB camunda
PROCESS_VARIABLES
customerId
...
searchField
4711
...
4711#camunda#Berlin#...
1
2
Native
Query:
3
Custom
Query:
4
Java API –
results are
camunda „Task“
entities
Own MyBatis
mapping – result
can be anything.
Called via
custom code.
19. Example
TaskInfo
- taskId
- customerId
- companyName
- contractId
- productName
- …
Your DB
camunda
PROCESS_VARIABLES
customerId
contractId
productId
4711
0815
42
5
TaskInfo Entity
(or ProcessInstanceInfo)
20. The challenge:
−Indexes cost space and performance in writing data
−We provide a generic database schema without knowing what you
exactly do with it
−We constantly work on the right balance of too many and too less
indexes
What you can do:
−Check indexes and slow query log
−Add index where appropriate for your situation (perfectly OK with
us, you do not loose support!)
−As Enterprise Customer you can always discuss/validate changes
with support
Example: create index PROC_DEF_ID_END_TIME ON
ACT_HI_PROCINST (PROC_DEF_ID_,END_TIME_)
(History) Queries
21. You can also customize history
Custom History
(e.g.
ElasticSearch)
Different History Levels:
- NONE
- ACTIVITY
- AUDIT
- FULL
- CUSTOM (own Filter written
in Java, e.g. „only variable
X“, „not process Y“, …)
Example for custom log level:
https://github.com/camunda/camunda-
bpm-examples/tree/master/process-
engine-plugin/custom-history-level
23. Asynchronous Continuation involve Jobs
Jobs are stored in the database
Job Executor can be configured
−Number of Worker Threads
−Number of Jobs fetched with one database query
−Size of in-memory Queue
−Lock Time, Retry Behavior, …
Job Execution can be distributed over a Cluster
Optimizing is not a straight forward task, hard to give general advise
If you need to improve: Measure and benchmark configurations in
your environment!
Job Execution
24. The good news: We did big performance improvements in Camunda
BPM 7.2!
Improved First Level Cache (throughput increased by up to 90% if
async Service Tasks are executed in a row)
Improved locking to have less Optimistic Lock Exceptions and more
Jobs acquired per Acquisition. Results in bigger Clusters getting
possible.
Job Execution in Camunda BPM 7.2
25. Recap:
Added log level “CUSTOM” for History
First Level Cache
Job Executor Acquisition Locking
Plus:
Added flush ordering (comparable to Hibernate) to minimize risk of
deadlocks
Summary: Performance Improvements in 7.2
28. Recommendation: Measure! No guessing.
camunda engine
Process
Application
External
Load
Generator
e.g. JMeter,
HP Load Runner,
CURL, …
REST
„close to production“
environment
29. - Measure
- JobExecutor Horizontal Scalability
- Impact of 1st level cache reuse
- Improvements Version 7.1.0 vs. Version 7.2.0
- Environment: Amazon AWS Cloud (EC2 & RDS)
Benchmark
40. What is Horizontal Scalability?
Scale up the number of transactions executed by adding more
processing nodes to the system. [*]
[*] http://en.wikipedia.org/wiki/Scalability#Horizontal_and_vertical_scaling (Adapted)
Horizontal Scalability
transactions /
sec
nodes
41. The current Situation
Scale number of Process Engine Nodes (JVMs)
Up to a certain point
Limited possibilities for scaling the shared
relational Database. In a sense this can
only be scaled “up”, not “out”.
Shared
Relational
Database
Process
Engine
Process
Engine
Process
Engine
42. Which way to go?
Distributed
Datastore
Process
Engine
Process
Engine
Process
Engine
Distributed Datastore.
Use a database which is itself a
distributed system and can be
scaled horizontally.
- Apache Cassandra,
- Apache HBase,
- Distributed Caches
(Hazelcast, …)
- ...
Sharding and partitioning.
Distribute the state over multiple
Datastores.
- Multiple instances of
PostgreSQL
- Each “DB” is a Mongo DB
shard
- No “DB” at all: use a
filesystem journal?
- ...
Key Difference: on the right hand side, the process engine itself is “distributed”
in the sense that it is aware of the distribution and sharding.
43. The problem with Distributed Datastores
(In the context of process engines)
1. Consistency guarantees offered by these databases (eventual consistency, ACID vs.
BASE, ...) often do not match the requirements of BPMN process execution. See:
conflicting concurrent transactions:
a. Racing incoming signals (E.g.: Two Messages targeting the same event instance arrive at the
same time)
b. Joins & Synchronization (E.g.: Gateways, Multi Instance, ...)
c. Cancel Activity instance (E.g.: Interrupting Message Boundary Event)
1. Data Representation and Network Latency / Overhead: Process instance state is
composite:
a. Token state / active activity instances
b. Variables
c. Task Information, …
Challenge is to find a data representation which does not lead to distribution of the state of a
single process instance across the cluster while still supporting the required access patterns.
2. Significant differences between individual technologies while there are no
industry standards in place yet. (Different with SQL).
44. Sharding => Distributed yet Local
Scale horizontally...
Each “shard / node” maitains its state
locally
Partitioning workflow instance state
- Each process instance lives inside a single shard /
partition
=> local data consistency easy to guarantee,
=> easy to access efficiently
=> Support range of different persistence engines
(Relational Database, Non-Relational Databases, …)
45. Proces
s
Engine
Flexible Architecture
...
Reality @
zalando 2014
Proces
s
Engine
Proces
s
Engine
The simplest case
A single process engine node
running on top of a
conventional database.
A medium Scenario
Horizontally scale on top of a
conventional database.
Massive Compute Cluster
500 Nodes ?
All of this should be possible with one
unified architecture!
46. No more Search!
The catch
“Find Process Instance
for order with ID 43543242”
??
???
47. Human Workflow (Build Task Lists)
History: Monitoring, Reporting, …
Message Correlation
When is „Search“ required?
48. Message Correlation
The Problem to solve
Workflow Instance State
for order with ID 435345
Incoming Message:
“customer cancelled Order
with ID 435345”
49. Yes, but for non-workflow execution Use Cases
Use Search Index?
(A)sync
Updates
Search Index
(Near Realtime)
Tasklist
Queries,
Monitoring,...
50. Vision
HistoryTasksCore Process Execution
Signal / Cancel Activity Instance by Id
Correlate Message
Query for List of Tasks Monitoring,
Reports
Real Time, Strongly Consistent
Horizontally scalable through sharding
Multiple persistence technologies possible
Near Real Time, Eventually Consistent
Use best technology for the Job.
Async Event Stream
51. But still...
HistoryTasksCore Process Execution
Signal / Cancel Activity Instance by Id
Correlate Message
Query for List of Tasks Monitoring,
Reports
In the simplest case!
52. Learning #4:
You can do true horizontal
clustering with the engine which
exists today!
There is no need for No-SQL
persistence in the core engine.
54. Camunda BPM Performance is already awesome
However: We are continuously improving
performance
There are strategies to solve specific performance
challenges
There is no limit in scalability
Summary
55. Start now!
Open Source Edition
• Download:
www.camunda.org
• Docs, Tutorials etc.
• Forum
• Meetings
Enterprise Edition
• Trial:
www.camunda.com
• Additional Features
• Support, Patches etc.
• Consulting, Training
http://camunda.com/bpm/consultation/
info@camunda.com | US +1.415.800.3908 | DE +49 30 664040 900