Joram Barrez, Principal Software Engineer at Flowable, explains how to run Flowable on MongoDB.
It was presented at the Flowfest 2018 in Barcelona, Spain
2. Transactions
• Flowable relies on the transactional semantics of a
relational db
• “Atomically” moving from one stable state to another
• This doesn’t free you from forgetting about service failures,
but understanding the transactional model of Flowable sure
makes it easier to write resilient processes
• MongoDB 4.0 added support for transactions (June)
2
3. MongoDB
• Open-source NoSql JSON document store
• Short history
• Started in 2007 by 10gen as component of their PaaS
• Was known in the early days (2.2 versions and before) as the
dev/null db
• Acquired WiredTiger end of 2014
• WiredTiger default storage engine in 3.2
• WiredTiger enables transactional semantics (ACID) on multi-
document operations in 4.0 (*)
3* “Path to transactions” series on https://www.youtube.com/user/MongoDB/videos
4. Flowable – MongoDB
• All code: https://github.com/flowable/flowable-mongodb
4
Service call
Command Interceptor/
Commands
Agenda / operations
EntityManagers
DataManagers
Engine core logic
Low-level data access
High-level data functions
5. Implementation
• Replace the lowest layer
• MongoDB’s transactions follow a familiar programming
model
• Concept of clientSession
• Matches Flowable’s low-level session concept nicely
5
7. Implementation
• Replace all Datamanager interface implementations with a
MongoDB counterpart
• alpha releases
• Gather interest/feedback
• Using the existing test suite to validate the implementation
• Completed -> beta / stable release
• (Almost) 1-1 translation of the relational data structure
• Optimizations along the way
• MongoDb-specific structure optimization surely will follow
7
9. Challenges
• Taking joins for granted
• Denormalization needed
• Way more work as a developer to guarantee data consistency
• E.g simple example: see ‘latest’ of Process definition
• Exchange writes/updates for faster reads
9
10. Luckily
• Over the past years
• We’ve made Flowable a lot faster by keeping in mind that one
exchange over a network is extremely expensive
• Denormalization, prefetching, entity counts
10
14. Results
• Although the graphs seem to indicate a relative large
difference, we’re talking about sub-ms differences!
• Relational db’s have not been idling
• See our recent performance benchmarks
• https://blog.flowable.org/2018/03/05/flowable-6-3-0-
performance-benchmark/
• https://blog.flowable.org/2018/03/13/async-history-
performance-benchmark/
14
15. Conclusion
• The transactional support in MongoDB is impressive
• Data consistency perspective
• Performance perspective
• Using Flowable on MongoDB is a valid alternative
15
16. Current limitations
• Read/Write to primary only
• Adding replica nodes seemed to have a negative effect
• Even though read/write to primary (current MongoDB transactions
limitation)
• MongoDB transactions are still under development
16https://www.youtube.com/watch?v=dQh03YLkmyg
17. Future work
• MongoDB is designed for horizontal scale
• (Yes, (for example) postgres has partitioning, but …)
• Sharded clusters + Flowable à interesting use cases
• Shard by tenant
• Shard on process definition key
• BigData use cases … like ML!
17
18. Machine Learning
• Process/Case engines are in a prime position
• End-user data through forms
• Service invocation data
• (Semi-)Structured models
18
19. Machine Learning
• MongoDB being “BigData” (e.g better suited for streaming,
reactive, etc.) opens up use cases for ML
• Demo
• Run processes a lot from start to end
• Feed historical data into ML
• See if human work is repetitive and suggest optimizations
19
21. Machine Learning
1. Look for Human Decision patterns
2. Gather possible data inputs and backtrack
22. Machine Learning
1. Look for Human Decision patterns
2. Gather possible data inputs and backtrack
3. Use machine learning (Spark decision tree algorithm) to
calculate potential patterns in the data
1. i.e. which data at the start leads to a certain path later on
(within certain % of confidence)