Presented by David Hammer, Robert Breske The GVL aggregates millions of usage reports for musical and cinematographic works from various sources every year. To cope with this amount of incoming data, a transparent and audit-proof data aggregation system was desired to be built. The solution was a DMN rule-based approach embedded in camunda's proccess engine framework, altogether with an audit-proof storage of the process history data. This system is capable of simulating the process flow for any processed data record at any later point in time, in case of a need for revision or validation. The GVL currently runs this scalable system with a troughput of more than one million daily process instances.

1. Rule-Based Mass
Data Processing with
DMN and Camunda
Society of
collective rights
management
21.09.2018

2. 2
Robert Breske
robert.breske@cimt-ag.de
developer and software consultant
5+ years project experience
David Hammer
david.hammer@gvl.de
business architect
15+ years overall experience
Society of
collective rights
management

3. 3
Performing artists
Producers of sound recordings
Event organisers
Music video clip producers
Rights users
Distribution
License
revenues
Permission to
administrate rights
Licensing

4. GVL matching business
• Documentation and maintainance of product repertoire
• Matching of around 20 million usages each year
• Clarification of rights conflicts
• Product data deduplication
4

5. 5
Product data from
rights owners
Usage reports from
radio and tv stations
Reported
participations by
artists
Data core
Data reception
and pre-processing
Distribution system
Product fusion process
300M+ Distinct master
data objects
Usage reports from
other royalties
Product data from
other sources

6. 6
Search Evaluate Modify Audit
Product fusion process

7. 7
Search Evaluate Modify Audit
Product fusion process

8. 8
Search Evaluate Modify Audit
BPMN and
DMN
Camunda Cockpit
history view
Product fusion process

9. Core principles of the solution
History simulation & Synchronous performance
9

10. 10

11. 11
[1]
Runtime
database
History
database

12. 12
[1]
Runtime
database
History
database

13. 13
[1]
Runtime
database
History
database
inputVariableA inputVariableB softwareVersion
Custom audit
database

14. 14
• Full history can be rebuild at any point in time
• All process input data is stored in audit database
• Re-execution (=simulation) of process provides
„full history“

15. 15
[1]
• History level set to null
• Process input saved to
denormalized audit database
Runtime
database
History
database
inputVariableA inputVariableB softwareVersion
Custom audit
database
• Rebuild desired history only by
re-executing the process instance
in a sandbox (=simulation)

16. 16
[1]
• History level set to null
• Process input saved to
denormalized audit database
Runtime
database
History
database
inputVariableA inputVariableB softwareVersion
Custom audit
database
• Rebuild desired history only by
re-executing the process instance
in a sandbox (=simulation)

17. 17
Runtime
database
Runtime
memory
Significant write-through costs
for process variables

18. 18
No asynchronous points to
prevent variable updates on
database

19. 19
Bad practice
Fail forward allows for error
recovery even though all steps
are executed synchronously in
happy path
No asynchronous points to
prevent variable updates on
database

20. 20
Bad practice
Fail forward allows for error
recovery even though all steps
are executed synchronously in
happy path
No asynchronous points to
prevent variable updates on
database

21. 21
Runtime
database
Runtime
memory
Significant write-through costs
for process variables
DB update triggered
only in failure scenario

22. 22
Runtime
database
Runtime
memory
Significant write-through costs
for process variables
DB update triggered
only in failure scenario

23. The result
23

24. 24
Search Evaluate Modify Audit
Product fusion process

25. 25
Convert all data processing
rules into DMN tables and
calculate an overall score

26. 26
Make data processing
decisions based on these
results

27. 27
• Review all processes from
audit database in a custom
cockpit plugin
• Simulate the history if
necessary based on
revision id

28. 28
Review every single
decision and rule in
Camunda cockpit history

29. 29
Review every single
decision and rule in
Camunda cockpit history

30. Process instances per day
0
200000
400000
600000
800000
1000000
1200000
1400000
0 5 10 15 20 25
30
We stopped scaling at 1M+
process instances per day

31. Process instances per day
0
200000
400000
600000
800000
1000000
1200000
1400000
0 5 10 15 20 25
31
We stopped scaling at 1M+
process instances per day