Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models

Marlon Dumas
Marlon DumasProfessor at University of Tartu | Co-Founder at Apromore à University of Tartu and Apromore Pty Ltd
••
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models David Chapela-Campa1, Ismail Benchekroun2, Opher Baron2, Marlon Dumas1, Dmitry Krass2, and Arik Senderovich3 21st International Conference on Business Process Management (BPM 2023) 1 University of Tartu, Estonia 2 University of Toronto, Canada 3 York University, Canada
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 2 Introduction
Business Process Simulation (BPS) 3 BPS allows users to address “what-if” analysis questions. What would be the cycle time of the process if the rate of arrival of new cases doubles? Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Business Process Simulation (BPS) 4 BPS models can be manually created by modeling experts. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Business Process Simulation (BPS) 5 BPS models can be manually created by modeling experts. Use of process mining techniques to automatically discover BPS models from business process event logs. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Business Process Simulation (BPS) 6 How to assess the quality of a BPS model? Automatic assessment. Useful to detect the sources of deviations. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 7 Proposed Framework
Quality of a BPS model 8 How to assess the quality of a BPS model? Comparing an event log with a BPS model. Variety of different BPS models formats. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log
Quality of a BPS model 9 How to assess the quality of a BPS model? Generate K simulated event logs. Compare individually and report the average and confidence interval. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 9 Process event log K simulated event logs
Quality of a BPS model 10 A BPS model can be very accurate in one aspect (e.g., control-flow), yet very different in another (e.g., processing times). Three main dimensions: control-flow, temporal, congestion. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 11 Proposed Framework Control-flow measures
Control-flow: Control-Flow Log Distance 12 Control-Flow Log Distance (CFLD): given two event logs L1 and L2, (minimum) average distance to transform each case in L1 into another case in L2, such that each case in L1 is paired to a different case in L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Camargo, M., Dumas, M., Rojas, O.G.: Discovering generative models from event logs: data-driven simulation vs deep learning. PeerJ Comput. Sci. 7, e577 (2021) Process event log Simulated event log
Control-flow: Control-Flow Log Distance 13 Control-Flow Log Distance (CFLD): given two event logs L1 and L2, (minimum) average distance to transform each case in L1 into another case in L2, such that each case in L1 is paired to a different case in L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H
Control-flow: Control-Flow Log Distance 14 Control-Flow Log Distance (CFLD): given two event logs L1 and L2, (minimum) average distance to transform each case in L1 into another case in L2, such that each case in L1 is paired to a different case in L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H 0
Control-flow: Control-Flow Log Distance 15 Control-Flow Log Distance (CFLD): given two event logs L1 and L2, (minimum) average distance to transform each case in L1 into another case in L2, such that each case in L1 is paired to a different case in L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H 0 0 0.2
Control-flow: Control-Flow Log Distance 16 Control-Flow Log Distance (CFLD): given two event logs L1 and L2, (minimum) average distance to transform each case in L1 into another case in L2, such that each case in L1 is paired to a different case in L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models CFLD = 0+0+0.75+0+0.2 5 = 0.19
Control-flow: N-Gram Distance 17 N-Gram Distance (NGD): given two event logs L1 and L2, and a positive integer 𝑛, difference in the frequencies of the 𝑛-grams observed in both L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Leemans, S.J.J., Syring, A.F., van der Aalst, W.M.P.: Earth movers’ stochastic conformance checking. In: BPM Forum 2019. LNBIP, vol. 360, pp. 127–143. Springer (2019) Process event log Simulated event log
Control-flow: N-Gram Distance 18 N-Gram Distance (NGD): given two event logs L1 and L2, and a positive integer 𝑛, difference in the frequencies of the 𝑛-grams observed in both L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H N = 3
Control-flow: N-Gram Distance 19 N-Gram Distance (NGD): given two event logs L1 and L2, and a positive integer 𝑛, difference in the frequencies of the 𝑛-grams observed in both L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H N = 3
Control-flow: N-Gram Distance 20 N-Gram Distance (NGD): given two event logs L1 and L2, and a positive integer 𝑛, difference in the frequencies of the 𝑛-grams observed in both L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H N = 3
Control-flow: N-Gram Distance 21 N-Gram Distance (NGD): given two event logs L1 and L2, and a positive integer 𝑛, difference in the frequencies of the 𝑛-grams observed in both L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H N = 3
Control-flow: N-Gram Distance 22 N-Gram Distance (NGD): given two event logs L1 and L2, and a positive integer 𝑛, difference in the frequencies of the 𝑛-grams observed in both L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 0 1 2 3 4 5 6 _ _ A _ A B _ A C _ A E A B C A C B A E F B C D C B D E F G F G H F G I C D _ Process event log Simulated event log
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 23 Proposed Framework Temporal measures
Process event log Simulated event log Temporal: Absolute Event Distribution 24 Absolute Event Distribution (AED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2. How different they are distributed through the event log. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Process event log Simulated event log Temporal: Absolute Event Distribution 25 Absolute Event Distribution (AED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2. How different they are distributed through the event log. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Temporal: Absolute Event Distribution 26 Absolute Event Distribution (AED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2. How different they are distributed through the event log. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 06-10-2022 10am – 11am 07-10-2022 11am – 12pm
Temporal: Absolute Event Distribution 27 Absolute Event Distribution (AED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2. How different they are distributed through the event log. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Earth mover's distance
Temporal: Circadian Event Distribution 28 Circadian Event Distribution (CED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2, for each day of the week. How different they are distributed through each day of the week. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Monday Tuesday Wednesday Thursday 10am – 11am Friday 11am – 12pm
Temporal: Circadian Event Distribution 29 Circadian Event Distribution (CED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2, for each day of the week. How different they are distributed through each day of the week. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EMD Monday Monday
Temporal: Relative Event Distribution 30 Relative Event Distribution (RED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2, with respect to the start of their case. How different they are distributed within each process case. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 00:00:00 01:01:47
Temporal: Relative Event Distribution Relative Event Distribution (RED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2, with respect to the start of their case. How different they are distributed within each process case. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EMD
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 32 Proposed Framework Congestion measures
Congestion: Case Arrival Rate 33 Case Arrival Rate (CAR): given two event logs L1 and L2, distance between how the case arrivals are distributed in L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 06-10-2022 10am – 11am
Congestion: Case Arrival Rate 34 Case Arrival Rate (CAR): given two event logs L1 and L2, distance between how the case arrivals are distributed in L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EMD
Congestion: Cycle Time Distribution 35 Cycle Time Distribution (CTD): given two event logs L1 and L2, distance between the distribution of cycle times in L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 01:07:02
Congestion: Cycle Time Distribution 36 Cycle Time Distribution (CTD): given two event logs L1 and L2, distance between the distribution of cycle times in L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EMD
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 37 Evaluation
Evaluation 38 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ1: Are the proposed measures able to discern the impact of different known modifications to a BPS model? EQ2: Is the N-Gram Distance’s performance significantly different from the CFLD’s performance? No modifications Control-flow Gateway probabilities Case arrival rate Activity durations Resource contention Working calendars Extraneous delays
Evaluation 39 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
Evaluation 40 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ1: Are the proposed measures able to discern the impact of different known modifications to a BPS model?
Evaluation 41 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ2: Is the N-Gram Distance’s performance significantly different from the CFLD’s performance? Kendall rank correlation coefficient 1.0
Evaluation 42 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique?
Evaluation 43 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique? 4 real-life processes: each split into disjoint training and test.
Evaluation 44 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique? Automatically discover BPS model with SIMOD and Service Miner.
Evaluation 45 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique?
Evaluation 46 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique?
Evaluation 47 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique?
Evaluation 48 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique?
Evaluation 49 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ4: Does the 1-WD report the same insights in real-life scenarios as the EMD?
Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 50 Conclusion
Conclusion 51 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Proposed a framework to measure the quality of a BPS model: decomposing into three perspectives (control-flow, temporal, and congestion), and defined measures for each of these perspectives. The measures proved their ability to detect the alterations in their corresponding perspectives. Beyond capturing the quality of BPS model and identifying the sources of discrepancies, the measures can also assist in eliciting areas for improvement in these techniques. The presented computationally efficient alternatives led to similar conclusions.
Future Work 52 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Explore the applicability of the proposed measures to other process mining problems, e.g., concept drift detection and variant analysis. Studying how to assess the quality of BPS models in the context of object-centric event logs. Study other quality measures for BPS models adapted from the field of generative machine learning, for example, by using a discriminative model that attempts to distinguish between data generated by the BPS model and real data.
1 sur 52

Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models

••
Télécharger pour lire hors ligne
Données & analyses

Business Process Simulation (BPS) is an approach to analyze the performance of business processes under different scenarios. For example, BPS allows us to estimate what would be the cycle time of a process if one or more resources became unavailable. The starting point of BPS is a process model annotated with simulation parameters (a BPS model). BPS models may be manually designed, based on information collected from stakeholders and empirical observations, or automatically discovered from execution data. Regardless of its origin, a key question when using a BPS model is how to assess its quality. In this paper, we propose a collection of measures to evaluate the quality of a BPS model w.r.t. its ability to replicate the observed behavior of the process. We advocate an approach whereby different measures tackle different process perspectives. We evaluate the ability of the proposed measures to discern the impact of modifications to a BPS model, and their ability to uncover the relative strengths and weaknesses of two approaches for automated discovery of BPS models. The evaluation shows that the measures not only capture how close a BPS model is to the observed behavior, but they also help us to identify sources of discrepancies. Presentation delivered by David Chapela-Campa at the BPM'2023 conference, Utrecht, September 2023.

Marlon Dumas
Marlon DumasProfessor at University of Tartu | Co-Founder at Apromore à University of Tartu and Apromore Pty Ltd

Recommandé

Walking the Way from Process Mining to AI-Driven Process Optimization par
Walking the Way from Process Mining to AI-Driven Process OptimizationWalking the Way from Process Mining to AI-Driven Process Optimization
Walking the Way from Process Mining to AI-Driven Process OptimizationMarlon Dumas
672 vues•27 diapositives
Traditional BI VS Self Service BI par
Traditional BI VS Self Service BITraditional BI VS Self Service BI
Traditional BI VS Self Service BIVisual_BI
632 vues•68 diapositives
Learning Accurate Business Process Simulation Models from Event Logs via Auto... par
Learning Accurate Business Process Simulation Models from Event Logs via Auto...Learning Accurate Business Process Simulation Models from Event Logs via Auto...
Learning Accurate Business Process Simulation Models from Event Logs via Auto...Marlon Dumas
421 vues•22 diapositives
Introduction to Business Process Monitoring and Process Mining par
Introduction to Business Process Monitoring and Process MiningIntroduction to Business Process Monitoring and Process Mining
Introduction to Business Process Monitoring and Process MiningMarlon Dumas
1.7K vues•207 diapositives
MLOps - The Assembly Line of ML par
MLOps - The Assembly Line of MLMLOps - The Assembly Line of ML
MLOps - The Assembly Line of MLJordan Birdsell
1.7K vues•19 diapositives
Business Analyst Training in Hyderabad par
Business Analyst Training in HyderabadBusiness Analyst Training in Hyderabad
Business Analyst Training in HyderabadUgs8008
993 vues•18 diapositives

Contenu connexe

Tendances

Process Mining - Chapter 9 - Operational Support par
Process Mining - Chapter 9 - Operational SupportProcess Mining - Chapter 9 - Operational Support
Process Mining - Chapter 9 - Operational SupportWil van der Aalst
2K vues•25 diapositives
Machine Learning Applications in Credit Risk par
Machine Learning Applications in Credit RiskMachine Learning Applications in Credit Risk
Machine Learning Applications in Credit RiskQuantUniversity
13.4K vues•57 diapositives
Data Driven Risk Assessment par
Data Driven Risk AssessmentData Driven Risk Assessment
Data Driven Risk AssessmentResolver Inc.
1.2K vues•42 diapositives
MLOps for production-level machine learning par
MLOps for production-level machine learningMLOps for production-level machine learning
MLOps for production-level machine learningcnvrg.io AI OS - Hands-on ML Workshops
478 vues•12 diapositives
Enterprise BPM Framework par
Enterprise BPM Framework Enterprise BPM Framework
Enterprise BPM Framework Frank Luyckx
4K vues•30 diapositives
Process Mining and Predictive Process Monitoring par
Process Mining and Predictive Process MonitoringProcess Mining and Predictive Process Monitoring
Process Mining and Predictive Process MonitoringMarlon Dumas
2.1K vues•46 diapositives

Tendances(20)

Process Mining - Chapter 9 - Operational Support par Wil van der Aalst
Process Mining - Chapter 9 - Operational SupportProcess Mining - Chapter 9 - Operational Support
Process Mining - Chapter 9 - Operational Support
Wil van der Aalst•2K vues
Machine Learning Applications in Credit Risk par QuantUniversity
Machine Learning Applications in Credit RiskMachine Learning Applications in Credit Risk
Machine Learning Applications in Credit Risk
QuantUniversity•13.4K vues
Data Driven Risk Assessment par Resolver Inc.
Data Driven Risk AssessmentData Driven Risk Assessment
Data Driven Risk Assessment
Resolver Inc.•1.2K vues
MLOps for production-level machine learning par cnvrg.io AI OS - Hands-on ML Workshops
MLOps for production-level machine learningMLOps for production-level machine learning
MLOps for production-level machine learning
cnvrg.io AI OS - Hands-on ML Workshops•478 vues
Enterprise BPM Framework par Frank Luyckx
Enterprise BPM Framework Enterprise BPM Framework
Enterprise BPM Framework
Frank Luyckx•4K vues
Process Mining and Predictive Process Monitoring par Marlon Dumas
Process Mining and Predictive Process MonitoringProcess Mining and Predictive Process Monitoring
Process Mining and Predictive Process Monitoring
Marlon Dumas•2.1K vues
MLOps in action par Pieter de Bruin
MLOps in actionMLOps in action
MLOps in action
Pieter de Bruin•1.6K vues
Machine Learning Model Deployment: Strategy to Implementation par DataWorks Summit
Machine Learning Model Deployment: Strategy to ImplementationMachine Learning Model Deployment: Strategy to Implementation
Machine Learning Model Deployment: Strategy to Implementation
DataWorks Summit•3.3K vues
Microsoft Power BI | Brief Introduction | PPT par Sophia Smith
Microsoft Power BI | Brief Introduction | PPTMicrosoft Power BI | Brief Introduction | PPT
Microsoft Power BI | Brief Introduction | PPT
Sophia Smith•3.3K vues
BPM (Business Process Management) Introduction par Integrify
BPM (Business Process Management) IntroductionBPM (Business Process Management) Introduction
BPM (Business Process Management) Introduction
Integrify•27.8K vues
Predictive Analytics - An Overview par MachinePulse
Predictive Analytics - An OverviewPredictive Analytics - An Overview
Predictive Analytics - An Overview
MachinePulse•32.5K vues
Enterprise Architecture Management (EAM) I Best Practices I NuggetHub par RichardNowack
Enterprise Architecture Management (EAM) I Best Practices I NuggetHubEnterprise Architecture Management (EAM) I Best Practices I NuggetHub
Enterprise Architecture Management (EAM) I Best Practices I NuggetHub
RichardNowack•340 vues
What is MLOps par Henrik Skogström
What is MLOpsWhat is MLOps
What is MLOps
Henrik Skogström•2.6K vues
Data Science, Machine Learning and Neural Networks par BICA Labs
Data Science, Machine Learning and Neural NetworksData Science, Machine Learning and Neural Networks
Data Science, Machine Learning and Neural Networks
BICA Labs•9K vues
Introduction to Azure Synapse Webinar par Peter Ward
Introduction to Azure Synapse WebinarIntroduction to Azure Synapse Webinar
Introduction to Azure Synapse Webinar
Peter Ward•222 vues
Getting Started with Azure AutoML par Vivek Raja P S
Getting Started with Azure AutoMLGetting Started with Azure AutoML
Getting Started with Azure AutoML
Vivek Raja P S•164 vues
Master Data Management - Aligning Data, Process, and Governance par DATAVERSITY
Master Data Management - Aligning Data, Process, and GovernanceMaster Data Management - Aligning Data, Process, and Governance
Master Data Management - Aligning Data, Process, and Governance
DATAVERSITY•11.9K vues
MLOps Virtual Event: Automating ML at Scale par Databricks
MLOps Virtual Event: Automating ML at ScaleMLOps Virtual Event: Automating ML at Scale
MLOps Virtual Event: Automating ML at Scale
Databricks•685 vues
The Decision Model and Notation (DMN) standard - a worked example par Aviva
The Decision Model and Notation (DMN) standard - a worked exampleThe Decision Model and Notation (DMN) standard - a worked example
The Decision Model and Notation (DMN) standard - a worked example
Aviva•5.5K vues
Master Data Management's Place in the Data Governance Landscape par CCG
Master Data Management's Place in the Data Governance Landscape Master Data Management's Place in the Data Governance Landscape
Master Data Management's Place in the Data Governance Landscape
CCG•1.2K vues

Similaire à Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models

Process Capability par
Process CapabilityProcess Capability
Process CapabilityThomas Abraham
7K vues•41 diapositives
Process Mining and Predictive Process Monitoring par
Process Mining and Predictive Process MonitoringProcess Mining and Predictive Process Monitoring
Process Mining and Predictive Process MonitoringMarlon Dumas
3.3K vues•40 diapositives
Statistical Process Control - SPC par
Statistical Process Control - SPCStatistical Process Control - SPC
Statistical Process Control - SPCPrasenjit Puri
153 vues•25 diapositives
Six Sigma Dfss Application In Data Accarucy par
Six Sigma Dfss Application In Data AccarucySix Sigma Dfss Application In Data Accarucy
Six Sigma Dfss Application In Data Accarucyxyhfun
529 vues•28 diapositives
Process Mining and Predictive Process Monitoring in Apromore par
Process Mining and Predictive Process Monitoring in ApromoreProcess Mining and Predictive Process Monitoring in Apromore
Process Mining and Predictive Process Monitoring in ApromoreMarlon Dumas
713 vues•34 diapositives
A process to improve the accuracy of mk ii fp to cosmic charles symons par
A process to improve the accuracy of mk ii fp to cosmic charles symonsA process to improve the accuracy of mk ii fp to cosmic charles symons
A process to improve the accuracy of mk ii fp to cosmic charles symonsIWSM Mensura
349 vues•30 diapositives

Similaire à Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models(20)

Process Capability par Thomas Abraham
Process CapabilityProcess Capability
Process Capability
Thomas Abraham•7K vues
Process Mining and Predictive Process Monitoring par Marlon Dumas
Process Mining and Predictive Process MonitoringProcess Mining and Predictive Process Monitoring
Process Mining and Predictive Process Monitoring
Marlon Dumas•3.3K vues
Statistical Process Control - SPC par Prasenjit Puri
Statistical Process Control - SPCStatistical Process Control - SPC
Statistical Process Control - SPC
Prasenjit Puri•153 vues
Six Sigma Dfss Application In Data Accarucy par xyhfun
Six Sigma Dfss Application In Data AccarucySix Sigma Dfss Application In Data Accarucy
Six Sigma Dfss Application In Data Accarucy
xyhfun•529 vues
Process Mining and Predictive Process Monitoring in Apromore par Marlon Dumas
Process Mining and Predictive Process Monitoring in ApromoreProcess Mining and Predictive Process Monitoring in Apromore
Process Mining and Predictive Process Monitoring in Apromore
Marlon Dumas•713 vues
A process to improve the accuracy of mk ii fp to cosmic charles symons par IWSM Mensura
A process to improve the accuracy of mk ii fp to cosmic charles symonsA process to improve the accuracy of mk ii fp to cosmic charles symons
A process to improve the accuracy of mk ii fp to cosmic charles symons
IWSM Mensura•349 vues
Multi-Perspective Comparison of Business Processes Variants Based on Event Logs par Marlon Dumas
Multi-Perspective Comparison of Business Processes Variants Based on Event LogsMulti-Perspective Comparison of Business Processes Variants Based on Event Logs
Multi-Perspective Comparison of Business Processes Variants Based on Event Logs
Marlon Dumas•557 vues
Advantages Of Linear Programming Models par Brenda Torres
Advantages Of Linear Programming ModelsAdvantages Of Linear Programming Models
Advantages Of Linear Programming Models
Brenda Torres•3 vues
Robust PID Controller Design for Non-Minimum Phase Systems using Magnitude Op... par IRJET Journal
Robust PID Controller Design for Non-Minimum Phase Systems using Magnitude Op...Robust PID Controller Design for Non-Minimum Phase Systems using Magnitude Op...
Robust PID Controller Design for Non-Minimum Phase Systems using Magnitude Op...
IRJET Journal•41 vues
Comprehensive acquisition management methodology rev6 par clifpt
Comprehensive acquisition management methodology rev6Comprehensive acquisition management methodology rev6
Comprehensive acquisition management methodology rev6
clifpt•44 vues
3D-DRESD AC par Marco Santambrogio
3D-DRESD AC3D-DRESD AC
3D-DRESD AC
Marco Santambrogio•408 vues
Se notes par Noor Ul Hudda Memon
Se notesSe notes
Se notes
Noor Ul Hudda Memon•1.7K vues
Gestión de la calidad sem 2 par youffre
Gestión de la calidad sem 2Gestión de la calidad sem 2
Gestión de la calidad sem 2
youffre•118 vues
Lpg detector final ppt par Agara Mudhalvan
Lpg detector final pptLpg detector final ppt
Lpg detector final ppt
Agara Mudhalvan•147 vues
Practical Guides on Programming with Big Number Library in Scientific Researches par theijes
Practical Guides on Programming with Big Number Library in Scientific ResearchesPractical Guides on Programming with Big Number Library in Scientific Researches
Practical Guides on Programming with Big Number Library in Scientific Researches
theijes•31 vues
Unbiased, Fine-Grained Description of Processes Performance from Event Data par Vadim Denisov
Unbiased, Fine-Grained Description of Processes Performance from Event DataUnbiased, Fine-Grained Description of Processes Performance from Event Data
Unbiased, Fine-Grained Description of Processes Performance from Event Data
Vadim Denisov•34 vues
BUSINESS PROCESS MODELED WITH BPMN AND CTL MODEL CHECKING par IJCI JOURNAL
BUSINESS PROCESS MODELED WITH BPMN AND CTL MODEL CHECKINGBUSINESS PROCESS MODELED WITH BPMN AND CTL MODEL CHECKING
BUSINESS PROCESS MODELED WITH BPMN AND CTL MODEL CHECKING
IJCI JOURNAL•2 vues
DECISIONS IN A SUPPLY CHAIN MODELING FOR COMPARATIVE EVALUATION STRATEGIES IN... par IAEME Publication
DECISIONS IN A SUPPLY CHAIN MODELING FOR COMPARATIVE EVALUATION STRATEGIES IN...DECISIONS IN A SUPPLY CHAIN MODELING FOR COMPARATIVE EVALUATION STRATEGIES IN...
DECISIONS IN A SUPPLY CHAIN MODELING FOR COMPARATIVE EVALUATION STRATEGIES IN...
IAEME Publication•267 vues
Decisions in a supply chain modeling for comparative evaluation strategies in... par IAEME Publication
Decisions in a supply chain modeling for comparative evaluation strategies in...Decisions in a supply chain modeling for comparative evaluation strategies in...
Decisions in a supply chain modeling for comparative evaluation strategies in...
IAEME Publication•294 vues
Software Architecture in the age of Cloud Computing par Jaroslav Gergic
Software Architecture in the age of Cloud ComputingSoftware Architecture in the age of Cloud Computing
Software Architecture in the age of Cloud Computing
Jaroslav Gergic•525 vues

Plus de Marlon Dumas

Discovery and Simulation of Business Processes with Probabilistic Resource Av... par
Discovery and Simulation of Business Processes with Probabilistic Resource Av...Discovery and Simulation of Business Processes with Probabilistic Resource Av...
Discovery and Simulation of Business Processes with Probabilistic Resource Av...Marlon Dumas
63 vues•15 diapositives
Business Process Optimization: Status and Perspectives par
Business Process Optimization: Status and PerspectivesBusiness Process Optimization: Status and Perspectives
Business Process Optimization: Status and PerspectivesMarlon Dumas
544 vues•18 diapositives
Learning When to Treat Business Processes: Prescriptive Process Monitoring wi... par
Learning When to Treat Business Processes: Prescriptive Process Monitoring wi...Learning When to Treat Business Processes: Prescriptive Process Monitoring wi...
Learning When to Treat Business Processes: Prescriptive Process Monitoring wi...Marlon Dumas
35 vues•19 diapositives
Process Mining and Data-Driven Process Simulation par
Process Mining and Data-Driven Process SimulationProcess Mining and Data-Driven Process Simulation
Process Mining and Data-Driven Process SimulationMarlon Dumas
299 vues•29 diapositives
Modeling Extraneous Activity Delays in Business Process Simulation par
Modeling Extraneous Activity Delays in Business Process SimulationModeling Extraneous Activity Delays in Business Process Simulation
Modeling Extraneous Activity Delays in Business Process SimulationMarlon Dumas
148 vues•42 diapositives
Prescriptive Process Monitoring Under Uncertainty and Resource Constraints par
Prescriptive Process Monitoring Under Uncertainty and Resource ConstraintsPrescriptive Process Monitoring Under Uncertainty and Resource Constraints
Prescriptive Process Monitoring Under Uncertainty and Resource ConstraintsMarlon Dumas
92 vues•23 diapositives

Plus de Marlon Dumas(20)

Discovery and Simulation of Business Processes with Probabilistic Resource Av... par Marlon Dumas
Discovery and Simulation of Business Processes with Probabilistic Resource Av...Discovery and Simulation of Business Processes with Probabilistic Resource Av...
Discovery and Simulation of Business Processes with Probabilistic Resource Av...
Marlon Dumas•63 vues
Business Process Optimization: Status and Perspectives par Marlon Dumas
Business Process Optimization: Status and PerspectivesBusiness Process Optimization: Status and Perspectives
Business Process Optimization: Status and Perspectives
Marlon Dumas•544 vues
Learning When to Treat Business Processes: Prescriptive Process Monitoring wi... par Marlon Dumas
Learning When to Treat Business Processes: Prescriptive Process Monitoring wi...Learning When to Treat Business Processes: Prescriptive Process Monitoring wi...
Learning When to Treat Business Processes: Prescriptive Process Monitoring wi...
Marlon Dumas•35 vues
Process Mining and Data-Driven Process Simulation par Marlon Dumas
Process Mining and Data-Driven Process SimulationProcess Mining and Data-Driven Process Simulation
Process Mining and Data-Driven Process Simulation
Marlon Dumas•299 vues
Modeling Extraneous Activity Delays in Business Process Simulation par Marlon Dumas
Modeling Extraneous Activity Delays in Business Process SimulationModeling Extraneous Activity Delays in Business Process Simulation
Modeling Extraneous Activity Delays in Business Process Simulation
Marlon Dumas•148 vues
Prescriptive Process Monitoring Under Uncertainty and Resource Constraints par Marlon Dumas
Prescriptive Process Monitoring Under Uncertainty and Resource ConstraintsPrescriptive Process Monitoring Under Uncertainty and Resource Constraints
Prescriptive Process Monitoring Under Uncertainty and Resource Constraints
Marlon Dumas•92 vues
Robotic Process Mining par Marlon Dumas
Robotic Process MiningRobotic Process Mining
Robotic Process Mining
Marlon Dumas•333 vues
Accurate and Reliable What-If Analysis of Business Processes: Is it Achievable? par Marlon Dumas
Accurate and Reliable What-If Analysis of Business Processes: Is it Achievable?Accurate and Reliable What-If Analysis of Business Processes: Is it Achievable?
Accurate and Reliable What-If Analysis of Business Processes: Is it Achievable?
Marlon Dumas•51 vues
Process Mining: A Guide for Practitioners par Marlon Dumas
Process Mining: A Guide for PractitionersProcess Mining: A Guide for Practitioners
Process Mining: A Guide for Practitioners
Marlon Dumas•141 vues
Optimización de procesos basada en datos par Marlon Dumas
Optimización de procesos basada en datosOptimización de procesos basada en datos
Optimización de procesos basada en datos
Marlon Dumas•386 vues
Process Mining and AI for Continuous Process Improvement par Marlon Dumas
Process Mining and AI for Continuous Process ImprovementProcess Mining and AI for Continuous Process Improvement
Process Mining and AI for Continuous Process Improvement
Marlon Dumas•207 vues
Prescriptive Process Monitoring for Cost-Aware Cycle Time Reduction par Marlon Dumas
Prescriptive Process Monitoring for Cost-Aware Cycle Time ReductionPrescriptive Process Monitoring for Cost-Aware Cycle Time Reduction
Prescriptive Process Monitoring for Cost-Aware Cycle Time Reduction
Marlon Dumas•362 vues
Mine Your Simulation Model: Automated Discovery of Business Process Simulatio... par Marlon Dumas
Mine Your Simulation Model: Automated Discovery of Business Process Simulatio...Mine Your Simulation Model: Automated Discovery of Business Process Simulatio...
Mine Your Simulation Model: Automated Discovery of Business Process Simulatio...
Marlon Dumas•654 vues
On the Road to AI-Infused Process Execution par Marlon Dumas
On the Road to AI-Infused Process ExecutionOn the Road to AI-Infused Process Execution
On the Road to AI-Infused Process Execution
Marlon Dumas•270 vues
Process Mining 2.0: From Insights to Actions par Marlon Dumas
Process Mining 2.0: From Insights to ActionsProcess Mining 2.0: From Insights to Actions
Process Mining 2.0: From Insights to Actions
Marlon Dumas•742 vues
Process Mining in Action: Self-service data science for business teams par Marlon Dumas
Process Mining in Action: Self-service data science for business teamsProcess Mining in Action: Self-service data science for business teams
Process Mining in Action: Self-service data science for business teams
Marlon Dumas•407 vues
Enterprise Systems par Marlon Dumas
Enterprise SystemsEnterprise Systems
Enterprise Systems
Marlon Dumas•797 vues
Process Mining Meets Causal Machine Learning: Discovering Causal Rules From E... par Marlon Dumas
Process Mining Meets Causal Machine Learning: Discovering Causal Rules From E...Process Mining Meets Causal Machine Learning: Discovering Causal Rules From E...
Process Mining Meets Causal Machine Learning: Discovering Causal Rules From E...
Marlon Dumas•437 vues
Identifying Candidate Routines for Robotic Process Automation From Unsegmente... par Marlon Dumas
Identifying Candidate Routines for Robotic Process Automation From Unsegmente...Identifying Candidate Routines for Robotic Process Automation From Unsegmente...
Identifying Candidate Routines for Robotic Process Automation From Unsegmente...
Marlon Dumas•444 vues
Analitica de datos e inteligencia artificial para procesos de negocios par Marlon Dumas
Analitica de datos e inteligencia artificial para procesos de negociosAnalitica de datos e inteligencia artificial para procesos de negocios
Analitica de datos e inteligencia artificial para procesos de negocios
Marlon Dumas•356 vues

Dernier

Listed Instruments Survey 2022.pptx par
Listed Instruments Survey 2022.pptxListed Instruments Survey 2022.pptx
Listed Instruments Survey 2022.pptxsecretariat4
130 vues•12 diapositives
Penetration testing by Burpsuite par
Penetration testing by BurpsuitePenetration testing by Burpsuite
Penetration testing by BurpsuiteAyonDebnathCertified
5 vues•19 diapositives
Custom Tag Manager Templates par
Custom Tag Manager TemplatesCustom Tag Manager Templates
Custom Tag Manager TemplatesMarkus Baersch
30 vues•17 diapositives
Product Research sample.pdf par
Product Research sample.pdfProduct Research sample.pdf
Product Research sample.pdfAllenSingson
35 vues•29 diapositives
Infomatica-MDM.pptx par
Infomatica-MDM.pptxInfomatica-MDM.pptx
Infomatica-MDM.pptxKapil Rangwani
12 vues•16 diapositives
VoxelNet par
VoxelNetVoxelNet
VoxelNettaeseon ryu
20 vues•21 diapositives

Dernier(20)

Listed Instruments Survey 2022.pptx par secretariat4
Listed Instruments Survey 2022.pptxListed Instruments Survey 2022.pptx
Listed Instruments Survey 2022.pptx
secretariat4•130 vues
Penetration testing by Burpsuite par AyonDebnathCertified
Penetration testing by BurpsuitePenetration testing by Burpsuite
Penetration testing by Burpsuite
AyonDebnathCertified•5 vues
Custom Tag Manager Templates par Markus Baersch
Custom Tag Manager TemplatesCustom Tag Manager Templates
Custom Tag Manager Templates
Markus Baersch•30 vues
Product Research sample.pdf par AllenSingson
Product Research sample.pdfProduct Research sample.pdf
Product Research sample.pdf
AllenSingson•35 vues
Infomatica-MDM.pptx par Kapil Rangwani
Infomatica-MDM.pptxInfomatica-MDM.pptx
Infomatica-MDM.pptx
Kapil Rangwani•12 vues
VoxelNet par taeseon ryu
VoxelNetVoxelNet
VoxelNet
taeseon ryu•20 vues
Dr. Ousmane Badiane-2023 ReSAKSS Conference par AKADEMIYA2063
Dr. Ousmane Badiane-2023 ReSAKSS ConferenceDr. Ousmane Badiane-2023 ReSAKSS Conference
Dr. Ousmane Badiane-2023 ReSAKSS Conference
AKADEMIYA2063•5 vues
Inawisdom Quick Sight par PhilipBasford
Inawisdom Quick SightInawisdom Quick Sight
Inawisdom Quick Sight
PhilipBasford•8 vues
Customer Data Cleansing Project.pptx par Nat O
Customer Data Cleansing Project.pptxCustomer Data Cleansing Project.pptx
Customer Data Cleansing Project.pptx
Nat O•6 vues
Enhancing Financial Sentiment Analysis via Retrieval Augmented Large Language... par patiladiti752
Enhancing Financial Sentiment Analysis via Retrieval Augmented Large Language...Enhancing Financial Sentiment Analysis via Retrieval Augmented Large Language...
Enhancing Financial Sentiment Analysis via Retrieval Augmented Large Language...
patiladiti752•8 vues
AZConf 2023 - Considerations for LLMOps: Running LLMs in production par SARADINDU SENGUPTA
AZConf 2023 - Considerations for LLMOps: Running LLMs in productionAZConf 2023 - Considerations for LLMOps: Running LLMs in production
AZConf 2023 - Considerations for LLMOps: Running LLMs in production
SARADINDU SENGUPTA•9 vues
Construction Accidents & Injuries par Bisnar Chase Personal Injury Attorneys
Construction Accidents & InjuriesConstruction Accidents & Injuries
Construction Accidents & Injuries
Bisnar Chase Personal Injury Attorneys•6 vues
apple.pptx par honeybeeqwe
apple.pptxapple.pptx
apple.pptx
honeybeeqwe•6 vues
Employees attrition par MaryAlejandraDiaz
Employees attritionEmployees attrition
Employees attrition
MaryAlejandraDiaz•7 vues
6498-Butun_Beyinli_Cocuq-Daniel_J.Siegel-Tina_Payne_Bryson-2011-259s.pdf par 10urkyr34
6498-Butun_Beyinli_Cocuq-Daniel_J.Siegel-Tina_Payne_Bryson-2011-259s.pdf6498-Butun_Beyinli_Cocuq-Daniel_J.Siegel-Tina_Payne_Bryson-2011-259s.pdf
6498-Butun_Beyinli_Cocuq-Daniel_J.Siegel-Tina_Payne_Bryson-2011-259s.pdf
10urkyr34•7 vues
CRM stick or twist.pptx par info828217
CRM stick or twist.pptxCRM stick or twist.pptx
CRM stick or twist.pptx
info828217•11 vues
Oral presentation (1).pdf par reemalmazroui8
Oral presentation (1).pdfOral presentation (1).pdf
Oral presentation (1).pdf
reemalmazroui8•6 vues
Best Home Security Systems.pptx par mogalang
Best Home Security Systems.pptxBest Home Security Systems.pptx
Best Home Security Systems.pptx
mogalang•9 vues
Data Journeys Hard Talk workshop final.pptx par info828217
Data Journeys Hard Talk workshop final.pptxData Journeys Hard Talk workshop final.pptx
Data Journeys Hard Talk workshop final.pptx
info828217•11 vues
Inawsidom - Data Journey par PhilipBasford
Inawsidom - Data JourneyInawsidom - Data Journey
Inawsidom - Data Journey
PhilipBasford•9 vues

Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models

  • 1. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models David Chapela-Campa1, Ismail Benchekroun2, Opher Baron2, Marlon Dumas1, Dmitry Krass2, and Arik Senderovich3 21st International Conference on Business Process Management (BPM 2023) 1 University of Tartu, Estonia 2 University of Toronto, Canada 3 York University, Canada
  • 2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 2 Introduction
  • 3. Business Process Simulation (BPS) 3 BPS allows users to address “what-if” analysis questions. What would be the cycle time of the process if the rate of arrival of new cases doubles? Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
  • 4. Business Process Simulation (BPS) 4 BPS models can be manually created by modeling experts. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
  • 5. Business Process Simulation (BPS) 5 BPS models can be manually created by modeling experts. Use of process mining techniques to automatically discover BPS models from business process event logs. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
  • 6. Business Process Simulation (BPS) 6 How to assess the quality of a BPS model? Automatic assessment. Useful to detect the sources of deviations. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
  • 7. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 7 Proposed Framework
  • 8. Quality of a BPS model 8 How to assess the quality of a BPS model? Comparing an event log with a BPS model. Variety of different BPS models formats. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log
  • 9. Quality of a BPS model 9 How to assess the quality of a BPS model? Generate K simulated event logs. Compare individually and report the average and confidence interval. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 9 Process event log K simulated event logs
  • 10. Quality of a BPS model 10 A BPS model can be very accurate in one aspect (e.g., control-flow), yet very different in another (e.g., processing times). Three main dimensions: control-flow, temporal, congestion. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log
  • 11. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 11 Proposed Framework Control-flow measures
  • 12. Control-flow: Control-Flow Log Distance 12 Control-Flow Log Distance (CFLD): given two event logs L1 and L2, (minimum) average distance to transform each case in L1 into another case in L2, such that each case in L1 is paired to a different case in L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Camargo, M., Dumas, M., Rojas, O.G.: Discovering generative models from event logs: data-driven simulation vs deep learning. PeerJ Comput. Sci. 7, e577 (2021) Process event log Simulated event log
  • 13. Control-flow: Control-Flow Log Distance 13 Control-Flow Log Distance (CFLD): given two event logs L1 and L2, (minimum) average distance to transform each case in L1 into another case in L2, such that each case in L1 is paired to a different case in L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H
  • 14. Control-flow: Control-Flow Log Distance 14 Control-Flow Log Distance (CFLD): given two event logs L1 and L2, (minimum) average distance to transform each case in L1 into another case in L2, such that each case in L1 is paired to a different case in L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H 0
  • 15. Control-flow: Control-Flow Log Distance 15 Control-Flow Log Distance (CFLD): given two event logs L1 and L2, (minimum) average distance to transform each case in L1 into another case in L2, such that each case in L1 is paired to a different case in L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H 0 0 0.2
  • 16. Control-flow: Control-Flow Log Distance 16 Control-Flow Log Distance (CFLD): given two event logs L1 and L2, (minimum) average distance to transform each case in L1 into another case in L2, such that each case in L1 is paired to a different case in L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models CFLD = 0+0+0.75+0+0.2 5 = 0.19
  • 17. Control-flow: N-Gram Distance 17 N-Gram Distance (NGD): given two event logs L1 and L2, and a positive integer 𝑛, difference in the frequencies of the 𝑛-grams observed in both L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Leemans, S.J.J., Syring, A.F., van der Aalst, W.M.P.: Earth movers’ stochastic conformance checking. In: BPM Forum 2019. LNBIP, vol. 360, pp. 127–143. Springer (2019) Process event log Simulated event log
  • 18. Control-flow: N-Gram Distance 18 N-Gram Distance (NGD): given two event logs L1 and L2, and a positive integer 𝑛, difference in the frequencies of the 𝑛-grams observed in both L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H N = 3
  • 19. Control-flow: N-Gram Distance 19 N-Gram Distance (NGD): given two event logs L1 and L2, and a positive integer 𝑛, difference in the frequencies of the 𝑛-grams observed in both L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H N = 3
  • 20. Control-flow: N-Gram Distance 20 N-Gram Distance (NGD): given two event logs L1 and L2, and a positive integer 𝑛, difference in the frequencies of the 𝑛-grams observed in both L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H N = 3
  • 21. Control-flow: N-Gram Distance 21 N-Gram Distance (NGD): given two event logs L1 and L2, and a positive integer 𝑛, difference in the frequencies of the 𝑛-grams observed in both L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Process event log Simulated event log A B C D A B C D A C B D A E F G H A E F G I A B C D A C B D A E F G A E F G H A E F G H N = 3
  • 22. Control-flow: N-Gram Distance 22 N-Gram Distance (NGD): given two event logs L1 and L2, and a positive integer 𝑛, difference in the frequencies of the 𝑛-grams observed in both L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 0 1 2 3 4 5 6 _ _ A _ A B _ A C _ A E A B C A C B A E F B C D C B D E F G F G H F G I C D _ Process event log Simulated event log
  • 23. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 23 Proposed Framework Temporal measures
  • 24. Process event log Simulated event log Temporal: Absolute Event Distribution 24 Absolute Event Distribution (AED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2. How different they are distributed through the event log. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
  • 25. Process event log Simulated event log Temporal: Absolute Event Distribution 25 Absolute Event Distribution (AED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2. How different they are distributed through the event log. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
  • 26. Temporal: Absolute Event Distribution 26 Absolute Event Distribution (AED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2. How different they are distributed through the event log. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 06-10-2022 10am – 11am 07-10-2022 11am – 12pm
  • 27. Temporal: Absolute Event Distribution 27 Absolute Event Distribution (AED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2. How different they are distributed through the event log. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Earth mover's distance
  • 28. Temporal: Circadian Event Distribution 28 Circadian Event Distribution (CED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2, for each day of the week. How different they are distributed through each day of the week. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Monday Tuesday Wednesday Thursday 10am – 11am Friday 11am – 12pm
  • 29. Temporal: Circadian Event Distribution 29 Circadian Event Distribution (CED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2, for each day of the week. How different they are distributed through each day of the week. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EMD Monday Monday
  • 30. Temporal: Relative Event Distribution 30 Relative Event Distribution (RED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2, with respect to the start of their case. How different they are distributed within each process case. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 00:00:00 01:01:47
  • 31. Temporal: Relative Event Distribution Relative Event Distribution (RED): given two event logs L1 and L2, distance between the time series of the events in L1 and L2, with respect to the start of their case. How different they are distributed within each process case. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EMD
  • 32. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 32 Proposed Framework Congestion measures
  • 33. Congestion: Case Arrival Rate 33 Case Arrival Rate (CAR): given two event logs L1 and L2, distance between how the case arrivals are distributed in L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 06-10-2022 10am – 11am
  • 34. Congestion: Case Arrival Rate 34 Case Arrival Rate (CAR): given two event logs L1 and L2, distance between how the case arrivals are distributed in L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EMD
  • 35. Congestion: Cycle Time Distribution 35 Cycle Time Distribution (CTD): given two event logs L1 and L2, distance between the distribution of cycle times in L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 01:07:02
  • 36. Congestion: Cycle Time Distribution 36 Cycle Time Distribution (CTD): given two event logs L1 and L2, distance between the distribution of cycle times in L1 and L2. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EMD
  • 37. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 37 Evaluation
  • 38. Evaluation 38 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ1: Are the proposed measures able to discern the impact of different known modifications to a BPS model? EQ2: Is the N-Gram Distance’s performance significantly different from the CFLD’s performance? No modifications Control-flow Gateway probabilities Case arrival rate Activity durations Resource contention Working calendars Extraneous delays
  • 39. Evaluation 39 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models
  • 40. Evaluation 40 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ1: Are the proposed measures able to discern the impact of different known modifications to a BPS model?
  • 41. Evaluation 41 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ2: Is the N-Gram Distance’s performance significantly different from the CFLD’s performance? Kendall rank correlation coefficient 1.0
  • 42. Evaluation 42 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique?
  • 43. Evaluation 43 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique? 4 real-life processes: each split into disjoint training and test.
  • 44. Evaluation 44 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique? Automatically discover BPS model with SIMOD and Service Miner.
  • 45. Evaluation 45 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique?
  • 46. Evaluation 46 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique?
  • 47. Evaluation 47 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique?
  • 48. Evaluation 48 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ3: Given two BPS models discovered by existing automated BPS model discovery techniques in real-life scenarios, are the proposed measures able to identify the strengths and weaknesses of each technique?
  • 49. Evaluation 49 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models EQ4: Does the 1-WD report the same insights in real-life scenarios as the EMD?
  • 50. Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models 50 Conclusion
  • 51. Conclusion 51 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Proposed a framework to measure the quality of a BPS model: decomposing into three perspectives (control-flow, temporal, and congestion), and defined measures for each of these perspectives. The measures proved their ability to detect the alterations in their corresponding perspectives. Beyond capturing the quality of BPS model and identifying the sources of discrepancies, the measures can also assist in eliciting areas for improvement in these techniques. The presented computationally efficient alternatives led to similar conclusions.
  • 52. Future Work 52 Can I Trust My Simulation Model? Measuring the Quality of Business Process Simulation Models Explore the applicability of the proposed measures to other process mining problems, e.g., concept drift detection and variant analysis. Studying how to assess the quality of BPS models in the context of object-centric event logs. Study other quality measures for BPS models adapted from the field of generative machine learning, for example, by using a discriminative model that attempts to distinguish between data generated by the BPS model and real data.

Notes de l'éditeur

  1. So, the first thing we need to know is, what is business process simulation?
  2. BPS aims to replicate the execution of a process, to mimic the behavior of the process, in a certain scenario (set of resources, etc.) analyzing its performance (KPIs) This allows users… The starting point is a BPS model… A process model annotated with a set of simulation parameters that define the scenario (resources, calendars, activity durations…). [NEXT]
  3. BPS models may be manually created based on information collected via interviews or empirical observations.. Or [NEXT…]
  4. they may be automatically discovered from execution data recorded in process-aware information systems (event logs) Regardless of the origin, a key question when using a BPS model is… [NEXT]
  5. how to assess its quality? Several approaches have been proposed to address this problem. However, these approaches are either manual and qualitative or they produce a single number that does not allow one to identify the source(s) of deviations between the BPS model and the observed reality
  6. First we need to decide what to compare when assessing the quality of a BPS model. What we are comparing is a BPS model, with a PROCESS What we usually have is…
  7. …an event log! Now, the first thing we asked ourselves was: should we compare a BPS model against a event log? But it is true that BPS models do not follow a standard structure… They can be formed by queue systems, but less of more models (resources, more complex waiting times), and they will change during time with new research. Thus, what we can do is simulate an event log out of the BPS model
  8. and compare log to log. K runs and compute the avg and conf int
  9. Abstract event logs into time-series or histograms and compare them
  10. We have two event logs, we are focusing on the control-flow, so the first step is to… [NEXT]
  11. obtain the activity sequences of each event log. Then, we compute the Damerau-Levenshtein (string edit distance) distance between each pair of cases… [NEXT]
  12. For exampe, [comment examples], we repeat this for each case Once we have all the pairings computed, we compute the matching between cases of one log to another (such as each case in one log is matched to one case in the other event log, with no repetitions… [NEXT]
  13. While minimizing the sum of distances using the Hungarian algorithm for optimal alignment. Finally, the CFLD measure is the average of these distances… [NEXT]
  14. The computational complexity of computing the DL-distance for all possible pairings is O(N2 ×MTL3) where N is the number of traces in the logs (assuming both logs have an equal number of cases, which holds in our setting) and MTL is the maximum trace length. Since all pairings are put into a matrix to compute the optimal alignment of cases (the one that minimizes the total sum of distances), CFLD’s memory complexity is quadratic on the number of cases. The optimal alignment of traces using the Hungarian algorithm has a cubic complexity on the number of cases.
  15. In the same way than for the CFLD, we are focusing on the control-flow, so the first step is to obtain the activity sequences of each event log. Leemans et al. measure the quality of a stochastic process model by mapping the model and a log to their Directly-Follows Graph (DFG), viewing each DFG as a histogram, and measuring the distance between these histograms. We note that the histogram of 2-grams of a log is equal to the histogram of its DFG. Given this observation, we generalize the approach of to n-grams, noting that the histogram of n-grams of a log is equal to the (n-1)th-Markovian abstraction of the log. Then… [NEXT]
  16. Let’s assume a size of N=3, so the N-grams are 3-grams (sequences of three activites). We compute all 3-grams observed in both logs, considering two dummy activities in the start and end of each trace. Then… [NEXT]
  17. We measure the frequency of each N-gram in each log…
  18. We measure the frequency of each N-gram in each log…
  19. And compute the sum of absolute differences between them, normalized by the sum of frequencies of all n-grams (value between 0-1). NGD is considerably more efficient than CFLD, as the construction of the histogram of n-grams is linear on the number of events in the log, and the same goes for computing the differences between the n-gram histograms.
  20. For the temporal measures, we first do the opposite of the control-flow, we abstract from the control-flow information… [NEXT]
  21. retaining only the events (in this case start and end)… [NEXT]
  22. Then we discretize these events into bins of 1h in the following way. Obtaining a time-series with the number of events happening in each hour of the process timeline.
  23. Once we have the temporal distribution (not a probabilistic distribution, but just the events occurring in the timeline), we compare both time-series with the EMD to measure the distance. We measure the trend.
  24. The same process is followed for the next measure, but in this case discretized to weekdays. In this way, we measure the seasonality of the events happening in the process.
  25. The temporal distribution of events of each day of the week is compared and then we compute the average distance of the 7 days.
  26. Finally, for the third one, we focus on how the events are distributed within their corresponding trace. For this, we compute the time from the case arrival to the event happening and bin it in hours.
  27. Finally, for the third one, we focus on how the events are distributed within their corresponding trace. For this, we compute the time from the case arrival to the event happening and bin it in hours.
  28. For the case arrival rate we want to measure how different the arrival of cases is. So, we retain only the events denoting the arrival of each case (start of first activity instance). Then we build the distribution in the same way than the previous metrics… [NEXT]
  29. and compare with EMD.