Comparing the Effectiveness of Reasoning Formalisms for Partial Models
•
1 like•683 views
Pooya Saadatpanah, Michalis Famelis, Jan Gorzny, Nathan Robinson, Marsha Chechik, Rick Salay University of Toronto September 30th, 2012 MoDeVVa’12
![Comparing the
Effectiveness
of Reasoning
Formalisms for
Uncertainty in software modeling
Partial Models
Saadatpanah,
Famelis,
Gorzny,
Robinson,
Chechik, Salay
• Uncertainty: pervasive in MDE
Introduction
Designer
Uncertainty • Models with uncertainty:
Modeling • Represent choice among many possibilities
Uncertainty
• Can be refined to many different classical models
Property
Checking
Process
Verification
Technologies • Our goal:
Experiments
Handle models with uncertainty in MDE
Results without having to remove it [MoDeVVa’11].
Conclusion
2 / 32](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Recommended
Recommended
More Related Content
Viewers also liked
Viewers also liked (6)
Similar to Comparing the Effectiveness of Reasoning Formalisms for Partial Models
Similar to Comparing the Effectiveness of Reasoning Formalisms for Partial Models (20)
More from Michalis Famelis
More from Michalis Famelis (6)
Recently uploaded
Recently uploaded (20)
Comparing the Effectiveness of Reasoning Formalisms for Partial Models
- 1. Comparing the Effectiveness of Reasoning Formalisms for Partial Models Saadatpanah, Famelis, Gorzny, Comparing the Effectiveness of Reasoning Robinson, Chechik, Salay Formalisms for Partial Models Introduction Designer Uncertainty Pooya Saadatpanah, Michalis Famelis, Jan Gorzny, Modeling Nathan Robinson, Marsha Chechik, Rick Salay Uncertainty Property Checking University of Toronto Process Verification Technologies September 30th, 2012 Experiments MoDeVVa’12 Results Conclusion 1 / 32
- 2. Comparing the Effectiveness of Reasoning Formalisms for Uncertainty in software modeling Partial Models Saadatpanah, Famelis, Gorzny, Robinson, Chechik, Salay • Uncertainty: pervasive in MDE Introduction Designer Uncertainty • Models with uncertainty: Modeling • Represent choice among many possibilities Uncertainty • Can be refined to many different classical models Property Checking Process Verification Technologies • Our goal: Experiments Handle models with uncertainty in MDE Results without having to remove it [MoDeVVa’11]. Conclusion 2 / 32
- 3. Comparing the Effectiveness of Reasoning Formalisms for Existing Work Partial Models Saadatpanah, Famelis, Gorzny, Robinson, Chechik, Salay Introduction Designer Uncertainty Modeling Uncertainty Property Checking Process Verification Technologies Experiments Results Conclusion 3 / 32
- 4. Comparing the Effectiveness of Reasoning Formalisms for In This Paper Partial Models Saadatpanah, Famelis, Gorzny, Robinson, Chechik, Salay Introduction Designer Uncertainty Modeling Uncertainty Property Checking Process Verification Technologies Experiments Results Conclusion 4 / 32
- 5. Comparing the Effectiveness of Reasoning Formalisms for Outline Partial Models Saadatpanah, Famelis, Introduction Gorzny, Robinson, Chechik, Salay What is uncertainty? Introduction Designer How to represent uncertainty with partial models (MAVO). Uncertainty Modeling Uncertainty Process for checking properties Property Checking Process Alternative verification technologies Verification Technologies Experiments Experiments Results Results Conclusion Conclusion 5 / 32
- 6. Comparing the Effectiveness of Reasoning Formalisms for Introduction to Uncertainty Partial Models Saadatpanah, Famelis, Gorzny, Robinson, Chechik, Salay What the designer knows. Introduction Designer Uncertainty Modeling Uncertainty Property Checking Process Verification Technologies Experiments Results Conclusion 6 / 32
- 7. Comparing the Effectiveness of Reasoning Formalisms for Introduction to Uncertainty Partial Models Saadatpanah, Famelis, Gorzny, Robinson, Chechik, Salay What the designer does not know. Introduction Designer Uncertainty Modeling Uncertainty Property Checking Process Verification Technologies Experiments Results Conclusion 6 / 32
- 8. Comparing the Effectiveness of Reasoning Formalisms for Introduction to Uncertainty Partial Models Saadatpanah, Famelis, Gorzny, Robinson, Chechik, Salay What the designer does not know. Introduction Designer Uncertainty Modeling Uncertainty Property Checking Process Verification Technologies Experiments Results Conclusion 6 / 32
- 9. Comparing the Effectiveness of Reasoning Formalisms for Introduction to Uncertainty Partial Models Saadatpanah, Famelis, Gorzny, Robinson, Chechik, Salay What the designer does not know. Introduction Designer Uncertainty Modeling Uncertainty Property Checking Process Verification Technologies Experiments Results Conclusion 6 / 32
- 10. Comparing the Effectiveness of Reasoning Formalisms for Uncertainty: a Set of Possible Partial Models Saadatpanah, Refinements. Famelis, Gorzny, Robinson, Chechik, Salay Introduction Designer Uncertainty Modeling Uncertainty If we remove all uncertainty, we have a concrete refinement. Property Checking Process Verification Technologies Experiments Results Conclusion 7 / 32
- 11. Comparing the Effectiveness of Reasoning Formalisms for Modeling Uncertainty with Partial Partial Models Saadatpanah, Models Famelis, Gorzny, Robinson, Explicating uncertainty in a partial model. Chechik, Salay Introduction Designer Uncertainty Modeling Uncertainty Property Checking Process Verification Technologies In a refinement, a May element is optional. Experiments In a refinement, a Set element can be multiplied to many copies. Results In a refinement, a Variable element can be unified with some other. Conclusion In a refinement, an Open world model can be expanded with some other elements. Partial models: Syntactic annotations of the points of uncertainty. 8 / 32
- 12. Comparing the Effectiveness of Reasoning Formalisms for Modeling Uncertainty with Partial Partial Models Saadatpanah, Models Famelis, Gorzny, Robinson, Explicating uncertainty in a partial model. Chechik, Salay Introduction Designer Uncertainty Modeling Uncertainty Property Checking Process Verification Technologies In a refinement, a May element is optional. Experiments In a refinement, a Set element can be multiplied to many copies. Results In a refinement, a Variable element can be unified with some other. Conclusion In a refinement, an Open world model can be expanded with some other elements. Partial models: Syntactic annotations of the points of uncertainty. 8 / 32
- 13. Comparing the Effectiveness of Reasoning Formalisms for Modeling Uncertainty with Partial Partial Models Saadatpanah, Models Famelis, Gorzny, Robinson, Explicating uncertainty in a partial model. Chechik, Salay Introduction Designer Uncertainty Modeling Uncertainty Property Checking Process Verification Technologies In a refinement, a May element is optional. Experiments In a refinement, a Set element can be multiplied to many copies. Results In a refinement, a Variable element can be unified with some other. Conclusion In a refinement, an Open world model can be expanded with some other elements. Partial models: Syntactic annotations of the points of uncertainty. 8 / 32
- 14. Comparing the Effectiveness of Reasoning Formalisms for Modeling Uncertainty with Partial Partial Models Saadatpanah, Models Famelis, Gorzny, Robinson, Explicating uncertainty in a partial model. Chechik, Salay Introduction Designer Uncertainty Modeling Uncertainty Property Checking Process Verification Technologies In a refinement, a May element is optional. Experiments In a refinement, a Set element can be multiplied to many copies. Results In a refinement, a Variable element can be unified with some other. Conclusion In a refinement, an Open world model can be expanded with some other elements. Partial models: Syntactic annotations of the points of uncertainty. 8 / 32
- 15. Comparing the Effectiveness of Reasoning Formalisms for Modeling Uncertainty with Partial Partial Models Saadatpanah, Models Famelis, Gorzny, Robinson, Explicating uncertainty in a partial model. Chechik, Salay Introduction Designer Uncertainty Modeling Uncertainty Property Checking Process Verification Technologies In a refinement, a May element is optional. Experiments In a refinement, a Set element can be multiplied to many copies. Results In a refinement, a Variable element can be unified with some other. Conclusion In a refinement, an Open world model can be expanded with some other elements. Partial models: Syntactic annotations of the points of uncertainty. 8 / 32
- 16. Comparing the Effectiveness of Reasoning Formalisms for Outline Partial Models Saadatpanah, Famelis, Introduction Gorzny, Robinson, Chechik, Salay What is uncertainty? Introduction Designer How to represent uncertainty with partial models (MAVO). Uncertainty Modeling Uncertainty Process for checking properties Property Checking Process Alternative verification technologies Verification Technologies Experiments Experiments Results Results Conclusion Conclusion 9 / 32
- 17. Comparing the Effectiveness of Reasoning Formalisms for Property Checking in Partial Partial Models Saadatpanah, Models Famelis, Gorzny, Robinson, Chechik, Salay Introduction Designer Uncertainty Modeling Uncertainty Property Checking Process Verification Technologies Experiments Results Conclusion 10 / 32
- 18. Comparing the Effectiveness of Reasoning Formalisms for Property Checking in Partial Partial Models Saadatpanah, Models Famelis, Gorzny, Robinson, Chechik, Salay Introduction Designer Uncertainty Modeling Uncertainty Property Checking Process Verification Technologies Experiments Results Conclusion 10 / 32
- 19. Comparing the Effectiveness of Reasoning Formalisms for Property Checking in Partial Partial Models Saadatpanah, Models Famelis, Gorzny, Robinson, Chechik, Salay Introduction Designer Uncertainty Modeling Uncertainty Property Checking Process Verification Technologies Experiments Results Conclusion 10 / 32
- 20. Comparing the Effectiveness of Reasoning Formalisms for Property Checking in Partial Partial Models Saadatpanah, Models Famelis, Gorzny, Robinson, Chechik, Salay Introduction Designer Uncertainty Modeling Uncertainty Property Checking Process Verification Technologies Experiments Results Conclusion 10 / 32
- 21. Comparing the Effectiveness of Reasoning Formalisms for Verification Technologies I Partial Models Saadatpanah, Famelis, Gorzny, Robinson, • Alloy Chechik, Salay • Lightweight formal methods Introduction • Model finder based on SAT Designer • First order logic specifications expressed in relational logic Uncertainty • Grounded to CNF representation Modeling • Finds counter examples Uncertainty Property Checking Process • Constraint Satisfaction Problem (Minizinc/Flatzinc) Verification Technologies • Assign value to variables to satisfy all constraints Experiments • Constraint modeling language Results • Easily translatable to the form required by other CSP Conclusion solvers 11 / 32
- 22. Comparing the Effectiveness of Reasoning Formalisms for Verification Technologies II Partial Models Saadatpanah, Famelis, Gorzny, Robinson, Chechik, Salay • Satisfiability Modulo Theory (Z3) • Constraint satisfaction search with richer theories Introduction • Theorem prover Designer Uncertainty • Check the satisfiability logical formulas Modeling Uncertainty Property Checking Process • Answer Set Programming (Clingo=Gringo+Clasp) Verification • Answer set solvers Technologies • Conflict-driven nogood learning Experiments • Normal logic programs Results Conclusion 12 / 32
- 23. Comparing the Effectiveness of Reasoning Formalisms for Outline Partial Models Saadatpanah, Famelis, Introduction Gorzny, Robinson, Chechik, Salay What is uncertainty? Introduction Designer How to represent uncertainty with partial models (MAVO). Uncertainty Modeling Uncertainty Process for checking properties Property Checking Process Alternative verification technologies Verification Technologies Experiments Experiments Results Results Conclusion Conclusion 13 / 32
- 24. Comparing the Effectiveness of Reasoning Formalisms for Experimental Setup Partial Models Saadatpanah, Famelis, Gorzny, Robinson, Chechik, Salay Introduction Designer Uncertainty Modeling Uncertainty Property Checking Process Verification Technologies Experiments Results Conclusion 14 / 32
- 25. Comparing the Effectiveness of Reasoning Formalisms for Experimental Setup Partial Models Saadatpanah, Famelis, Gorzny, Robinson, Chechik, Salay Introduction Designer Uncertainty Modeling Uncertainty Property Checking Process Verification Technologies Experiments Results Conclusion 15 / 32
- 26. Comparing the Effectiveness of Reasoning Formalisms for Random Input Generation Partial Models Saadatpanah, Famelis, Gorzny, Robinson, • Meta-model : directed graphs Chechik, Salay • Minimal meta-model Introduction • A few constraints Designer • Most difficult one for solvers Uncertainty Modeling Uncertainty • Randomly decorated with MAVO annotations. Property Checking Process • Parameters are based on real case studies. Verification • Graph density Technologies • Percentage of MAVO annotated elements Experiments • Percentages of M-, S- and V-annotated elements Results Conclusion • 3 Model Size : Small, Medium, Large, X-Large 16 / 32
- 27. Comparing the Effectiveness of Reasoning Formalisms for Experimental Setup Partial Models Saadatpanah, Famelis, Gorzny, Robinson, Chechik, Salay Introduction Designer Uncertainty Modeling Uncertainty Property Checking Process Verification Technologies Experiments Results Conclusion 17 / 32
- 28. Comparing the Effectiveness of Reasoning Formalisms for Relational Encoding I Partial Models Saadatpanah, Famelis, Gorzny, Robinson, Chechik, Salay • Relational Algebra used in DBMSs Introduction Designer • Directly translatable into the different formalisms Uncertainty Modeling Uncertainty • Intermediate representation Property Checking • FOL semantics of MAVO Process • Reasoning formalisms Verification Technologies Experiments • Meaningful comparison Results • Most efficient encoding in each formalism : impossible! Conclusion • Solution : common encoding 18 / 32
- 29. Comparing the Effectiveness of Reasoning Formalisms for Relational Encoding II Partial Models Saadatpanah, Famelis, Gorzny, Robinson, Chechik, Salay (Very) high level overview. Introduction • The metamodel encoded Designer as a schema. Uncertainty • Partial model FOL Modeling Uncertainty semantics encoded as Property Checking constraints over the Process schema. Verification Technologies • Creating a concrete Experiments refinement populates the Results database. Conclusion 19 / 32
- 30. Comparing the Effectiveness of Reasoning Formalisms for Experimental Setup Partial Models Saadatpanah, Famelis, Gorzny, Robinson, Chechik, Salay Introduction Designer Uncertainty Modeling Uncertainty Property Checking Process Verification Technologies Experiments Results Conclusion 20 / 32
- 31. Comparing the Effectiveness of Reasoning Formalisms for Translation To Formalisms Partial Models • Alloy Saadatpanah, Famelis, • Relations : Alloy signatures Gorzny, Robinson, • Instances : Atoms Chechik, Salay • MAVO constraints : quantified predicates over signatures • Bound is required Introduction Designer • CSP Uncertainty • Relations : Finite set of Integers Modeling • Instances : Integers Uncertainty • MAVO constraints : cardinality and intersection of sets Property Checking • Bound is required Process • SMT Verification • Relations : Uninterpreted boolean functions Technologies • Instances : Abstract values Experiments • MAVO constraints : Quantified logic over truth table of Results functions Conclusion • ASP • Program rules for both instances and relations • Bound is required 21 / 32
- 32. Comparing the Effectiveness of Reasoning Formalisms for Experimental Setup Partial Models Saadatpanah, Famelis, Gorzny, Robinson, Chechik, Salay Introduction Designer Uncertainty Modeling Uncertainty Property Checking Process Verification Technologies Experiments Results Conclusion 22 / 32
- 33. Comparing the Effectiveness of Reasoning Formalisms for Properties Checked Partial Models Saadatpanah, Famelis, Gorzny, • Inspired by real metamodel constraints. Robinson, Chechik, Salay • No transitive closure, since it is expensive to check. Introduction Designer Uncertainty P1: There exists a node with a self-loop. Modeling Uncertainty P2: All nodes have outgoing edges. Property Checking P3: All nodes have outgoing or incoming edges. Process Verification P4: For all pairs of nodes n1 , n2 there exists at most one edge e Technologies e such that n1 → n2 Experiments Results P5: For every pair of nodes n1 , n2 , n1 = n2 there exist two edges e1 e2 Conclusion e1 , e2 such that n1 → n2 and n2 → n1 . 23 / 32
- 34. Comparing the Effectiveness of Reasoning Formalisms for Experimental Setup Partial Models Saadatpanah, Famelis, Gorzny, Robinson, Chechik, Salay Introduction Designer Uncertainty Modeling Uncertainty Property Checking Process Verification Technologies Experiments Results Conclusion 24 / 32
- 35. Comparing the Effectiveness of Reasoning Formalisms for Experimental Parameters Partial Models Saadatpanah, • Bound (2, 4, 6) Famelis, Gorzny, • Solvers (except SMT) use bound for grounding expressions Robinson, Chechik, Salay to atoms. • How many times can an ’S’-annotated element be Introduction replicated in a refinement. Designer Uncertainty • Repetitions Modeling Uncertainty • 5 times Property Checking • Cutoff time/memory Process • less than 10 minutes Verification Technologies • less than 5 gigabyte Experiments • otherwise timeout Results • What we measure Conclusion • How long does it take for each solver to return an answer • A score out of 1200 • if timeout : zero! 25 / 32
- 36. Comparing the Effectiveness of Reasoning Formalisms for Findings Partial Models Saadatpanah, Famelis, Gorzny, Robinson, Chechik, Salay Introduction Designer Uncertainty Modeling Uncertainty Property Checking Process Verification Technologies Experiments Results Conclusion 26 / 32
- 37. Comparing the Effectiveness of Reasoning Formalisms for Findings Partial Models Saadatpanah, Famelis, Gorzny, Robinson, Chechik, Salay Introduction Designer Uncertainty Modeling Uncertainty Property Checking Process Verification Technologies Experiments Results Conclusion 26 / 32
- 38. Comparing the Effectiveness of Reasoning Formalisms for Findings Partial Models Saadatpanah, Famelis, Gorzny, Robinson, Chechik, Salay Introduction Designer Uncertainty Modeling Uncertainty Property Checking Process Verification Technologies Experiments Results Conclusion 26 / 32
- 39. Comparing the Effectiveness of Reasoning Formalisms for SMT the champion? Partial Models Saadatpanah, Famelis, Gorzny, Robinson, Chechik, Salay • Unaffected by bounds! Introduction • Works at higher level of abstraction Designer (Theory of uninterpreted functions.) Uncertainty Modeling • Unaffected by expensive grounding phase. Uncertainty Property Checking Process Verification Caveat: Technologies Experiments • SMT can theoretically return “I don’t know”. Results • (However: we didn’t observe that.’) Conclusion 27 / 32
- 40. Comparing the Effectiveness of Reasoning Formalisms for Threats to Validity Partial Models Saadatpanah, Famelis, Gorzny, Randomly generated graphs. Robinson, Chechik, Salay • Tuned the generator with realistic graph properties. Introduction • Values of graph properties from case studies. Designer Uncertainty Modeling Uncertainty Fairness of comparisons. Property • Common encoding to level the field. Checking Process Verification Technologies Choice of specific reasoning engines Experiments Results • When available: winners of competitions. Conclusion • CSP: most convenient input language. 28 / 32
- 41. Comparing the Effectiveness of Reasoning Formalisms for Summary Partial Models Saadatpanah, Famelis, Gorzny, Robinson, Chechik, Salay Research Question What is the most efficient formalism for verifying models Introduction Designer containing uncertainty? Uncertainty Modeling Uncertainty Property Checking Process Verification Technologies Experiments Results Conclusion 29 / 32
- 42. Comparing the Effectiveness of Reasoning Formalisms for Summary Partial Models Saadatpanah, Famelis, Gorzny, Robinson, Chechik, Salay Introduction Designer Uncertainty Modeling Uncertainty Property Checking Process Verification Other results: Technologies • Framework for running experiments (now full tool support). Experiments Results • Random generator for arbitrary type graphs. Conclusion • Relational encoding. • Translations of the RA encoding to different formalisms. 29 / 32
- 43. Comparing the Effectiveness of Reasoning Formalisms for Future Work Partial Models Saadatpanah, Famelis, Gorzny, Robinson, Chechik, Salay • Implement symmetry breaking in the SMT encoding. Introduction Designer Uncertainty Modeling Uncertainty • Experiment with properties that require transitive closure. Property Checking Process Verification Technologies Experiments • Experiment with partial models containing OW. Results Conclusion 30 / 32
- 44. Questions?
- 45. Comparing the Effectiveness of Reasoning Formalisms for Bibliography I Partial Models Saadatpanah, Famelis, Gorzny, Robinson, Chechik, Salay Introduction Famelis, M., Ben-David, S., Chechik, M., and Salay, R. (2011). “Partial Models: A Position Paper”. Designer In Proceedings of MoDeVVa’11, pages 1–6. Uncertainty Famelis, M., Chechik, M., and Salay, R. (2012). Modeling Uncertainty “Partial Models: Towards Modeling and Reasoning with Uncertainty”. In Proceedings of ICSE’12. Property Checking Salay, R., Famelis, M., and Chechik, M. (2012). Process “Language Independent Refinement using Partial Modeling”. In Proceedings of FASE’12. Verification Technologies Experiments Results Conclusion 32 / 32