This document summarizes the key discussion points and research questions identified by a working group on model-based engineering and verification and validation. The group identified 8 thematic categories for focusing research: 1) bridging the gap between models and formal verification techniques, 2) refining existing V&V methodologies for model-based development, 3) relating design-time models to runtime behavior, 4) determining appropriate properties to verify, 5) verifying model transformations, 6) handling informal, formal and incomplete modeling, 7) benchmarking and comparing V&V tools, and 8) leveraging domain-specific languages. For each category, the document outlines the current status and poses open research questions to guide future work at the intersection of

1. Research Questions for Validation
and Verification in the Context of
Model-Based Engineering
Catherine Dubois
Michalis Famelis
Martin Gogolla
Leonel Nobrega
Ileana Ober
Martina Seidl
Markus Voelter
ENSIIE, Evry, France
University of Toronto,
University of Bremen,
University of Madeira, Funchal,
University of Toulouse,
Johannes Kepler University Linz, Austria
Voelter Ingenieurburo, Heidenheim, Germany
October 1st , 2013
MoDeVVa 2013, Miami, USA

2. Introduction
Abstraction techniques are one of the promising paths for
the future advances in the field of verification.
– Clarke, Emerson, Sifakis: Turing Lecture 2008
V&V crucial for MBE
– Uncover hidden properties and errors
– Verify transformations
– Ensure quality, etc.
2

3. About
• This paper is the result of the working group on V & V at
Dagstuhl-Seminar 13182 held in May 2013.
• Dagstuhl: scientific retreat
in western Germany
• Seminar topic:
Meta-Modeling Model-
Based Engineering Tools
• Three break-outs: Informal
Modeling, Compositionality,
Modeling and V&V
3

4. 4
Our goal:
Identify main areas in the synergy between MBE and
V&V where we need to focus research. What is the
status, what are the research questions?
• Two-day
collaborative
brainstorming
workshop.
• Culminated in a
plenary
presentation
…and this report

5. Disclaimer
• No claim of completeness or exhaustiveness
• Represents the informed opinions of the authors
• May have missed existing answers to some
questions
• Is this the right level of granularity / level of
detail?
5

6. Thematic Categories
1. Gap between Models and V&V Formalisms
2. Need to Refine Existing Methodologies
3. Design-time vs. Runtime
4. Properties
5. Model Transformations
6. Informal vs. Formal vs. Incomplete Modeling
7. Comparison and Benchmarking
8. Domain-Specific Languages
6

7. Models and V&V Formalisms (context)
Model Property
Model' Property'
Designer Level
(can be Domain-
Specific)
Verification
formalism level
Verification engine
Verification
feed-back
7
Transformations
and traceability
mechanisms
But behavioral
semantics often leads
to non-bijective
correspondences
between
design time and
runtime artifacts.
V&V tool configuration

8. Models and V&V Formalisms
(questions)
• How to express properties at the level of models in a way
understandable to clients?
• How to formulate models and properties in a single language
transparent to clients?
• How to report the V&V results and diagnostics in an appropriate
form to clients?
• How to bridge the gap between formally expressed and verified
properties on one side and client attention on the other side?
• Can modeling language extensions help in making explicit the
“needs” of V&V machines?

9. Refining Existing Methodologies
(context)
• Integrating V&V in development can support early V&V
similar to how debugging is offered by IDEs.
• Generic methodologies identify points where V&V can be
used.
• A Model-based development methodology would need to
allow variations based on application domain, nature of
project, etc.
• Goal: better identify which V&V activities are meaningful at
the various phases of design; take full advantage of V&V
engines.
9

10. Refining Existing Methodologies
(questions)
• How do we integrate V&V in the overall development
and modeling process?
– On the technical level of tool exchange?
– On the methodological level of using the right technique at
the right time for the right task?
• When are techniques like animation, execution,
symbolic evaluation, testing, simulation, proving or test
case generation used most efficiently during
development?
– For which model and model transformation properties can
they be employed?
10

11. Design-time vs. Runtime (context)
• Models are specified at design time
• During execution these models are instantiated
• The dynamic nature of structure during execution
makes it difficult to understand and represent
runtime information
• Existing modeling environments offer limited
support for precisely specifying instantiation and
snapshots
11

12. Design-time vs. Runtime (questions)
 During the V&V phase, how do we obtain an initial model
instantiation?
 How do we obtain large and meaningful instantiations?
 How do we connect design time and runtime artifacts?
 How do we deal with the issue of scalability in the context
of V&V ?
 How do we handle time and space concerns w.r.t. design
time and runtime artifacts?
 How do we automatically or semi-automatically manage
the V&V machine configuration?
12

13. Properties (context)
• Model and model transformation properties relevant in V&V:
– consistency, reachability, dependence, minimality, conformance, safety,
liveness, deadlock freeness, termination, confluence, correctness
• Confusion caused by:
– Different kinds according to the nature of the model (static/ dynamic),
its level of abstraction, etc.
– Many tools and techniques (potentially complementary)
Main challenge:
What kind of property to verify on which model at what stage with what
kind of technique?
13

14. Properties (questions)
• What are the benefits and trade-offs between expressing
properties on more abstract modeling levels in contrast to
expressing them on more concrete levels?
• How do we find the right techniques for uncovering static and
dynamic model properties?
• Which techniques are appropriate for uncovering static modeling
language inherent properties, which for static model-specific
properties?
• Which techniques are appropriate for uncovering dynamic generic
properties, which for dynamic model-specific properties?
14

15. Model Transformations (context)
• Core component of MBE
• Many applications:
– Maintaining inter-model consistency
– Semantics definition of (domain-specific) modelling languages
– …
• Challenge: Verification of model transformations
– Proving correctness, termination, confluence
– What are the differences to “normal” code?
– Is the higher abstraction level beneficial to V&V?
15

16. Model Transformations (questions)
• What verification techniques are meaningful for verifying
model transformations?
• How do we analyse properties like confluence and
termination?
• How do we analyse correctness of model transformations
w.r.t. a transformation contract?
• How do we infer a transformation contract from a model
transformation?
16

17. Informal vs. Formal vs. Incomplete
Modeling (context)
• During V&V: switch on or off particular model elements (in class diagrams, e.g.,
multiplicities); configure constraints by negating, deactivating or activating them
(in class diagrams, e.g., class invariants)
• Different types of granularity (a) all model elements may be relaxed (b) only a
manual model element selection can be considered for relaxation (c) a semi-
automatic element selection for relaxation may be offered
• Ultimate vision: sliders on the user interface to gradually go from a strict, formal
model through various intermediate levels to a totally relaxed and informal model;
fewer formal model parts activated means more informality in the model
• Minimal formal frame for test case construction must be preserved: e.g., for class
diagrams of central classes and associations and for state charts of central states
and transitions
• For a completely informal model no formal scenario (no test cases) can be
formulated
17

18. Informal vs. Formal vs. Incomplete
Modeling (questions)
• How do we leverage informal assumptions found in sketches for
exploratory V&V?
• Are informal sketches close enough to V&V at all?
• What are appropriate relaxation mechanisms for different degrees
of formality?
• How do we handle incomplete or partial models w.r.t. V&V?
• How do we deactivate and activate model units?
• How do we handle the exploration of model properties and
alternatives?
18

19. Comparison and Benchmarking
(context)
• Benchmarking can boost research:
– Fair comparison of tools
– Impartial benchmark selection which covers the spectrum
of interesting test cases
– Clear documentation of outcomes; reproducibility
– Publicity makes it easy to identify progress, problems
• However, in MBE:
– No common standards
– No community platform for benchmarks
– Not clear what metrics are relevant for measuring
“improvement”
19

20. Comparison and Benchmarking
(questions)
• How to compare existing V&V tools w.r.t.
functionality, coverage, scalability,
expressiveness, executing system (i.e., for models
at runtime)?
• Which criteria are appropriate for comparison?
• Can we globally compare fairly at all?
– Broad and diverse spectrum of V&V machines:
B, Coq, HOL/Isabelle, SAT, SMT, CSP solvers, Relational logic,
enumerative techniques
20

21. Domain-Specific Languages
(Context)
• Most verification tools have hard to use input languages,
alien to normal developers.
– Hence, verification tools are often not used.
• MBE approaches become more and more mainstream.
• Models can simplify analysis and verification, because of
the higher degree of domain semantics they express.
• Potential to exploit the two approaches synergistically
– From the high-level models, we can automate the
generation of the input to the verification tools.
21

22. Domain-Specific Languages
(Questions)
• How to define DSLs close to domain concepts but still allow
generation of meaningful input for V&V tools?
• V&V tools need the specification of properties.
– How to express them at the domain level in a user-friendly way?
– Can the property specifications be integrated with the same
DSL and/or model used for describing the to-be-verified system
without creating self-fulfilling prophecies?
• How to bring V&V feedback back to the domain level and
express it in terms of the DSL-level input?
• Can incremental languages extensions help with making
programs expressed in general-purpose languages more
checkable? For example, the semantics of a specific
extension construct may enable the generation of very rich
inputs to the verification tool.
22

23. Thematic Categories
1. Gap between Models and
V&V Formalisms
2. Need to Refine Existing
Methodologies
3. Design-time vs. Runtime
4. Properties
5. Model Transformations
6. Informal vs. Formal vs. Incomplete
Modeling
7. Comparison and Benchmarking
8. Domain-Specific Languages
23

24. Conclusion
• Report-back from working group on MBE and V&V at
Dagstuhl in May 2013.
• Main areas in the synergy of MBE and V&V.
• For each: described status and identified research
questions.
• Our hope:
Spark discussions and debate, help focus
research in MBE and V&V.
24

25. Questions?
25