MBT2013 ASDSpec Spec Explorer

1. Rachid Kherrazi
6-4-2014 Grenoble
USING FORMAL SPECIFICATIONS TO SUPPORT
MODEL BASED TESTING
ASDSPEC: A TOOL COMBINING THE BEST OF TWO TECHNIQUES

2. PEOPLE INVOLVED
• Rachid Kherrazi
– Senior Consultant @ Nspyre
– Domain
• Process and Product Improvement
• RAMS (Reliability Availability Maintenance and Safety)
• Model Based Testing and Model Driven Engineering
• Also : Arjan van Der Meer & Marc Hamilton
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014 2

3. ABOUT NSPYRE
MARKET SEGMENTS
• High Tech
• Traffic & Infra
• Industry
AREA’S OF EXCELLENCE
(Model Based) Systems Engineering /Model Driven
Engineering / Model Based Testing /Industrial Automation /
Simulation / Big Data / Mobile Solutions
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014 3

4. CONTENTS
1) MDSD Introduction
2) MDE with ASD:Suite
3) MBT with MS Spec Explorer
4) ASDSpec
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014 4

5. V-MODEL AND MODEL DRIVEN SYSTEM DEVELOPMENT
Requirements
Analysis
System Functional
Analysis
Design Synthesis
SW Analysis &
Design
SW Implementation & Unit
Test
Module Integration
& Test
(Sub-)System
Integration &Test
Model/RequirementsRepository
Model Driven System
Development
System Acceptance
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014 5

6. V-MODEL AND MODEL DRIVEN SYSTEM DEVELOPMENT
Requirements
Analysis
System Functional
Analysis
Design Synthesis
SW Analysis &
Design
SW Implementation & Unit
Test
Module Integration
& Test
(Sub-)System
Integration &Test
Model/RequirementsRepository
Model Based System
Engineering
System Acceptance
Some of related technology
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014 6

7. V-MODEL AND MODEL DRIVEN SYSTEM DEVELOPMENT
Requirements
Analysis
System Functional
Analysis
Design Synthesis
SW Analysis &
Design
SW Implementation & Unit
Test
Module Integration
& Test
(Sub-)System
Integration &Test
Model/RequirementsRepository
Model Driven
Engineering
System Acceptance
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014
Some of related technology
7

8. V-MODEL AND MODEL DRIVEN SYSTEM DEVELOPMENT
Requirements
Analysis
System Functional
Analysis
Design Synthesis
SW Analysis &
Design
SW Implementation & Unit
Test
Module Integration
& Test
(Sub-)System
Integration &Test
Model/RequirementsRepository
System Acceptance
Model Based Testing
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014
Some of related technology
8

9. IntegrationDesign
Coding
Requirements System test
Test case specification
(design of logical test case)
(reduction of number of test cases by application of test techniques
Test execution
(reporting)
Test case generation
(design of physical test case)
(selection of input values and calculation of expected results)
PAGE 9
V-MODEL FOR TESTING
3 main steps in test process
Module
Subsystem
System
9

10. MBT IS THE AUTOMATION OF TEST CASE
GENERATION
Manual Automatic
Test specification
Test execution
Test generation
Modelbasedtesting
Traditionalmanualtesting
Traditionalautomatictesting
Manual
Scripting
Testing skills + Scripting skills + Modeling skills
Manual
Modeling
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014 10

11. PERCEIVED BENEFITS
• Increased productivity (increased automation)
• Better test script maintenance
• Improved product reliability (new type of bugs, Increased test coverage)
• Agility ( Easily react to new feature changes, Reusability of test semantics,
Early test engagement, Drive quality upstream)
• Increased employee satisfaction (challenging, new horizon, fun)
• Question: How can we improve MBT e.g. increase further the productivity?
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014 11

12. MBT IS THE AUTOMATION OF TEST CASE
GENERATION
Manual Automatic
Test specification
Test execution
Test generation
Modelbasedtesting
Traditionalmanualtesting
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014 12
Testing skills
Manual
Modeling
MBTnext
Semi-automatic
Modeling
Manual
Scripting
+ Scripting skills Modeling skills
Traditionalautomatictesting

13. IDEA: USING BEST OF BOTH WORLDS
• Automatic generation of (partial) Spec Explorer test model from
existing ASD interface model
Benefits
• Less effort for model creation
– reuse of existing work
• Testing of complete system including
– Legacy code
– External components
– Data combination testing
– Interaction testing
• Results: High Quality, Reduced cost
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014
Statistical Dynamic
ASDspec
13

14. USE CASE: CONTAINER TERMINAL
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014
• Multiple components
• Components need to interact to function
• Controller needed to coordinate interaction
14

15. Method
OVERVIEW OF PROJECT STAGES
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014 15
Use case
/SUT
Goal
Technique
Tool
Results
Stage 1
Container Terminal (CT)
Development of control
software for the CT
Model Driven Engineering +
hand written sw
ASD:Suite
Create design Models +
interface models in ASD
Generate code
Productivity (code
generation) c.t. trad. dev
-
Testing complete system,
Interaction, data, external
code
Stage 2
Container Terminal (CT)
Verification of the
developed control software
of the CT (generated +
Hand written)
Model based Testing
Spec Explorer
Create Test Model in Spec
Explorer, generate test
suite
Test Productivity
Modeling skills,
complexity costs
Stage 3
Container Terminal (CT)
Verification of the developed control
software of the CT (generated + Hand
written)
Model based Testing
ASDspec & Spec Explorer
Generate Test Model from existing ASD
interface Model , complete Test Model ,
generate test suite
+ Productivity (code generation, partial test
generation)
Testing complete system, Interaction, data,
external code
(benefits only in case of existing of ASD
Models)

16. CONTENTS
1) MDSD Introduction
2) MDE with ASD:Suite
3) MBT with MS Spec Explorer
4) ASDSpec
Company
Tool
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014 16

17. ASD (ANALYTICAL SOFTWARE DESIGN)
• Model Driven Engineering (code generation)
• Component Based Development
• Models are verified mathematically at design
time (formal methods)
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014 17

18. COMPONENT BASED
DEVELOPMENT ASD-STYLE
Code Generation 2014 18
Client
A
B C
Interface A
Interface B Interface C
Client
component
ASD
component
Used
component
Implemented
service
Used
service
Component:
Decomposition
Specification
Design
Verification
Generation
Interface model
specifies
behaviour
Design model
implements
behaviour

For each component a interface mode and design model are created in ASD

19. ASD: WORKFLOW
• Designer defines behavior in component models
• ASD:suite verifies models using model checking
• ASD:suite generates implementation code
19Code Generation 2014
But some shortcoming
• No support for
Legacy/External code
• No support for indirect
component interaction
• Limited data
interaction
 Testing is needed.

20. RESULTS:
ASD-CONTAINER TERMINAL
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014 20
Higher overall productivity compared to traditional development.
Main remaining problems:
- Testing interaction and complete system.
- Debugging of third party library (some bugs found in manual written
code, legacy code)
Productivity

21. CONTENTS
1) MDSD Introduction
2) MDE with ASD:Suite
3) MBT with MS Spec Explorer
4) ASDSpec
Company
Tool
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014 21

22. MBT WITH SPEC EXPLORER
Explore &
Analyze
Remodel
Generate
C# Model
(or other .Net
Language)
Model Graph
Test Suite
Execute
Nunit
http://www.nunit.com/
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014 22

23. MODELING IN SPEC EXPLORER
[TypeBinding("CraneComponent")]
class Crane
{
public Cranestates Cranestatevar;
internal ICrane_NIModel ICrane_NIimpl;
internal ICrane ICraneimpl;
[Rule(Action = "new CraneComponent(craneNr)",
ModeTransition = "crane->crane0")]
Crane(int craneNr)
{
ICraneimpl = new ICraneCranecomponent(this);
craneList.Add(this);
}
State declaration
Initialization
Used components
Spec Explorer
represents models as
annotated C# code
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014 23

24. MODELING IN SPEC EXPLORER
[Rule(Action = "this.PlaceContainer(height)", ModeTransition = "crane4->crane4")]
public void ICrane_PlaceContainer(GripperHeightEnum height)
{
switch (basecomponent.Cranestatevar) {
case Cranestates.Positioned: {
basecomponent.Cranestatevar
= Cranestates.PlacingContainer;
basecomponent.ICrane_NIimpl.PlaceContainer(height);
return;
}
default:{
Condition.IsTrue(false);
throw new InvalidOperationException();
}
};
}
Every model method
represents an action, its
conditions and its effects.
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014 24

25. MBT-VISUALIZATION
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014
Completed models can
be explored to
discover all possible
behaviour
25

26. MBT- VISUALIZATION
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014
Explored model
contains all possible
action sequences
26

27. TEST GENERATION
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014
Spec Explorer
creates test
sequences to cover
explored behaviour
27

28. GENERATED TEST SCRIPTS
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014
Nunit test scripts are
used to implement
and execute tests
28

29. RESULTS:
SPEC EXPLORER-CONTAINER TERMINAL
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014 29
Significantly less effort than traditional automated testing
• Model Based Testing + Software Analysis
• Support of data combination testing
• Support of Model composition, incremental
However some remaining problems:
• Modeling still comparatively expensive
• Modeling effort, complexity and skills (experienced tester needed)

30. HOW TO COMBINE BOTH BENEFITS
ASD generates verified
software components
Spec Explorer generates
automated software tests
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014
Complications
• Modeling needed
• MBT skills needed
Complications
• Legacy code
• External code
• Component
Interaction
• Limited support of
data Client
A
B C
Test
script
A
B C
 
Test
model
30

31. CONTENTS
1) MDSD Introduction
2) MDE with ASD:Suite
3) MBT with MS Spec Explorer
4) ASDSpec
Company
Tool
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014 31

32. OUR SOLUTION: ASDSPEC
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014
ASDSpec generates Test Model (push button) from existing ASD
interface models
Tester need to refine manually the generated model (add behavior,
data, slicing,…)
32

33. COMBINING THE BEST OF BOTH WORLDS
Automatic generation of partial Spec Explorer test model from existing ASD model
(push button)
Benefits
• Effortless model creation
– reuse of existing work
• Testing of complete system including
– Legacy code
– External components
– Data combination testing
– Interaction testing
Results:
– High Quality, Reduced cost
– Adds dynamic testing to ASD
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014
Statistical Dynamic
ASDspec
33

34. ASDSPEC: WORKFLOW
ASDspec
ASD
Interface
Model
Spec
Explorer
Model
Cord
Script
Spec
Explorer
Test
Cases
Reuse existing ASD interface models to generate
Spec Explorer MBT models automatically
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014 34

35. ASDSPEC: DATA AND MODEL COMPOSITION
ASDspec
ASD
Interface
Model
Spec
Explorer
Model
Cord
Script
Spec
Explorer
Test
Cases
Add data
interaction
Add model
composition
Generated code can be extended with any Spec
Explorer feature
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014 35

36. GENERAL MDSD ARCHITECTURE
Model
Reverse
Engineering
Model
Repository
Model
Code,
Doc, etc.
Model
Editor(s)
Model
Checker
Model Transformation
M2M M2T
Metamodel(s)
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014 37

37. ASDSPEC ARCHITECTURE
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014
ASDSpec
makes use of
existing MDSD
architecture
Model
Reverse
Engineering
Model
Repository
Spec
Explorer
Model
SpecExplorer
Cord
Model
Editor(s)
Model
Checker
Model Transformation
M2M M2T
Metamodel(s)
ASD
Interface
models
38

38. ASDSPEC: TECHNOLOGIES
QVTO
ASD
XML
SE
model
Cord
script
Acceleo
Acceleo
ASD
EMF
SE
EMF
Cord
EMF
QVTO
QVTO
ASDSpec
Model transformations bridge gap between ASD
and Spec Explorer
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014 39

39. RESULTS:
ASDSPEC- CONTAINER TERMINAL
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014 40
Spec Explorer ASD + ASDSpec
Approach Generate test suite Generate test model
Techniques MBT MBT
Effort/complexity Medium Low
Test cases 89 93
Perceived effectiveness Medium High
Bugs Some bugs in hand written code
/HAL
All known bus found
• Low effort model creation
- Easy reuse of existing work
• Testing of complete system
- Legacy code, data interaction
• High Quality, Cost Reduction

40. TRADITIONAL SW DEVELOPMENT
TRADITIONAL MANUAL TESTING
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014
Design spec
Hand written code
Detail design
External code
Interface spec
Legacy code
Integration testing
System testing
Unit testing
Manual
Test suite
41
Summary

41. TRADITIONAL SW DEVELOPMENT
TRADITIONAL AUTOMATED TESTING
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014
Design spec
Hand written code
Detail design
External code
Interface spec
Legacy code
Integration testing
System testing
Unit testing
Testexecution
Tool
Manual
scripted
Test suite
42
Summary

42. ASD BASED SW DEVELOPMENT
TRADITIONAL AUTOMATED TESTING
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014
Design spec
Design model
Generated codeHand written code
Interface model
External code
Interface spec
Legacy code
Integration testing
System testing
Unit testing
Manual
scripted
Test suite
Testexecution
Tool
43
Summary

43. ASD BASED SW DEVELOPMENT
MODEL BASED TESTING
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014
Design spec
Design model
Generated code
Manual Test
model
Interface model
External code
Interface spec
Config,
data
Legacy code
Integration testing
System testing
Unit testing
Generated
Test suite
Manual
scripted
Test suite
Testexecution
Tool
MBT
tool
Hand written code
44
Summary

44. ASD BASED SW DEVELOPMENT
MODEL BASED TESTING+ASDSPEC
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014
Design spec
Design model
Generated code
Generated
Test model
Interface model
External code
Interface spec
Config,
data
Legacy code
Integration testing
System testing
Unit testing
Generated
Test suite
Manual
scripted
Test suite
Testexecution
Tool
MBT
tool
Hand written code
45
Summary

45. TOOL STATUS
• Tool currently in prototype phase
• as an Eclipse plugin
• For now available on request
• Future steps
– Support data handling/configuration aspects in
ASDSpec, instead of relying on manual additions in
Spec Explorer,
– Supporting other MBT tools.
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014 46

46. CONCLUSIONS
• MDE and MBT technologies have matured a lot in the
latest years
• It is a matter of time….. Evolution…..
Nineth Workshop on Model-Based Testing April 6, 2014 ETAPS 2014 47

47. Using Formal Specifications to
Support Model Based Testing
ASDSpec: A Tool Combining the
Best of Two Techniques
ASD
Static Dynamic
ASDspec
Static Dynamic
ASDspec