MODELS 2018 - invited talk at Industry Day

1. .lusoftware verification & validation
VVS
Automated and Scalable Solutions
for Software Testing:
The Essential Role of Model-Driven
Engineering
Lionel C. Briand
MODELS 2018, Industry Day

2. Objectives
2
• Not a scientific presentation, but an experience report
• Reflecting on more than 20 years of collaborative research
with industry at the intersection of test automation and
model-driven engineering
• Domains: Automotive, satellite, aerospace, energy, finance,
government …
• Main message: Modeling is an essential component of many
test automation solutions in numerous industrial contexts

3. Software Testing
3
SW Representation
(e.g., specifications)
SW Code
Derive Test cases
Execute Test cases
Compare
Expected
Results or properties
Get Test Results
Test Oracle
[Test Result==Oracle][Test Result!=Oracle]
• Automation!
• Context
• Trade-off
• White-box vs. black-box
• Sampling strategy
• Constraint solving
• Expected results
• General properties
• Observability
• Trace analysis
• Repeatability
• Controllability
• Early testing

4. Research Project Examples

5. Cyber-Physical Systems (CPSs)
5

6. Testing Cyber-physical Systems:
Challenges
6
complex
systems
interacting
with complex
environment
hybrid
behavior
intertwining of
software and
hardware
complex
requirements
uncertainty

7. CPS Development Process
7

8. Simulink vs. SysML
• Two commonly used, industry-strength and complementary
modeling languages for CPS
• Simulink
• Functional and algorithmic computations (MiL-level)
• E.g., controllers, plant and environment models
• SysML
• Software architecture modeling (SiL-level) plus aspects of hardware
• E.g., State machines integrating different controllers, real-time behaviors
8

9. Model Testing

10. Context
10
Attitude Determination and Control System
(ADCS)
Earth

11. CPS Development Process
11

12. Model Testing
12
• Testing the implemented system on the deployment platform
is necessarily limited
• Shift the bulk of testing from implemented systems
to models of such systems and their environments:
• Enables early testing of CPSs (in the design space)
• Is Scalable
• Can execute a large number of test scenarios (simulations)
• Efficiently explore the design space
• Guides the testing process at later stages by identifying high-risk
test scenarios

13. Requirements
• Efficient model execution
• To execute thousands of test cases within practical time
• No user intervention
• Co-Simulation of heterogeneous models
• Software (SysML) and function (Simulink) models
• Controllability
• To emulate, as precisely as possible, the expected runtime behavior of the
future deployed system and its environment
• Observability
• To enable automated failure detection based on temporal and timing
properties
13

14. Our Approach
• A modeling methodology
• To specify testable models of a CPS (relying on SysML)
• To integrate software models with function (Simulink) models capturing
hardware and environment
• A co-simulation framework
• To execute software and function models in a synchronized way
• To enable the generation of adequate execution traces (oracles, guiding test
generation)
• Entirely generated from models
14

15. CPS Simulation & Test
15
Test
inputs
CPS SysML
Models
CPS Function
Models
Integrates
Modeling
Co-Simulation
Matlab
Runtime
C Code
Code
Generation Executes
Execution
Traces
Calls
Data
Test input:
- initial state of satellite (speed,
attitude, orbit, date),
- a sequence of time-stamped
external events (tele-commands)

16. 16
Requirements-Driven Testing

17. Context
International Electronics
& Engineering (IEE)
IEE develops real-time embedded systems:
• Automotive safety sensing systems, e.g., seat
occupancy status
• Automotive comfort & convenience systems,
17

18. Traceability
18
• In many domains, various types of traceability
are required.
• For example, in automotive (ISO 26262),
traceability between requirements and system
tests: requirements-driven testing.
• Many requirements, many tests, therefore many
traces …
• Automation is required.

19. Objectives
• Support generation test cases from requirements
• Capture and create traceability information between test
cases and requirements
• Requirements are captured through use cases
• Use cases are used to communicate with customers and the
system test team
• Complete and precise behavioral models are not an option:
too difficult and expensive (no model-based testing)
19

20. Strategy
• Analyzable use case specifications
• Automatically extract test model from the use case
specifications using Natural Language Processing
• Minimize modeling, domain modeling only
• No behavioral modeling
20

21. Errors.size() == 0
Status != null
t > 0 && t < 50
Constraints
Domain Model Test Cases
Test Scenarios
21
THE ACTOR SEND
THE SYSTEM VALI
THE SYSTEM DIS
THE ACTOR SEND
THE ACTOR SEND
THE SYSTEM VALI
THE SYSTEM DIS
THE ACTOR SEND
THE ACTOR SEND
THE SYSTEM VALI
THE SYSTEM DIS
THE ACTOR SEND
Use Cases
UMTG
Based on Natural Language
Processing
Traceability

22. • RUCM is based on a (1) template, (2) restriction rules,
and (3) specific keywords constraining the use of
natural language in use case specifications
• RUCM reduces ambiguity and facilitates automated
analysis of use cases
• Conformance is supported by a tool based on NLP
• Trade-off between analyzability and natural language
Restricted Use Case Modeling:
RUCM
22

23. RUCM
Use Case Name: Identify Occupancy Status
Actors: AirbagControlUnit
Precondition: The system has been initialized
. . .
Basic Flow
1. The seat SENDS occupancy status TO the system.
2. INCLUDE USE CASE Classify occupancy status.
3. The system VALIDATES THAT the occupant class for airbag control is valid.
4. The system SENDS the occupant class for airbag control TO AirbagControlUnit.
Specific Alternative Flow
RFS 3
1. IF the occupant class for airbag control is not valid THEN
Postcondition: The occupant class for airbag control has been sent.
Postcondition: The previous occupant class for airbag control has been sent.
[Yue et al. TOSEM’13]

24. 24
Basic Flow
1. The seat SENDS occupancy status TO the system.
2. INCLUDE USE CASE Classify occupancy status.
3. The system VALIDATES THAT
the occupant class for airbag control is valid and
the occupant class for seat belt reminder is valid.
4. The system SENDS the occupant class for airbag control TO
AirbagControlUnit.
5. The system SENDS the occupant class for seat belt reminder TO
SeatBeltControlUnit.
6. The System Waits for next execution cycle.
Postcondition: The occupant class for airbag control and the
occupant class for seat belt reminder have been sent.
INPUT STEP
INCLUDE STEP
CONDITIONAL STEP
OUTPUT STEP
OUTPUT STEP
INTERNAL STEP
POSTCONDITION
DOMAIN ENTITY
CONSTRAINT
CONSTRAINT
DOMAIN ENTITY
DOMAIN ENTITY

25. Case Study
• BodySense, embedded system for detecting occupancy
status in a car
• Evaluation:
• Cost of additional modelling (Constraints)
• Effectiveness in terms of covered scenarios
compared to current practice at IEE
• Keep in mind changes and repeated testing
25

26. Costs of Additional Modeling
26
Use Case Steps Use Case
Flows
OCL
Constraints
UC1 50 8 9
UC2 44 13 7
UC3 35 8 8
UC4 59 11 12
UC5 30 8 5
UC6 25 6 12
5 to 10 minutes to write each constraints
=> A maximum of 10 hours in total

27. Effectiveness: scenarios covered
27
0
5
10
15
20
25
30
35
40
UC1 UC2 UC3 UC4 UC5 UC6
Scenarios Covered By Engineer Scenarios Covered By UMTG
100%
100%
100%
100% 100%
100%
81%
77%
100%
86%
50% 67%
It is hard for engineers to capture
all the possible scenarios
involving error conditions.

28. Generating OCL Constraints
28
• Constraints may be a challenge in practice
• NLP: Semantic Role Labeling
• Determine the role of words in a sentence
(e.g., affected actor)
• Match words with corresponding concepts in
the domain model
• Generate an OCL formula based on patterns

29. Semantic Role Labeling (SRL)
“no error has been detected”
Error.allInstances()->forAll( i | i.isDetected = false)
A1
“The system detects temperature errors
TemperatureError.allInstances()->forAll( i | i.isDetected = true)
A1
A1A0
A1: actor that is affected by the
activity described in a sentence
A0: actor that performs
an activity
A1 verb
verb
verb
verb

30. Empirical Evaluation
• Case study: BodySense, embedded system for detecting
occupancy status in a car
• Evaluation:
• Automatically generate the OCL constraints required
to automatically derive executable test cases
• Automatically generate executable test cases
30

31. OCL generation: Precision and Recall
• 88 OCL constraints to be generated
• OCLGen generates 69 constraints
• 66 correct, only 3 incorrect
• Very high precision
• High Recall
31
# Correctly generated constraints
# Generated constraints
=precision = = 0.97
66
69
# Correctly generated constraints
# Constraints to be generated
=recall = = 0.7566
88

32. Results: Limiting Factors
• Imprecise specifications
• “The system VALIDATES THAT the temperature is valid“
• Inconsistent terminology
• “The system VALIDATES THAT the occupancy status is valid“
BodySense.allInstances()->forAll( i | i.temperature < 200 )
BodySense.allInstances()->forAll( i | i.occupancyStatus <> Empty )
32

33. Schedulability Analysis and
Stress Testing
33

34. Case Study
34
Drivers
(Software-Hardware Interface)
Control Modules
Alarm Devices
(Hardware)
Multicore Architecture
Real-Time Operating System
System monitors gas leaks and fire in
oil extraction platforms

35. Problem and Context
• Schedulability analysis encompasses techniques that try to
predict whether (critical) tasks are schedulable, i.e., meet
their deadlines
• Stress testing runs carefully selected test cases that have
a high probability of leading to deadline misses
• Stress testing is complementary to schedulability analysis
• Testing is typically expensive, e.g., hardware in the loop
• Finding stress test cases is difficult
35

36. Finding Stress Test Cases is Hard
36
0
1
2
3
4
5
6
7
8
9
j0, j1 , j2 arrive at at0 , at1 , at2 and must
finish before dl0 , dl1 , dl2
J1 can miss its deadline dl1 depending on
when at2 occurs!
0
1
2
3
4
5
6
7
8
9
j0 j1 j2 j0 j1 j2
at0
dl0
dl1
at1 dl2
at2
T
T
at0
dl0 dl1
at1
at2
dl2

37. Challenges and Solutions
• Ranges for arrival times of all tasks form a very large input
space
• Task interdependencies and properties constrain what
parts of the space are feasible
• Task architecture captured by UML MARTE profile.
• Solution: We re-expressed the problem as a constraint
optimization problem and used a combination of constraint
programming (IBM CPLEX) and meta-heuristic search (GA)
37

38. Constraint Optimization
38
Constraint Optimization Problem
Static Properties of Tasks
(Constants)
Dynamic Properties of Tasks
(Variables)
Performance Requirement
(Objective Function)
OS Scheduler Behaviour
(Constraints)

39. Solution Overview
39
UML Modeling
(e.g., MARTE)
Constraint Optimization
Optimization Problem
(Find arrival times that maximize the
chance of deadline misses)
System Platform
Solutions
(Task arrival times likely to lead to
deadline misses)
Deadline Misses Analysis
System Design
Design Model (Time and
Concurrency Information)
INPUT
OUTPUT
Stress Test Cases
Constraint
Prog. and Genetic
Algorithms

40. Combining CP and GA
40
A:12 S. Di Alesio et al.
Fig. 3: Overview of GA+CP: the solutions x , y and z in the initial population of GA evolve into

41. Summary
• We provided a solution for generating stress test cases by combining
meta-heuristic search and constraint programming
• Meta-heuristic search (GA) identifies high risk regions in the
input space
• Constraint programming (CP) finds provably worst-case
schedules within these (limited) regions
• Achieve (nearly) GA efficiency and CP effectiveness
• Our approach can be used both for stress testing and
schedulability analysis (assumption free)
41

42. Beyond CP Systems
• Models play a role in test automation in other domains than
cyber-physical systems
42
Bridging the gap between requirements models (business
processes) and Behavior-Driven Development

43. Conclusions
43

44. The Role of Models
• No effective and scalable test automation is possible, in many
contexts, without models: Guiding test generation, generating oracles
• Requirements (e.g., use case specifications)
• Architecture (e.g., task properties and dependencies)
• Behavior of system and environment (e.g., state and timing
properties)
• Business processes
• Diverse domains: Cyber-physical systems, finance, e-government …
44

45. Models are not Enough
• Models are not enough for test automation
• Optimization and metaheuristic search
• Natural language processing
• Simulation
• Constraint solving
• …
• Test automation solutions are necessarily multidisciplinary
45

46. Modeling is a Trade-off
• Modeling is costly and a challenge to many
• Effective tool support, e.g., QA, is important
• Upfront investment in tooling and training
• Benefits can mostly be obtained from exploiting models
for automation of costly and error-prone tasks, e.g.,
testing in the large
• Scope and depth of modeling is a trade-off
• This trade-off is driven by contextual factors
46

47. Open Problems
• Testing cannot be exhaustive: Strategies to explore the test input
space
• Metaheuristic search, efficient and effective constraint solving …
• Early testing: How to find significant problems in early artifacts,
e.g., requirements, architecture
• Executable models, co-simulation …
• Oracles: Striking the right balance between cost and
effectiveness
• Scalable trace analysis, DSL for expressing trace properties,
probabilistic oracles for non-deterministic systems …
47

48. Industry-Academia
Collaborations
• Crucial to address relevant and open problems, develop
scalable and applicable solutions.
48
Adapted from [Gorschek et al. 2006]
Problem
Formulation
Problem
Identification
State of the
Art Review
Candidate
Solution(s)
Initial
Validation
Training
Realistic
Validation
Industry Partners
Research Groups
1
2
3
4
5
7
Solution
Release
8
6

49. 49

50. Acknowledgments
• Shiva Nejati
• Fabrizio Pastore
• Chunhui Wang
• Stefano Di Alesio
• …
50

51. Selected References
• L. Briand et al. “Testing the untestable: Model testing of complex software-intensive systems”,
IEEE/ACM ICSE 2016, V2025
• C. González et al., “Model Testing of Cyber-Physical Systems”, ACM/IEEE MODELS 2018
• C. Wang et al., “Automatic Generation of System Test Cases from Use Case Specifications”, ACM
ISSTA 2017
• C. Wang et al., “Automated Generation of Constraints from Use Case Specifications to Support
System Testing”, IEEE ICST 2018
• S. Di Alesio et al. “Combining genetic algorithms and constraint programming to support stress
testing of task deadlines”, ACM Transactions on Software Engineering and Methodology (TOSEM),
2015
• More on: https://wwwen.uni.lu/snt/people/lionel_briand?page=Publications
51

52. .lusoftware verification & validation
VVS
Automated and Scalable Solutions
for Software Testing:
The Essential Role of Model-Driven
Engineering
Lionel C. Briand
MODELS 2018