Best Paper Presentation at Profes 2015

1. Rudolf Ramler1 and Michael Felderer2
1 Software Competence Center Hagenberg, Hagenberg, Austria
2 Institute of Computer Science, University of Innsbruck, Austria
December 04, 2015
PROFES 2015, Bolzano
A Process for Risk-Based Test Strategy
Development and Its Industrial Evaluation

2. Motivation: ISO/IEC/IEEE 29119
Slide 2Risk‐Based Test Strategy Development and Industrial Evaluation
http://softwaretestingstandard.org/

3. Risk Concept in Software Testing
Slide 3Risk‐Based Test Strategy Development and Industrial Evaluation
Probability (P) Impact (I)
Risk Value
Risk Item
Probability Criteria Impact Criteria
Risk LevelTest Planning
Test Design
Test Execution
Test Evaluation
Risk‐Based Test Strategy

4. Slide 4Risk‐Based Test Strategy Development and Industrial Evaluation
Defect
Classification
Identification of
Impact Factors
1. Definition of
Risk Items
2. Probability
Estimation
3. Impact
Estimation
4. Computation of
Risk Values
6. Definition of
Test Strategy
7. Refinement of
Test Strategy
Defect
Management
Requirements
Management
5. Determination
of Risk Levels
Quality
Management
Identification of QA/
Testing Techniques Proces Step
Preparation Step
Related Process
Legend
Process for Risk‐Based
Test Strategy Development

5. Definition of Risk Items (1), Probability (2) and Impact Estimation (3)
Slide 5Risk‐Based Test Strategy Development and Industrial Evaluation
Probability
Impact
Derive impact from
weighted impact
factors in fishbone
diagram!
Derive probability
from defect
distribution!

6. Computation of Risk Values (4)
Slide 6Risk‐Based Test Strategy Development and Industrial Evaluation
Component Rank
A 5 5
21 2 1
105 1 0 5
6
B 29 2 9
50 5 0
1.450 1 4 5 0
1
C 25 2 5
10 1 0
250 2 5 0
4
D 18 1 8
46 4 6
828 8 2 8
2
E 14 1 4
8 8
112 1 1 2
5
F 13 1 3
50 5 0
650 6 5 0
3
Estimated Probability Estimated Impact Computed Risk (= P * I)

7. Determination of Risk Levels (5)
Slide 7Risk‐Based Test Strategy Development and Industrial Evaluation

8. Definition and Refinement of Test Strategy (6), (7)
Slide 8Risk‐Based Test Strategy Development and Industrial Evaluation
Components A B C D E F
I II III IV Risk level I IV II III I III
Unit testing (100% branch coverage) X X
Code reviews X X X X X X X
Manual testing of use cases (base flow) X X
Manual testing of use cases (base + alternative flows) X X X X X
Exploratory testing X X X X X
Automated smoke/regression tests X X X X X
Beta test phase at selected customers X X X X X X X
Risk level
Testing techniques

9. • Case study with five software development companies
• Research questions
(RQ1) Is the outcome of the process useful?
(RQ2) Is conducting the process easy?
(RQ3) Is the outcome of the process representative?
• Iterative data collection and analysis in joint research transfer
project, in each iteration
(1) Joint workshop to discuss and prepare steps of the process
(2) Application of steps to create practical results for company
(3) Questionnaire to collect feedback from each company
(4) Semi‐structured interviews with key members of companies
Evaluation
Slide 9Risk‐Based Test Strategy Development and Industrial Evaluation

10. Case A Case B Case C Case D Case E
Domain ERP software Access
systems
Document
management
Training and
recruitment
Payment
systems
Core
business
Software
product and
service
Hardware incl.
embedded
software
Software
product and
custom
development
Service
including
software
platform and
operation
Solution
development
incl. software
and hardware
Employees 15 40 10 40 15
Software
Releases
2 to 4 releases
per year;
service
releases on
demand
Adjusted to
hardware
product cycles
(years)
New releases
every one and
four weeks
On demand Delivery as
custom
projects
Overview of Case Companies
Slide 10Risk‐Based Test Strategy Development and Industrial Evaluation

11. 1.
2.
3.
4.5.
6.
7.
A B C D E Avg
high
low
Perceived Usefulness (RQ1)
Slide 11Risk‐Based Test Strategy Development and Industrial Evaluation
Usefulness of ... A B C D E Avg sdev
1. Definition of Risk Items 2 1 1 1 2 1.4 0.49
2. Probability Estimation 2 2 1 1 1 1.4 0.49
3. Impact Estimation 1 2 1 1 1 1.2 0.40
4. Computation of Risk Values 1 5 1 2 1 2.0 1.55
5. Determination of Risk Levels 1 1 1 2 1 1.2 0.40
6. Definition of Test Strategy 1 1 1 3 1 1.4 0.80
7. Refinement of Test Strategy 1 2 3 2 2 2.0 0.63

12. 1.
2.
3.
4.5.
6.
7.
A B C D E Avg
high
low
Perceived Ease of Use (RQ2)
Slide 12Risk‐Based Test Strategy Development and Industrial Evaluation
Ease of Use of ... A B C D E Avg sdev
1. Definition of Risk Items 1 2 3 5 2 2.6 1.36
2. Probability Estimation 2 1 3 1 1 1.6 0.80
3. Impact Estimation 1 5 3 4 2 3.0 1.41
4. Computation of Risk Values 1 1 1 3 1 1.4 0.80
5. Determination of Risk Levels 1 4 1 1 1 1.6 1.20
6. Definition of Test Strategy 2 4 2 2 4 2.8 0.98
7. Refinement of Test Strategy 1 1 3 2 4 2.2 1.17

13. 1.
2.
3.
4.5.
6.
7.
A B C D E Avg
high
low
Perceived Representativeness (RQ3)
Slide 13Risk‐Based Test Strategy Development and Industrial Evaluation
Representativeness of ... A B C D E Avg sdev
1. Definition of Risk Items 1 4 1 1 1 1.6 1.20
2. Probability Estimation 1 5 1 2 2 2.2 1.47
3. Impact Estimation 1 5 1 1 2 2.0 1.55
4. Computation of Risk Values 1 5 1 4 1 2.4 1.74
5. Determination of Risk Levels 1 2 1 4 3 2.2 1.17
6. Definition of Test Strategy 1 4 1 3 1 2.0 1.26
7. Refinement of Test Strategy 4 2 2 2 2.5 0.87

14. • Process for risk‐based test strategy development
• Evaluation in research transfer project with five companies
• Overall positive result for usefulness of the process
• Overall ease of use perceived as moderate
• Most effort caused by Impact Estimation, Definition of Test
Strategy and Definition of Risk Items
• Overall representativeness is high with one notable exception
• Future work
• Refinement of process steps Impact Estimation and Definition
of Test Strategy
• Additional evaluations especially long‐turn studies
Conclusions
Slide 14Risk‐Based Test Strategy Development and Industrial Evaluation

15. Slide 15Risk‐Based Test Strategy Development and Industrial Evaluation
Track on Systematic Literature Reviews and Mapping Studies in SE