Paper presented during Postgraduate Research Workshop at Software Testing Conference 2012 (SOFTEC2012)

1. A Regression Analysis Approach for
Building a Prediction Model for
System Testing Defects
(Paper No: S1-3)
Muhammad Dhiauddin bin Mohamed Suffian
Faculty of Computer Science & Information System
mdhiauddin2@live.utm.my
AP Dr. Suhaimi Ibrahim
Advanced Informatics School
suhaimiibrahim@utm.my

2. Presentation Outline
•
•
•
•
•
Introduction
Related Works
Research Methodology
Findings and Discussion
Conclusion and Recommendation

3. Introduction
• Defect prediction is very significant to the independent
testing team
– ensure all potential field defects could be successfully contained
within system testing phase
– defects could be prevented from escaping to the end-users
– achieve the target of zero known post release defects for the software
delivered to end-users
• Common understanding of defect  forecast defects in
software [1][2]
• Defect prediction for system testing
– predict failures in system testing instead of defects [3]
– predict remaining defects in software release as part of test process
simulation [4]

4. Introduction (cont.)
• Motivation to undertake this research effort:
Assigning
appropriate
number of test
engineers across
multiple test
projects
• Re-align test
execution to
meet deadline
• Action plan
when actual ≠
prediction
• Right test
scenarios to
capture
predicted
defects
• Better root
cause analysis
• Decision by
management on
software release
• Stability of
whole
development
process

5. Introduction (cont.)
• Objectives:
– To analyze existing techniques of building
prediction model for system testing defects
– To build prediction model for system testing
defects using statistical approach
– To evaluate the proposed prediction model based
on specified acceptance criteria.

6. Related Works
• Defect terminology:
– any flaw in the system or even in the system’s
components could cause the system to
malfunction [5]
– deviation from its specification: physical software
or work products [6]
– imperfection in the process as well as work
product besides software [7]
– defect generated from V & V activities

7. Related Works (cont.)
•
Approaches to defect prediction:
– Term is used interchangeably with defect estimation to describe the proactive process
of characterizing defects found in software in producing high quality product [8]
– size and complexity metrics: McCabe’s cyclomatic complexity as well as lines of code
(LOC) e.g. Defect = 4.86 + 0.018 Lines of Code [6]
– categorized into project management, work product assessment and process
improvement [7]
– used Rayleigh Model to predict defect density at different phases of project life cycle ]9]
– combination of product and project metrics via regression analysis [10]
– used mathematical distributions as quality prediction model as part of software fault
prediction techniques [11]
– Used development information as important factor for the prediction and model quality
[12]
– applying object-oriented metrics for predicting faults in open source software [13]
– several inputs can be used to simulate system test phase in SDLC [14]
– applied statistical method in Six Sigma to predict defect density [15]
– used defect decay model to predict remaining defects in on-going testing process [16]

8. Related Works (cont.)
•
Issues:
– Strength: Easy to use, efficient, effective and able to indicate the process
performances; Weakness: need to have sampling, require stable process and
does not account for changes [7]
– Critiques [6]:
•
•
•
•
•
unknown relationship between defect and failures
problems with multivariate statistical approach,
problems of using size and complexity metrics as sole predictors of defects
problems in statistical methodology and data quality
false claims about software decomposition
• Measuring the success:
– measuring the percent of faults found in the identified files [17]
– help in maintenance resource planning as well as software insurance [18]

9. Research Methodology
Source of data:
ONE applied
R&D
organization
V-shaped
process model
Metrics from V&V:
•requirement review
•design review
•test plan review
•test cases review
•code inspection & unit testing
•system testing
Software type:
•Web-based
•Componentbased
Language:
•PHP
•.NET
•Java

10. Findings and Discussion
Prediction of
System Testing
Defects

11. Findings and Discussion(cont.)
Initial data set
Initial regression analysis result

12. Findings and Discussion (cont.)
Revised data set

13. Findings and Discussion (cont.)

14. Findings and Discussion (cont.)
Verification result
Selected prediction model for initial implementation
Functional Defects
–
= 4.00 - 0.204 Requirement Error - 0.631 Coding Error +
1.90 KLOC – 0.140 Requirement Page + 0.125 Design Page
0.169 Total Test Cases + 0.221Total Effort Days

15. Conclusion and Recommendation
•
Achievement:
– Demonstrated the successful construction of prediction model for system
testing defects by applying regression analysis approach
– Demonstrated the ability to predict defects for system testing by using metrics
in requirement, design and coding phase
•
Future works:
– To predict non-functional defects such as performance, security and usability
defects
– To predict defects based on severity of defects i.e. critical, major and minor
defects
– To incorporate more factors in building similar model such as function point,
programming languages, and number of classes
– To develop software tool that could dynamically generate the latest prediction
equation in real time and assist in prediction activity

16. THANK YOU