3. Building the Model A. Define Phase
The Define Phase primarily involves establishing the project
In Design for Six Sigma (DfSS), the research is organized accord- definition and acknowledging the fundamental needs of the re-
ing to DMADV phases: Define (D), Measure (M), Analyze (A), De- search. A typical software production process which applies the V-
sign (D) and Verify (V). DfSS seeks to sequence proper tools and Model software development life cycle is used as the basis for this
techniques to design in more value during new product devel- research. Throughout the process, the testing team is involved in
opment, while creating and using higher quality data for key de- all review sessions for each phase, starting from planning until
velopment decisions. Achievement of DfSS is observed when the the end of the system integration testing phase. The test lab is
products or services developed meet customer needs rather than involved in reviewing the planning document, the requirement
competing alternatives. In general, DfSS-DMADV phases are de- analysis document, the design document, the test planning doc-
scribed as below: ument and the test cases (see Figure 1). The software production
process is governed by project management, quality manage-
• Define - identify the project goals and customer (internal
and external) requirements ment, configuration and change management, integral and sup-
port as well as process improvement initiatives, in compliance to
• Measure - determine customer needs and specifications; CMMI®. From the figure, the area of study is the functional or sys-
benchmark competitors and industry
tem test phase, where defects are going to be predicted. There-
• Analyze – study and evaluate the process options to meet fore, only potential factors/predictors in the phases prior to the
customer needs system testing phase are considered and investigated.
• Design – detailing the process to meet customer needs
• Verify – confirm and prove the design performance and abil-
ity to meet customer needs
Figure 1: Typical Software Production Process
Two (2) schematic diagrams are produced, which are a high level are defect containment in the test phase, customer satisfaction,
schematic diagram (Figure 2) and a detailed schematic diagram quality of the process being imposed to produce the software and
(Figure 3). The high level schematic diagram deals with estab- project management. There are two aspects involved related to
lishing the Big Y or business target, little Ys, vital Xs and the goal these little Ys: potential number of defects before the test phase
statement against the business scorecard. In this research, Big which is the research interest, and the number of defects after
Y is to produce software with zero-known post-release defects, completing the test phase.
while for little Ys, elements that contribute to achieving the Big Y
www.testingexperience.com The Magazine for Professional Testers 53
4. Big Y Zero-Known Post
Release Defects
Little ys
Defect Contain- Customer Project
Quality of process
ment in Test Phase satisfaction Management
Vital Xs
Potential # of Level of People Timeline
defect before test satisfaction capability Allocation
# of defect after Process Rescue
test Effectiveness Allocation
Figure 2: Schematic Diagram
For the detailed schematic diagram (or detailed Y-X tree), possible test, and these are summarized in a Y to X tree diagram as shown
factors that contribute to the test defect prediction are defined in the figure below. The highlighted factors are selected for fur-
from the Vital X, which is the potential number of defects before ther analysis.
Test Defect
Prediction
Software Historical
Knowledge Test Process Errors Fault Project
Complexity Defect
Require- Developer Test Case De- Require- Require- Defect Project Project
ment Pages Knowledge sign Coverage ment Error ment Fault Severity Domain Thread
Design Tester Targeted Total Design Defect Type/
Design Error Component
Pages Knowledge Test Cases Fault Category
Programming Test Defect
Language
CUT Error CUT Fault Application
Automation Validity
Test Case Execu- Test Plan Integration Total PR (Defects)
Code Size tion Productivity Error Fault Raised
Total Effort in Test Test Cases Test Case
Design Phase Error Fault
Total Effort in
Phases Prior to
System Test Factors to consider
Figure 3: Detailed Y-X Tree Diagram
54 The Magazine for Professional Testers www.testingexperience.com
5. B. Measure Phase
known results of PASS and FAIL are identified. Then three test
A Measurement System Analysis (MSA) is conducted to validate engineers are selected to execute the test cases at random. This
that the processes of discovering defects are repeatable and re- is repeated three times for every engineer. The outcome is pre-
producible, thus eliminating human errors. Ten test cases with sented in Figure 4 below:
Tester 1 Tester 2 Tester 3
TCD ID TC Result
1 2 3 1 2 3 1 2 3
TC1 PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS
TC2 FAIL FAIL FAIL FAIL FAIL FAIL FAIL PASS PASS PASS
TC3 FAIL FAIL FAIL FAIL FAIL FAIL FAIL PASS PASS PASS
TC4 PASS FAIL FAIL FAIL FAIL FAIL FAIL PASS PASS PASS
TC5 PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS
TC6 PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS
TC7 PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS
TC8 FAIL FAIL FAIL FAIL FAIL FAIL FAIL FAIL FAIL FAIL
TC9 PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS
TC10 PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS
Figure 4: Data for Analysis
The data is then used to run the MSA, where the result is com- As the MSA is passed, the operational definition is prepared con-
pared against the Kappa value. Since the overall result of all ap- sisting of type of data to be gathered, measurement to be used,
praisers against standard is above 70% (as required by Kappa’s responsibilities, and mechanism to obtain those data so that the
value), the MSA is considered as a PASS as presented in Figure 5 data can be used for next DfSS phase.
below:
Overall Result for MSA is PASS since Kappa value is greater than 0.7 or 70%
Figure 5: Overall Result of MSA
C. Analyze Phase analysis via manual stepwise regression analysis, i.e. the predic-
tors are added and removed during the regression until a strong
The data gathered from the Measure Phase is used to perform
statistical result is obtained. The figure involves the P-value of
a first round of regression analysis using the data shown in Fig-
each predictor against the defects, the R-squared (R-Sq.) value and
ure 6, which was collected from thirteen projects. The predictors
the R-squared adjusted (R-Sq. (adj.)) value. These figures demon-
used include requirement error, design error, Code and Unit Test
strated the goodness of the equation and how well it can be used
(CUT) error, Kilo Lines of Code (KLOC) size, targeted total test cases
for predicting the defects. The result of the regression analysis is
to be executed, test plan error, test cases error, automation per-
presented in Figure 7.
centage, test effort in days, and test execution productivity per
staff day. The regression is done against the functional defects as
the target. MINITAB software is used to perform the regression
56 The Magazine for Professional Testers www.testingexperience.com
6. Project Req. Er- Design CUT KLOC Total Test Plan Test Case Automa- Test Test Ex- Func-
Name ror Error Error Test Error Error tion % Effort ecution tional
Cases Produc- Defects
tivity
Project A 5 22 12 28.8 224 0 34 0 6.38 45.8000 19
Project B 0 0 1 6.8 17 0 6 0 9.36 17.0000 1
Project C 9 10 14 5.4 24 4 6 0 29.16 5.8333 4
Project D 7 12 2 1.1 25 4 9 0 13.17 7.0000 0
Project E 11 29 3 1.2 28 4 12 0 14.26 3.4000 3
Project F 0 2 7 6.8 66 1 7 0 32.64 31.0000 16
Project G 3 25 11 4 149 5 0 0 7.15 74.5000 3
Project H 4 9 2 0.2 24 4 0 0 18.78 7.6667 0
Project I 7 0 1 1.8 16 1 3 0 9.29 2.6818 1
Project J 1 7 2 2.1 20 1 4 0 6.73 1.9450 0
Project K 17 0 3 1.4 13 1 4 0 8.44 6.5000 1
Project L 3 0 0 1.3 20 1 7 0 14.18 9.7500 1
Project M 2 3 16 2.5 7 1 6 0 8.44 1.7500 0
Figure 6: Data for Regression Analysis
Figure 7: Regression Analysis Result
Project Req. Design CUT KLOC Req. Design Total Test Total Test Func- All
Name Error Error Error Page Page Test Cases Effort Design tional Defects
Cases Error Effort Defects
Project A 5 22 12 28.8 81 121 224 34 16.79 15.20 19 19
Project B 0 0 1 6.8 171 14 17 6 45.69 40.91 1 1
Project C 9 10 14 5.4 23 42 24 6 13.44 13.44 4 4
Project D 7 12 2 1.1 23 42 25 9 4.90 9.90 0 0
Project E 11 29 3 1.2 23 54 28 12 4.72 4.59 3 3
Project F 0 2 7 6.8 20 70 88 7 32.69 16.00 16 27
Project G 3 25 11 4 38 131 149 0 64.00 53.30 3 3
Project H 4 9 2 0.2 26 26 24 0 5.63 5.63 0 0
Project I 17 0 3 1.4 15 28 13 4 9.13 7.88 1 1
Project J 61 34 24 36 57 156 306 16 89.42 76.16 25 28
Project K 32 16 19 12.3 162 384 142 0 7.00 7.00 12 12
Project L 0 2 3 3.8 35 33 40 3 8.86 8.86 6 6
Project M 15 18 10 26.1 88 211 151 22 30.99 28.61 39 57
Project N 0 4 0 24.2 102 11 157 0 41.13 28.13 20 33
Figure 8: New Set of Data for Regression
www.testingexperience.com The Magazine for Professional Testers 57
7. D. Design Phase
Based on the result, it has been demonstrated that for this round
of regression, possible predictors are design error, targeted total Further analysis is conducted during the Design Phase due to the
test cases to be executed, test plan error, and test effort in days, need to generate a prediction model that both is practical and
in which the P-value for each predictor is less than 0.05. As overall makes sense from the viewpoint of software practitioners using
model equation, this model portrays strong characteristics of a logical predictors, to filter the metrics to contain only valid data,
good model via high percentage of R-Sq. and R-Sq. (adjusted) val- and to reduce the model to have only coefficients that have a logi-
ues: 96.7% and 95.0% respectively. From the regression, this equa- cal correlation to the defect. As a result, a new set of data is used
tion is selected for study in the next phase: to refine the model as shown in Figure 8.
Using the new set of data, new regression results are presented
Defect = - 3.04 + 0.220 Design Error + 0.0624 Targeted Total
in Figure 9.
Test Cases - 2.30 Test Plan Error + 0.477 Test Efforts
Figure 9: New Regression Result
From these latest results, it can be demonstrated that significant Based on the the verification result, it is clearly shown that the
predictors for predicting defects are requirement error, CUT er- model is fit for use and can be implemented to predict test de-
ror, KLOC, requirement page, design page, targeted total test cas- fects for the software product. This is justified by the predicted
es to be executed, and test effort in days from the phases prior to defects number, which falls within 95% Prediction Interval (PI). As
system test. The P-value for each factor is less than 0.05, while the the test defect prediction model equation is finalized, the next
R-Sq. and R-Sq. (adjusted) values are 98.9% and 97.6% respective- consideration is to emphasize on the control plan with regard to
ly, which results in a stronger prediction equation. The selected its implementation in the process, i.e. when the actual number
equation is as below: of defects found is lower or greater than the prediction. Figure 11
below summarizes the control plan:
Functional Defects (Y) = 4.00 - 0.204 Requirement Error - 0.631
CUT error + 1.90 KLOC - 0.140 Requirement Page + 0.125 Design Actual Defects < Predicted Actual Defects > Predicted
Page - 0.169 Total Test Cases + 0.221 Test Effort Defects Defects
Perform thorough testing Re-visit the errors captured
during ad-hoc test during requirement, design
E. Verify Phase and CUT phase
The selected model in the Design Phase is now verified against Perform additional test Re-visit the errors captured
new projects that have yet to go to the System Test phase. The strategy that relates to the during test case design
actual defects found after the System Test has been completed discovery of more functional
defects
are compared against the predicted defects to ensure that actual
defects fall between 95% prediction intervals (PI) of the model as Re-visit the model and the
presented in the last column in Figure 10. factors used to define the
model
No Predicted Actual 95% CI 95% PI
Figure 11: Action Plan for Test Defect Prediction Model
Functional Functional (min, max) (min, max)
Defects Defects
1. 182 187 (155, 209) (154, 209)
2. 6 1 (0, 2) (0, 14)
3. 1 1 (0, 3) (0, 6)
Figure 10: Verification Result for the Prediction Model
58 The Magazine for Professional Testers www.testingexperience.com