Jamie Mitchell explores perhaps the most underused test technique in our arsenal—white-box testing. Also known as structural testing, white-box requires some programming expertise, access to the code, and an analysis tool. If you only employ black-box testing, you could easily ship a system having tested only 50 percent or less of the code base. Not good! Although you might believe that the developers have performed sufficient unit and integration testing, how do you know that they have achieved the level of coverage your project requires? Jamie describes the levels of code coverage that the business and your customers may need—from statement coverage to modified condition/decision coverage. Leading you through examples of pseudocode, Jamie explains when you should strive to achieve different code coverage target levels. Even if you have no personal programming experience, understanding structural testing will make you a better tester. So, join Jamie for some code-diving!

1. T14
Test Techniques
5/2/2013 1:30:00 PM
White-box Testing: When Quality Really
Matters
Presented by:
Jamie Mitchell
Jamie Mitchell Consulting, Inc.
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

2. Jamie Mitchell
Jamie Mitchell has more than thirty years of experience developing and testing both hardware and
software. In 1991, Jamie moved from hardware to the dark side - software. He was a pioneer in the test
automation field, working with a variety of vendor, open source, and custom-built tools since Windows 3.0.
Jamie's specialty is increasing the productivity of automation and test teams through innovative ideas and
custom tool extensions. In addition, Jamie provides training, mentoring, and expert technical support in all
aspects of testing and automation. Jamie is the coauthor (with Rex Black) of Advanced Software Testing,
Volume 3: Guide to the ISTQB Advanced Certification as an Advanced Technical Test Analyst.

3. 4/19/2013
Structural Testing
When Quality Really Matters
Jamie L Mitchell
Jamie Mitchell Consulting Inc.
When Software Really Needs to Work
Copyright (c) Jamie Mitchell Consulting Inc.
2
1

4. 4/19/2013
Why Structure-based Testing?
StructureExperience-based techniques are not thorough
enough:
◦ Error-guessing
◦ Exploratory testing
◦ Defect- and taxonomy-based
Black-box testing depends on how good your test
basis model is:
◦ Requirements
◦ Use cases
◦ Functional specifications
How well have you tested?
Is testing less than half of your code sufficient?
Copyright (c) Jamie Mitchell Consulting Inc.
3
WhiteWhite-Box Test Basis
Design documents and models
◦ Architecture
◦ Functional design
◦ Low-level design
Object models
Class interface documents
Code
Copyright (c) Jamie Mitchell Consulting Inc.
4
2

5. 4/19/2013
Today’s Discussion Targets
Control-flow analysis: selecting certain
paths through the software to test, based
on desired coverage
Data-flow analysis: a control-flow
technique that scrutinizes the life-cycle of
data variables
Copyright (c) Jamie Mitchell Consulting Inc.
5
ControlControl-flow Testing
Select certain paths through the
software
◦ Based on desired level of coverage
Based on statements
Based on decisions
Based on loops
Based on paths
Copyright (c) Jamie Mitchell Consulting Inc.
6
3

6. 4/19/2013
Statement Coverage
Execute every line of code
Tools available for measurement
◦ Run tool while executing tests
◦ Evaluate coverage
◦ Create more tests to reach areas missed
Also called instruction or code coverage
Copyright (c) Jamie Mitchell Consulting Inc.
7
Decision Coverage
Statement coverage may miss bugs
◦ One test needed for statement coverage
(a = 5, b = 4)
Passes
z = 0;
if (a > b) then
z = 12;
x = 72 / z
Select inputs to force each decision to execute
both possible ways (T/F)
◦ Now two test cases are needed for coverage
(a = 5, b = 4)
(a = 4, b = 5)
Passes
Finds bug
Copyright (c) Jamie Mitchell Consulting Inc.
8
4

7. 4/19/2013
Condition Coverage
Decisions may be arbitrarily complex
if (a > b) then…
if (A || B) && (C == D) || (!E) && (F) then…
Condition testing requires that each atomic
condition must be evaluated both true and false
Note that condition coverage – by itself – may
not be as strong as decision coverage
if (a AND b) then
(a = F, b = T
(a = T, b = F
F)
F)
Copyright (c) Jamie Mitchell Consulting Inc.
9
Decision / Condition Coverage
A combination of Condition and Decision
1. Each atomic condition must be evaluated
both ways AND
2. Decision coverage must be satisfied
if (a AND b) then
(a = F, b = T
(a = T, b = F
(a = T, b = T
F)
F)
T)
Is that enough coverage?
Copyright (c) Jamie Mitchell Consulting Inc.
10
5

8. 4/19/2013
Modified Condition / Decision
Usually called MC/DC
◦ For each condition in the predicate, there is at
least one test where the outcome is TRUE only
because of that condition being TRUE
◦ For each condition in the predicate, there is at
least one test where the outcome is FALSE only
because of that condition being FALSE
Usually results in N+1 test cases where N is
number of conditions in predicate
Required by some safety standards to ensure
that enough testing has been done
Ex: FAA DO/178B Level A (catastrophic) software
Copyright (c) Jamie Mitchell Consulting Inc.
11
MC/DC Coverage (2)
if ((a || b) && c) then …
Decision/Condition can be achieved with
(a = T, b = T, c = T)
(a = F, b = F, c = F)
T
F
Note that (b) never independently affects
the outcome of the condition
Copyright (c) Jamie Mitchell Consulting Inc.
12
6

9. 4/19/2013
MC/DC Coverage (3)
if ((a || b) && c) then …
Following test set will achieve MC/DC coverage
1.
2.
3.
4.
(a = T,
(a = F,
(a = F,
(a = T,
b = F,
b = F,
b = T,
b = F,
c = T)
c = T)
c = T)
c = F)
T
F
T
F
Copyright (c) Jamie Mitchell Consulting Inc.
13
Coverage Based on Loops
Loops make testing …interesting…
The number of possible paths may
approach infinite depending on how many
loops are executed
Basic coverage criteria
◦ Test when no loops are made
◦ Test when a single loop is made
◦ Test when n loops are made (where n is either
the expected max or a large number)
Copyright (c) Jamie Mitchell Consulting Inc.
14
7

10. 4/19/2013
Loops (2)
This loop can calculate a factorial (n!)
for (i = 1; i <= n; i++)
f *= i;
Use (n = 0) for zero loops
Use (n = 1) for one time through
Determining the max requires thought
◦ In this case, (f) gets very large
◦ Use (n = 12); 13 would cause an overflow
Copyright (c) Jamie Mitchell Consulting Inc.
15
Nested Loops
Nested loops are really hard to test
Boris Beizer* had some advice
1.
2.
3.
4.
5.
Start at the innermost loop, setting all outer loops to
their minimum iteration setting
Test boundaries for innermost loop while keeping outer
loops at their minimum
If you have done outermost loop goto 5. Else move
one loop outward and test as in step 2 with all inner
loops set to their typical values
Continue outward until all loops done
Test the boundaries for all loops simultaneously
* Software Testing Techniques, 2nd Ed. by Boris Beizer
Copyright (c) Jamie Mitchell Consulting Inc.
16
8

11. 4/19/2013
Path Testing
Full path testing in a non-trivial system is
impossible
◦ The number of paths approaches infinite
◦ This is due to loops as seen previously
We can identify independent, non-looping
paths which cover all the edges and nodes of
a control-flow graph
◦ These are called basis paths
◦ Testing the basis paths can guarantee decision
(and statement) coverage
◦ McCabe cyclomatic complexity uses this method
◦ Tools are available for calculating this value
Copyright (c) Jamie Mitchell Consulting Inc.
17
Cyclomatic Complexity Example
Complexity = 3
Tests needed
◦ 1-2-6-7-8
◦ 1-3-4-7-8
◦ 1-3-5-8
Example from “A Complexity Measure” by Thomas
McCabe, IEEE Transactions on SW Engineering, Vol.
SE-2, No 4, December 1976
Copyright (c) Jamie Mitchell Consulting Inc.
18
9

12. 4/19/2013
DataData-flow Analysis (1)
Programs create, set, read, evaluate and destroy
data
The life cycle of data can be analyzed to find
defects that control-flow techniques miss
Some defect examples
◦
◦
◦
◦
Incorrect assignment of a value
Incorrect input statement
Failure to define a variable before use
Incorrect path taken due to incorrect value in a
control predicate
◦ Use of a variable after it is destroyed or out of scope
◦ Redefinition of a variable before it can be used.
Look for consecutive touches to each variable
Copyright (c) Jamie Mitchell Consulting Inc.
19
DataData-flow Analysis (2)
Dependent on the language used
We look for anomalies in data life-cycle
◦ Not all anomalies are defects
◦ While this is nominally a static technique, it
will not find all data defects (often data is
manipulated or assigned dynamically)
Sometimes called DUK testing
◦D
◦U
◦K
Define
Use
Kill
Copyright (c) Jamie Mitchell Consulting Inc.
20
10

13. 4/19/2013
DataData-flow Testing Combinations (1)
Anomaly Explanation
1.
~d
first define
Allowed
2.
du
define-use
Allowed, normal case
3.
dk
define-kill
Potential bug; data was never used
4.
~u
first use
Potential bug; data was used without definition.
It may be a global variable, set outside the
routine
5.
ud
use-define
Allowed: data used and then re-defined
6.
uk
use-kill
Allowed
7.
~k
first kill
Potential bug; data is killed before definition
8.
ku
kill-use
Serious defect; data is used after being killed
Copyright (c) Jamie Mitchell Consulting Inc.
21
DataData-flow Testing Combinations (2)
Anomaly Explanation
9.
kd
kill-define
Usually allowed. Data is killed and then redefined. Some theorists believe this should be
disallowed
10. dd
define-define
Potential bug; double definition
11. uu
use-use
Allowed; normal case. Some do not bother
testing this pair since no redefinition occurred
12. kk
kill-kill
Potential bug
13. d~
define last
Potential bug; dead variable? May be a global
variable which is used in other context
14. u~
use last
Allowed. Variable was used in this routine but
not killed off
15. k~
kill last
Allowed; normal case
Copyright (c) Jamie Mitchell Consulting Inc.
22
11

14. 4/19/2013
Function to Test
1.
public static double calculateBill (int Usage) {
2.
double Bill = 0;
3.
if(Usage > 0) {
4.
Bill = 40;
5.
6.
if(Usage > 100) {
7.
if(Usage <= 200) {
8.
Bill = Bill + (Usage - 100) * 0.5;
9.
}
10.
else {
11.
Bill = Bill + 50 + (Usage - 200) * 0.1;
12.
if(Bill >= 100) {
13.
Bill = Bill * 0.9;
14.
}
15.
}
16.
}
17.
}
18.
19.
return Bill;
}
Copyright (c) Jamie Mitchell Consulting Inc.
23
Variable DUK Patterns
Usage data-flow
information
Bill data-flow
information
Pattern
Explanation
1.
~d (1)
normal case
1. ~d (2)
Pattern
normal case
Explanation
2.
du (1-3)
normal case
2. dd (2-4)
suspicious
3.
uu (3-6)(6-7)
(7-8)(7-11)
normal case
3. du (2-18)(4-8)
(4-11)(11-12)
normal case
4.
uk (6-19)
(8-19)(11-19)
normal case
4. ud (8-8)(11-11)(13-13)
acceptable
5. uu (12-13)(12-18)
normal case
5.
k~ (19)
normal case
6. uk (18-19)
normal case
7. k~ (19)
normal case
Copyright (c) Jamie Mitchell Consulting Inc.
24
12

15. 4/19/2013
Summary
Many testers simply use black-box and
experience-based techniques
Some systems require far deeper testing
This presentation barely touches the
many advanced techniques that can be
used in testing
Learning these techniques can give you an
advantage in your career
Copyright (c) Jamie Mitchell Consulting Inc.
25
13