An introductory lecture to white box testing

1. Università degli Studi dell’Aquila
L20: White Box/Structural Testing
Henry Muccini
DISIM, University of L’Aquila
www.henrymuccini.com, henry.muccini@univaq.it

2. The material in these slides may be freely reproduced and
distributed, partially or totally, as far as an explicit
reference or acknowledge to the material author is
preserved.
Partially based on material from Alex Orso, Mauro Pezze’,
Michal Young, and Andreas Zeller

3. AGENDA
Recap
Fault, Error, Failure chain
White Box Testing
Statement Coverage
Branch Coverage
Basic Condition Coverage
MC/CD Coverage

4. Recap
“Systematic” testing vs. “ad hoc”testing
→Repeatable, Measurable
Software Testing Process
→Test Selection, Test Execution, Oracle, Adequacy
Type of testing
→Unit,Integration, System
→Performance, Stress
→Regression
→WB, BB

5. Fundamental Testing Questions
Test Case:
How is test described / recorded?
Test Criteria:
What should we test?
Test Oracle:
Is the test correct?
Test Adequacy:
How much is enough?
Test Process:
Is testing complete and effective?
Test Driver: used to execute the system on the SUT
Stubs: to reproduce missing objects, for integration testing

6. Fault, Error, Failure
Fault: anomaly in source code
→ may or may not produce a failure
→ a “bug”
Error: inappropriate development action
→ action introduced by a fault
→ Cause of a fault
Failure: incorrect/unexpected behavior or output
→ incident is the symptoms revealed by execution
→ failures are usually classified

7. THE FAULT-FAILURE MODEL
Error Failure
Fault
affects the
delivered service
sw internal state
must be is corrupted
activated (latent or manifest)
A fault will manifest itself as a failure if all of the 3 following conditions hold:
1) EXECUTION: the faulty piece of code is executed
2) INFECTION: the internal state of the program is corrupted (error state)
3) PROPAGATION: the error propagates to program output
PIE MODEL (Voas, IEEE TSE Aug. 1992)

8. WHITE BOX/STRUCTURAL TESTING

9. How to test this
Java program?

10. Selection of test suite is based on some
elements in the code (Test) Inputs
Assumption: Executing the faulty
“element” is a necessary condition for
revealing a fault Source
Code
There exist several examples
→ Control flow (statement, branch, basis path,
path) Output
→ Condition (simple, multiple) Internal behavior
→ Loop
→ Dataflow (all-uses, all-du-paths)
→ Fault based (mutation)

11. Idea:
→ A program is a graph
→ A graph can be traversed, by using some inputs
→ A test suite is adequate if I guarantee certain
coverage
─ Metric: percentage of coverage achieved
Objective: Cover the software structure

12. Control Flow
→ The partial order of statement execution, as defined by the
semantics of the language
Data Flow
→ The flow of values from definitions of a variable to its uses
Graph representation of control flow and
data flow relationships

13. 1 function P return INTEGER is
2 begin
3 X, Y: INTEGER;
4 READ(X); READ(Y);
5 while (X > 10) loop
6 X := X – 10;
7 exit when X = 10;
8 end loop;
9 if (Y < 20 and then X mod 2 = 0) then
10 Y := Y + 20;
11 else
12 Y := Y – 20;
13 end if;
14 return 2*X + Y;
15 end P;

14. 6 7 10
F
T T T
F 9 T
2,3,4 5 9a 9´
9b 14
F F
12

15. printSum(int a, int b) { Test this case
int result = a + b;
if (result > 0)
printcol(“red”, result); and this one
else if (result < 0)
printcol(“blue”, result);
[else do nothing] and this one!
}

16. Defined in terms of
→ test requirements
Result in
→ test specifications
→ test cases
Selection VS adequacy/evaluation criteria

17. test specifications
printSum(int a, int b) { a+b>0
int result = a + b;
if (result > 0)
printcol(“red”, result); a + b < 0
else if (result < 0)
printcol(“blue”, result);
[else do nothing] a + b == 0
}

18. test cases
printSum(int a, int b) { a == 3
b == 9
int result = a + b;
if (result > 0)
a == -5
printcol(“red”, result); b == -8
else if (result < 0)
printcol(“blue”, result);
a == 0
[else do nothing] b == 0
}

19. Test requirements: Statements in program
Cstmts = (number of executed statements)
(number of statements)

20. printSum(int a, int b) {
int result = a + b;
if (result > 0)
printcol(“red”, result);
else if (result < 0)
printcol(“blue”, result);
}
Coverage: 0%

21. a == 3
b == 9
printSum(int a, int b) {
int result = a + b;
if (result > 0)
printcol(“red”, result);
else if (result < 0)
printcol(“blue”, result);
}
Coverage: 71%

22. a == 3 a == -5
b == 9 b == -8
printSum(int a, int b) {
int result = a + b;
if (result > 0)
printcol(“red”, result);
else if (result < 0)
printcol(“blue”, result);
}
Coverage: 100%

23. Test hypothesis:
→By executing a path once, potential faults related to it will
be revealed (independently from the input)
Ideally:
→To exhaustively cover all possible paths along the program
graph representation
Approximations to path coverage
→Coverage criteria

24. Only about 1/3 of NASA statements were executed
under test before software was released (Stucki 1973)
Microsoft reports 80-90% statement coverage
Boeing must get 100% statement coverage (feasible)
for all software
Usually can about 85% coverage; 100% is harder
→ Unreachable code; dead code
→ Complex sequences
→ Not enough resources

26. a == 3 a == -5
b == 9 b == -8
printSum(int a, int b) {
int result = a + b;
if (result > 0)
printcol(“red”, result);
else if (result < 0)
printcol(“blue”, result);
}
Coverage: 100%

27. a == 3 a == -5
b == 9 b == -8
printSum(int a, int b) {
int result = a + b;
if (result > 0)
printcol(“red”, result);
else if (result < 0)
printcol(“blue”, result);
[missing branch]
}
Coverage: 100%

28. Test requirements: Branches in the program
Cbranches = (number of traversed branches)
(number of branches)

29. a == 3 a == -5
b == 9 b == -8
printSum(int a, int b) {
int result = a + b;
if (result > 0)
printcol(“red”, result);
else if (result < 0)
printcol(“blue”, result);
[missing branch]
}
Coverage: ?

30. a == 3 a == -5
b == 9 b == -8
printSum(int a, int b) {
int result = a + b;
if (result > 0)
printcol(“red”, result);
else if (result < 0)
printcol(“blue”, result);
[missing branch]
}
Coverage: 88%

31. a == 3 a == -5 a == 0
b == 9 b == -8 b == 0
printSum(int a, int b) {
int result = a + b;
if (result > 0)
printcol(“red”, result);
else if (result < 0)
printcol(“blue”, result);
[missing branch]
}
Coverage: 100%

33. entry • Consider test cases
{(x=5,y=5), (x=5, y=-5)}
1. void main() {
3
2. float x, y;
3. read(x); 4
4. read(y); !(X=0 X=0 ||
5. if(x==0)||(y>0) ||y>0) y>0
6. y = y/x;
7 6
7. else x = y+2;
8
8. write(x);
9. write(y);
9
10.}
exit

34. entry • Consider test cases
{(x=5,y=5), (x=5, y=-5)}
1. void main() {
3 • The test suite is adequate
2. float x, y;
for branch coverage, but
3. read(x); 4 does not reveal the fault
4. read(y); !(X=0 X=0 ||
||y>0) y>0
at statement 6
5. if(x==0)||(y>0)
• Predicate 5 can be true
6. y = y/x;
7 6 or false operating on only
7. else x = y+2;
8 one condition
8. write(x);
9. write(y);
10.}
9
⇓
exit
Basic condition coverage

36. Test requirements: Truth values assumed by
basic conditions
Cbc = (number of boolean values assumed by all basic conditions)
(number of boolean values of all basic conditions)
x y
T T
F F

37. entry • Consider test cases
{(x=0,y=-5), (x=5, y=5)}
1. void main() {
3 • The test suite is
2. float x, y;
adequate for basic
3. read(x); 4 condition coverage
4. read(y); !(X=0 X=0 ||
x y
5. if(x==0)||(y>0) ||y>0) y>0
6. y = y/x; T T
7 6
7. else x = y+2; F F
8
8. write(x);
• but is not adequate
9. write(y);
9 for branch coverage.
10.}
exit
⇓
Branch and condition
coverage

38. Test requirements: Branches and truth values
assumed by basic conditions
if ( ( a || b ) && c ) { … }
a b c Outcome
T T T T
F F F F

39. Key idea: Test important combinations of
conditions, avoiding exponential blowup
A combination is “important” if each basic
condition is shown to independently affect the
outcome of each decision

40. MC/DC criterion: For each basic condition C, there are
two test cases in which the truth values of all
conditions except C are the same, and the compound
condition as a whole evaluates to True for one of those
test cases and False for the other

41. ( a && b && c )
Test case a b c outcome
1 True True True True
2 True True False False
3 True False True False
4 True False False False
5 False True True False
6 False True False False
7 False False True False
8 False False False False

42. ( a && b && c )
Test case a b c outcome
1 True True True True
2 True True False False
3 True False True False
4 True False False False
5 False True True False
6 False True False False
7 False False True False
8 False False False False

43. MC/DC criterion: For
( a && b && c ) each basic condition
Test case a b c outcome C, there are two test
1 True True True True cases in which the
2 True True False False truth values of all
3 True False True False conditions except C
4 True False False False are the same, and the
5 False True True False compound condition
6 False True False False as a whole evaluates
7 False False True False
to True for one of
8 False False False False
those test cases and
False for the other

44. ( a && b && c )
Test case a b c outcome
1 True True True True
2 True True False False
3 True False True False
4 True False False False
5 False True True False
6 False True False False
7 False False True False
8 False False False False
1 True True True True
5 False True True False

45. ( a && b && c )
Test case a b c outcome
1 True True True True
2 True True False False
3 True False True False
4 True False False False
5 False True True False
6 False True False False
7 False False True False
8 False False False False
1 True True True True
5 False True True False

46. ( a && b && c )
Test case a b c outcome
1 True True True True
2 True True False False
3 True False True False
4 True False False False
5 False True True False
6 False True False False
7 False False True False
8 False False False False
1 True True True True
5 False True True False

47. ( a && b && c )
Test case a b c outcome
1 True True True True
2 True True False False
3 True False True False
4 True False False False
5 False True True False
6 False True False False
7 False False True False
8 False False False False
1 True True True True
5 False True True False
3 True False True False

48. ( a && b && c )
Test case a b c outcome
1 True True True True
2 True True False False
3 True False True False
4 True False False False
5 False True True False
6 False True False False
7 False False True False
8 False False False False
1 True True True True
5 False True True False
3 True False True False

49. ( a && b && c )
Test case a b c outcome
1 True True True True
2 True True False False
3 True False True False
4 True False False False
5 False True True False
6 False True False False
7 False False True False
8 False False False False
1 True True True True
5 False True True False
3 True False True False

50. ( a && b && c )
Test case a b c outcome
1 True True True True
2 True True False False
3 True False True False
4 True False False False
5 False True True False
6 False True False False
7 False False True False
8 False False False False
1 True True True True
5 False True True False
3 True False True False
2 True True False False

51. Tradeoff between number of required test
cases and thoroughness of the test
Required by both US and European quality
standards in aviation

53. Path coverage: cover each path in the program
Loop coverage (given n): cover all paths that contain at
most n iterations of any loop in the program
Data-flow coverage: cover data-flow relations in the
program (du pairs)
Mutation coverage: cover (kill) mutated versions of the
program

54. Most widely used as adequacy criteria because fully
automatable!
Not all graph paths correspond to actual program paths
→some paths could be not executable
→Conditions may not be satisfiable due to interdependent
conditions
→Paths may not be executable due to interdependent decisions
The testing output is strongly related to the testing
inputs