This presentation describes a simple formula to estimate the average defect level (% of none met requirements) of a software product throughout its production cycles based on the probability of finding an unsuccessfully implemented requirements and the probability that this type of defect gets fixed.

1. The dynamic interaction of passed and failed
requirements during software testing
Copyrights (c) 2011-2013 Pragmatic
1
Cohesion Consulting

2. Motivation
• In my professional experience, I have
repeatedly witnessed systemic issues during
Software Verification that prompted me to
develop the model presented in this paper.
• It is my hope that the non-intuitive results
that the model yields help software
development team better understand certain
dynamic behavior embedded in the software
development process.
Copyrights (c) 2011-2013 Pragmatic
2
Cohesion Consulting

3. Requirements Statuses
• We are interested in 2 simple sets of
requirements. All requirements within each
set share one of the following statuses:
– The first one represents requirements that have
been implemented i.e. requirements for which the
development team has created actual software
code. Let’s call this set of requirements ISR.
– The second one represents requirements that
have failed testing after they were implemented
by the development team. Let’s call this second
set of requirements FSR.
Copyrights (c) 2011-2013 Pragmatic
3
Cohesion Consulting

4. Requirement States Probabilities
• Let’s call SFR the probability that a
requirement that has been implemented fails
testing. When this failure happens the
requirement becomes part of the FSR set
otherwise it remains in the ISR set.
• Let’s call SFXR the probability that a failed
requirement gets fixed. When the fix takes
place, the requirement is no longer part of the
FSR set but it moves into the ISR set.
Copyrights (c) 2011-2013 Pragmatic
4
Cohesion Consulting

5. Requirement States Probabilities
• The following probability constraints exist:
0 <= SFR <= 1
0 <= SFXR <=1
• We can derive a probability transition matrix that
captures the possible transitions of a requirement
from one set to the other:
From ISR to ISR, probability = 1-SFR
From ISR to FSR, probability = SFR
From FSR to ISR, probability = SFXR
From FSR to FSR, probability = 1-SFXR
Copyrights (c) 2011-2013 Pragmatic
5
Cohesion Consulting

6. Transition probability matrix
Let M be the transition probability matrix,
M=
To ISR To FSR
From ISR 1-SFR SFR
From FSR SFXR 1-SFXR
Copyrights (c) 2011-2013 Pragmatic
6
Cohesion Consulting

7. Two-Step Transition probabilities
• The following question is particularly
interesting:
– given the transition probability matrix M and the
fact that a requirement has been added to the ISR
set, what is the probability that this same
requirement will be in the ISR set two transitions
in the future?
Copyrights (c) 2011-2013 Pragmatic
7
Cohesion Consulting

8. Two-Step Transition probabilities
• The problem is not as obvious as it seems;
here is is why:
– When a Requirement enters set ISR it has a SFR
probability of being moved to set FSR and a 1-SFR
probability of staying in ISR. Once in FSR, it has a
SFXR probability of moving into ISR and a 1-SFXR
probability of staying in FSR.
– We can represent the possible paths with the
following tree
Copyrights (c) 2011-2013 Pragmatic
8
Cohesion Consulting

9. Two-Step Transition probabilities
ISR P(ISR,ISR,ISR) = (1-SFR).(1-SFR)
1-SFR
ISR
SFR
1-SFR
FSR P(ISR,ISR,FSR) = (1-SFR).SFR
ISR
ISR P(ISR,FSR,ISR) = SFR.SFXR
SFR SFXR
FSR
1-SFXR
FSR P(ISR,FSR,FSR) = SFR.(1-SFXR)
Copyrights (c) 2011-2013 Pragmatic
9
Cohesion Consulting

10. Two-Step Transition probabilities
• So the probability that a Requirement that
started in ISR is in ISR after two transitions is:
P(ISR,ISR,ISR) + P(ISR,FSR,ISR) = (1-SFR).(1-SFR) +
SFR.SFRX
• All other similar two-step transition
probabilities can be calculated by M2
Copyrights (c) 2011-2013 Pragmatic
10
Cohesion Consulting

11. Two-Step Transition probabilities
M2 =
To ISR To FSR
From ISR (1-SFR).(1-SRF)+SFR.SFXR (1-SFR).SFR+SRF.(1-SFXR)
From FSR SFXR.(1-SFR)+(1-SFXR).SFXR SFXR.SFR+(1-SFXR).(1-SFXR)
Copyrights (c) 2011-2013 Pragmatic
11
Cohesion Consulting

12. Steady State Analysis
• We are interested in knowing in the long run,
the probability P1 that any given requirement
will be in the ISR set. The probability that such
requirement will be in the FSR set is P2=1-P1.
• Markov chains can help us solve this problem
by calculating the Steady State probabilities of
Matrix M:
P1 = SFXR/(SFR+SFXR) [e1]
P2 =1- SFXR/(SFR+SFXR) [e2]
Copyrights (c) 2011-2013 Pragmatic
12
Cohesion Consulting

13. Steady State Analysis- Example 1
• if:
P1 = SFXR/(SFR+SFXR) [e1]
• Then P1 = 0.5 = 50% when SFR = SFXR.
• This implies that even if there is a low
probability that a requirement that has been
implemented fails testing (SFR); if an equally
low fixing rate takes place, given sufficient
time, a requirement gets closer and closer to
having a 50% chance of being in the set that
failed testing!
Copyrights (c) 2011-2013 Pragmatic
13
Cohesion Consulting

14. Steady State Analysis – Example 2
• Let’s look at another illustration:
– P1 is the probability of having in the long run, a
non-failed requirement. Let’s say that we want P1
to be 90%. If we know that an implemented
requirement has an SFR=20% probability of failing
testing then what should SFXR (the rate at which
failed requirements are fixed) be to assure
P1=90%?
Copyrights (c) 2011-2013 Pragmatic
14
Cohesion Consulting

15. Steady State Analysis – Example 2
• From [e1] we can say that:
[e1] -> 0.9 = SFXR/(0.2+SFXR)
[e1] -> 0.9.(0.2+SFXR) = SFXR
[e1] -> 0.18 + 0.9.SFXR = SFXR
[e1] -> 0.18/(1-0.9) = SFXR
[e1] -> SFXR = 1.8 > 1 which is impossible!
Copyrights (c) 2011-2013 Pragmatic
15
Cohesion Consulting

16. Steady State Analysis – Example 2
• If SFXR was equal to its maximum value of
100% then the maximum value of P1 can only
be 1/(1+0.2) = 83.3%=P1
• This result shows that there is a clear upper
bound to the probability that in the long run a
requirement will not fail testing given the
value of SFR. SFR is the probability that a given
requirement that has just been implemented
fails testing.
Copyrights (c) 2011-2013 Pragmatic
16
Cohesion Consulting

17. Steady State Analysis – Example 2
• The only way to achieve P1 = 90% when SFXR
= 100% is to decrease SFR from 20% so that:
0.9 = 1/(SFR+1)
SFR + 1 = 1/0.9
SFR = 1/0.9 – 1
SFR = 11.11%
• This lower value of SFR implies a better quality
of software code that properly implements
more requirements.
Copyrights (c) 2011-2013 Pragmatic
17
Cohesion Consulting

18. Transient Analysis
• It is possible to graph the values of P1 and P2
over several transitions, this is called transient
analysis.
• Transient analysis can show how fast the
steady states values for P1 and P2 are
reached.
• Transient analysis can also show instability in
P1 and P2 values over several transitions due
to higher values of SFR and SFRX (see
examples 4 and 5 in subsequent slides).
Copyrights (c) 2011-2013 Pragmatic
18
Cohesion Consulting

19. Transient Analysis – Example 1
• SFR = 10% and SFXR = 50%
Copyrights (c) 2011-2013 Pragmatic
19
Cohesion Consulting

20. Transient Analysis – Example 2
• SFR = 10% and SFXR = 10%
Copyrights (c) 2011-2013 Pragmatic
20
Cohesion Consulting

21. Transient Analysis – Example 3
• SFR = 10% and SFXR = 1%
Copyrights (c) 2011-2013 Pragmatic
21
Cohesion Consulting

22. Transient Analysis – Example 4
• SFR = 90% and SFXR = 90%
Copyrights (c) 2011-2013 Pragmatic
22
Cohesion Consulting

23. Transient Analysis – Example 5
• SFR = 75% and SFXR = 100%
Copyrights (c) 2011-2013 Pragmatic
23
Cohesion Consulting

24. For Comments and Questions contact didier@pragmaticohesion.com
Copyrights (c) 2011-2013 Pragmatic
24
Cohesion Consulting