ACCU 2010 Talk on various costs surrounding a move to agile testing. Demonstrates use of "cycle time" and "cost of delay"

1. The Costs of Agile Testing
Schalk W. Cronjé
ysb33r@gmail.com
ACCU 2010
1 / 45
© Schalk W. Cronjé

2. Agile Today
ACCU 2010
2 / 45
© Schalk W. Cronjé

3. Agile Today
● Mature
● Well-known methodologies agile
today
● Embraced by many
– Even if only by lip service
● Misunderstood by many
● Too easy to tick the boxes
than to deliver value
● Tyranny of the urgent
– Discipline
ACCU 2010
3 / 45
© Schalk W. Cronjé

4. Institutions
● Sets of internalised rules Institution
supported by values
– Has tacit influence
– Rules will be contested
● Shape understanding of Organisations
social meaning + order
– Provides a framework for
performance
● Shapes of rights + duties People
– Political authority
– Economic opportunities
An organisation can become "institutionalised"
Institutions don't last forever
ACCU 2010
4 / 45
© Schalk W. Cronjé

5. Scott's Model
interpret
Societal Institutions innovate
invent, error
negotiate
diffuse,
impose
Organisational Fields
invent,
negotiate
diffuse,
impose
Organisations invent,
sanction negotiate
behaviour
diffuse,
impose
Actors
Limitations of cognitive / social rationality (groups/individuals)
Selective perception
ACCU 2010
5 / 45
© Schalk W. Cronjé

6. "Excellent system qualities are a
continuous management and engineering
challenge, with no perfect solutions"
Tom Gilb, Overload 85, June 2008
ACCU 2010
6 / 45
© Schalk W. Cronjé

7. The Economic View
ACCU 2010
7 / 45
© Schalk W. Cronjé

8. "You are not here to produce software,
you are here to provide value
to the business"
Schalk W. Cronjé
ACCU 2010
8 / 45
© Schalk W. Cronjé

9. Why take an economic view?
● It helps to quantify the effects of multiple
interacting variables
● It helps us to understand that the customer is
not the only judge of value
● By using an economic framework it can allow
us to maximise value, including
– cycle time
– product cost
– development expense
● It helps to communicate with non-technical
decision makers
ACCU 2010
9 / 45
© Schalk W. Cronjé

10. Cost of Quality
● Appraisal costs
– Discovering condition of hardware & 3rd-party
software components
● Internal failure costs
– Defects found before shipment
● External failure costs
– Defects found after shipment
● Prevention costs
– Costs for preventing all of the above
ACCU 2010
10 / 45
© Schalk W. Cronjé

11. Batch Development
Feature 1 Dev Feature 1 QA
Feature 2 Dev Feature 2 QA
Feature 3 Dev Feature 3 QA
Feature 4 Dev Feature 4 QA
delivered
build
ACCU 2010
11 / 45
© Schalk W. Cronjé

12. Batch Development
Total Time = devt1 .. devt4 + qat1 …qat4 + fixdelay + fixtime + retest time
Feature 1 Dev Feature 1 QA
Feature 2 Dev Feature 2 QA
Feature 3 Dev Feature 3 QA
Feature 4 Dev Feature 4 QA
delivered
build
defect trickle feed
Bug Feature 5+ QA
DB Retest fixes QA
delivered
build
Feature 5+ Dev
ACCU 2010
12 / 45
© Schalk W. Cronjé

13. Time Factors
Building test
understanding infrastructure
specs
Feature 1 Dev Feature 1 QA
Feature 2 Dev Feature 2 QA
Feature 3 Dev Feature 3 QA
Feature 4 Dev Feature 4 QA
basic build
verification
delivered
build
time from raising defect until it is
available for testing
defect trickle feed time to
re-test
Bug Feature 5+ QA
DB Retest fixes QA
delivered
build
Feature 5+ Dev
ACCU 2010
13 / 45
© Schalk W. Cronjé

14. Feedback
● Faster feedback makes learning faster and
more efficient
● Co-location improves communication
● Faster feedback provides a sense of control
● Large batches leads to slower feedback
ACCU 2010
14 / 45
© Schalk W. Cronjé

15. Optimising Batch Size
Total
● U-curve optimisation Cost
Holding cost
● Holding cost includes delay
Cost
in time to learn
Transaction cost
● Transaction cost includes
overhead in releasing a build Batch size
for testing
Optimal Batch Size
_________________________________________________
√ 2 x Cost per Batch x Total features in project
--------------------------------------------------
Holding cost per feature
ACCU 2010
15 / 45
© Schalk W. Cronjé

16. "If you only quantify one thing,
quantify the cost of delay"
Don Reinertsen
ACCU 2010
16 / 45
© Schalk W. Cronjé

17. The Attention Principle
"Time counts more than money"
Don Reinertsen
ACCU 2010
17 / 45
© Schalk W. Cronjé

18. Building an Agile Testing
Model
ACCU 2010
18 / 45
© Schalk W. Cronjé

19. Simple Flow Model
Specify Develop QA
Isn't this
Kanban waterfall?
Board
Ready Spec Spec Dev Dev QA QA Released
Complete Complete Complete
ACCU 2010
19 / 45
© Schalk W. Cronjé

20. Reducing Learning Time
● Writing test specifications after the
development is costly in time
– Knowledge decay
– Leads to incomplete designs
● Move the test specification up-front before any
development starts
– This is part of requirements discovery
– It broadens the perspective of the programmer
– Allows us to distinguish between automated checks
and human testing
ACCU 2010
20 / 45
© Schalk W. Cronjé

21. Reducing Learning Time
Specify Develop QA
Test specification
+Time to specify -Time to re-work
+Time to develop -Time to re-test
ACCU 2010
21 / 45
© Schalk W. Cronjé

22. Quantifying Learning Time
● Can you convert time saved to a monetary
value? Before After
Avg cycle days to spec 2 4
Avg cycle days to develop 8 10
Avg cycle days to test 13 8
Avg rework days during 5 1
test
Total 23 22
Cost $8,118.00 $7,872.00
(*) Data in graph is from an insignificant data set and provides no conclusive proof. It is for illustrative purposes only
ACCU 2010
22 / 45
© Schalk W. Cronjé

23. Change the Flow Model
Specify Develop & Check Verify
● Ensure the all automated checks are included
in the development stage
● Free up a Verification stage for exploratory
testing and validating that process has been
fulfilled.
ACCU 2010
23 / 45
© Schalk W. Cronjé

24. Change the Flow Model
Specify Develop & Check Verify
+Time to develop -Time to re-work
+More testing people -Time to re-test
ACCU 2010
24 / 45
© Schalk W. Cronjé

25. The Role of the Tester
● Up-front involvement to build in quality
● Help to build test infrastructure up-front during
the development phase
● Work with developer to critically review what
is being automated.
ACCU 2010
25 / 45
© Schalk W. Cronjé

26. Apply Lean Principles
Specify Develop & Check Verify
● Make the system pull-based
● Reduce the batch size
● Limit the work-in-progress
● One feature end-to-end
● If the flow breaks, fix it immediately
ACCU 2010
26 / 45
© Schalk W. Cronjé

27. Apply Lean Principles
Specify Develop & Check Verify
-Average Cycle time
+Time to develop
-Inherited delay cost
+Time to test
-Waiting time
ACCU 2010
27 / 45
© Schalk W. Cronjé

28. Team Composition
ACCU 2010
28 / 45
© Schalk W. Cronjé

29. The Classic Dev-QA Team
● This is the easiest team to adapt
● Requires the will to change to a new working
practices
● If teams are distributed then groups of
programmers and testers must be co-located.
ACCU 2010
29 / 45
© Schalk W. Cronjé

30. The "No Testers" Team
● All development + testing done by same
people.
● Requires T-skilled people
● Initial slow down of cycle time due to learning
of new skills OR reduction in quality delivered
ACCU 2010
30 / 45
© Schalk W. Cronjé

31. Paired Teams
● Pair up people with different skills to work on
same feature
– Feature teams are not a new concept
● Could be perceived to have higher person
cost per feature
– Instead of distributing the person cost over multiple
queues, the cost is combined into a single queue
– Can actually lead to reduced cycle time.
ACCU 2010
31 / 45
© Schalk W. Cronjé

32. Real-world measurements
ACCU 2010
32 / 45
© Schalk W. Cronjé

33. Real-world measurements
Specify Develop & Check Verify
M1 M2 NF1 NF2
READY 4 6 5 1h
SPEC 1 0.5 6 0.5
SPEC/Completed 2 0.5 1 0.5
DEV + Check 2 1 13 4
DEV + Check / 3 6 4 18
Completed
VERIFY 3 7 8 1
VERIFY/Completed 2 5 6 18
Blockages 0.5 1h 2 1h
Total 17 26 43 42
Average cycle days per feature
ACCU 2010
33 / 45
© Schalk W. Cronjé

34. What is wrong?
M1 M2 NF1 NF2
READY 4 6 5 1h
SPEC 1 0.5 6 0.5
SPEC/Completed 2 0.5 1 0.5
DEV + Check 2 1 13 4
DEV + Check / 3 6 4 18
Completed
VERIFY 3 7 8 1
VERIFY/Completed 2 5 6 18
Blockages 0.5 1h 2 1h
Total 17 26 43 42
Average cycle days per feature
ACCU 2010
34 / 45
© Schalk W. Cronjé

35. What is wrong?
Excessive long time between feature availability and testing. Is there enough
testing bandwidth? Or is this team just cheating?
M1 M2 NF1 NF2
READY 4 6 5 1h
SPEC 1 0.5 6 0.5
SPEC/Completed 2 0.5 1 0.5
DEV + Check 2 1 13 4
DEV + Check / 3 6 4 18
Completed
VERIFY 3 7 8 1
VERIFY/Completed 2 5 6 18
Blockages 0.5 1h 2 1h
Total 17 26 43 42
Average cycle days per feature
ACCU 2010
35 / 45
© Schalk W. Cronjé

36. What is wrong?
M1 M2 NF1 NF2
READY 4 6 5 1h
SPEC 1 0.5 6 0.5
SPEC/Completed 2 0.5 1 0.5
DEV + Check 2 1 13 4
DEV + Check / 3 6 4 18
Completed
VERIFY 3 7 8 1
VERIFY/Completed 2 5 6 18
Blockages 0.5 1h 2 1h
Total 17 26 43 42
Average cycle days per feature
ACCU 2010
36 / 45
© Schalk W. Cronjé

37. What is wrong?
Is this 2 days due to re-work? Is there a problem in the development phase?
M1 M2 NF1 NF2
READY 4 6 5 1h
SPEC 1 0.5 6 0.5
SPEC/Completed 2 0.5 1 0.5
DEV + Check 2 1 13 4
DEV + Check / 3 6 4 18
Completed
VERIFY 3 7 8 1
VERIFY/Completed 2 5 6 18
Blockages 0.5 1h 2 1h
Total 17 26 43 42
Average cycle days per feature
ACCU 2010
37 / 45
© Schalk W. Cronjé

38. How do these teams compare?
M1 M2 NF1 NF2
READY 4 6 5 1h
SPEC 1 0.5 6 0.5
SPEC/Completed 2 0.5 1 0.5
DEV + Check 2 1 13 4
DEV + Check / 3 6 4 18
Completed
VERIFY 3 7 8 1
VERIFY/Completed 2 5 6 18
Blockages 0.5 1h 2 1h
Total 17 26 43 42
Average cycle days per feature
ACCU 2010
38 / 45
© Schalk W. Cronjé

39. Exercise
● Analyse the previous listed teams and
comment on their process (with liberty)
● Is it possible to identify points where cost of
delay can be calculated?
● How would you identify the holding costs and
transaction costs?
● Can you improve testing/quality control and
what is the expected and hidden costs?
ACCU 2010
39 / 45
© Schalk W. Cronjé

40. The End Game
ACCU 2010
40 / 45
© Schalk W. Cronjé

41. The institution hits back
speaking at public
Societal Institutions conferences,
success publishing articles
propagates from
known champions
slow diffusion
or no diffusion
Organisational Fields
invent,
negotiate
diffuse,
impose
attempting to
Organisations introduce change
sanction
behaviour
imposing existing
failed processes
Actors
Limitations of cognitive / social rationality (groups/individuals)
Selective perception
ACCU 2010
41 / 45
© Schalk W. Cronjé

42. Making Testing more Agile
● Reduce learning time - bring test specification
forward
● Pair up developers and testers
● Be intelligent about what is being automated.
ACCU 2010
42 / 45
© Schalk W. Cronjé

43. Investing in Agile Testing
● Building infrastructure
● Good testers demand equivalent
remuneration to good programmers
● Put measurements in place
● Cross-train people to create T-skilled sets
● Smooth the Flow
● Recognise that transition will take months,
even a year
ACCU 2010
43 / 45
© Schalk W. Cronjé

44. Cost of Agile Testing
● Appraisal costs
● Internal failure costs
● External failure costs
● Prevention costs
– Checks against external deliverables
– Automating checks (TDD)
– Building the capacity for addressing critical
issues without too much interruption to other
development.
– Training & Practising the mind-set
ACCU 2010
44 / 45
© Schalk W. Cronjé

45. Summary
● Balanced teams reduce costs
● Shared problem-solving can reduce feature
cycle time
● Reduced cycle time will have a direct
influence on delay cost.
● Following a process does not make you
testing agile
● Measure to improve
● You will operate within an institutional context
ACCU 2010
45 / 45
© Schalk W. Cronjé