Based on hands-on Agile experience acquired over multiple delivery projects and Agile coaching/consulting assignments, Vatsala and Aman share common Agile Testing dilemmas, and possible solutions, tying them to the principle of "moving testing upstream". Presented at Next Generation Testing conference in Bangalore, India (July 2014).

1. Breaking Free
of
Agile Testing Dilemmas!
Vatsala Singh
Aman King

2. What’s a dilemma?

3. di·lem·ma noun də-ˈle-mə
a situation in which you have to
make a difficult choice
http://www.m-w.com/dictionary/dilemma

4. What makes Agile Testing
a dilemma?

5. Few Reasons
“Continuous delivery of valuable software”
“Welcome changing requirements”
“Face-to-face conversation”
“Working software is the primary measure of progress”

6. Agile Manifesto
February 2001, Utah, USA
http://agilemanifesto.org/principles.html

7. Lean
Muda (無駄)
7 wastes
“Whenever defects
occur, extra costs
are incurred…
should not be
passed on to the
consumer and
should be taken as
a loss.”
2001
2003
http://en.wikipedia.org/wiki/The_Toyota_Way
http://en.wikipedia.org/wiki/Lean_software_development

8. “Cease dependence on inspection to achieve
quality. Eliminate the need for inspection on
a mass basis by building quality into the
product in the first place.”
W. Edwards Deming
(1900 – 1993)
https://www.deming.org/theman/theories/fourteenpoints

9. Cost of Change:
In requirement phase x1
in analysis x10
in design x100
and so on
1981
http://www.academia.edu/1554649/Software_Engineering_Economics_by_Barry_W._Boehm

10. Dilemma!
In essence…
testing a developed product
is already too late?!

11. 0
20
40
60
80
100
120
Requirements Design & Implementation Testing
Cost of Defect
Testing Upstream
Move testing upstream!

12. Analyst
Business
Developer QA
Requirements Testing
Design &
Implementation
Testing
$$$$$$
$$$$$$

13. Testing Upstream
Testing
Testing
Analyst
Business
Developer QA
Requirements Testing
Testing
Design &
Implementation
$$$ $$$$$$ $$$$$$
$$$$$$

14. Iterations
Analyst
Business
Developer QA
Requirements
Design &
Implementation
Testing
Testing
Testing
Testing
$ $$ $$$$

15. Iterations
Analyst
Business
Developer QA
Requirements
Design &
Implementation
Testing
Testing
Testing
Testing
How?
How?
How?

17. Business Facing
Technology Facing

18. Business Facing
Technology Facing
SupportProgramming
CritiqueProduct

19. Agile Testing Quadrants
http://www.exampler.com/old-blog/2003/08/21/#agile-testing-project-1
Brian Marick

20. Quadrant Dilemmas
•Automation?
•Manual?
What
•Requirements?
•Design & Implementation?
•Testing?
When
•Business Analysts?
•Developers?
•QAs?
Who

21. Agile Testing Quadrants
Business Facing
Technology Facing
SupportProgramming
CritiqueProduct
Desk-check
Showcase / UAT
Exploratory Testing
Usability Testing
Performance Tests
Security Tests
Unit Tests
Component Tests
Integration Tests
Paper Prototyping / UX
Functional UI Tests
Acceptance Tests
Automated + Manual Manual
Automated Automated + Manual
What

22. Agile Testing Quadrants
Business Facing
Technology Facing
SupportProgramming
CritiqueProduct
Design & Implementation
Requirements
Design & Implementation
Design & Implementation
Testing
Post-Deployment
Design & Implementation
Testing
Post-Deployment
When

23. Agile Testing Quadrants
Business Facing
Technology Facing
SupportProgramming
CritiqueProduct
Business Analysts
QAs
Developers
QAs
End-users
Business Analysts
QAs
Developers
QAs
Who

24. http://agiletester.ca/
2009

25. End to end –
business facing
Localized –
technology
facing
Test Pyramid
Unit
Service
UI
http://martinfowler.com/bliki/TestPyramid.html

26. Test Pyramid
Broaderscope
Fasterfeedback
Unit
Service
UI

27. Anti-pattern:
Ice-cream Cone
Unit
Service
UI

28. A closer look at
common dilemmas

29. PracticePrinciple
Moving testing
upstream
TDD
ATDD
Effective
manual testing
Timely
feedback

30. Practice SolutionProblem
ATDD

31. Practice SolutionProblem
ATDD
Needs close
collaboration
Slow tests
Test
maintenance
Test data
Flaky tests

32. Practice SolutionProblem
ATDD
Slow tests
Test
maintenance
Test data
Flaky tests
Story kickoff
Design tests
before
development
Definition of
“Done”
No
“stabilization”
phase
Needs close
collaboration

33. Practice SolutionProblem
ATDD
Test
maintenance
Test data
Flaky tests
Needs close
collaboration
Test pyramid
Parallelization
Headless
browser
testing
Slow tests

34. Practice SolutionProblem
ATDD
Slow tests
Test data
Flaky tests
Needs close
collaboration
Test
maintenance
Test pyramid
Tests
consolidation
Test reviewers/
champions

35. Practice SolutionProblem
ATDD
Slow tests
Test
maintenance
Flaky tests
Needs close
collaboration
Test data
Subset of
prod data
Setup/
teardown
scripts

36. Practice SolutionProblem
ATDD
Slow tests
Test
maintenance
Test data
Needs close
collaboration
Flaky tests
Conditional
waits
Test data
availability
Dependent
systems check
Stubbed
dependencies

37. Practice SolutionProblem
Efficient
manual testing

38. Practice SolutionProblem
Efficient
manual testing
Balance
automation vs
manual
Too many
manual
scenarios
Lack of
domain
knowledge

39. Practice SolutionProblem
Efficient
manual testing
Balance
automation vs
manual
Too many
manual
scenarios
Lack of
domain
knowledge
Devs
collaborate on
automation
Manual test
what cannot
be automated
Automate in
parallel to
development
Dev : QA ratio

40. Practice SolutionProblem
Efficient
manual testing
Balance
automation vs
manual
Too many
manual
scenarios
Lack of
domain
knowledge
Scope to
current
iteration
Consolidate
manual
scenarios
More
automation

41. Practice SolutionProblem
Efficient
manual testing
Balance
automation vs
manual
Too many
manual
scenarios
Lack of
domain
knowledge
Pair with
Business
Analyst
Run Client
Showcases
Training

42. Practice SolutionProblem
Timely
feedback

43. Practice SolutionProblem
Missing CI
practices
Lack of CI
environments
“Scheduled”
CI builds
Timely
feedback
Delayed
testing

44. Practice SolutionProblem
Lack of CI
environments
“Scheduled”
CI builds
Timely
feedback
Delayed
testing
CI tool setup
Training
Automate in
parallel to
development
Missing CI
practices

45. Practice SolutionProblem
“Scheduled”
CI builds
Timely
feedback
Delayed
testing
Provision
dedicated VMs
Infrastructure
as Code
Database
schema
on demand
Missing CI
practices
Lack of CI
environments

46. Practice SolutionProblem
Timely
feedback
Delayed
testing
Trigger build
per commit
Pull out slow
stages into
parallel builds
Composite
builds
Missing CI
practices
Lack of CI
environments
“Scheduled”
CI builds

47. Practice SolutionProblem
Timely
feedback
Manual
desk-check
testing
CI deployment
pipeline
No
“stabilization”
phase
Missing CI
practices
Lack of CI
environments
“Scheduled”
CI builds
Delayed
testing

48. But how do we know
we’re making progress?

49. Metrics
Customer-reported defect count
Average time between faulty commit and defect detection
Runtime of tests
Lead time of the entire process

50. What’s bothering you
today?
Thank you
slideshare.net/amanking
thoughtworks.com