There's a recording of this talk at Agile 2012 here: http://www.youtube.com/watch?v=v-L_2y6g5DI Creating automated end-to-end functional acceptance tests is hard. Maintaining them over time is harder. Some agilistas even claim that the cost outweighs the benefit. In this lecture we present five principles for creating valuable, maintainable acceptance test suites. We discuss practices such as layering acceptance tests to reduce coupling between the test harness, and talk about how teams should be organized in order to efficiently manage acceptance test driven development. The core of the talk discusses how to manage the evolution of acceptance tests by organizing them as scenarios rather than as suites of story tests. Finally we show how to manage data for acceptance tests.

1. Maintainable Acceptance Tests
Badrinath Janakiraman
Jez Humble
Agile 2012 Dallas
@badrij | badri@thoughtworks.com
@jezhumble | jez@thoughtworks.com
http://thoughtworks-studios.com/
Thursday, August 16, 12

2. what to expect
• Creating high quality acceptance tests
• How to structure a maintainable acceptance test
suite
• Patterns for effective teamwork
• Managing test data
Thursday, August 16, 12

3. what to take away
• Quality is everybody’s responsibility
• High quality test suites are continuously curated
- by testers and developers working together
• Test code needs to receive the same care as
production code
• Exhaustive story-level testing is not a good
basis for maintainable acceptance suites
Thursday, August 16, 12

4. different kinds of tests
Business facing
AUTOMATED MANUAL
Showcases
Support programming
Functional acceptance
Usability testing
tests
Critique project
Exploratory testing
Unit tests Non-functional
Integration tests acceptance tests
System tests (performance, scaling, ...)
AUTOMATED MANUAL / AUTOMATED
Technology facing
Diagram invented by Brian Marick
Thursday, August 16, 12

5. UI
Mike Cohn:
Succeeding with Agile
Service
Unit
Thursday, August 16, 12

6. End to End
Business Facing
Localized
Technology Facing
Thursday, August 16, 12

7. principle 0
Writing good acceptance tests is
hard.
(good: when the tests are green,
we know the software works)
Thursday, August 16, 12

8. mingle
2006 2012
20 tests 3000 tests
500 LOC 50k LOC
2 minutes 12 hours
• actual running time: 55 minutes
• 7-10 times a day
• for 6 years, across 4 offices now
Thursday, August 16, 12

9. why do test suites decay?
for the same reasons code does
we don’t pay enough attention to expressing intent
only testers care about maintaining tests
Thursday, August 16, 12

10. principles
Thursday, August 16, 12

11. principle 1
Tests are first-class citizens of
your project
Thursday, August 16, 12

12. preventing decay in test code
treat test code as production c0de
refactor relentlessly
don’t repeat yourself
don’t repeat yourself
use record-playback tools to build your suite
Thursday, August 16, 12

13. preventing decay of intention
given-when-then is insufficient
separate intention from mechanics
express the test as steps of a user's journey
Thursday, August 16, 12

14. a solution
use natural language to express intentions
use a general purpose programming language to
express test mechanics
use a tool that allows you to operate in either
domain transparently
Thursday, August 16, 12

15. Thursday, August 16, 12

16. page object
https://gist.github.com/3345556
public class LoginPage {
private final SeleniumSession browser;
public LoginPage(Selenium browser){
this.browser = browser;
}
public HomePage loginAs(String user, String password){
browser.type('#login', login);
browser.type('#password', password);
browser.submit('#login-form');
return new HomePage(this.browser);
}
public HomePage loginExpectingError(String user, String password){
browser.type('#login', login);
browser.type('#password', password);
browser.submit('#login-form');
return new LoginPage(this.browser);
}
}
Thursday, August 16, 12

17. Acceptance Criteria
Customer Tester
Test implementation
Developer Tester
Thursday, August 16, 12

18. tester / quality analyst
...is a role, not a person
...is not a failed developer
...advocates for the user and makes the quality of the
system transparent
...should not be primarily working on manual
regression testing
...should be focussed on exploratory testing &
maintaining automated acceptance tests
Thursday, August 16, 12

19. remember
passing acceptance tests are necessary (but
insufficient) for “done”
encapsulate!
the acceptance tests are owned by - and the
responsibility of - the team
Thursday, August 16, 12

20. principle 2
always interact with the system
under test the same way a user
would
Thursday, August 16, 12

21. browser based tests are unreliable
"the test fails in CI, but when I run the app,
everything seems fine"
usually an indication that test mechanics and user
interaction patterns differ
ajax based tests?
JS heavy applications, which need non-zero
processing time to modify the UI
Thursday, August 16, 12

22. some solutions
Test your application the way a user might use it.
Understand when behavior is asynchronous and
account for it explicitly
Don’t use bare sleeps: poll
If it’s hard to write the test, you need to have a
conversation with the team
Thursday, August 16, 12

23. some solutions
wait-utils (https://github.com/itspanzi/WaitUtils)
for ajax tests, if your JS framework provides a pre-
and post-call hook, intercept those to count the
number of active calls before proceeding
Thursday, August 16, 12

24. var AjaxTracker = { https://gist.github.com/3315690
PENDING_REQUESTS: $A([]),
onCreate: function(request){
if (!request.url.match(/gadgets/js//)) {
this.PENDING_REQUESTS.push(request.url);
}
},
onComplete: function(request){
this.PENDING_REQUESTS = this.PENDING_REQUESTS.without(request.url);
},
onException: function(request, exception){
try {
this.onComplete(request);
}catch(e){
if (Prototype.isFireBugsEnabled) {
console.log("Got Exception on request: " + request.url);
console.log(e);
throw(e);
}
}
},
allAjaxComplete: function(includeCardSummary){
var requests = this.PENDING_REQUESTS.reject(function(url) {
return url.match(/cards/card_summary/) || url.match(/also_viewing/);
});
return requests.size() == 0;
}
};
Ajax.Responders.register(AjaxTracker);
Thursday, August 16, 12

25. remember
make time to go back and refactor your
tests
use layers and encapsulation: separate high
level intent and low level mechanics
use page object to interact with SUT; run
against service layer where possible
Thursday, August 16, 12

26. principle 3
continuously curate the structure
of your test suites
Thursday, August 16, 12

27. #1312
#1301 As a... I want... So that...
As a... I want... So that... #1310
As a... I want... So that... Given... Given...
Given... Given... #1306 When... When...
When... When... As a... I want... So that... Then... Then...
Given... Given...
Then... Then... When... #1304
When...
#1315 Given... Given... AsThen...
a... I want... So that...
Then...
When... When...
As a... I want... So that...
Then... Then... Given... Given...
Given... Given... #1308 When... When...
When... When... As a... I want... So that...
Then... Then...
#1307 Then... Then... #1313
As a... I want... So that... Given... Given... As a... I want... So that...
#1317 When... When...
Then...#1311
As a... I want... Then...
So that...
Given... Given... #1303 Given... Given...
When... When... As a... Given... So that... As a... I want... So When...
that... When...
I want... Given...
Then... Then... When... When... Then... Then...
Given... Given...
Given...
Then... Given...
Then... When... When...
When... When...
Then... Then...
Then... Then...
#1305
#1309 As a... I want... So that... #1302
As a... I want... So that... As a... I want... So that...
#1318
Given... Given... I want... So that...
Given... #1316
Given... When...
As a...
When... Given... #1314Given...
When... As a... I want... So that...
When... When... As a... I want... So that...
When...
Then... Then...
Given... Given... Then...
Then... Then... Then...
Given... Given... When... When... Given... Given...
When... When... Then... Then... When... When...
Then... Then... Then... Then...
Thursday, August 16, 12

28. journey tests
#1612
As a customer
I want a gift wrapping option
So that I don’t have to wrap
Buy Product them and post them myself Buy Product
Search product catalogue Search product catalogue
Add product to cart Add product to cart
Check out Check out
Create new account Create new account
Provide address details Provide address details
Provide credit card details Provide credit card details
Complete order Select gift wrapping option
Verify order created Complete order
Verify credit card debited Verify order created
Verify email sent Verify gift wrapping option
Verify credit card debited
Verify email sent
Thursday, August 16, 12

29. some solutions
identify user journeys
( journey: the path a persona takes through the
application to achieve an end goal)
most applications have very few distinct personas
most stories in iterative development are
enhancements to existing journeys
Thursday, August 16, 12

30. features
Basic shopping cart functionality
Searching for a product
- searching for a word should bring up all products which have that word in their name
- searching for a phrase should bring up all products which have any of the words in their
name
- searching for a quoted phrase should bring up all products which have all words in the
the quoted phrase in their name
Paginating search results
- return 25 results per page by default
- if there are fewer than 25 results, do not show pagination links
- provide a "previous" link on every page other than the first page of results
- provide a "next" link on every page other than the last page of results
- if user supplies a page number which is less than 1 in the URL, stay on the first page
- if the user supplies a page number greater than the last page of results, stay on the
last page
Gift-wrap option
Thursday, August 16, 12

31. story tests
Story tests for search
- test that searching for "friends" brings back 782 results
-- results should include how to win friends and influence people
- test that searching for dead friends brings back 8900 results
-- results should include <how to win friends and influence people>
-- results should include <The Zombie Survival Guide: Complete Protection from the
Living Dead>
- test that searching for "dead friends" brings back 57 results
-- results should include <all my friends are dead>
Story tests for pagination
- with a catalog of 3 products, I should see no pagination links
- with a catalog of 25 products, I should see no pagination links
- with a catalog of 26 products, I should see 1 link to page two, along with a next link
but no previous link
- with a catalog of 26 products, on page 2, I should see one product, with a link to
page one, a previous link but no next link
Story tests for gift wrapping
Thursday, August 16, 12

32. journey tests
Journey of user buying a book
- Login as user "bob"
- Search for <"my friends" dead>
- Make sure that 3 pages of results show
- Verify that "All My Friends Are Dead" by "Avery Monson" is on
the first page
- Add two copies of the book to the shopping cart
- Gift wrap one of them
- Proceed to checkout
Thursday, August 16, 12

33. more solutions
extract journeys from your acceptance tests
make them fast and run them first
do test the most likely path that the team, business
and UX folk agree upon
do not test every possible path through the system
extract negative tests and edge cases into a
regression suite which runs after your journey tests
Thursday, August 16, 12

34. build quality in
“Cease dependence on
mass inspection to achieve
quality. Improve the
process and build quality
into the product in the first
place”
W. Edwards Deming
Thursday, August 16, 12

35. why cross-functional teams?
output of testing is not just bug reports
feedback from testing to product design
feedback from test framework to system architecture
developers and testers share knowledge and skills
Thursday, August 16, 12

36. principle 4
everyone owns acceptance tests
Thursday, August 16, 12

37. when acceptance tests break
Triage to find root cause
1. There was an environmental problem
2. There is a bug with the test
3. An assumption changed
4. The test actually caught a bug
Fix the problem
Add a guard to prevent it happening again
Optimise your test suite: detect failures fast
Optimise your process for time to fix tests
Thursday, August 16, 12

38. intermittent failures
flaky tests are worse than useless
quarantine flaky tests - but not forever
http://martinfowler.com/articles/
nonDeterminism.html
Thursday, August 16, 12

39. external systems
not all tests should call the external system
parameterize connections to external systems
Run integration smoke tests before full
acceptance suite
Thursday, August 16, 12

40. impersonator pattern
create a proxy from SUT to external system
cache results from integration smoke tests
run integration smoke tests before acceptance suite
periodically expire cache
only run acceptance suite if integration smoke tests
pass!
Thursday, August 16, 12

41. principle 5
acceptance tests are responsible
for managing their own test data
Thursday, August 16, 12

42. test data
Test-specific data
Test reference data
Application reference data
Ensure tests are independent
Don’t use production data dumps (except for
performance testing and staging)
Thursday, August 16, 12

43. recap
1. treat acceptance tests like production code
2. always interact with the SUT like a user would
3. continuously curate your user journeys
4. collective ownership of acceptance tests
5. acceptance tests own their data
Thursday, August 16, 12

44. take-aways
• quality is everybody’s responsibility
• high quality test suites are continuously curated
- by testers and developers working together
• test code needs to receive the same care as
production code
• exhaustive story-level testing is not a good
basis for maintainable acceptance suites
Thursday, August 16, 12

45. questions
@badrij | badri@thoughtworks.com
@jezhumble | jez@thoughtworks.com
http://continuousdelivery.com/
© 2011 ThoughtWorks, Inc.
http://thoughtworks-studios.com/
Thursday, August 16, 12