10. Minimize # of end-to-end tests
• AATs should be a small portion of your test suite
• Is your new feature entirely new?
• Balance high # of unit tests + selected end-to-end &
acceptance
11. Create them for journeys, not stories
• Too heavy, slow and expensive to create for each story
• Must be tracked outside of stories
18. Start with a story
User story
As an internet user,
I want to search for “ALT.NET”
And get valid results
• Acceptance criteria 1: I get at least 100k results
• Acceptance criteria 2: Results ordered by relevance
…
Scenarios
Feature
23. Communication
• Helps specify behavior of the system in plain text
• Provides a medium for non-tech and devs to agree on
Are we
talking
about
the right
system?
36. When Acceptances Tests catches a bug
• See why bug got through unit/integration tests
• Add unit, integration tests
• Prune AAT?
Unit
37. Concise Specs
• Declarative
• Reuse steps
• Concise, many methods, fat fixtures
Bad:
• Click on Log In button
• Click username box and type ‘myUsername’ and click ‘password’ box and type
‘myPassword’
• Click on link for Transfer Payment
• Click box for amount and type 400
• Click ‘Transfer’ button
• Assert success message
Good:
• Login as ‘MyAccountName’
• Navigate to Transfer Payment page
• Transfer 400 dollars
• Assert success message
38. Organize your tests…
Use Page Objects
Keeps each page’s tests
• Encapsulated
• Readable
• Reusable
39. public class GoogleHomepage: PageBase, IGoogleHomepage {
// Flow for this page
}
Domain
Service
UI
GoogleHomepage
+search(query)
+clickSearch()
+clickGettingLucky Data Access
GoogleHomepageService
+search(query)
+clickSearch()
+clickGettingLucky
IGoogleHomepage
+search(query)
+clickSearch()
+clickGettingLucky
Page
Objects
42. Keeps changes minimal
• Changes in the application should require minimal
changes in the tests, and not in the specs
43. Easier to write new tests
• Constructing page objects
• Can be used to create a DSL
44. One way
[When(@"I log in with the teacher account '(.*)'")]
public void WhenILogInWithTheTeacherAccount(string orgCode)
{
var driver = new FirefixDriver();
driver.Navigate().GoToUrl("http://www.google.com");
var userTb = Driver.FindElement(By.Name("username”));
var passwordTb = Driver.FindElement(By.Id("password"));
userTb.SendKeys(username);
passwordTb.SendKeys(password);
var loggedInLink = Driver.FindElement(By.LinkText("Log Out"));
Assert.That(loggedInLink != null);
}
45. Better way – page objects
public class LogInPage : PageBase
{
public PageBase LogIn(string username, string password)
{
Driver.FindElement(By.Id("UserName")).SendKeys(username);
Driver.FindElement(By.Id("Password")).SendKeys(password);
Driver.FindElement(By.CssSelector("input[type="submit"]")).Click();
return GetInstance<AccountLandingPage>(Driver);
}
public void HasExpectedTab(string tabName)
{
var link = Driver.FindElement(By.LinkText(tabName));
Assert.AreEqual(tabName, link.Text);
}
}
48. Developer tips
• Don't run in same assembly as other tests, so these can
be run separately since they're very slow (i.e. nightly)
• Navigation
• Create different levels of suites depending on depth/level
of feedback desired:
• Smoke, Current iteration/sprint, Regression
• Run AATs as close to the real environment as possible
49. Gherkin
Specs shouldn’t have much setup code
Given I am on the homepage
And I click login
And I enter username of ‘username’ and password of ‘password’
When clicking ‘Log In’
Then take me to my landing page
Given I am on the homepage
When logging in with ‘username’ and ‘password’
Then take me to my landing page
50. UI tests
• Sometimes tests will fail if the page doesn’t have enough
time to load
• Capture screen shots when tests fail
• Run nightly
• Run across multiple machines
51. Anti-patterns
• Developers writing acceptance tests by themselves, for
themselves
• Tying AATs to an implementation - unit tests are
implementation specific, AATs are NOT
• Too much repetition in the gherkin
• Too many UI tests
• UI tests are failing too easily when changing the UI
53. Where we can be
• Reliable tests
• Reusable test code (even a DSL)
• Separation of concerns
• Short and clear tests
• Fast test creation
• Quick regression feedback
• Tests that are understood by non-technical people
• Fewer bugs
54. Resources:
Tools used here:
• SpecFlow – specflow.org
• PhantomJS – phantomjs.org
• Selenium WebDriver – seleniumhq.org
Books:
• Specification By Example, Gojko Adzic
• Continuous Delivery, Jez Humble, David Farley
• Growing Object-Oriented Software, Guided By Tests, Steve Freeman, Nat Pryce
Articles:
• Automated Acceptance Tests, http://www.thoughtworks.com/insights/articles/automated-acceptance-tests
• BDD with SpecFlow and ASP.NET MVC, http://blog.stevensanderson.com/2010/03/03/behavior-driven-
development-bdd-with-specflow-and-aspnet-mvc/
• Using Gherkin Language in SpecFlow,
https://github.com/techtalk/SpecFlow/wiki/Using-Gherkin-Language-in-SpecFlow
• BDD with SpecFlow, MSDN, http://msdn.microsoft.com/en-us/magazine/gg490346.aspx
• Using SpecFlow with the Page Object, http://blogs.lessthandot.com/index.php/EnterpriseDev/application-lifecycle-
management/using-specflow-to
• Problems with Acceptance Tests, http://xprogramming.com/xpmag/problems-with-acceptance-testing
• Top 10 reasons why teams fail with Acceptance Testing, http://gojko.net/2009/09/24/top-10-reasons-why-teams-
fail-with-acceptance-testing/
• Maintaining Automated Acceptance Tests (ThoughtWorks), http://www.youtube.com/watch?v=uf0EVbH5hdA
• Creating Maintainable Automated Acceptance Test Suites, Jez Humble,
• http://www.youtube.com/watch?v=v-L_2y6g5DI
Intro
My experience with AATs
Why – seen benefits and pitfalls, ways to mitigate the cons
Bad legacy code in mission critical system, not unit testable
…
Useful in general in a test suite for many reasons…
Ask Q’s
Acceptance Tests
Fits their requirements. The story is complete.
Automated, pipeline, run
Readable by whole spectrum
What kind of tests do you run with your specs?
UI – smoke, sanity, UI goo
Functionality – Integration tests
Run close to production; like a user would.
Mostly unit tests
HARD to maintain, slow SO….
Unit test everything thru JavaScript
Most RELEVANT test cases for AATs (can’t hit all alt paths)
Shouldn’t manually regression test
Tests are written before developer starts
Whoever writes user stories
Dev/Tester/BA
BEFORE iteration starts
All know definition of User Story?
Should be turned into specs before iteration starts (bus & )
-Communication with business stakeholders, BAs, QA
-Unit tests don’t account for everything, seams
-Gaps in coverage, unit tests not hitting right test cases,
-Full regression doesn’t always happen
-Manual testing lapse
Mistakes in testing, non-testing, non-regressing
Allow testers to do more exploratory and higher level things, less boring
One of the things that takes the longest to get software delivered, is making sure everything works functionally and nothing that worked broke