Many teams struggle with the implementation of user story acceptance criteria and having a shared understanding about the expected story outcomes. This often results in missed stakeholder expectations, ad-hoc assumptions made by the team during implementation and conflict between team members and the product owner around testing.
This session shows how specification-by-example and acceptance test driven development will address team conflict, missed stakeholder expectations and overall increasing the level of clarity on the project end-to-end. The presentation will cover the theory behind ATDD and case-studies and practical experience from real projects.
The talk was held at the ALM summit 3 in Redmond, January 2013. Recording of the talk can be found here: http://channel9.msdn.com/Events/ALM-Summit/ALM-Summit-3/Implementing-ATDD-and-Specification-By-Example
7. Purpose of user stories?
• Describe user needs
• Unit of prioritization
• Unit of planning
• Reminder for a discussion
• Mechanism to defer detail
8. Requirement levels
Impact Mapping
Story Mapping
Specification-By-Example
Why?
How?
Acceptance
Criterion
Epic
Deliverable, Outcome
Impact, Goal
Easier to define upfront Harder to define upfront
Reminder
for a
discussion
Bug
report
Isolated,
formalized
example
User Activity
User Story
Example
Code
9. Collecting acceptance criteria
“I would try to put a book into
the shopping cart …”
“I would try to remove a book
from the shopping cart…”
“I’d check whether the shopping cart
is empty, when I enter the shop …”
“I would try to add the same book
again to the shopping cart …”
Books can be placed into shopping cart.
Books can be removed from shopping cart.
Shopping cart should be empty when
entering the shop.
Adding same book again to shopping cart
should increase quantity.
As a potential customer
I want to collect books in a shopping cart
So that I can order several books at once.
“Imagine this story is already implemented:
What would you try out?”
11. Examples for user stories
UI wire frames,
existing UI
rules, key examples
existing artifacts,
mock-ups
12. Discussion of acceptance criteria
Original idea for the illustration:George Dinwiddie
http://blog.gdinwidiee.com
public void TestInitialOrderDiscount()
{
Customer newCustomer = new Customer();
Order newOrder = new Order(newCustomer);
newOrder.AddBook(
Catalog.Find(“ISBN-0955683610”)
);
Assert.Equals(33.75,
newOrder.Subtotal);
}
Register as “bart_bookworm”
Go to “/catalog/search”
Enter “ISBN-0955683610”
Click “Search”
Click “Add to Cart”
Click “View Cart”
Verify “Subtotal” is “$33.75”
We would like to encourage new users to buy in our shop.
Therefore we offer 10% discount for their first order.
13. Specification-By-Example
Examples …
– make abstract descriptions
better understandable
– are usually not formally
exchanged or documented
Brian Marick
Examples Tests
Requirements
consist of
describe verify
fulfillment of
14. Discussion of acceptance criteria
Original idea for the illustration:George Dinwiddie
http://blog.gdinwidiee.com
public void TestInitialOrderDiscount()
{
Customer newCustomer = new Customer();
Order newOrder = new Order(newCustomer);
newOrder.AddBook(
Catalog.Find(“ISBN-0955683610”)
);
Assert.Equals(33.75,
newOrder.Subtotal);
}
Register as “bart_bookworm”
Go to “/catalog/search”
Enter “ISBN-0955683610”
Click “Search”
Click “Add to Cart”
Click “View Cart”
Verify “Subtotal” is “$33.75”
We would like to encourage new users to buy in our shop.
Therefore we offer 10% discount for their first order.
15. Illustrated with shared examples
Original idea for the illustration:George Dinwiddie
http://blog.gdinwidiee.com
Given the user has not ordered yet
When the user adds a book with the price of USD 37.50 into the shopping cart
Then the shopping cart sub-total should be USD 33.75.
16. Discover new aspects
Original idea for the illustration:George Dinwiddie
http://blog.gdinwidiee.com
Actually, this not quite right:
Books on sale should be excluded.
18. Abstract acceptance criteria
As a shop visitor
I want to collect books in my shopping basket
so that I can purchase multiple books at once.
Books can be added to the shopping basket
Books can be removed from the shopping basket
Shopping basket is initially empty
The same book can be added multiple times to the shopping basket
19. Formalizing examples
As a shop visitor
I want to collect books in my shopping basket
so that I can purchase multiple books at once.
Books can be added to the shopping basket
Given my shopping basket is empty
When I add the book “Harry Potter” to my shopping basket
Then my shopping basket should contain 1 copy of “Harry Potter”
20. Formalizing examples
As a shop visitor
I want to collect books in my shopping basket
so that I can purchase multiple books at once.
Books can be added to the shopping basketThe same book can be added multiple times to the shopping basket
When I add the book “Harry Potter” to my shopping basket
Then my shopping basket should contain 2 copies of “Harry Potter”
Given my shopping basket contains 1 copy of “Harry Potter”
21. Example structure
The same book can be added multiple times to the shopping basket
When I add the book “Harry Potter” to my shopping basket
Then my shopping basket should contain 2 copies of “Harry Potter”
Given my shopping basket contains 1 copy of “Harry Potter”
Title: Describes isolated, focused intention=abstract acceptance criterion
Arrange: Context, initial state of the system
Act: Execution of the feature
Assert: Assertion of observable behavior
And I should see the warning: “Book already existed in basket”
Triple-A
constraint
“Checks”
Steps can be chained up
Evolution of a domain specific
test and specification language.
23. Purpose of examples
• Shared understanding:
acceptance criteria
• Documentation:
system details
• Regression-tests:
violated assumptions
24. Continuous validation
When I add the book “Harry Potter” to my shopping basket
Then my shopping basket should contain 2 copies of “Harry Potter”
Given my shopping basket contains 1 copy of “Harry Potter”
„Step Definitions“ are binding individual steps
to an automatable interface of the application.
Automation does not necessarily have to bind to the UI.
Automation of system is supported/evolving with development.
SystemUI Automation
25. Demo
Gherkin automation for .NET
• Visual Studio plugin (VS-Gallery)
• NuGet Package
http://www.specflow.org
27. Test automation becomes expensive, when …
• trying to automate manual tests
• making tests unreadable when
automating them
• automating only after
completing implementation
structure
readability
when to test
28. Structure
Manual tests
Asserts Multiple combined features
Structure ACT-ASSERT-
ACT-ASSERT-
ACT-ASSERT-
…
Dependent on existence of other features
Long test path with high chance to break
Cause and impact of an error hard to
trace.
Automated Check
Single aspect of a single feature
ARRANGE –
ACT –
ASSERT
Independent of other features
Short test path with lower chance to break
Cause and impact of an error easier to
trace.
30. Readability
// Go to web page 'http://localhost:40001/' using new browser instance
BrowserWindow localhostBrowser = BrowserWindow.Launch(
new System.Uri(this.RecordedMethod1Params.Url));
// Click 'Register found item' link
Mouse.Click(uIRegisterFoundItemHyperlink, new Point(56, 9));
// Click 'Save' button
Mouse.Click(uISaveButton, new Point(44, 14));
int foundItemNo1 = int.Parse(uIFundNr127Pane.InnerText.Substring(9));
// Click 'Register found item' link
Mouse.Click(uIRegisterFoundItemHyperlink, new Point(63, 7));
// Click 'Save' button
Mouse.Click(uISaveButton, new Point(34, 11));
int foundItemNo2 = int.Parse(uIFundNr128Pane.InnerText.Substring(9));
Assert.IsTrue(foundItemNo1 + 1 == foundItemNo2);
// Click 'Close' button
Mouse.Click(uICloseButton, new Point(26, 11));
31. A readable test case
Scenario: New found items should receive a consecutive number for the
current year
Given the previous found item of the
current year had the number 145
When I register a new found item
Then the last found item of the
current year should have the number 146
32. When to test
Acceptance criteria
(ATDD, BDD)
UnitTests
(TDD)
business view
technical view
Exploratory tests
Workflow tests
Performance, Scalability,
Usability,Security, …
definingtheproduct
criticizingtheproduct
New dimension: defining the product
Synergy: Specification of requirements and tests
AgileTesting Quadrants: Brian Marick
33. Manual testing is still necessary!
User
journeys
Acceptance-
criteria
Units
exploratory
testing
Manual Check
after story done
Main
success paths
Undiscovered
acceptance criteria
No/(few) manual
regression checks
Few paths
are enough
More time
for exploration
34. Cross-functional work
Sprint 1 Sprint 2 Sprint 3
short iteration
US4
Plan
Implement &
autom. test
US5
Plan
Implement &
autom. test
US2
Plan
Implement &
autom. test
US3
Plan
Implement &
autom. test
US6
Plan
Implement &
autom. test
US1
Plan
Implement &
autom. test
US7
Plan
Implement &
autom. test
US8
Plan
Implement &
autom. test
US9
Plan
Implement &
autom. test
ExploratoryTests
Specification and test
Collaboration for defining
acceptance criteria
Collaboration for
automation
Preventing bugs
instead of finding them!
Extension of
“Test Cases”
Limit
WIP
Collaboration
in manual
testing
52. Higher level scenario structure
Story Mapping
Why?
How?
Acceptance
Criterion
Epic
Deliverable, Outcome
Impact, Goal
User Activity
User Story
Example
Code
Easier to define upfront Harder to define upfront
53. Story Maps
• Original concept by Jeff Patton
• Support iterative product design
• Optimized for desired outcome or
deliverable the system should support
54. Building story maps
Get popular books
delivered fast and
conveniently
Find
book
Collect
books
Order
Wait for
delivery
Receive
delivery
time
browse
best
sellers
shopping
basket
enter
address
delivery
slip
delivery
notification
pay with
credit
card
search
book by
title
wish list
inquiry
order
status
desired outcome
or deliverable
user
activities
system
features
necessity
55. Walking
skeleton
Enabling build–measure-learn
Get popular books
delivered fast and
conveniently
Find
book
Collect
books
Order
Wait for
delivery
Receive
delivery
time
browse
best
sellers
enter
address
delivery
slip
pay with
credit
card
search
book by
title
wish list
inquiry
order
status
necessity
shopping
basket
delivery
notification
manual
work-
around
omitted
steps
desired outcome
or deliverable
user
activities
system
features
63. User Stories vs. Features
Product/Sprint Backlog
User Story 1
AccCrit 1
AccCrit 2
User Story 2
AccCrit 3
AccCrit 4
Living Documentation
Feature 1
AccCrit 1
AccCrit 2
Feature n
AccCrit 4
AccCrit m
User Story n
AccCrit 5
AccCrit m
AccCrit 3
AccCrit 5
„Done“
• Future options of the system
• Organized/refined according to
priority, value, effort, risk, ...
• Next possible increments of
the product (units of work)
• Current state of the system
• Organized/refined for
functional overview
• Versioned and maintained
together with source code
64. Living documentation
• Link business
readable
automation
from source
code to story
maps
• Feed execution
results into
story maps
• Sync Story
Story Map
requirements
withTFS
work items
66. Collaboration over specification
Documentation after conversation
• Collaborative discovery
• Should trigger new questions
• Infinite number of examples
• Readability and ability to automate
67. Level of automation
Controller
Business Layer
Data Layer
Model
View
Browser automation
Trigger behavior
through controller
Assert behavior
on model, db, ..
Setup pre-conditions
through service
interfaces
Out-of-process
In-process
68. Test execution performance
• Grouping tests
– Current WIP
– Completed work
• Database
– In-memory
– Templates for setup
• Parallel execution
• Smart execution order
• Level of automation
69. Internal vs. external DSL
Example Source: Liz Keogh
https://github.com/lunivore/tictactoe-java/blob/master/
scenarios/com/lunivore/tictactoe/scenarios/Three_in_a_row_wins.java
70. Non-functional acceptance criteria
Given there are 100,000 users registered on the system
When I create a new account
Then I should be taken to my dashboard within 5ms
When 1000 users are hitting the homepage simultaneously
Then each user should get a response within 2ms
Matt Wynne
http://blog.mattwynne.net/2012/03/13/using-cucumber-for-load-testing
77. Questions
Got curious about Specification-By-Example?
Learned something new about using SpecFlow?
Christian Hassa,TechTalk
(ch@techtalk.at, @chr99ha)
Join the BoF Session
„Specification-By-Example (ATDD, BDD) in practice“
Today at 15h15 (right after this talk)