To support efficiently ATDD or BDD at the speed and scale required in Agile and DevOps projects, your acceptance criteria need to: - Be understandable by business experts, Product owners, developers (defining the business terminology) - Facilitates test authoring as well as maintenance (based on a semantic and not just text) - Can be transformed automatically into code for test execution

1. Laurent PY
CEO, Smartesting
Laurent.py@smartesting.com
@py_laurent
www.smartesting.com
Acceptance testing
in DevOps

2. Our long and painful
journey
towards DevOps

3. Overview of our development process 2004/06

Product: CertifyIt, eclipse plug-in in JAVA (Model-Based-Testing)
– Waterfall process
– 1 release every 6 months
– Few tests and no TDD
– 1 month (x5 people) to do release testing before
deployment
– Poor quality impacting customers’ feedback and adoption
3

4. Tasting agility end of 2006

Product: CertifyIt, eclipse plug-in in JAVA (Model-Based-Testing)
– First XP and then Scrum process
– We do TDD
– Continuous integration (code & unit test)
– 1 customer release every 3 months (1 operation release every 3 weeks)
– 1 man/month to do release testing before deployment
– Good feedback from customers!
4

5. Now we are experiencing DevOps - 2012

Product: Zest, testing platform in the cloud
– Still Scrum process
– We do TDD and…
– …Acceptance Testing Driven Development (ATDD), 100% automated
– Acceptance tests are part of the CI process
– We do several deployments a day (≈10)
5

6. What we’ve learnt

Shorter iterations (1 to 4 weeks) lead to
massive test automation completed by
exploratory testing

Acceptance tests become the specification

Testing starts earlier in the development
process: Shift left!
Req
Management
& Definition
Test
Planning
Execution
Defect
management

7. What we’ve learnt

To achieve this level of speed (DevOps context),
acceptance tests should be:
– Understandable to both developers and business
experts to strengthen alignment and enable the ‘shift
left’
– Maintainable to manage efficiently changes in
requirements and keep pace with continuous
deployment
– Automated to enable rapid execution and feed-back

ATDD with Business Domain Language
and refactoring capabilities
7

8. ATDD & Refactoring
My view as product owner & tester:
Acceptance tests need to be continually
reviewed and refactored just like code!!!

9. ATDD & Refactoring
My view as product owner & tester:
Acceptance tests need to be continually
reviewed and refactored just like code!!!
Martin Fowler

10. Acceptance testing
driven development

11. Shift left
Daily meeting
Product Backlog
Sprint Backlog
Product at the end of the
iteration
Sprint
1 to 4 weeks
Acceptance testing
Shift left
Acceptance testing
Scrum team
PO
developers
testers
Scrum
master

12. Acceptance Testing Driven Development (ATDD)

Acceptance test is a powerful artifact to improve
communication

Test as the definition of ‘STOP’

Written prior to development by tester

Based on a business DSL (domain specific Language)

Confirmed with stakeholders

Mostly automated
Test in natural language
Test fixture
Code

13. Acceptance Testing Driven Development (ATDD)

Benefits:
– Improve communication and collaboration between project
stakeholders
– Shared understanding of what a successful implementation
means
– Better coverage of business expectations
– Faster feed back

Challenges:
– New methodology that requires rigor and discipline
– Find the right balance between people/process/tool

14. Continuous acceptance testing

15. Zest: create acceptance tests with DSL

Key features:
–
–
Suggestion to promote the reuse of Action Word and avoid duplication
–
Refactoring: when an Action Word is modified, impacts are performed automatically across
the tests
–
Optimization: inspection features enable to continuously optimize the acceptance tests
–

Define progressively your Action Words and your business domain language for test
authoring
Create scripts and accelerate test automation (Ruby, Java, XML…)
Integrations with:
Watir, Appium…
15

16. Collaboration through acceptance tests & DSL
Tester
Create acceptance
Tests
Product Owner
Validate acceptance
tests
Developer
Automate acceptance
tests
• DSL
• Refactoring capabilities
16

17. Define new business entities…
Define progressively your business terminology with
Action Words. Enable collaboration based on acceptance
tests.

18. …or define business entities right from the tests
Promote test steps into Action Words (what developers
call extract function). This is refactoring!

19. Evils of duplication

20. A fundamental principle

21. Reuse, reuse and reuse Action Words!
Suggestions
Create and maintain consistent scenarios for
your project

22. Analyse and optimize continuously your tests
When duplications are identified, refactoring
options are suggested!

23. Add, remove, modify Action Words and Scenarios
Add parameters to Action Word
Propagate automatically to
the scenarios
Test refactoring enables to perform automatically
impact analysis and test maintenance tasks

24. Generate scripts
Use of Action Words significantly decrease the
cost of automation and accelerate the overall testing
cycle

25. So your acceptance tests are…
Understandable
Definition of business domain tests enable
better alignment of the team
• DSL
• Refactoring capabilities
Can be automated
Good design based on Action Words
streamlines the test automation phase
Maintainable
Refactoring and optimization features
dramatically accelerate maintenance

26. And remember
Acceptance tests need to be
continually reviewed and
refactored just like code!!!

27. Give a try
Zest-testing.com
Laurent PY
CEO, Smartesting
Laurent.py@smartesting.com
@py_laurent
www.smartesting.com