Overall in this speech, I will be detailing the key patterns to build a test assembly unit to support continuous deployment using agile testing process, specifications by examples, specification test automation pyramid etc. During this walk-thru, experience reports/examples will be shared appropriately. a) Raw Materials - The highlight of this section will be the talk on approach towards creating whole specification bundle. Areas like team collaboration, stake-holders involvement, product development stages involved, context to the team etc,. will be covered. b) Processing unit - We follow a filtering mechanism using which the raw material(specifications) gets filtered and consumed during the development process. The talk will be mainly concentrated on the consumption of specifications by different team members like stakeholders, DEVs, QAs and BAs. In this stage, the raw materials get consumed/processed with additional layer of automated tests. test automation pyramid will also be detailed. c) Laboratory tests - This will be aimed at providing a glimpse of the key technique Dev-box testing also known as dev-check, Desk-check etc. This will be followed by an explanation on the best practices to implement this technique and consumption of the raw materials (specifications) in this stage. The topic will also cover the decision tree to move the processed product to different stages. d) Integrated line - Exploratory testing and building safety net, will be implemented as part of this validation. I also wish to take you through the best practices followed. e) Packaging and Delivery - The areas under this section will be test implementation, integration as part of CI and key benefits of following this approach. I will also try to explain the approach towards continuous deployment like weekly, monthly etc.

1. World Conference –
Next Generation Testing
08 – 12 July, Le-Meridien, Bangalore
Specifications
Test Assembly Unit For Enterprise
Testing
www.unicomlearning.com
Sathyan Sethumadhavan
Thoughtworks

2. www.unicomlearning.com
We implemented this for…
• Enterprise publishing house in Brazil
• Ongoing product development - for the last 20
months
• Team size of 70 - Distributed across India and Brazil
• Continuous delivery – 2 weeks deployment
• Platform supports multiple browsers and mobile OS
• High level sanity regression cycle
• Test pyramid – 800+ tests (not including Unit) –
Tests across different layers

3. www.unicomlearning.com
Terms I will be using…
• Specifications with Example – collaborative approach to defining requirements and business-
oriented functional tests - illustrating requirements using realistic examples instead of abstract
• Continuous Delivery - Pattern language used in software development to automate and improve
the process of software delivery
• Developer Box Testing – Developer, tester and BA to come and sit at the developer’s machine to
demo the changes
• Acceptance Tests – Test conducted to determine if the requirements of a specification or contract
are met
• Test driven development - Automated test case that defines a desired improvement or new
function, then produces the minimum amount of code to pass that test
• Exploratory Testing - Approach to software testing that is concisely described as simultaneous
learning, test design and test execution

4. www.unicomlearning.com
The Approach is….
Test Driven
Development
(TDD)
Acceptance Test
Driven Development
(ATDD)
Acceptance
Tests
Specification
s Assembly
Unit
Behavior Driven
Development
(BDD)

5. www.unicomlearning.com
Requirements as Tests
EXAMPLES
REQUIREMENTS
TESTS
Can
Become
Elaborat
e
Verify
Living
Documents
Support
Continuous
Delivery
Confidence on
Product
Quality
~No
Communication
Gap

6. www.unicomlearning.com
Requirements as Tests

7. www.unicomlearning.com
Specifications Test Assembly Unit
Raw
Material
s
Processi
ng Unit
Laborat
ory
Tests
Integrate
d
Pipeline
Packagi
ng and
Delivery

8. www.unicomlearning.com
?????????
• Possibly I didn’t ….
– System Integration points
– End user experience
– Technical architecture

9. www.unicomlearning.com
Raw Materials
Negligible
Requirement
Gaps
Requirements
with No
Ambiguity
All
boundary
flows
covered
Same language used
to detail the
requirements to the
whole team
All Team
Workshop
Three
Amigos
Ad-Hoc
Conversatio
n
Write +
Review

10. www.unicomlearning.com
???????
• Possibly I didn’t….
– Possible data / boundary conditions
– Different user flows
– Performance requirements
– Integration points

11. www.unicomlearning.com
Processing Unit
RSpec + API
(with mocks)
Filtered &
Categorized
Altered
Referenc
ed
Each line of code is
validated using
tests
Boundary
flows/conditions
validated

12. www.unicomlearning.com
?????
• Possibly I could ….
– Validate it early
– Validate the happy paths

13. www.unicomlearning.com
Laboratory Tests
Spinach +
Headless Webkit
RSpec + API
Acceptance
Scenarios
Ad-Hoc
Scenarios
Dev Box
Testing
RSpec +
API
Developer box
validation– Before
each check-in
Early functional
validation with
minimal mocks

14. www.unicomlearning.com
??????
• Possibly I could spent time on….
– Exploring different user flows
– Sanity on different browsers
– Create automated tests, so that I don’t need to repeat the
verification again as product evolves

15. www.unicomlearning.com
Integrated Line
Cucumber +
Capybara Web-
driver
Spinach + Headless
Web-kit
RSpec + API
Exploratory
Testing
End to End
Business
Scenarios
Different
Browsers
Parallel
Pipelines &
Safety Net
Runs against
each successful
build
Parallel test runs
for multi - browser
and UI test-suite
Specifications
referred -
Different user
flows

16. www.unicomlearning.com
Packaging and Delivery
Tests as
part of
Continuous
deployment
No Manual
Regression
Cycle
Tests as Living
Documentation

17. www.unicomlearning.com
Reporting & Metrics
Deployment Readiness
• Test Automation Coverage
• Test run execution
Product Quality
• Root-cause Trend
• Defect Type Trend

18. www.unicomlearning.com
Reporting Contd…

19. www.unicomlearning.com
You could implement this for both…..
Greenfield
Existing
Structure
Waterfall
Model
Agile
Methodologies
Need based
development

20. www.unicomlearning.com
Let’s Re-cap
Process Side
• No translation of requirements throughout
the development cycle
• Fail fast – 95% of functionalities gets
validated before check-in
• Spent more time on exploratory testing
and automation
• Build a test automation pyramid
• Don’t count defects – Analyze/Fix the root
cause
• Tests as living documents
Technical Side
• Refer specifications data to create Unit
testing
• Create test technical stack with a) full
mocks b) less mocks and headless c) No
mocks UI tests
• Continuously run automated tests – be it
a part of continuous integration
• Deploy daily, weekly or bi-weekly

21. www.unicomlearning.com
Possible Framework Stack
Ruby
RSpec
Spinac
h
Cucum
ber
Rest
clientHeadle
ss
web-kit
Web-
driver
Capyb
ara
Calaba
sh
Java
Junit
Cucum
ber-
jvm
Rest
Assure
d
Phanto
mJs
Webdri
ver
Fluentl
enium
Calaba
sh-
driver

22. Tools
• Twist - http://studios.thoughtworks.com/twist-agile-test-
automation/
• Concordion – http://www.concordion.org
• Cucumber – http://cukes.info
• FitNesse - http://fitnesse.org
• Jbehave - http://jbehave.org
• GreenPepper: http://www.greenpeppersoftware.com.
• Robot Framework: http://www.robotframework.org
• SpecFlow: http://www.specflow.org
• TextTest: http://www.texttest.org
www.unicomlearning.com

23. References
① Specification by example - Gojko Adzic
② Projects implemented SDD in real-time
③ http://geekcubo.com/2012/09/specifications-test-
automation-pyramid/
④ http://geekcubo.com/2011/12/specifications-enterprise-
testing/
www.unicomlearning.com

24. Thank You
satnair@thoughtworks.com
sathyan.s.nair@gmail.com
Geekcubo.com/agileworld
World Conference – Next Generation Testing
UNICOM Trainings & Seminars Pvt. Ltd.
contact@unicomlearning.com