Continuous Delivery without Test Automation @STPCon, San Diego
1. Continuous Delivery without
Test Automation
Maaret Pyhäjärvi
Email: <maaret@iki.fi> | Twitter: maaretp
Maaret Pyhäjärvi
Nimeä | Attribution (Finland)
http://creativecommons.org/licenses/by/1.0/fi/
http://creativecommons.org/licenses/by/1.0/fi/deed.en
2. A Bit of Background
• 3 years into a web based (.NET/C#)
product
• 1:8 tester:developer ratio
• Missing agile technical practices:
test automation non-existent
• Scrum-like monthly releases for first
2 years
– Releases late
– Releases not tested well enough
• Negative cycle of failing with
estimates eating team morale
• Jira in a central role with estimates
and burndown
5. The Main Driver for Change:
Testing
Scheduled release
Feature 1
Feature 2
Feature 3
Feature 4
Testing
Feature 1
Feature 2
Feature 3
Feature 4
R1 R2 R3
Done
includes
Tested
Schedule
over
Quality
SCRUM + SCHEDULED DELIVERY with continuous integration
KANBAN + CONTINUOUS DELIVERY
6. Enablers that Made Timing Just
Right
• Henri Karhatsu and #NoEstimates –
experience report
• Availability of Git in Visual Studio without
plugins
• Problems with schedules in Scrum
• Lean and value delivery focus in Product
Management
7. Changing How We Managed Our
Work: Setting up Jira Kanban
Removed
hour &
story point
estimates
Agreed in
WIP for
each
phase
Agreed on
swarming
Updated
issue types
8. Branches and Test
Environments
Feature Test
Environment
Feature branches
on-demand builds
“Developers think
this can be tested”
Development
Test
Environment
Integration branch
build-on-checkin
“Developers think
this can be
released”
Acceptance
Test
Environment
Master branch with
fixes and merges
from integration
on-demand builds
“Release next
morning”
9. Continuous Deployment but
Significant Lead Times
Mon Tue Wed Thu Fri Mon Tue Wed Thu Fri
Change to
Master
Integration
to master
Feature to
integration
Developers drive the decision on what
they want feedback on
11. Exploratory Testing:
Better tests, better testers!
• An approach to software testing
– Emphasized freedom and responsibility
of an individual in a process where
continuous optimization of value of
information is important
• “Any testing process that involves
simultaneous (mutually supporting) learning,
test design, and execution.” – James Bach
& Cem Kaner
• Disciplined, planned and controlled testing
that emphasizes continuous learning
• Research in Finland, Itkonen et al. 2007
– No significant difference in results for
preplanned test cases and exploratory testing
– More false alarms with test cases
– Comparing overall effort: significantly more in
test-case based testing
Unknown
territory
Test-related
learning
Design of new
tests
Test execution
Result
interpretation
11
12. Exploratory Testing Enables
Continuous Deployment
• Every Jira task gets planned for
– Sometimes we go with developer
testing only
– Sometimes we test extensively
• Exploratory Planning
– No set test cases
– Talking to developers and reflecting
to a model of of use created through
earlier explorations
– Actionable information first –principle
– Production monitoring is an option for
getting information
13. Exploratory Testing:
Learning & Modeling
”A day’s work”
Vision (“Sandbox”) Current Charter
Other Charters Details
Bug
Reports
Perception of
quality and
coverage
Quality
ReportDebriefing
Tester
Test
Manager
Past
Results
Obstacles
Outlook
Feelings
?
#
xCharter backlog of the future
testing
Out of
budget
Next in
importance!
#, ?, x, +
20:20:60
Session sheets of the past testing
Idea of
exploration
Metrics
summary
Coachin
g
13
Playbooks
Coverage outlines
14. Changes
• Active discussion about schedules and
merging, and needs of testing in the
branches
• More pairing for testing the features
• More group work on defining the features
• Introducing Flowdock due to increased
need to collaborate; integrating logs,
emails
• Focus on getting better (scope of test
automation; refactoring; pairing and group
work; individuals’ skills)
15. To End This With: Key
Observations on Ideas that Guide
Our Test Design
16. Testers don’t break
your code, they
break your illusions
about the code.
-- adapted from James Bach
Notes de l'éditeur
Mention background, Altom and seeking work in San Diego area for love.
Fast as in few hours. Fast is embedded in responding to change: some change comes only from testing the value in real production environment.
Just saying we can take best of the context we have and start living to the values.
Value in production
With 9 developers, it’s possible to end up having a feature per day ready for production…
We want it to work in production. And with developer-tester perspectives, we’ve managed to make that true.
We had recruited a tester because we knew (the devs) that they couldn’t see things from all the necessary angles. So they had asked for the feedback.
Mention: exploratory regression testing, performance testing and test automation.
Happy agile team with satisfied customers with collaboration and smart exploratory testing, not automation
Remember to talk about tacit knowledge and the fact that there’s skills and heuristics that are teachable that testers tend to specialize on…
Automation: db tests, selenium, unit tests
The way we actually test is still the same: there’s no agile testing, there’s just testing in agile context.
Happy agile team with satisfied customers with collaboration and smart exploratory testing, not automation