Gain a deeper understanding of what Exploratory Testing (ET) is, the essential elements of the practice with practical tips and techniques, and finally, ideas for integrating ET into the cadence of an agile process

1. Dare to Explore…
Discover ET!

2. Outline
• What Exploratory Testing (ET) is and isn’t
• Dissecting Exploratory Testing
– Learning
– Designing
– Executing
• Fitting it within the Agile Context
• Adoption Experience
• Q & A

3. What does Exploratory
Testing mean to YOU?

4. Testing Pyramid
Integration
Tests
Unit Tests
Developer-centric
Are we building the
code right?
Business-centric
Are we building the
right code?
Adaptation of Mike Cohn's test automation pyramid
Exploratory
Testing
GUI
Tests
Acceptance Tests
(API layer)
verification

5. What is Exploratory Testing?
Test-related learning
Test design
Test execution
Test result interpretation

6. Exploratory Testing is a mindset
Embrace
continuous
improvement of
test quality
Enabling choice, not
constraining it
Tests are experiments
for learning
Push the
boundaries,
don’t execute
routine checks
Embrace
continuous
learning

7. Exploratory Testing…
IS NOT IS
Doing random stuff to see
what happens
A rigorous investigative
practice
Repetitive Adaptive
Replacement for
predesigned testing
Complimentary to
conventional techniques
About producing
comprehensive test cases
for later use
Creative & more free-style

8. Why ET?

9. Two sides of Testing
Tested
Checked
(scripted manual +
automated)
Explored
“No matter how many tests we write, no matter
how many cases we execute, we always find the
most serious bugs when we go off the script”
Explore It! Reduce Risk and Increase Confidence with Exploratory Testing by Elizabeth
Hendrickson

10. Checking
• Designing scripts with
expected results upfront
• Verifies what was working
still works
• Well documented and
repeatable
Exploring
• Learning, designing and
executing at the same time
• Tests the system
capabilities, boundaries
• Can be sparsely
documented and hard to
repeat

11. Exploration with a general mission without specific step-
by-step instructions
Effectiveness relies on the tester’s knowledge, skills,
and experience
Simultaneously learning about the software under test
while designing and executing tests, using feedback
from the last test to inform the next.
Characteristics of ET

12. ET is a fit where…
• Need for manual testing approach that’s adaptable
because the SUT changes rapidly
• Requirements change often, and the specifications are
vague or incomplete
• Writing detailed test scripts for a manual test effort
doesn’t make sense
• Need to provide rapid feedback on a new product or
feature

13. Key testing problem
• We must test a product whose purpose is
emerging and/or changing
• Coded without fully specified
requirements upfront
Agile fit?

14. ET: Nuts & Bolts

15. The Flow…
• Focus the exploration
• Learn about the system/features
• Design tests with interesting
variations
• Attack and report
Unscripted doesn’t mean unprepared

16. Test charters bring Focus
Charters define the area(s) we are testing
and the kind of vulnerabilities we’re looking
for
- Features
- Suspected risky areas of the system
- Coupling points
Meandering without direction/purpose is
getting lost

17. Sample Charter
• Target: What are you exploring? Feature, component, interface etc.
• Resources: Data set, technique, configuration etc.
• Information: What are you hoping to find? Is it security
vulnerability, performance, capability, usability etc.
Explore <target>
With <resources>
To discover <information>
Explore It! Reduce Risk and Increase Confidence with Exploratory Testing by Elizabeth Hendrickson
Explore the Foo Form Fee
Payment functionality
using active accounts in the
Demo environment
to discover how the payment
interface responds.
Template Example

18. Learn the general shape
What does the system do?
Input, Store, Compute, Output
“I know that I do not know” ~ Socrates

19. Design tests with interesting
variations
The narrower the view, the wider the ignorance
• Varying sequences of actions and varying timing
• Test error handling by making required resources
unavailable or locked, or to break connections
• Inventing personae to generate extreme scenarios
• ...

20. Use heuristics to guide test
design on the fly
• Test Heuristics Cheat Sheet:
(Elisabeth Hendrickson, James Lyndsay, and Dale Emery)
http://testobsessed.com/wp-content/uploads/2011/04/
testheuristicscheatsheetv1.pdf
• How To Break Software: “17 Attacks”
(Alan A. Jorgensen, James A. Whittaker)

21. Ideas for Executing & Recording
All code is guilty, until proven innocent.
• Explore Early, Explore Often
• Time-box the session
• Line up the team
• Surfacing potential problems/risks quickly is the goal
• Work in pairs or groups, whenever possible
• Be distracted by anomalies and new ideas
• Avoid test repetition
• Document your findings in an appropriate way (low
overhead is key)

22. Running ET Sessions

23. Moving Quality Concerns
Upstream
Previous State
Bug bash and IV&V executed just before the release
Target State
Using a whole team approach to exploring early and
often, we hope to surface and address risks sooner, and
that can lead to better software, fewer surprises, and
greater confidence

24. Approach
• Synchronized Cadence across 8 Agile
teams
• Test Charter Autonomy
• Time-boxed Exploratory Sessions
- Explore & Record Observations
- Summarize and Report

25. Current
Enterprise
Comprised
of
many
disparate
data
sources
and
systems,
which
leads
to:
Inconsistent
Data
Data
is
defined,
captured
and
interpreted
inconsistently
throughout
the
Enterprise
Ad-­‐Hoc
and
Near
Real-­‐Time
Analysis
Limita<ons
Limited
ability
to
conduct
ad-­‐hoc
and
near
real-­‐>me
analysis,
delaying
>mely
decision-­‐making
Cross-­‐Func<onal
Domain
Limita<ons
Inability
to
provide
a
cross-­‐func>onal
view
across
data
domains
easily
or
completely
Lack
of
Agility
in
Response
to
Evolving
Business
Model
Lack
of
agility
to
adjust
data
procurement
and
delivery
in
response
to
a
rapidly
evolving
business
model
Lack
of
Data
Across
the
Enterprise
Data
gaps
exist
due
to
limited
understanding
of
current
and
an>cipated
data
usage
and
applica>on
across
the
Enterprise
Notes From
the Trenches
Challenges and
solutions transitioning
a traditional test team
into an exploratory
test team on a large,
heavily interfaced
system.
I’ve a feeling we’re not in Kansas anymore
Challenge: Changing a mindset
Solution: Framework, cadence, training, discussions, and metrics
Why do you want access to my system?
Challenge: You need “visas” to explore
Solution: You need allies, and strong ones
Yet another build!!!?
Challenge: Continuous Delivery means continuous deployments
Solution: Test in parallel with and after production deployments
What is truth?
Challenge: Acceptance criteria vs. Desired Outcome
Solution: Find someone who cares
Boldly going where no one has gone before
Challenge: Indigenous population may be indigent
Solution: “Marketing” brochure and chocolate chip cookies

26. Concluding thoughts

27. Benefits
Unleashes tester’s creativity and skill
Ability to perform focused testing without
comprehensive documentation
Rapid flow of feedback to the team
Based
on
the
summary
by
Itkonen
and
Rau>ainen

28. Challenges
Managing test coverage can be difficult
Quality of testing highly dependent on
expertise and skill of tester
Repeatability of found defects challenging at
times

29. Summary
• Surface and address risks sooner
• Can help identify the big bugs
• Adaptive
• Complementary to scripted testing
• Agile fit

30. 30

31. Raj Indugula
raj.indugula@lithespeed.com
John Hughes
jhughes@bstonetech.com

32. Resources and References
• Bach, James. “What Is Exploratory Testing?”
• Marick, Brian. “A Survey of Exploratory Testing”
• Hendrickson, Elisabeth. “Explore It!”
• Jorgensen, Alan. “How to Break Software (with examples)”
• Kaner, Cam, Tinkham, Andy. “Learning Styles and Exploratory
Testing”
• Bolton, Michael, et al. “Exploratory Testing Dynamics”
• Itkonen, Juha. “Do test cases really matter? An experiment
comparing test case based and exploratory testing”
32