Talk given at Agile Testing Days USA 2022
You know your own product team needs to improve when it comes to testing and quality, yet you are not sure how to tackle this challenge? Many teams out there feel the need to grow a quality mindset, considering this to be a major stumbling block in their environments. At my current company, tech leadership had a clear mission: improve the testing and quality culture of our product teams by leveling up knowledge, skills and practices. The problem: how to get there?
Experiments to the rescue! In a complex environment like ours, no one could tell what would work, so we had to try things out. I designed an experiment to increase transparency, raise awareness, and especially drive experimentation in individual teams to grow a quality culture. Together with a newly formed working group, I supported four pilot teams to design, run and evaluate their own measurable experiments in their own specific context. These teams tried various approaches like exploratory testing, pair and ensemble testing, or documenting architectural decisions.
In this talk I will share our lessons learned, the outcome and impact of these experiments, and what influenced the next experiments to try. Get inspired by our continuous journey towards making experiments a part of team culture across the company!
Takeaways:
- Understand why small, frugal experiments are crucial in complex environments
- Learn how to design your own measurable experiments, in your team or globally
- Get tips on how to run and evaluate experiments
- Gain insights on how to learn from experiment results and decide on next steps
- Start a movement towards an experiment-driven quality culture
20. OUR HYPOTHESIS
With an experimental group of 4 product teams, we believe that
• creating transparency where they currently are regarding testing and
quality as well as identifying problems,
• raising awareness and knowledge of the options at hand to probe the
problems, and
• running first experiments to learn what works in their context to enable
them to rinse and repeat,
will result in increased maturity of each team with regards to testing and
quality.
…
@lisihocke
21. OUR HYPOTHESIS
…
We'll know we have succeeded when each of the 4 teams
• has a clear, shared test strategy in place reflecting the current
situation as well as identified problems,
• has gained further knowledge and skills regarding testing and quality
through a workshop, and
• has adopted 1 new practice in their testing and quality culture that
proved valuable in their context.
@lisihocke
23. EXIT CRITERIA
The experiment will stop and be re-evaluated in the following cases,
whichever comes first:
• No problem there! We find out that the chosen product teams
already have transparency, awareness and experiments in place to
improve their testing and quality.
• Yay, success :) The hypothesis success criteria are fulfilled as defined
above.
• Didn't work out - let's rethink it. It is November 1, 2019 (even
though success criteria are not fulfilled).
@lisihocke
27. @lisihocke
THE HIGH LEVEL PLAN
Find contributors
Align among contributors
Select product teams as pilot teams
28. @lisihocke
THE HIGH LEVEL PLAN
Find contributors
Align among contributors
Select product teams as pilot teams
Work with each team
29. @lisihocke
THE HIGH LEVEL PLAN
Find contributors
Align among contributors
Select product teams as pilot teams
Work with each team
Evaluate the hypothesis success criteria
35. @lisihocke
COMMUNICATION GUIDELINES TO APPROACH THE
PRODUCT TEAMS
Appreciate what's working.
It's an opportunity to learn.
Let's improve together.
It's about you, the team!
36. @lisihocke
COMMUNICATION GUIDELINES TO APPROACH THE
PRODUCT TEAMS
Appreciate what's working.
It's an opportunity to learn.
Let's improve together.
It's about you, the team!
We support you, custom-tailored to your needs.
37. @lisihocke
COMMUNICATION GUIDELINES TO APPROACH THE
PRODUCT TEAMS
Appreciate what's working.
It's an opportunity to learn.
Let's improve together.
It's about you, the team!
We support you, custom-tailored to your needs.
Senior tech leadership has our back!
57. @lisihocke
TEAM EXPERIMENTS
Review the currently known challenges
Pick one challenge to tackle next
Brainstorm potential solutions
Pick one option to try out
58. @lisihocke
TEAM EXPERIMENTS
Review the currently known challenges
Pick one challenge to tackle next
Brainstorm potential solutions
Pick one option to try out
Design the experiment
59. @lisihocke
TEAM EXPERIMENTS
Review the currently known challenges
Pick one challenge to tackle next
Brainstorm potential solutions
Pick one option to try out
Design the experiment
Run the experiment
60. @lisihocke
TEAM EXPERIMENTS
Review the currently known challenges
Pick one challenge to tackle next
Brainstorm potential solutions
Pick one option to try out
Design the experiment
Run the experiment
Evaluate the experiment
61. @lisihocke
TEAM EXPERIMENTS
Challenge: <short challenge description>
Hypothesis:
We believe that <doing X>
will result in <desired outcome>.
We'll know we have succeeded when <measurable criterion is
achieved>.
Experiment: <concrete details like planned actions, experiment run time,
exit criteria, etc.>
65. All 4 teams have a clear, shared test strategy in place
reflecting the current situation as well as identified problems
HYPOTHESIS EVALUATION
@lisihocke
66. All 4 teams have a clear, shared test strategy in place
reflecting the current situation as well as identified problems
All 4 teams have a clear, shared test strategy in place
reflecting the current situation as well as identified problems
→ YES ☺
HYPOTHESIS EVALUATION
@lisihocke
67. All 4 teams have a clear, shared test strategy in place
reflecting the current situation as well as identified problems
All 4 teams have a clear, shared test strategy in place
reflecting the current situation as well as identified problems
→ YES ☺
All 4 teams have gained further knowledge and skills
regarding testing and quality through a workshop
HYPOTHESIS EVALUATION
@lisihocke
68. All 4 teams have a clear, shared test strategy in place
reflecting the current situation as well as identified problems
All 4 teams have a clear, shared test strategy in place
reflecting the current situation as well as identified problems
→ YES ☺
All 4 teams have gained further knowledge and skills
regarding testing and quality through a workshop
All 4 teams have gained further knowledge and skills
regarding testing and quality through a workshop
→ YES ☺
HYPOTHESIS EVALUATION
@lisihocke
69. All 4 teams have a clear, shared test strategy in place
reflecting the current situation as well as identified problems
All 4 teams have a clear, shared test strategy in place
reflecting the current situation as well as identified problems
→ YES ☺
All 4 teams have adopted 1 new practice in their testing
and quality culture that proved valuable in their context
All 4 teams have gained further knowledge and skills
regarding testing and quality through a workshop
All 4 teams have gained further knowledge and skills
regarding testing and quality through a workshop
→ YES ☺
HYPOTHESIS EVALUATION
@lisihocke
70. All 4 teams have a clear, shared test strategy in place
reflecting the current situation as well as identified problems
All 4 teams have a clear, shared test strategy in place
reflecting the current situation as well as identified problems
→ YES ☺
All 4 teams have adopted 1 new practice in their testing
and quality culture that proved valuable in their context
All 4 teams have adopted 1 new practice in their testing
and quality culture that proved valuable in their context
→ Only 2
All 4 teams have gained further knowledge and skills
regarding testing and quality through a workshop
All 4 teams have gained further knowledge and skills
regarding testing and quality through a workshop
→ YES ☺
HYPOTHESIS EVALUATION
@lisihocke
73. @lisihocke
IMPACT WE HAD – AND DIDN’T HAVE
☺ Team conversations about testing and quality
☺ Raised awareness and increased knowledge and skills
74. @lisihocke
IMPACT WE HAD – AND DIDN’T HAVE
☺ Team conversations about testing and quality
☺ Raised awareness and increased knowledge and skills
☺ Teams inspired to improve their practices
75. @lisihocke
IMPACT WE HAD – AND DIDN’T HAVE
☺ Team conversations about testing and quality
☺ Raised awareness and increased knowledge and skills
☺ Teams inspired to improve their practices
Not the whole team was on board, silos remained
76. @lisihocke
IMPACT WE HAD – AND DIDN’T HAVE
☺ Team conversations about testing and quality
☺ Raised awareness and increased knowledge and skills
☺ Teams inspired to improve their practices
Not the whole team was on board, silos remained
Not all people opened up for new concepts, misconceptions prevailed
77. @lisihocke
IMPACT WE HAD – AND DIDN’T HAVE
☺ Team conversations about testing and quality
☺ Raised awareness and increased knowledge and skills
☺ Teams inspired to improve their practices
Not the whole team was on board, silos remained
Not all people opened up for new concepts, misconceptions prevailed
Teams struggled with experiments, fell back into everyday business
81. @lisihocke
A SECOND EXPERIMENT
We believe that if we learn about the testing strategy of product teams
and provide them peer feedback regarding our perspective on their
challenges, then this will
• help the teams build increased self-awareness regarding testing and
quality, and
• help us understand our current testing and quality culture.
We'll know we have proven the hypothesis when the teams as well as we
are able to explain the difference between their approach and our
suggestions and why it might differ from expectations.
83. A SECOND PLAN
Align within the initiative group on core values and select product teams
@lisihocke
84. A SECOND PLAN
Align within the initiative group on core values and select product teams
Interview, present and provide feedback on the teams’ testing strategy
@lisihocke
85. A SECOND PLAN
Align within the initiative group on core values and select product teams
Interview, present and provide feedback on the teams’ testing strategy
Evaluate what we learned in this experiment and propose the next steps
@lisihocke
88. @lisihocke
CORE VALUES TO BASE OUR FEEDBACK ON
Quality is Team Responsibility
Tackle the Unknown, Automate the Known
89. @lisihocke
CORE VALUES TO BASE OUR FEEDBACK ON
Quality is Team Responsibility
Tackle the Unknown, Automate the Known
Invest in Quality and Quality Will Work for You
97. @lisihocke
PRODUCT TEAMS: WE NEED YOU TO SOLVE OUR
PROBLEMS!
Or rather: we need support so we become aware of our
options and enablement so we can solve our own problems
101. Or rather: we need to know which impact we want to have in
our specific contexts and what information we need to have it
@lisihocke
TECH LEADS: WE NEED DATA & GLOBAL METRICS!
103. @lisihocke
→ UNDERSTAND NEEDS & FIND HELPFUL
INDICATORS
Experimenting together with tech leads on perceived quality
needs and indicators helping them make better decisions
104. @lisihocke
→ UNDERSTAND NEEDS & FIND HELPFUL
INDICATORS
Experimenting together with tech leads on perceived quality
needs and indicators helping them make better decisions
Shout-out to Michael Kutz, João Proença, Toyer Mamoojee,
Emanuil Slavov and Jenny Bramble
106. @lisihocke
CONTRIBUTORS: WE NEED QUALITY TO START
WITHIN THE TEAMS!
Or rather: we need to change the system so behaviors
contributing to quality get rewarded and valued
108. @lisihocke
→ SHAPE THE SYSTEM & REWARD BEHAVIORS
CONTRIBUTING TO QUALITY
Experimenting on shaping the system through leadership
workshops, building a foundation for quality to emerge
109. @lisihocke
→ SHAPE THE SYSTEM & REWARD BEHAVIORS
CONTRIBUTING TO QUALITY
Experimenting on shaping the system through leadership
workshops, building a foundation for quality to emerge
Shout-out to Shiva Krishnan
111. @lisihocke
QUALITY IS ON ALL OF US
Create transparency on the status quo, pain points and needs
→ The teams, the leadership, the system
112. @lisihocke
QUALITY IS ON ALL OF US
Create transparency on the status quo, pain points and needs
→ The teams, the leadership, the system
Raise awareness regarding the options at hand to improve
113. @lisihocke
QUALITY IS ON ALL OF US
Create transparency on the status quo, pain points and needs
→ The teams, the leadership, the system
Raise awareness regarding the options at hand to improve
Experiment in your context to grow a quality culture - together