Growing an Experiment-driven Quality Culture (TestIstanbul Conference 2021)
•
0 likes•139 views
Keynote given at TestIstanbul Conference 2021 You know your own product team needs to improve when it comes to testing and quality, yet you are not sure how to tackle this challenge? Many teams out there feel the need to grow a quality mindset, considering this to be a major stumbling block in their environments. At my current company, tech leadership had a clear mission: improve the testing and quality culture of our product teams by leveling up knowledge, skills and practices. The problem: how to get there? Experiments to the rescue! In a complex environment like ours, no one could tell what would work, so we had to try things out. I designed an experiment to increase transparency, raise awareness, and especially drive experimentation in individual teams to grow a quality culture. Together with a newly formed working group, I supported four pilot teams to design, run and evaluate their own measurable experiments in their own specific context. These teams tried various approaches like exploratory testing, pair and ensemble testing, or documenting architectural decisions. In this talk I will share our lessons learned, the outcome and impact of these experiments, and what influenced the next experiments to try. Get inspired by our continuous journey towards making experiments a part of team culture across the company! Takeaways: - Understand why small, frugal experiments are crucial in complex environments - Learn how to design your own measurable experiments, in your team or globally - Get tips on how to run and evaluate experiments - Gain insights on how to learn from experiment results and decide on next steps - Start a movement towards an experiment-driven quality culture
Recommended
Recommended
More Related Content
Similar to Growing an Experiment-driven Quality Culture (TestIstanbul Conference 2021)
Similar to Growing an Experiment-driven Quality Culture (TestIstanbul Conference 2021) (20)
More from Lisi Hocke
More from Lisi Hocke (18)
Recently uploaded
Recently uploaded (20)
Growing an Experiment-driven Quality Culture (TestIstanbul Conference 2021)
- 1. GROWING AN EXPERIMENT-DRIVEN QUALITY CULTURE LISI HOCKE @lisihocke
- 2. THIS IS A STORY OF EXPERIMENTATION @lisihocke
- 3. THIS IS A STORY OF EXPERIMENTATION @lisihocke
- 4. THIS IS A STORY OF EXPERIMENTATION @lisihocke
- 5. THIS IS A STORY OF EXPERIMENTATION @lisihocke
- 6. THIS IS A STORY OF EXPERIMENTATION @lisihocke
- 7. THIS IS A STORY OF EXPERIMENTATION @lisihocke
- 8. WHAT’S IN THE BOX? OUR STARTING POINT @lisihocke
- 9. @lisihocke THIS IS US AT FLIXMOBILITY
- 10. @lisihocke IT‘S TIME TO FOCUS ON QUALITY
- 12. THE CHALLENGE @lisihocke Quality is value
- 13. THE CHALLENGE @lisihocke Quality is value Lack of transparency
- 14. THE CHALLENGE @lisihocke Quality is value Lack of transparency Knowledge is not common
- 15. THE CHALLENGE @lisihocke Quality is value Lack of transparency Knowledge is not common The need to improve
- 16. THE CHALLENGE @lisihocke Quality is value Lack of transparency Knowledge is not common The need to improve Let’s tackle it before it tackles us
- 17. OUR MISSION @lisihocke Improve the testing & quality culture of our product teams by leveling up knowledge, skills and practices
- 18. LET‘S EXPERIMENT! AN EXPERIMENT OF EXPERIMENTS @lisihocke
- 20. OUR HYPOTHESIS With an experimental group of 4 product teams, we believe that • creating transparency where they currently are regarding testing and quality as well as identifying problems, • raising awareness and knowledge of the options at hand to probe the problems, and • running first experiments to learn what works in their context to enable them to rinse and repeat, will result in increased maturity of each team with regards to testing and quality. … @lisihocke
- 21. OUR HYPOTHESIS … We'll know we have succeeded when each of the 4 teams • has a clear, shared test strategy in place reflecting the current situation as well as identified problems, • has gained further knowledge and skills regarding testing and quality through a workshop, and • has adopted 1 new practice in their testing and quality culture that proved valuable in their context. @lisihocke
- 23. EXIT CRITERIA The experiment will stop and be re-evaluated in the following cases, whichever comes first: • No problem there! We find out that the chosen product teams already have transparency, awareness and experiments in place to improve their testing and quality. • Yay, success :) The hypothesis success criteria are fulfilled as defined above. • Didn't work out - let's rethink it. It is November 1, 2019 (even though success criteria are not fulfilled). @lisihocke
- 24. @lisihocke THE HIGH LEVEL PLAN
- 25. @lisihocke THE HIGH LEVEL PLAN Find contributors
- 26. @lisihocke THE HIGH LEVEL PLAN Find contributors Align among contributors
- 27. @lisihocke THE HIGH LEVEL PLAN Find contributors Align among contributors Select product teams as pilot teams
- 28. @lisihocke THE HIGH LEVEL PLAN Find contributors Align among contributors Select product teams as pilot teams Work with each team
- 29. @lisihocke THE HIGH LEVEL PLAN Find contributors Align among contributors Select product teams as pilot teams Work with each team Evaluate the hypothesis success criteria
- 31. @lisihocke COMMUNICATION GUIDELINES TO APPROACH THE PRODUCT TEAMS
- 32. @lisihocke COMMUNICATION GUIDELINES TO APPROACH THE PRODUCT TEAMS Appreciate what's working.
- 33. @lisihocke COMMUNICATION GUIDELINES TO APPROACH THE PRODUCT TEAMS Appreciate what's working. It's an opportunity to learn.
- 34. @lisihocke COMMUNICATION GUIDELINES TO APPROACH THE PRODUCT TEAMS Appreciate what's working. It's an opportunity to learn. Let's improve together.
- 35. @lisihocke COMMUNICATION GUIDELINES TO APPROACH THE PRODUCT TEAMS Appreciate what's working. It's an opportunity to learn. Let's improve together. It's about you, the team!
- 36. @lisihocke COMMUNICATION GUIDELINES TO APPROACH THE PRODUCT TEAMS Appreciate what's working. It's an opportunity to learn. Let's improve together. It's about you, the team! We support you, custom-tailored to your needs.
- 37. @lisihocke COMMUNICATION GUIDELINES TO APPROACH THE PRODUCT TEAMS Appreciate what's working. It's an opportunity to learn. Let's improve together. It's about you, the team! We support you, custom-tailored to your needs. Senior tech leadership has our back!
- 38. @lisihocke SELECT PILOT TEAMS – LET’S TALK!
- 39. @lisihocke SELECT PILOT TEAMS – LET’S TALK! Building quality in
- 40. @lisihocke SELECT PILOT TEAMS – LET’S TALK! Building quality in What's working
- 41. @lisihocke SELECT PILOT TEAMS – LET’S TALK! Building quality in What's working What's worrying
- 42. @lisihocke SELECT PILOT TEAMS – LET’S TALK! Building quality in What's working What's worrying Eagerness
- 43. @lisihocke SELECT PILOT TEAMS – LET’S TALK! Building quality in What's working What's worrying Eagerness Readiness
- 44. @lisihocke READY, SET GO! LET’S WORK WITH THE TEAMS
- 46. @lisihocke TRANSPARENCY Desired outcome: The team has a clear, shared test strategy in place reflecting the current situation as well as identified problems.
- 47. @lisihocke TRANSPARENCY: PAIN POINTS Shout-out to Ashley Hunsberger & Lisa Crispin Visualize current workflow & identify problems
- 48. @lisihocke TRANSPARENCY: RISK Shout-out to Beren van Daele & Marcel Gehlen RiskStorming to identify context-specific product risks
- 49. @lisihocke TRANSPARENCY: STATUS QUO Document current test strategy
- 51. @lisihocke AWARENESS Desired outcome: The team has gained further knowledge and skills regarding testing and quality through a workshop.
- 53. @lisihocke TEAM EXPERIMENTS Desired outcome: The team has adopted 1 new practice in their testing and quality culture that proved valuable in their context.
- 54. @lisihocke TEAM EXPERIMENTS Review the currently known challenges
- 55. @lisihocke TEAM EXPERIMENTS Review the currently known challenges Pick one challenge to tackle next
- 56. @lisihocke TEAM EXPERIMENTS Review the currently known challenges Pick one challenge to tackle next Brainstorm potential solutions
- 57. @lisihocke TEAM EXPERIMENTS Review the currently known challenges Pick one challenge to tackle next Brainstorm potential solutions Pick one option to try out
- 58. @lisihocke TEAM EXPERIMENTS Review the currently known challenges Pick one challenge to tackle next Brainstorm potential solutions Pick one option to try out Design the experiment
- 59. @lisihocke TEAM EXPERIMENTS Review the currently known challenges Pick one challenge to tackle next Brainstorm potential solutions Pick one option to try out Design the experiment Run the experiment
- 60. @lisihocke TEAM EXPERIMENTS Review the currently known challenges Pick one challenge to tackle next Brainstorm potential solutions Pick one option to try out Design the experiment Run the experiment Evaluate the experiment
- 61. @lisihocke TEAM EXPERIMENTS Challenge: <short challenge description> Hypothesis: We believe that <doing X> will result in <desired outcome>. We'll know we have succeeded when <measurable criterion is achieved>. Experiment: <concrete details like planned actions, experiment run time, exit criteria, etc.>
- 63. TIME TO EVALUATE LESSONS LEARNED @lisihocke
- 65. All 4 teams have a clear, shared test strategy in place reflecting the current situation as well as identified problems HYPOTHESIS EVALUATION @lisihocke
- 66. All 4 teams have a clear, shared test strategy in place reflecting the current situation as well as identified problems All 4 teams have a clear, shared test strategy in place reflecting the current situation as well as identified problems → YES ☺ HYPOTHESIS EVALUATION @lisihocke
- 67. All 4 teams have a clear, shared test strategy in place reflecting the current situation as well as identified problems All 4 teams have a clear, shared test strategy in place reflecting the current situation as well as identified problems → YES ☺ All 4 teams have gained further knowledge and skills regarding testing and quality through a workshop HYPOTHESIS EVALUATION @lisihocke
- 68. All 4 teams have a clear, shared test strategy in place reflecting the current situation as well as identified problems All 4 teams have a clear, shared test strategy in place reflecting the current situation as well as identified problems → YES ☺ All 4 teams have gained further knowledge and skills regarding testing and quality through a workshop All 4 teams have gained further knowledge and skills regarding testing and quality through a workshop → YES ☺ HYPOTHESIS EVALUATION @lisihocke
- 69. All 4 teams have a clear, shared test strategy in place reflecting the current situation as well as identified problems All 4 teams have a clear, shared test strategy in place reflecting the current situation as well as identified problems → YES ☺ All 4 teams have adopted 1 new practice in their testing and quality culture that proved valuable in their context All 4 teams have gained further knowledge and skills regarding testing and quality through a workshop All 4 teams have gained further knowledge and skills regarding testing and quality through a workshop → YES ☺ HYPOTHESIS EVALUATION @lisihocke
- 70. All 4 teams have a clear, shared test strategy in place reflecting the current situation as well as identified problems All 4 teams have a clear, shared test strategy in place reflecting the current situation as well as identified problems → YES ☺ All 4 teams have adopted 1 new practice in their testing and quality culture that proved valuable in their context All 4 teams have adopted 1 new practice in their testing and quality culture that proved valuable in their context → Only 2 All 4 teams have gained further knowledge and skills regarding testing and quality through a workshop All 4 teams have gained further knowledge and skills regarding testing and quality through a workshop → YES ☺ HYPOTHESIS EVALUATION @lisihocke
- 71. @lisihocke IMPACT WE HAD – AND DIDN’T HAVE
- 72. @lisihocke IMPACT WE HAD – AND DIDN’T HAVE ☺ Team conversations about testing and quality
- 73. @lisihocke IMPACT WE HAD – AND DIDN’T HAVE ☺ Team conversations about testing and quality ☺ Raised awareness and increased knowledge and skills
- 74. @lisihocke IMPACT WE HAD – AND DIDN’T HAVE ☺ Team conversations about testing and quality ☺ Raised awareness and increased knowledge and skills ☺ Teams inspired to improve their practices
- 75. @lisihocke IMPACT WE HAD – AND DIDN’T HAVE ☺ Team conversations about testing and quality ☺ Raised awareness and increased knowledge and skills ☺ Teams inspired to improve their practices Not the whole team was on board, silos remained
- 76. @lisihocke IMPACT WE HAD – AND DIDN’T HAVE ☺ Team conversations about testing and quality ☺ Raised awareness and increased knowledge and skills ☺ Teams inspired to improve their practices Not the whole team was on board, silos remained Not all people opened up for new concepts, misconceptions prevailed
- 77. @lisihocke IMPACT WE HAD – AND DIDN’T HAVE ☺ Team conversations about testing and quality ☺ Raised awareness and increased knowledge and skills ☺ Teams inspired to improve their practices Not the whole team was on board, silos remained Not all people opened up for new concepts, misconceptions prevailed Teams struggled with experiments, fell back into everyday business
- 78. @lisihocke TOO MUCH EFFORT TO SCALE THIS APPROACH
- 79. HOW TO SCALE? A LIGHTWEIGHT APPROACH @lisihocke
- 81. @lisihocke A SECOND EXPERIMENT We believe that if we learn about the testing strategy of product teams and provide them peer feedback regarding our perspective on their challenges, then this will • help the teams build increased self-awareness regarding testing and quality, and • help us understand our current testing and quality culture. We'll know we have proven the hypothesis when the teams as well as we are able to explain the difference between their approach and our suggestions and why it might differ from expectations.
- 83. A SECOND PLAN Align within the initiative group on core values and select product teams @lisihocke
- 84. A SECOND PLAN Align within the initiative group on core values and select product teams Interview, present and provide feedback on the teams’ testing strategy @lisihocke
- 85. A SECOND PLAN Align within the initiative group on core values and select product teams Interview, present and provide feedback on the teams’ testing strategy Evaluate what we learned in this experiment and propose the next steps @lisihocke
- 86. @lisihocke CORE VALUES TO BASE OUR FEEDBACK ON
- 87. @lisihocke CORE VALUES TO BASE OUR FEEDBACK ON Quality is Team Responsibility
- 88. @lisihocke CORE VALUES TO BASE OUR FEEDBACK ON Quality is Team Responsibility Tackle the Unknown, Automate the Known
- 89. @lisihocke CORE VALUES TO BASE OUR FEEDBACK ON Quality is Team Responsibility Tackle the Unknown, Automate the Known Invest in Quality and Quality Will Work for You
- 90. @lisihocke INTERVIEW TEAM MEMBERS ON THEIR TEAM’S TESTING STRATEGY
- 91. @lisihocke FEEDBACK SESSION & EXPERIMENT EVALUATION
- 92. @lisihocke FEEDBACK SESSION & EXPERIMENT EVALUATION
- 94. @lisihocke … YET THEN LIFE HAPPENED
- 95. GROWING A CULTURE BELIEFS & NEEDS @lisihocke
- 96. @lisihocke PRODUCT TEAMS: WE NEED YOU TO SOLVE OUR PROBLEMS!
- 97. @lisihocke PRODUCT TEAMS: WE NEED YOU TO SOLVE OUR PROBLEMS! Or rather: we need support so we become aware of our options and enablement so we can solve our own problems
- 98. @lisihocke → INSPIRE & BUILD ON PEOPLE’S CURIOSITY
- 99. @lisihocke → INSPIRE & BUILD ON PEOPLE’S CURIOSITY Experimenting with tech talks and personal conversations inside a domain, creating a “pull”
- 100. @lisihocke TECH LEADS: WE NEED DATA & GLOBAL METRICS!
- 101. Or rather: we need to know which impact we want to have in our specific contexts and what information we need to have it @lisihocke TECH LEADS: WE NEED DATA & GLOBAL METRICS!
- 102. @lisihocke → UNDERSTAND NEEDS & FIND HELPFUL INDICATORS
- 103. @lisihocke → UNDERSTAND NEEDS & FIND HELPFUL INDICATORS Experimenting together with tech leads on perceived quality needs and indicators helping them make better decisions
- 104. @lisihocke → UNDERSTAND NEEDS & FIND HELPFUL INDICATORS Experimenting together with tech leads on perceived quality needs and indicators helping them make better decisions Shout-out to Michael Kutz, João Proença, Toyer Mamoojee, Emanuil Slavov and Jenny Bramble
- 105. @lisihocke CONTRIBUTORS: WE NEED QUALITY TO START WITHIN THE TEAMS!
- 106. @lisihocke CONTRIBUTORS: WE NEED QUALITY TO START WITHIN THE TEAMS! Or rather: we need to change the system so behaviors contributing to quality get rewarded and valued
- 107. @lisihocke → SHAPE THE SYSTEM & REWARD BEHAVIORS CONTRIBUTING TO QUALITY
- 108. @lisihocke → SHAPE THE SYSTEM & REWARD BEHAVIORS CONTRIBUTING TO QUALITY Experimenting on shaping the system through leadership workshops, building a foundation for quality to emerge
- 109. @lisihocke → SHAPE THE SYSTEM & REWARD BEHAVIORS CONTRIBUTING TO QUALITY Experimenting on shaping the system through leadership workshops, building a foundation for quality to emerge Shout-out to Shiva Krishnan
- 110. @lisihocke QUALITY IS ON ALL OF US
- 111. @lisihocke QUALITY IS ON ALL OF US Create transparency on the status quo, pain points and needs → The teams, the leadership, the system
- 112. @lisihocke QUALITY IS ON ALL OF US Create transparency on the status quo, pain points and needs → The teams, the leadership, the system Raise awareness regarding the options at hand to improve
- 113. @lisihocke QUALITY IS ON ALL OF US Create transparency on the status quo, pain points and needs → The teams, the leadership, the system Raise awareness regarding the options at hand to improve Experiment in your context to grow a quality culture - together