Over the past twenty years, Mary Thorn has had the opportunity to work at many startups, creating several QA/test departments from scratch. For the past ten years, she has done this in agile software companies. Recently Mary moved from leading small agile test organizations to leading a large agile test organization. She has learned how to lead agile testers and agile testing in large contexts. Mary takes you through what she has learned, identifies the keys to transitioning your test organization as it grows, and discusses the techniques required to lead it through the changes. Agile testing is difficult, and training your testers to be consistent and interchangeable across large scale agile teams is even more difficult. And still more difficult is test automation at scale. Mary shares her experience in creating an automation strategy that works in a large scale context. Join Mary as she discusses her learnings from leading large agile test organizations.

1.
W10
Agile
Testing
5/10/17
13:45
Agile
Testing
at
Scale
Presented
by:
Mary
Thorn
Ipreo
Brought
to
you
by:
350
Corporate
Way,
Suite
400,
Orange
Park,
FL
32073
888-­‐-­‐-­‐268-­‐-­‐-­‐8770
·∙·∙
904-­‐-­‐-­‐278-­‐-­‐-­‐0524
-­‐
info@techwell.com
-­‐
http://www.starwest.techwell.com/

2.
Mary
Thorn
Chief
storyteller
of
The
Three
Pillars
of
Agile
Testing
and
Quality,
Mary
Thorn
is
director
of
agile
practices
at
Ipreo
in
Raleigh,
NC.
Mary
has
a
broad
agile
and
testing
background
that
spans
automation,
data
warehouses,
and
web-­‐based
systems
in
a
wide
variety
of
technologies
and
testing
techniques.
During
her
more
than
nineteen
years
of
experience
with
healthcare,
HR,
financial,
and
SaaS-­‐based
products,
Mary
has
held
manager-­‐
and
contributor-­‐level
positions
in
software
development
organizations.
A
strong
leader
in
agile
and
testing
methodologies,
Mary
has
direct
experience
leading
teams
through
agile
adoption
and
beyond.

3. Investment Banks. Investors. Investor Relations.
New York | London | Bethesda | Delhi | Boston | www.ipreo.com
Agile Testing At Scale
1

4. For internal use only
About Mary Thorn
l Chief Story Teller of the book “The Three Pillars of Agile Testing
and Quality” written by Bob Galen, Mary Thorn is Director of
Agile Practices at Ipreo in Raleigh, NC.
l Mary has a broad background that spans agile, testing and
automation, and web-based systems in a wide variety of
technologies and agile techniques.
l During her more than twenty years of experience in healthcare,
HR, financial, and SaaS-based products Mary has held manager
and contributor level positions in software development
organizations. A strong leader in agile and testing
methodologies, she has direct experience leading teams through
agile adoption and beyond.

5. For internal use only
The Problem
• 100 testers (20 onshore and 80 offshore)
• Transitioning from waterfall to scrum
• 25 Products
• Most products had 35-75 test cases that were workflows that had
anywhere from 10,000-75,000 verification points
• Automation was done by vendors and would take anywhere from 1-2
weeks to run serially.
• 2 Test Managers

6. For internal use only
Mission

7. For internal use only
Vision
è The QA team members are functioning as an equal part of the
overall Scrum Team
è They are Subject Matter Experts (SMEs) in the area of the
application.
è They are technically competent.
è QA members are accountable, committed, trust their
teammates, driven to results, and don’t fear conflict.
è They are empowered to be change agents.

8. For internal use only
What does success look like?

9. For internal use only
Three Pillars of Agile Quality
3 Pillars of Agile Quality
Copyright © 2013 RGCG, LLC
4
Development & Test
Automation
• Pyramid-based Strategy:
(Unit + Cucumber +
Selenium)
• Continuous Integration
• Attack technical
infrastructure in the Backlog
• Visual Feedback –
Dashboards
• Actively practice ATDD and
BDD
Software Testing
• Risk-based testing:
Functional & Non-Functional
• Test planning @ Release &
Sprint levels
• Exploratory Testing
• Standards – checklists,
templates, repositories
• Balance across manual,
exploratory & automation
Cross-Functional Team
Practices
• Team-based Pairing
• Stop-the-Line Mindset
• Code Reviews & Standards
• Active Done-Ness
• Aggressive Refactoring of
Technical Debt
• User Stories, “3 Amigo”
based Conversations
• Whole Team Ownership of “Quality”
• Building it ‘Right’; Building the ‘Right’ Thing
• Healthy – Agile Centric Metrics
• Center of Excellence or Community of Practice
• Strategic balance across 3 Pillars; Assessment, Recalibration, and Continuous Improvement
This document and all information contained herein, including descriptions and illustrations, are proprietary and confidential
information of Ipreo Holdings, LLC and its affiliates. Copyright ® 2009 Ipreo Holdings, LLC. All rights reserved. Patent Pending. 7

10. For internal use only
Foundation of the 3-Pillars
This document and all information contained herein, including descriptions and illustrations, are proprietary and confidential
information of Ipreo Holdings, LLC and its affiliates. Copyright ® 2009 Ipreo Holdings, LLC. All rights reserved. Patent Pending. 8
• Whole Team Ownership of
“Quality”
• Knowing the “Right” thing to
Build AND Building it “Right”
• Healthy – Agile Centric
Metrics
• Steering Required – CoE or
CoP
• Strategic balance across 3
Pillars; Assessment,
Recalibration, and
Continuous Improvement
• Whole team view includes building it right,
everyone tests, everyone demo’s, etc.
• Focus on features/stories, confirmation,
conversation, and getting them staged
properly OVER testing
• 4-tier metrics: Quality, Value, Prediction,
Team
• Agile strategies need light-handed “steering”;
establish a CoE (heavier weight) or a CoP
(lightweight)
• Consider finding an assessment framework
and then tying it to your strategy
measurement, recalibration, and continuous
improvement.
• Make the foundation visible thru information
radiators and metrics

11. For internal use only
Development and Automation Pillar
This document and all information contained herein, including descriptions and illustrations, are proprietary and confidential
information of Ipreo Holdings, LLC and its affiliates. Copyright ® 2009 Ipreo Holdings, LLC. All rights reserved. Patent Pending. 9
Development &
Test Automation
• Pyramid-based
Strategy: (Unit +
Cucumber + Selenium)
• Continuous Integration
• Attack technical
infrastructure in the
Backlog
• Visual Feedback –
Dashboards
• Actively practice ATDD
and BDD
A central part of agile adoption is focusing on CI, 3-
tiered Automation development, and Dashboards to
begin incrementally building coverage for faster
feedback on changes.
100% automation is NOT the Goal!
In the interim, Hardening or Stabilization Sprints and
having a risk-based Release Train concept help
It’s important that Test or QA not ‘own’ the tooling
or all of the automation efforts. The strategy can
come from QA, but the tactical automation
development is best left to the team.
Mature teams invest in Automation, Tooling, and
Technical Debt reduction as part of Done-ness and
continually add it to their backlogs

12. For internal use only
Software Testing Pillar
This document and all information contained herein, including descriptions and illustrations, are proprietary and confidential
information of Ipreo Holdings, LLC and its affiliates. Copyright ® 2009 Ipreo Holdings, LLC. All rights reserved. Patent Pending. 10
Software Testing
• Risk-based testing:
Functional & Non-
Functional
• Test planning @
Release & Sprint levels
• Exploratory Testing
• Standards – checklists,
templates, repositories
• Balance across manual,
exploratory &
automation
Exploratory Testing (SBET with pairing) can be an
incredibly effective way to establish a whole-team,
collaborative view towards quality and testing. It
also emerges new tests.
Leverage ‘plans’ as a whole-team collaboration-
conversation mechanism; at Sprint and Release
levels.
Do not measure testing or tester progress; instead,
measure throughput, output, sprint outcomes, and
done-ness escapes at a team level.
You need a balanced test team; not everyone needs
to be able to program. But everyone needs to be
passionately skilled testers with curiosity.
Agile testing is a Risk-Based play in every Sprint and
across a release sequence.

13. For internal use only
Cross-Functional Team Pillar
This document and all information contained herein, including descriptions and illustrations, are proprietary and confidential
information of Ipreo Holdings, LLC and its affiliates. Copyright ® 2009 Ipreo Holdings, LLC. All rights reserved. Patent Pending. 11
Cross-Functional
Team Practices
• Team-based Pairing
• Stop-the-Line Mindset
• Code Reviews &
Standards
• Active Done-Ness
• Aggressive Refactoring
of Technical Debt
• User Stories – 3 Amigo
based Conversations
One of the hardest areas to get ‘right’ culturally. It
needs leadership alignment from Quality/Testing to
Product to Development and a consistent voice of
whole-team approaches.
This is where LEAN Thinking lives, where whole-
team collaboration happens, where professionalism
and craftsmanship are held dear.
I like the view of testers becoming the VOC,
champions of quality, and consistent questioners of
what is being build. Are we solving the right
problems…as simply as possible. Notions of Minimal
Viable Product / Feature help with focus.
And yes Virginia, there ARE standards, templates,
and a focus on x-team consistency!

14. ©2016
Ipreo.
All
rights
reserved.
None
of
Ipreo,
its
affiliates,
or
any
of
their
third
party
providers
is
liable
for
any
informa?onal
errors,
incompleteness,
or
delays,
or
for
any
ac?ons
or
decisions
taken
by
any
person
as
a
result
of
any
informa?on
contained
herein.
12
2016
Roadmap
Personnel&
Training
Software
TestingPillar
Cross
Functional
Pillar
Automation
Pillar
Q1 Q2 Q3 Q4
SOP Rolled Out
Upskilling testers in test agile testing techniques and test automation
Risk Based Testing
Test Cases written in Specflow
Exploratory Testing Rolled out
Test case gap Analysis Complete
Technical Debt is being attacked for Critical and High Risk man/auto stories
Hiring Managers/testers
Define/Baseline Metrics Performance testing plan in place Executing Performance testing plan
Automation Framework rolled out to all scrum teams
Definition of Done is in DNA
Automation is Green

15. For internal use only
Roadmap
è Transparency around areas where
improvement is needed within QA, especially
around areas of automation.
è It helps define what success looks like for the
QA personnel for the year, and it helps define
individual objectives around this.
è Defines the strategy discussed above.
è It actually makes you look like you know what
you are doing.

16. For internal use only
Roadmap – Who inputs to them?
è Your QA team(s) or Scrum team
è You as a leader (your experience, instincts, etc.)
è Your organizational directives
è Retro

17. For internal use only
Roadmap cont.
What are examples of what would be in a roadmap?
è Build automation framework
è Automate X smoke test
è Automate X regression test
è Define/implement performance testing strategy
è Train/implement BDD
è Train/implement exploratory testing
è Write manual regression test cases for x
è Create QA Standard Operating Procedure document
è Migrate away from Quality Center/ALM and save the company
$100k/year(BEST roadmap item ever)

18. For internal use only
I have one now what?

19. For internal use only
I have one now what?
è Once you have your roadmap, you need to work with your
product owner organization to create stories, and get them
prioritized and injected across your entire team’s product
backlogs. Make sure that you put all of your work in a
“business context”, explaining what it will do for “them” or the
“customer”.
è Another approach is to allocate a specific percentage of each
backlog to this sort of work.
è If you do not have a global agreement of some kind, you have
to allocate your work to the product backlogs on a situation-
by-situation basis.

20. For internal use only
Roadmap for Sale

21. For internal use only
Strategy(eat your own dog food)

22. For internal use only
Cross Cutting Strategy
3 Pillars of Agile Quality
Copyright © 2013 RGCG, LLC
4
Development & Test
Automation
• Pyramid-based Strategy:
(Unit + Cucumber +
Selenium)
• Continuous Integration
• Attack technical
infrastructure in the Backlog
• Visual Feedback –
Dashboards
• Actively practice ATDD and
BDD
Software Testing
• Risk-based testing:
Functional & Non-Functional
• Test planning @ Release &
Sprint levels
• Exploratory Testing
• Standards – checklists,
templates, repositories
• Balance across manual,
exploratory & automation
Cross-Functional Team
Practices
• Team-based Pairing
• Stop-the-Line Mindset
• Code Reviews & Standards
• Active Done-Ness
• Aggressive Refactoring of
Technical Debt
• User Stories, “3 Amigo”
based Conversations
• Whole Team Ownership of “Quality”
• Building it ‘Right’; Building the ‘Right’ Thing
• Healthy – Agile Centric Metrics
• Center of Excellence or Community of Practice
• Strategic balance across 3 Pillars; Assessment, Recalibration, and Continuous Improvement

23. For internal use only
©2016
Ipreo.
All
rights
reserved.
None
of
Ipreo,
its
affiliates,
or
any
of
their
third
party
providers
is
liable
for
any
informa?onal
errors,
incompleteness,
or
delays,
or
for
any
ac?ons
or
decisions
taken
by
any
person
as
a
result
of
any
informa?on
contained
herein.
21
3
Pillars
Tes4ng
Assessment
Developmen
t and Test
Automation
Pillar
Software
Testing
Pillar
Cross
Functional
Team Pillar
Whole Team
Ownership
Test
Leadership
Q2-2015
Team Health
Team Target
Development
and Test
Automation Pillar
Software Testing
Pillar
Cross Functional
Team Pillar
Whole Team
Ownership
Test Leadership
Q3-2016
Team
Health

24. For internal use only
©2016
Ipreo.
All
rights
reserved.
None
of
Ipreo,
its
affiliates,
or
any
of
their
third
party
providers
is
liable
for
any
informa?onal
errors,
incompleteness,
or
delays,
or
for
any
ac?ons
or
decisions
taken
by
any
person
as
a
result
of
any
informa?on
contained
herein.
22
Sev
1
&
2
Defect
Trend
0
1
2
3
4
5
6
7
8
9
10
Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sep
BSP
CSP
IL
Q3 Threshold

25. For internal use only
©2016
Ipreo.
All
rights
reserved.
None
of
Ipreo,
its
affiliates,
or
any
of
their
third
party
providers
is
liable
for
any
informa?onal
errors,
incompleteness,
or
delays,
or
for
any
ac?ons
or
decisions
taken
by
any
person
as
a
result
of
any
informa?on
contained
herein.
111
50
74
0
20
40
60
80
100
120
CSP
BSP
IL
CSP
BSP
IL
23
Average
Age
of
All
Open
Defects
90 Day Threshold

26. For internal use only
©2016
Ipreo.
All
rights
reserved.
None
of
Ipreo,
its
affiliates,
or
any
of
their
third
party
providers
is
liable
for
any
informa?onal
errors,
incompleteness,
or
delays,
or
for
any
ac?ons
or
decisions
taken
by
any
person
as
a
result
of
any
informa?on
contained
herein.
24
Automa4on
Pyramid
Q1 %
Change Direction Q2 %
Change Direction Q3 %
Change Direction Q4 %
Change Direction
Manual
Test
Cases 1195 0% − 1551 23% é 516 -­‐201% ê
Automated
UI
Tests 247 0% − 321 23% é 338 5% é
Automated
Integration
Tests 252 0% − 535 53% é 923 42% é
Automated
Unit
Tests 1959 0% − 1940 -­‐1% ê 3141 38% é

27. For internal use only
Doneness/Policies

28. For internal use only
Retrospectives
• Discovery Retro
• Quarterly QA Retro
• Technical Debt Retro
• Post Prod Sev 1 or 2 Retro

29. For internal use only
Supporting and Protecting the Team

30. For internal use only
Relationship building
è Project Management – Scrum Masters
è Architecture and Business Analysis
è Development
è Testing
è UX Design
è Product Owners
è DevOps or Technical Operational Team(s)
è Documentation
è Customer Facing Operational Team(s)

31. For internal use only
Hiring

32. For internal use only
People Management

33. For internal use only
Continuous Improvement
è Trainings – All 3 Pillars
• Automation
• Testing Practices
• Team(IE Five Dysfunctions)
• Agile
è 10% time
è Management and Leadership

34. For internal use only
Wrapping it Up