Vladimir Primakov - Qa management in big agile teams

1. QA Management in BIG Agile teams
Volodymyr Prymakov

2. Speaker info
Volodymyr Prymakov, Ukraine
Senior QA Manager at Ciklum, TCoE
Head of Performance QA Unit at TCoE
•14 years in QA
• 45 projects experience
• Certified: ISTQB Advanced Test Analyst &
Manager, ICAgile CP, SAFE
/in/vladimirprimakov/

3. Agenda
1. Big Team, What is it?
2. Process (Development and QA)
3. Infrastructure
4. Collaboration
5. Transparency
6. QA Team organization
7. Automation
8. Questions

4. What is a big team?

5. What is a big team?
Team

6. What is a big team?
Team Big team? - Expected!

7. What is a big team?
Team Big team? in Reality….

8. What is a big team?
Most of the problems in a big team
relates to
cross-team work and
common dependencies

9. Big agile team is a team of >50 people
1 PO
1 Tech Lead
5 Developers
2 QAs
1 PO
1 Tech Lead
5 Developers
2 QAs
1 PO
1 Tech Lead
5 Developers
2 QAs
1 PO
1 Tech Lead
5 Developers
2 QAs
1 PO
1 Tech Lead
5 Developers
2 QAs
1 PO
1 Tech Lead
5 Developers
2 QAs
PO Manager
Dev Manager
Architectors
QA Manager, Leads
SMSMSM
Program Manager
Release Manager
Dev. Teams
Leadership Team Release train

10. Products
1. Big (A lot of functionality or sub-products)
1. Complex (Architecture and infrastructure)
1. (Quite often) Monolete architecture or
interdependable components
1. A lot internal and external integrations.
2. A lot of functionality interdependencies.
High Regression risk!
3. Many end-users (sometimes, in many
countries)

11. PROCESS

12. Process - Overlapped Releases
Supported by the same teams
MASTER BRANCH
STABLE BRANCH

13. Process - Overlapped Releases
Supported by the same teams
STABLE BRANCH
MASTER BRANCH

14. Process - Overlapped Releases
Supported by the same teams
MASTER BRANCH
Bug Hotfixes
STABLE BRANCH

15. Process - Overlapped Releases
Supported by the same teams
MASTER BRANCH
Bug Hotfixes
STABLE BRANCHRegression Problems

16. Process - Overlapped Releases
!!!Delay because of merging,
regression and other problems
Supported by the same teams

17. Process - Overlapped Releases
!!!Delay because of merging,
regression and other problems
Supported by the same teams
Overlappingactivities

18. Process - Overlapped Releases
!!!More Delay
Supported by the same teams
Overlappingactivities
Overlappingactivities

19. Process - Overlapped Releases
!!!More Delay
SHIFTED RELEASES
Supported by the same teams
Overlappingactivities
Overlappingactivities

20. Process - Overlapped Releases
Supported by the same teams
PROBLEMS:
● No Buffer for human factor and
unpredictable problems

21. Process - Overlapped Releases
Supported by the same teams
PROBLEMS:
● No Buffer for human factor and
unpredictable problems
● Overlapping activities
● Big context switching

22. Process - Overlapped Releases
Supported by the same teams
PROBLEMS:
● No Buffer for human factor and
unpredictable problems
● Overlapping activities
● Big context switching
● Complicated branching
strategy and risky hot-fixing
process
● Not compatible with
Continuous Delivery

23. Alternative Process - Overlapped Releases
Regression and release support by Another team
PROBLEMS:
● Lack of new functionality
(implementation) knowledge
● Knowledge Transfer required
● Not enough expertise for bug
fixing
● Extra Collaboration needed
● Not Enough Capacity
● Motivation problem
System Dev & QA Team

24. STRAITFORWARD Release Pipeline
MASTER BRANCH
NOW

25. STRAITFORWARD Release
Pipeline
MASTER BRANCH
FUTURE

26. STRAITFORWARD Release Pipeline
BENEFITS:
● SIMPLE and STRAIGHTFORWARD
● NO WASTE of TIME FOR
KNOWLEDGE TRANSFER
● NO CONTEXT SWITCHING
● RELEASES EVERY 2 WEEKS
● COMPATIBLE with CONTINUOUS
DELIVERY

27. Process Effect on Quality
NEW PROCESS
Bug Rate
Story
Points
Delivered
OLD PROCESS
STILL GOOD VELOCITY
30-40% LESS BUGS

28. STABILIZATION Period
● No Feature Merging in
master or release
branch
● Only Blocker and
Critical bugs fixing
● Regression testing
● Bug Retest.

29. Release management
FOR OTHER PRODUCTS
Monitor Release
Weight Impact
Prepare in advance
Test and Retest it!
Releases for Integrated Systems (internal and 3-rd parties)

30. Late Code Merging
Late code merging leads to bugs, no time
for testing and release delays!

31. Late Code Merging
To Avoid Merging Problems:
● Size stories optimally

32. Late Code Merging
To Avoid Merging Problems:
● Size stories optimally
● Merge as often as possible

33. Late Code Merging
To Avoid Merging Problems:
● Size stories optimally
● Merge as often as possible
● Early Deliver and Test (Automatically)

34. Late Code Merging
To Avoid Merging Problems:
● Size stories optimally
● Merge as often as possible
● Early Deliver and Test (Automatically)
● Deadline for merging + buffer for testing

35. Late Code Merging
To Avoid Merging Problems:
● Size stories optimally
● Merge as often as possible
● Early Deliver and Test (Automatically)
● Deadline for merging + buffer for testing
● Communicate, manage, and revert risky merges.

36. Regression testing approach (e.g.)
Country 2
Country 1
Country 3
Country 4
Markets

37. Regression testing approach (e.g.)
Fast
Feedback
about product
quality state.
Scope:
P1 Test cases
Country 2
Country 1
Country 3
Country 4
Markets Regression Cycle 1
Mostly p1, p2
bugs

38. Regression testing approach (e.g.)
Fast
Feedback
about product
quality state.
Scope:
P1 Test cases
Country 2
Country 1
Country 3
Country 4
Markets Regression Cycle 1 Regression Cycle 2
Cross-Platform
Compatibility
testing
Scope:
Exploratory
testing
Mostly p1, p2
bugs
Mostly p3 or lower
priority bugs

39. Regression testing approach (e.g.)
Fast
Feedback
about product
quality state.
Scope:
P1 Test cases
Country 2
Country 1
Country 3
Country 4
Markets Regression Cycle 1 Regression Cycle 2
Cross-Platform
Compatibility
testing
Scope:
Exploratory
testing
Regression Cycle 3
Validate
Quality after
Bug-fixes
verifications
Scope:
P1, P2 test
cases
Mostly p1, p2
bugs
Mostly p3 or lower
priority bugs
p1, p2, p3 or lower
priority bugs

40. Regression testing approach (e.g.)
Country 2
Country 1
Country 3
Country 4
Markets Regression Cycle 1 Regression Cycle 2
All other
platforms
Regression Cycle 3
Platform 2
(Chrome)
Platform 1 (IE)
Platform 1 (IE)
Platform 2
(Chrome)
Platform 1 (IE)
Platform 2
(Chrome)
Platform 2
(Chrome)
Platform 2 (IE)
Cross Platform Testing approach - Sprint N

41. Regression testing approach (e.g.)
Country 2
Country 1
Country 3
Country 4
Markets Regression Cycle 1 Regression Cycle 2
All other
platforms
Regression Cycle 3
Platform 1 (IE)
Platform 2
(Chrome)
Platform 2
(Chrome)
Platform 1 (IE)
Platform 2
(Chrome)
Platform 1 (IE)
Platform 1 (IE)
Platform 2
(Chrome)
Cross Platform Testing approach - Sprint N + 1

42. Regression testing approach (e.g.)
Type of Test Scope Outcome
Smoke End-to-end flow (most
popular booking)
Blocker bugs
Acceptance Alternative End-to-End
flows
P1 and P2 bugs
UAT User Acceptance Test
scenarios
P1, P2, P3
Some bugs may be missed
Regression P1 P1 test cases P1 and P2 bugs
Regression P2 P2 test cases P3 bugs
Exploratory Exploratory test cases P1, P2, P3
Testing Order!

43. System Intergration Testing
● Obligatory test integrations with
real life internal and 3d party
services
● Do confirmation testing for the
rest of functionality if needed
(Exploratory, Cross-platforms, or
more wider testing)

44. Cross team Bug Escalation
Priority
p3, p4, less
QA Coordinator /
Release Coordinator
Product
Backlog

45. Cross team Bug Escalation
Priority
p3, p4, less
Priority
p1, p2
QA Coordinator /
Release Coordinator
Product
Backlog
Assign to current FixVersionTESTER

46. Cross team Bug Escalation
Priority
p3, p4, less
Priority
p1, p2
TEAM
QA Coordinator /
Release Coordinator
Product
Backlog
Assign to current FixVersionTESTER
ASSIGN to A CERTAIN TEAM
Assign to Developer and Resolve Quickly!Is this bug produced by the team?
Does the team has expertise to fast fix it?
Or does the team has capacity for bug fixing?

47. Cross team Bug Escalation
Priority
p3, p4, less
Priority
p1, p2
TEAM
QA Coordinator /
Release Coordinator
ASSIGN to A CERTAIN TEAM
Assign to Developer and Resolve Quickly!
Product
Backlog
Assign to current FixVersionTESTER

48. Cross team Bug Escalation
Priority
p3, p4, less
Priority
p1, p2
TEAM
TECH LEADS
QA Coordinator /
Release Coordinator
ASSIGN to A CERTAIN TEAM
Assign to Developer and Resolve Quickly!
Product
Backlog
Assign to current FixVersion
Which team produced the bug?
Which team has better expertise to fix it?
Which team has more capacity for bug fixing?
TESTER

49. Cross team Bug Escalation
Priority
p3, p4, less
Priority
p1, p2
TEAM
TECH LEADS
QA Coordinator /
Release Coordinator
ASSIGN to A CERTAIN TEAM
Assign to Developer and Resolve Quickly!
Product
Backlog
Assign to current FixVersionTESTER

50. Cross team Bug Escalation
Priority
p3, p4, less
Priority
p1, p2
TEAM
TECH LEADS
QA Coordinator /
Release Coordinator
ASSIGN to A CERTAIN TEAM
Assign to Developer and Resolve Quickly!
Product
Backlog
Assign to current FixVersionTESTER
SCRUM MASTERS
Which team has more capacity for bug
fixing?

51. Cross team Bug Escalation
Priority
p3, p4, less
Priority
p1, p2
TEAM
TECH LEADS
QA Coordinator /
Release Coordinator
ASSIGN to A CERTAIN TEAM
Assign to Developer and Resolve Quickly!
Product
Backlog
Assign to current FixVersionTESTER
SCRUM MASTERS

52. Cross team Bug Escalation
Priority
p3, p4, less
Priority
p1, p2
TEAM
TECH LEADS
QA Coordinator /
Release Coordinator
ASSIGN to A CERTAIN TEAM
Assign to Developer and Resolve Quickly!
Product
Backlog
Assign to current FixVersionTESTER
SCRUM MASTERS
POs
Which team has lower priority functionality
which can wait for the sake of bug fixing?

53. Cross team Bug Escalation
Priority
p3, p4, less
Priority
p1, p2
TEAM
TECH LEADS
QA Coordinator /
Release Coordinator
ASSIGN to A CERTAIN TEAM
Assign to Developer and Resolve Quickly!
Product
Backlog
Assign to current FixVersionTESTER
SCRUM MASTERS
POs

54. Cross team Bug Escalation
Priority
p3, p4, less
Priority
p1, p2
TEAM
TECH LEADS
QA Coordinator /
Release Coordinator
ASSIGN to A CERTAIN TEAM
Assign to Developer and Resolve Quickly!
Product
Backlog
Assign to current FixVersionTESTER
SCRUM MASTERS
POs
FACILITATE

55. Align QA approaches across teams

56. Align QA approaches across teams
• DoD for test cases
• DoD for Automation scripts
• Test suite Structure
• Test plan and run structure in test management
system
• Regression approach
• Bug Reporting
• Release readiness reporting
• and many others

57. Agreeing project process/approach
Agree cross-team
approaches/processes on all levels

58. Agreeing project process/approach
Agree cross-team
approaches/processes on all levels
Involve all concerned parties in the
discussions

59. Agreeing project process/approach
Agree cross-team
approaches/processes on all levels
Involve all concerned parties in the
discussions
Align cross-team procedures

60. Agreeing project process/approach
Agree cross-team
approaches/processes on all levels
Be tolerant and persistent
Involve all concerned parties in the
discussions
Align cross-team procedures

61. Agreeing project process/approach
Agree cross-team
approaches/processes on all levels
Be tolerant and persistent
Document agreements
Involve all concerned parties in the
discussions
Align cross-team procedures

62. Infrastructure

63. Infrastructure problems
2h of infrastructure
downtime for the team in
size of 50 Devs and QAs
may cost 12.5 man days…
(several thousands USD)

64. Test Environments Requirements
SIMILAR to
PRODUCTION
AUTOMATED and
CONTROLLED
FAST and
POWERFUL
LESS
INTERDEPENDENT
ROBUST and
STABLE

65. Typical Infrastructure/environment
problems
Problem
Essential difference between environments (configuration,
integrations, data) leading to leaks of bugs to production
Uncontrolled changes and manual interventions
Configuration problems.
Slow and overloaded environment. Usage conflicts.
Integration issues (external and internal).
Services and DB compatibility.
Unreliable or absent test data.
Unpredictable downtimes. Connectivity and access issues.
Solution
Minimize the difference. At lease 1 test environment should
be 99% identical to production.
Automate Everything. Apply change control procedure. Limit
unauthorized access. Notify about changes in advance!
Increase Capacity. Build several test environments.
Don’t save on the environment. Use more powerful network.
Controlled 3d party releases. Component decoupling. Mocks
usage. Backward compatibility.
Mocking Data, Standard data sets usage, Agreeing test data
provision from 3-d parties, etc.
Plan B: Backups, alternative suppliers etc.

66. Environments (e.g.)
Testing Owners
Integrations
Hardware
capacity
DB

67. Devops should be part of teams
● ~1 Devop for 2 teams
● Open communication. Participation at
daily scrums and other sync-ups
● Proactive resolution of ongoing
infrastructure and environment
problems
● Devops backlog prioritization for sake
of teams’ needs

68. Cross team
Collaboration

69. Strategic Planning
Plans are hidden from teams

70. Strategic Planning

71. Strategic Planning
Strategic planning helps to build the right product
Product increment and other initiatives for a Quarter/Half-a-year

72. Sprint Planning (Devs)
Teams
Grooming

73. Sprint Planning (Devs)
Teams Cross-Team
Grooming Tech-leads
Preplanning

74. Sprint Planning (Devs)
Teams Cross-Team
Grooming
Common Risks
and
Dependencies
Identified
Tech-leads
Preplanning

75. Sprint Planning (Devs)
Teams Cross-Team Teams
Grooming Sprint
Planning
(adjusted)
Common Risks
and
Dependencies
Identified
Tech-leads
Preplanning

76. Sprint Planning (Devs)
Teams Cross-Team Teams
Grooming Sprint
Planning
(adjusted)
Common Risks
and
Dependencies
Identified
Sprint
Start
Tech-leads
Preplanning

77. Sprint Planning (QAs)
Cross-Team
QA Leads
Preplanning
Sprint Planning (QAs)

78. Sprint Planning (QAs)
Cross-Team
Additional sprint
testing scope
identified
QA Leads
Preplanning

79. Sprint Planning (QAs)
Cross-Team Teams
Sprint
Planning
(adjusted)
Additional sprint
testing scope
identified
QA Leads
Preplanning

80. Sprint Planning (QAs)
Cross-Team Teams
Sprint
Planning
(adjusted)
Additional sprint
testing scope
identified
Sprint
Start
QA Leads
Preplanning

81. Sprint Planning (QAs)
Cross-Team Teams
QA Leads
Preplanning
Sprint
Planning
(adjusted)
Additional sprint
testing scope
identified
Sprint
Start
Cross-Team
QA Sprint
Scope
Review
Next Slide =>

82. QA Sprint Scope Review & Planning
Participants
QA Manager,
QA Leads,
1 QA from
every team

83. QA Sprint Scope Review & Planning
Participants Goals
1. Review Sprint scope (Features)
2. Define Dependencies, risks, and impact
3. Adjust scope and type of testing
4. Define additional testing activities
5. Identify impact on automation and
performance scripts
QA Manager,
QA Leads,
1 QA from
every team

84. QA Sprint Scope Review & Planning
Participants Goals
1. Review Sprint scope (Features)
2. Define Dependencies, risks, and impact
3. Adjust scope and type of testing
4. Define additional testing activities
5. Identify impact on automation and
performance scripts
-------------------------------------------------
1. Discuss QA Automation coverage
increment and other achievements
2. QA automation scope adjusting
QA Manager,
QA Leads,
1 QA from
every team

85. QA Sprint Scope Review - Outcomes

86. Regular Team & Scrums
Teams Cross-Team
Scrum
Scrum of
Scrums
● Cross-team
Dependencies,
impediments, and help
needed
● Environmental Issues
● Overall Sprint Scope
covering (once)
● Are we ready for Code
Freeze?
● Discussing release
stoppers: Blocker and
critical bugs
Auditory:
Release Manager,
Scrum Masters,
Tech Leads,
QA Leads,
Devops Lead

87. Regular Cross Team Syncups
Dev Manager,
QA Manager,
Tech Leads,
DevOps Leads,
Release Coordinator
● Strategic Plans,
● Common
Approaches,
● Important Ongoing
Activities,
● Other Common
Questions and
Problems

88. Readiness Checkpoints
Ready for Regression?
Ready for Stage Testing?
Ready for UAT by
Business?
Ready For Production?
Regularly Sync-ups
on Product Readiness!
Release Manager,
Scum Masters,
PO Manager,
QA Leads/QA Coordinator,
(POs),
(Tech Leads)

89. Lessons Learned
Teams
Team
Retro
Sprint/Release
Finishes

90. Lessons Learned
Teams Cross-Team
Team
Retro
QAs
Retro
Sprint/Release
Finishes

91. Lessons Learned
Teams Cross-Team
Team
Retro
QAs
Retro
Release
Retro
Sprint/Release
Finishes
Cross-Team

92. Knowledge sharing
Spread Knowledge among the project!
!!!Record videos and/or document the stuff
● Business Domain
● Product Functionality
● Overall Architecture
● Component Technology
● 3-d party components
● Common approaches, etc.

93. Minimize frequency of sync-ups
Meet as often
as it benefits,
but do not
overdo

94. Minimize frequency of sync-ups
Meet as often
as it benefits,
but do not
overdo
!!!Avoid long
meetings at
essential
sprint/release
phases

95. Transparency

96. Scope understanding
Initiative/Epic level
Know and
align with
the
project
roadmap

97. Scope understanding
Sprint/Release level
Use Fix Version for
features and bugs
in advance to
track overall
release scope
(cross-team)

98. Realtime quality boards
Monitor Quality on daily bases or more often
1. Dynamic Jira Dashboards 2. Kanban Boards for Blocker and Critical bugs

99. Testing reporting
1. Report/Escalate on Blocker and
Critical bugs by a necessity.
2. Report regression testing results
on daily bases.
3. Involve all concerned parties in
the report.
4. Build quality
awareness/transparency on the
project!

100. Quality trends
Monitor and analyze Quality trends.
Apply Corrective actions if needed.

101. QA Team

102. TypicalQATeamStructure
QA Manager
QA Manual
Lead
Team 1
Senior/Middle
QA
Middle/Junior
QA
Team 2
Senior/Middle
QA
Middle/Junior
QA
Team 3
Senior/Middle
QA
Middle/Junior
QA
QA Manual
Lead
Team 4
Senior/Middle
QA
Middle
QA/Junior QA
Team 5
Senior/Middle
QA
Middle/Junior
QA
Team 6
Senior/Middle
QA
Middle/Junior
QA
QA Manual
Lead
Team 7
Senior/Middle
QA
Middle/Junior
QA
Team 8
Senior/Middle
QA
Middle/Junior
QA
Team 9
Senior/Middle
QA
Middle/Junior
QA
QA Automation
Lead
Core Automation
Team
Senior/Middle
Auto QA
Middle Auto
QA
Middle/Junior
Auto QA

103. QA Team Typical Roles
Defines and aligns QA Approaches on the project,
Strategic Planning and Reporting,
Monitor and adjust ongoing QA activities,
Resource planning, recruitment, competences, etc.
Escalation questions
QA
Manager
QA Approaches implementation,
Cross team ongoing QA work planning and coordination in a sprint (release),
Regular QA reporting,
People management in their teams (Mood, PDPs), etc.
QA Lead
Building and aligning QA automation approaches and expertize on the project,
Organizing the corresponding Knowledge Sharing,
Leading automation Framework development and support,
Automation team management.
QA
Automation
Lead

104. QA Component Lead role
1. Main knowledge holder / expert in the product area.
2. Responsible for the quality of the product area. Regularly
monitor its quality. Escalate problems if needed.
3. Define dependencies and quality risks in the area. Inform
others about them.
1. Defines manual and automation testing scope for the product
area.
2. Monitor and analyze automation testing results for the
product area. Execute automation tests if needed.
1. Plans and organizes regression and other kind of testing for
the product area.

105. QA Coordinator Role
1. Organize QA syncups and planning meetings.
2. Plans and organizes cross team testing activities
a. regression testing.
b. testing at production.
c. etc.
3. Monitor overall product quality state.
4. Escalate Blocker and Critical bugs if needed.
5. Prepare and send quality reports.
6. Play a representative role in release readiness
meetings.
Coordinate cross-team testing activities in a release.
Cycle the role
between QA leads.

106. Delegation on All levels
Other roles:
● QA Product Market Lead
● Platform statistics
management
● Devices Management
● A/B Testing management
● Payment Card and refund
Management
● Etc.

107. Automation

108. Why Automation?
Cross team
work
Regression Risk Automation
prevents the risk

109. Requirements to QA automation
All Levels Correct order Cross Platform

110. Requirements to QA automation
Sufficient
Coverage
Stable
and
Trustful
High Speed

111. Requirements to QA automation
Frequent
Executed on all
environments

112. Aligned with Manual QA
Manual QAAutomation QA

113. Aligned with Manual QA
Integration with Test Management system

114. Non-Functional Automation

115. Parting Words

116. Questions