This document summarizes the journey of defining and automating QA reports and metrics for a growing company handling multiple projects simultaneously. It describes how initially manual reporting was automated using an external tool to gather data from sources like JIRA and SonarQube to generate reports on Confluence. Challenges included different client tooling, prioritizing reports after releases, and tool maintenance. Over time, the experienced team focused less on governance and more on consulting. Metrics are now used selectively to measure process improvements rather than compliance. QA status is included in high-level project reports, with the team deciding if detailed QA reports are needed based on the project.

1. There and back again
Our journey with QA Reports and metrics
1
Zbyszek Moćkun

2. Intro
2

3. Agenda
3
01 The Beginning
02 An unexpected journey / Define QA Reports and metrics
03 Find your Black Arrow / Automate QA Reports
04 The battle of five army / Challenges
05 And back again / Few years later

4. The beginning
Long long time ago
4

5. Bigger, bigger and bigger
• New clients from pharmacy, insurance and banking
• Which required more documentation and more formal processes
• Do not confident with agile methodologies and exploratory testing approach
• Project duration from 2 to 6 months
• Fast growing company
• A lot of newcomers which do not understand our ways of working
• Inexpierence team
5

6. How to control several simultanuously projects?
• Governance on company level
• Deliver always high / expected quality
• Support inexperienced team / share knowledge
• Reporting projects status to Company Board
• Compare projects
• Identify good practices
• Understand project complexity
6

7. An unexpected journey
Define QA Reports and metrics
7

8. How to identify metrics?
• Metrics should encourage to ask questions not give the answers
• Measure the processes not people
• Qualitative and Quantitave
• Highlight important processes, activities, ways of working
• Each release has quality report
8

9. 9
1 x full width image & text
placeholder (no header bar)

10. 10
1 x full width image & text
placeholder (no header bar)

11. 11
1 x full width image & text
placeholder (no header bar)

12. 12
1 x full width image & text
placeholder (no header bar)

13. 13
1 x full width image & text
placeholder (no header bar)

14. 14
1 x full width image & text
placeholder (no header bar)

15. 15
1 x full width image & text
placeholder (no header bar)

16. Find your black arrow
Automate QA Reports
16

17. Identify the enemy - Time
Several projects * 2 reports per month * 4 hours per report
17

18. Automate QA Reports
• External application
• Which gather data from sources like JIRA, SonarQube, QMetry
• And generate a report on Confluence
18

19. The Battle of Five Army
Challenges
19

20. Test Status
Test Case coverage and execution status
vs
Session Based Testing
20

21. Transparency
Internal Quality
vs
External Quality
21

22. Clients’ tooling
Different tools
The same tools, different security policies
22

23. Maintenance
One more internal tool for maintenance
JIRA / Confluence / QMetry API changes as example
23

24. Priority decreasing
Reports after release
Still required some manual work (about 2h)
More and more CD projects
24

25. And back again
Few years later
25

26. Adulthood
• Initial objectives are not valid anymore
• Experienced team
• Which know what is important for us
• Which gained the clients and board trust
• From governance to consulting
• QA status as part of project reports
26

27. Be reliable, do not measure, just do it
• Test Automation coverage dilemma
• DoD ensure me that I automated necessary tests on different levels
• Why I should measure the metric if I knew the result?
27

28. Now
• Project’s team makes decision if QA Report is needed and choose
metrics
• Most popular case: only final QA Report before UAT (are we production
ready?)
• Measure process improvement
• For the last proces change (shift left) we measured the improvement rate
• Eg. as a result we measured that Internal Quality raised from 50% even to
90% acceptance rate
28

29. The Ring / QA Reports
• Transform successfully
• Learn how to deliver bigger projects for demanding clients
• Identify what is important for us
• Teach newcomers our ways of working
• And we have a tool for next transformation
29

30. And Back Again
30
Thank You