Making Numbers Count: Metrics That Matter

Making Numbers Count: Metrics That Matter
©2015 Professional Quality Assurance Ltd. (PQA) --http://www.pqatesting.com 1
6/11/2015
Better Software
Conference West
June 11, 2015
Las Vegas, Nevada
Mike Trites
Making Numbers Count
Metrics that Matter
About Me
 Mike Trites
‣ Senior Test Consultant
‣ Over 10 years of experience in software testing, working
with several clients in multiple industry verticals
 Certifications
‣ Black Box Software Testing Foundations – AST
‣ Certified Tester, Foundation Level – ISTQB
‣ Certified Tester, Advanced Level Test Manager – ISTQB
2©2015 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com

6/11/2015
Introduction
How is testing going?
“We’re on-track.”
“95% of test
cases so far
have passed.”
“We found 15
new defects
yesterday.”
What I’m here for…
 Illustrate the important role that metrics play in the software
testing process.
 Explain how to give context when communicating metrics.
 Describe how to use metrics to improve the testing process
for future projects.
 Identify some common pitfalls and how to avoid them.
 Identify a selection of metrics to:
‣ Measure progress during a test cycle.
‣ Determine when to stop testing.
‣ Measure the effectiveness of testing.

6/11/2015
What I’m NOT here for…
 Provide a “one size fits all” solution for you or your
organization.
 Why not?
‣ Metrics are situation / context specific.
‣ There is no silver bullet.
‣ There is no “right” answer.
‣ They will not solve all your problems.
Agenda
 Why Metrics are Important
 Defining and Capturing Metrics
 Reporting on Metrics
 Identifying Trends
 Misuse of Metrics

6/11/2015
Why Metrics are Important

6/11/2015
Why Metrics are Important
Metrics play an
important role in:
• Estimation
• Evaluating suspension
criteria
• Evaluating exit criteria
• Status reporting
• Process improvement
Process Improvement
"Those who cannot remember the past are condemned to
repeat it.”
- George Santayana
 What is “better”? How do we know if we’re better now than
we were?
 Estimates:
‣ Comparing planned effort and actual effort
‣ Estimate variance and what factors affected it

6/11/2015
Metrics for Process Improvement
Closure Period / Turn-
around Time
• Measures how long it
takes for a reported
defect to be rectified
Defect Ratio
• Useful for assessing the
effectiveness of
regression or
automation suites
Metrics for Process Improvement
Defects by Root
Cause
• Allows you to
proactively
avoid
anticipated
defects in the
future
Closed Defects
by Resolution
• Identifies
instances of
duplicate or
invalid defects
Defect Removal
Efficiency
• Identifies
defects that
are uncovered
after the
application is
released

6/11/2015
Defining and Capturing Metrics
Defining Metrics
Every phase of the testing process can be measured.
 Schedule / Budget
 Test Coverage:
‣ Requirements
‣ Configurations
‣ Code
‣ Automation
 Test Design
 Test Execution
 Defects

6/11/2015
Capturing Metrics
Expensive commercial tools can be
useful, but are not always necessary.
• Examples:
‣ Test Case Management Tool
‣ Defect Management Tool
‣ Time Tracking System
There are less expensive tools that can
be used to achieve the same results:
• Examples:
‣ Bugzilla
‣ JIRA
‣ Excel
Reporting on Metrics

6/11/2015
Metrics Need Context
 When metrics are used in isolation, their significance is left
open to interpretation.
 Stakeholders can draw the wrong conclusions if no context
is provided and they are instead left to interpret the data on
their own.
 Textual summary can provide context, but so can a good
chart.
What makes a good chart?
 Accurate – Data is not purposely skewed or omitted.
 Clear – Information is presented in a way that is
unambiguous and easy to interpret at a glance.
 Concise – Chart is not cluttered with unneeded or
distracting data.
 Includes any necessary:
‣ Titles
‣ Labels
‣ Data values
‣ Trend lines
‣ Annotations

6/11/2015
Medium
68%
Low
18%
High
14%
Medium
68%
Low
18%
High
14%
Colour Conventions
Defects by Severity
More meaningful with
appropriate colours
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Example: Test Progress
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%

6/11/2015
Example: Defects by Release
= 63 (49%)
= 56 (44%)
= 9 (7%)
Fixed / Planned
v0.1 v0.2 v0.3 v0.4 v0.5 v0.6 v0.7
Discovered
v0.1 1 9 2
v0.2 7 10 15 1 1 3
v0.3 11 17 1 3
v0.4 1 31 10 3
v0.5 1 1
Example: Scorecard
Execution
Progress
Pass
Rate
Blocked
Tests
Must-Fix Other
Component 1 100 % 100 % 0 0 0
Component 2 100 % 78 % 0 0 4
Component 3 87 % 80 % 0 1 1
Component 4 95 % 100 % 0 0 4
Component 5 75 % 92 % 1 1 0
Component 6 98 % 79 % 2 1 4
Overall 93 % 88 % 3 3 13
DefectsTest Execution

6/11/2015
http://dilbert.com/strips/comic/2007-05-16/
Dashboards
Identifying Trends

6/11/2015
Trends
 One measurement captures status at a point in time.
 When a series of data points are examined, trends in that
data start to emerge.
 Trends allow us to:
‣ Differentiate between systemic behaviour and temporary
anomalies.
‣ Make predictions about the future.
 Accuracy increases with more historical data.
0
5
10
15
20
25
30
35
Defects Detected
Trends
Day
# of
Defects
1
2
3
4
5 15
6
7
8
9
10
?
?
?

6/11/2015
0
5
10
15
20
25
30
35
Defects Detected
0
5
10
15
20
25
30
35
Defects Detected
Trends
Day
# of
Defects
1
2 4
3
4
5 15
6
7
8
9
10
0
5
10
15
20
25
30
35
Defects Detected
0
5
10
15
20
25
30
35
Defects Detected
Trends
Day
# of
Defects
1
2 4
3
4
5 15
6
7
8
9
10 1

6/11/2015
0
5
10
15
20
25
30
35
Defects Detected
Trends
Day
# of
Defects
1
2 4
3
4
5 15
6 2
7
8 8
9
10 1
0
5
10
15
20
25
30
35
Defects Detected
Trends
Day
# of
Defects
1 3
2 4
3 10
4 7
5 15
6 2
7 3
8 8
9 2
10 1

6/11/2015
0
5
10
15
20
25
30
Example: Open vs. Closed Defects
0
5
10
15
20
25
30
0
2
4
6
8
10
12
14
16
18
0
2
4
6
8
10
12
14
16
18
Example: Open Defects

6/11/2015
Misuse of Metrics
Intentional Manipulation
“There are three kinds of lies: lies, damned lies, and
statistics.”
- Benjamin Disraeli
 Metrics can be manipulated to make
any situation look good or bad.

6/11/2015
Misleading Graphs
An example of a good graph:
0
100
200
300
400
500
A B C D
430 455 448 465
Misleading Graphs
Inappropriate scale:
0
200
400
600
800
1000
A B C D

6/11/2015
Misleading Graphs
Scale does not start at 0:
420
425
430
435
440
445
450
455
460
465
A B C D
Tester 1
Tester 2
Tester 3
Tester 4
Misleading Graphs
Defects by Component

6/11/2015
11%
42%
5%42%
Tester 1
Tester 2
Tester 3
Tester 4
Defects by
Component
Misleading Graphs
Poor Choice of Format
12%
15%
14%
8%11%
17%
13%
10% Tester 1
Tester 2
Tester 3
Tester 4
Tester 5
Tester 6
Tester 7
Tester 8

6/11/2015
Better Choice of Format
0%
2%
4%
6%
8%
10%
12%
14%
16%
18%
12%
15% 14%
8%
11%
16%
13%
10%
Overloading
 Often, managers are most
interested in defect statistics.
 Does defect count measure:
‣ Tester effectiveness?
‣ Developer skill?
‣ Product quality?
‣ Effort remaining?
 One metric does not tell the entire story.

6/11/2015
Performance Assessment
 People optimize what we measure them against, at the
expense of what we don’t.
 For example, if we assess testers based on their defect
counts, how much time will testers spend:
‣ Documenting their tests?
‣ Coaching other testers?
‣ Researching and polishing defect reports to make their
defect easier to understand, assess and replicate?
‣ Logging variants of the same failure to increase defect
count?
Performance Assessment
0
100
200
300
400
500
Tester 1 Tester 2 Tester 3
Test Cases Executed

6/11/2015
Key Points to Remember
 Metrics can be captured for just about any testing activity.
 Metrics play an important role in:
‣ Estimation
‣ Status reporting
‣ Evaluating suspension and exit criteria
‣ Process improvement
 Metrics, like anything, should be planned in advance.
 Reporting needs should be identified before setting up
tools.
 To avoid misinterpretation, metrics need context.

6/11/2015
 Even when properly presented, too much information can
be overwhelming.
 It is best to limit the number of metrics and reports that are
presented to stakeholders to only what is essential.
 Capturing, analysing and reporting on metrics can be a
drain or waste of time if those metrics are not being used to
provide some value in return.
 Metrics influence tester performance, but not always in the
way you might think.
 Metrics should be used as a starting point for discussion.
 One metric does not tell the entire story.
Questions or Comments?

6/11/2015
Thanks for coming!
Contact Information:
©2015 Professional Quality Assurance Ltd. (PQA) -- http://www.pqatesting.com
Mike.Trites@pqatesting.com
ca.linkedin.com/in/miketrites
@mike_trites
www.pqatesting.com
49

Making Numbers Count: Metrics That Matter

Recommandé

Recommandé

Contenu connexe

En vedette

En vedette (12)

Similaire à Making Numbers Count: Metrics That Matter

Similaire à Making Numbers Count: Metrics That Matter (20)

Plus de TechWell

Plus de TechWell (20)

Dernier

Dernier (20)

Making Numbers Count: Metrics That Matter