Contenu connexe Similaire à Agile Metrics - ASTQB Workshop by Philip Lew - XBOSoft (20) Agile Metrics - ASTQB Workshop by Philip Lew - XBOSoft4. #AgileMetrics
@XBOSoft @philiplew
A Little About Me
• Work Experience
– Telecommunications network engineer
– Team Lead, Data warehousing product
development
– Software product manager, BI product
• Specialties
– Software quality process improvement
– Software quality evaluation and measurement
– Software quality in use / UX design
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
6. #AgileMetrics
@XBOSoft @philiplew
Session Spirit and Expectations
• I won’t read the slides…
• Slides for you as a take-away (reference)
• Slightly different than your handouts
– Always thinking of new examples and ideas,
you can write me and I’ll send these to you
• Interactive
– Lots of exercises and chances to learn
– Lots of material
• Only 3 hours time for the workshop
– May skip over material – use as reference
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
8. #AgileMetrics
@XBOSoft @philiplew
UlQmately
What
We
Want
to
Learn
Today
• Mistakes
people
make
in
agile
metrics
and
how
to
avoid
them.
• How
to
consistently
and
systemaQcally
improve
root
causes
of
low
velocity.
• How
to
reduce
rework.
• How
to
analyze
your
agile
process
and
determine
meaningful
metrics
to
present
to
management.
• How
to
get
to
velocity
AND
quality,
and
what
to
measure.
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
9. #AgileMetrics
@XBOSoft @philiplew
Software Quality Journey
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
SoFware
Quality
SoFware
TesQng
TesQng
Metrics
Why
and
What
Metrics
Goal
QuesQon
Metric
SoFware
Dev
Processes
SoFware
Quality
Models
Business&
Product
Processes
EvaluaQon
Frameworks
AGILE
METRICS
10. #AgileMetrics
@XBOSoft @philiplew
The
SoFware
Quality
Conundrum
• What
quesQons/mysteries
do
you
have
about
soFware
quality
metrics?
– SoFware
quality
is
too
nebulous
to
get
your
hands
around
– Everyone
has
a
different
idea
of
what
it
is,
so
difficult
to
measure
– What
can
we
measure
for
quality
in
agile?
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
14. #AgileMetrics
@XBOSoft @philiplew
“Software is Eating the World
And now mobile is too”
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
conventional
software
applications
Web sites E-commerce
Print
advertising
Telephone
books
Web 1.0
Fully functional
applications>>
Higher
complexity
User profiling,
segmentation,
personalization
data collection
Web 2.0
Information
integration
Semantic Web
Web 3.0 - Big data and Mobile
Web-basedapplicationsand
nowmobileaccessand
nativemobile
New ideas,
integrated
platforms,
more
components
Ubiquitous
Access
15. #AgileMetrics
@XBOSoft @philiplew
Quality Gains Importance
and Shifts Meaning
Quality
needs
more
emphasis
• ShiFs
in
user
expectaQons
and
business
models
that
enable
users
to
switch
applicaQons
quickly.
• ApplicaQons
that
have
low
usability
or
quality
are
easily
leF
for
another
• Many
alternaQves
available
with
another
click
– Short
trial
subscripQon
– Many
trials
are
free
– No
cost
factor
involved
in
switching
applicaQons
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
Development
cycle
16. #AgileMetrics
@XBOSoft @philiplew
The Problem
• Evaluating the quality of software
applications is difficult.
• Research and standards are not entirely
applicable
• Models and frameworks lack
implementation specifics so hard to use
and benefit from them.
• How do we objectively measure software
quality?
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
17. #AgileMetrics
@XBOSoft @philiplew
Motivation
Why is QA Measurement Important
• Quality
Assurance
measurement
provides
informaQon
for
improvement
in
your
tesQng,
your
team,
and
your
company.
– Team
• Is
your
QA
team
improving?
– Process
• Have
you
reached
the
highest
state
of
efficacy?
• Is
your
tesQng
as
thorough
as
it
can
be?
– Product
• Is
soFware
ready
for
release?
Can
we
stop
tesQng
now?
• What
parts
of
the
soFware
are
at
most
risk
?
– Money
• Are
you
able
to
do
everything
you
want
within
budget?
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
What questions do you have?
20. #AgileMetrics
@XBOSoft @philiplew
Exercise - Value of Metrics
Why
do
you
want
to
measure?
1. ______________________________________
2. ______________________________________
3. ______________________________________
4. ______________________________________
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
23. #AgileMetrics
@XBOSoft @philiplew
What is Software Quality?
• Hard to define and evaluate
• Perceived quality
• Transparent when present
• Noticeable when absent
• Many views
• User View
• Product View
• Developer/Coder View
• Installer View
• Management View
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
What other
viewpoints
could their
be?
24. #AgileMetrics
@XBOSoft @philiplew
What is Software Quality?
• The
meaning
of
the
quality
term
is
not
simple
and
atomic,
but
a
mulQdimensional
and
abstract
concept.
• Quality
can
not
be
measured
directly
–
at
least
in
a
not
very
trivial
way
• Common
pracQce
assesses
quality
by
means
of
the
quanQficaQon
of
lower
abstracQon
concepts,
such
as
agributes
of
enQQes
• Given
the
complexity
that
a
quality
concept
involves,
a
model
is
used
in
order
to
specify
the
quality
requirements.
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
26. #AgileMetrics
@XBOSoft @philiplew
Exercise
• How does your organization define
software quality?
• In your group, come up with a definition of
software quality that you can agree on.
– Can be one sentence to a short
paragraph.
• Be prepared to share with the class.
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
28. #AgileMetrics
@XBOSoft @philiplew
ISO
25010:
“DEGREE
TO
WHICH
THE
SOFTWARE
PRODUCT
SATISFIES
STATED
AND
IMPLIED
NEEDS
WHEN
USED
UNDER
SPECIFIED
CONDITIONS”
Quality
Standards
and
Models
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
29. #AgileMetrics
@XBOSoft @philiplew
Quality
of
a
SoFware
Product
(Three
Views
for
Quality)
Internal
Quality
• Measure
of
the
degree
to
which
a
set
of
staQc
agributes
of
a
soFware
product
saQsfy
stated
and
implied
needs
when
the
soFware
product
is
used
under
specified
condiQons.
External
Quality
• Measure
of
the
degree
to
which
a
soFware
product
enables
the
behavior
of
a
system
to
saQsfy
stated
and
implied
needs
when
the
system
including
the
soFware
is
used
under
specified
condiQons.
Quality
in
Use
• Degree
to
which
a
product
used
by
specific
users
meets
their
needs
to
achieve
specific
goals
with
effecQveness,
efficiency,
flexibility,
safety
and
saQsfacQon
in
specific
contexts
of
use.
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
What are some examples of quality measurements from these 3 perspectives?
ISO
25010
33. #AgileMetrics
@XBOSoft @philiplew
Evolution of SW Quality
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
Software
processes
Code
Process
quality
Software
Quality
(internal)
Software
Quality
(external)
Software
Quality
(in use)
Product Usability
CMMI
assessment
model
white box
testing
black box
testing
Usability testing
Usability heuristic
Usability logging
CMMI
Assessment
Company
Programmer Tester End User
Type of
quality
What is
measured
How
measured?
Who
measures?
ISO 9000 ISO 9241ISO 9126 ISO 25010
36. #AgileMetrics
@XBOSoft @philiplew
Typical View of the Quality Life Cycle
Development
Quality
Assurance
• The emphasis of most organizations is to eliminate defects in the
Development and QA phases of the Software Development Life
Cycle.
• Current quality models don’t model outside of development and QA
Goal: Eliminating Defects
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
37. #AgileMetrics
@XBOSoft @philiplew
Building
a
Quality
Product
or
Service
• Using modeling concepts of influence and
dependency that we covered earlier
• Quality of each phase or step has an
influence on the quality of the next phase
AND on the final result
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
38. #AgileMetrics
@XBOSoft @philiplew
End User Perception of Quality
• Dependent on good product, of course.
• But also influenced by many other factors
outside of development and quality
assurance.
• Other parts of the organization and overall
process influence the quality of the
product itself and user’s perception of
quality.
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
40. #AgileMetrics
@XBOSoft @philiplew
Measuring Quality
• Within each phase, we model to understand and
evaluate quality at that phase
• At each phase, define quality and develop quality
measures
• The lower level measurements will aggregate to
higher cumulative measurements
• Aggregation can be done with various weighting
schemes with flexibility according to the business
of the organization
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
41. #AgileMetrics
@XBOSoft @philiplew
Quality Tree
Customer
Perceived
Quality
Sales
Quality
Requirements
Quality
Product
Development
Quality
Customer
Service
Quality
Possible
to
have
a
high
quality
product,
but
with
poor
customer
perceived
quality!
Here’s
Why…
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
Each Component or Phase
Has its Own Quality which
aggregates and influences
overall customer perception of
quality
What weighting scheme would
you use and why?
42. #AgileMetrics
@XBOSoft @philiplew
Product Mgmt – Requirements
Quality Definition
Product
Management
Quality
Development
understanding
CommunicaQon
with
development
Complete
ness
Accuracy
Timeliness
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
What other attributes would be appropriate
to model product management quality?
43. #AgileMetrics
@XBOSoft @philiplew
CSR
QA
Devl
Reqm’t
Sales
PM View of the Quality LifeCycle
Customer Interactions
Defects are introduced at all phases of the lifecycle of a product.
Requirements:
• Vague, ambiguity measures
• Incomplete
• Poorly Written-not clearly understood
• Lack of understanding of underlying needs-don’t match customer wants
Goal: Improving Quality
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
44. #AgileMetrics
@XBOSoft @philiplew
Development and QA
Quality Definition
Development
Quality
Code
Re-‐work
Adherence
to
standards
Defects
Test
ware
ValidaQon
and
verificaQon
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
What other attributes would be
appropriate to model DEV-QA quality?
45. #AgileMetrics
@XBOSoft @philiplew
CSR
QA
Devl
Reqm’t
Sales
Dev View of the Quality LifeCycle
Customer Interactions
Defects are introduced at all phases of the lifecycle of a product.
Development:
• Coding errors
• Non conformance to corporate coding best practices
• Not extensible
• Poor performance
• Rework
• Recurring Defect
Goal: Improving Quality
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
46. #AgileMetrics
@XBOSoft @philiplew
CSR
QA
Devl
Reqm’t
Sales
QA View of the Quality LifeCycle
Customer Interactions
Defects are introduced at all phases of the lifecycle of a product.
Quality Assurance:
• No test cases
• Non-standard test cases
• Inaccurate test cases, incomplete
• Defects not written understandably
• Defects unverifiable
• Defects incomplete information
Goal: Improving Quality
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
47. #AgileMetrics
@XBOSoft @philiplew
Key
Modeling
Concepts
Take
Aways
• Breakdown concept (quality) to components
(levels of abstraction) to understand, evaluate,
measure, and then improve
– Hierarchical requirements tree
– Measurements for each factor or element
(characteristic)
• Relationships and dependency
– Quality at one phase influences quality at the
next phase
– Not only in an organization but also within a
process
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
48. #AgileMetrics
@XBOSoft @philiplew
Exercise
1. Does your organization have a model for
software quality?
2. For your small group, develop a model for
software quality.
3. Be prepared to share with the class.
a. Why you included certain things
b. Why you left out others
c. Are all components evenly weighted? Why
not?
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
51. #AgileMetrics
@XBOSoft @philiplew
Understand, Evaluate and
Improve
• If our end goal is improvement in what?
• To improve, we need to evaluate
• In order to evaluate, we must understand
what we are evaluating
• To do this… We need metrics
Can you think of other examples in our lives
where this applies? Where do you use metrics
to evaluate and improve?
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
52. #AgileMetrics
@XBOSoft @philiplew
Metrics in real life
Food
Eaten
Weight
Performance
Race
Results
• Calories
• Fat
• Carbohydrates
• Protein
• Time of day
• Vitamins
• …
• Blood pressure
• Cholesterol
• Blood glucose
• Red cell count
• White cell count
• Hematocrit
• Hemoglobin
• Body fat %
• …
• Placing
• …
• Effort/Power
• Heart rate/Watts
• Speed
• Time
Intelligence
Finesse
Context
• Training
• Sleep
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
You don’t just measure your weight right? You measure calories!!
53. #AgileMetrics
@XBOSoft @philiplew
Metrics - Goals
• Understand
how
QA,
tesQng,
and
its
processes
and
where
the
problems
are
• Evaluate
the
process
and
the
product
– Monitor
how
something
is
performing
– Analyze
the
informaQon
provided
to
drive
improvements
• Improve
– Predict
and
control
process
and
product
qualiQes
– Reuse
successful
experiences
– Take
correcQve
acQon
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
54. #AgileMetrics
@XBOSoft @philiplew
Getting Started
• What you choose to measure is up to your
measurement goals/objectives – First
exercise in class
• You can only measure improvement in
areas that you have quantified.
• Establishing a metrics and measurement
framework is necessary … More later on
frameworks coming up!
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
55. #AgileMetrics
@XBOSoft @philiplew
Measurement Helps Us Answer
Questions
• Create a organizational memory –
baselines of current practices-situation
• Determine strengths and weaknesses of
the current process and product
– What types of errors are most common?
• Develop reasoning for adopting/refining
techniques
– What techniques will minimize the
problems?
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
56. #AgileMetrics
@XBOSoft @philiplew
More Questions That Metrics Can
Help Answer (2/2)
• Assess the impact of techniques
– Does more/less functional testing reduce
defects?
– Do less defects shorten help desk call times?
• Evaluate the quality of the process/product
– Are we applying inspections appropriately?
– What is the reliability of the product before/
after delivery?
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
58. #AgileMetrics
@XBOSoft @philiplew
Need for Consistent
Terminologies
• To many organizations, errors can mean faults.
• Anomalies usually mean a class of faults that are unlikely
to cause failures but may eventually cause failures
indirectly.
– anomaly is a deviation from the usual, but it is not
necessary wrong.
• Defects normally refer collectively to faults and failures.
– Sometimes a defect is a particular class of fault.
• Bugs refer to faults occurring in the code.
• Crashes are a special type of incident, where the system
ceases to function.
• Define terms clearly
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
59. #AgileMetrics
@XBOSoft @philiplew
Defects - Get All the Info
• So
once
we
get
common
terms
on
incidents,
faults
and
defects,
we
want
to
record
its
key
elements,
so
that
we
can
then
invesQgate
causes
and
cures:
– Loca%on:
Where
did
the
problem
occur?
– Timing:
When
did
it
occur?
– Mode:
What
was
observed?
– Effect:
Which
consequences
resulted?
– Mechanism:
How
did
it
occur?
– Cause:
Why
did
it
occur?
– Severity:
How
much
was
the
user
affected?
– Cost:
How
much
did
it
cost?
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
60. #AgileMetrics
@XBOSoft @philiplew
Exercise
• Collecting key measurements/information
is critical to a metrics program.
• Discuss in your groups:
– Define the relevant terms in collecting defects,
faults, incidents, error, etc..
– List out what information are you collecting on
your defects and what is lacking?
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
63. #AgileMetrics
@XBOSoft @philiplew
Measures Versus Metrics
Measures
• To
be
executed
• Executed
• Passed
• Failed
• Re-‐executed
• Total
ExecuQons
• Total
Passes
• Total
Failures
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
Metrics
• TC % complete
• TC % passed
• % failures
• % defects
corrected
• % bad fixes
• % defect
discovery
64. #AgileMetrics
@XBOSoft @philiplew
© 2015 XBOSoft, Inc.- All Rights Reserved.
What is Measurement?
• “Measurement is the process by which numbers
or symbols are assigned to attributes of entities
in the real world in such a way as to characterize
them according to clearly defined rules.” [1]
• “Measurement is the empirical, objective
assignment of numbers, according to a rule
derived from a model or theory, to attributes of
objects or events with the intent of describing
them.” [1]
[1] Software Engineering Metrics: What Do
They Measure and How Do We Know? Cem
Kaner and Walter P. Bond
65. #AgileMetrics
@XBOSoft @philiplew
Measurement
• First component of a measurement is
it scale, the four levels of scales are
nominal scale, ordinal scale, interval scale,
and ratio scale.
• Second component is measurement
quality
• Third component is causality
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
66. #AgileMetrics
@XBOSoft @philiplew
Nominal Scale
• Classifies
or
categorizes
the
agribute
being
measured.
• An
example
of
this
would
be
the
classificaQons
male
and
female.
• Doesn’t
allow
for
comparisons
to
be
made
or
mathemaQcal
operaQons
to
be
preformed.
– So
the
expression
male
>
female
or
male
<
female
are
invalid.
– It
is
impossible
to
subtract
a
female
from
a
male.
• The
key
requirement
of
the
nominal
scale
is
that
each
data
item
can
fit
into
only
one
category.
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
67. #AgileMetrics
@XBOSoft @philiplew
Ordinal Scale
• Similar
to
the
nominal
scale
except
categories
can
be
ordered
and
comparisons
can
be
made.
• An
example
of
this
would
be
customer
saQsfacQon
surveys
which
oFen
require
an
answer
of
completely
dissaQsfied,
somewhat
dissaQsfied,
neutral,
somewhat
saQsfied,
or
completely
saQsfied.
• In
such
a
case,
there
is
a
natural
order
to
the
scale
unlike
nominal
scales.
• Like
nominal
scales
mathemaQcal
operaQons
sQll
can’t
be
directly
performed.
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
68. #AgileMetrics
@XBOSoft @philiplew
Interval Scale
• Indicates
exact
differences
between
measurement
points
unlike
the
previous
two
scales.
• Only
the
mathemaQcal
operaQons
add
and
subtract
can
be
applied
to
an
interval
scale
because
the
zero
point
on
the
scale
has
been
arbitrarily
decided
upon.
• For
example
20
degrees
is
hoger
than
10
degrees
but
20
degrees
may
not
be
twice
as
hot
as
10
degrees.
• The
key
requirement
of
the
interval
scale
is
that
units
be
clearly
defined.
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
69. #AgileMetrics
@XBOSoft @philiplew
Ratio Scale
• Ratio scale is the highest level of
measurement much the same as interval
scales
• Except zero is a non-arbitrary point
allowing mathematical operations such as
division.
• Examples of ratio scale measurements are
weight and length.
– 160 pounds is twice that of 80 pounds
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
70. #AgileMetrics
@XBOSoft @philiplew
Measurements Quality
• Two
facets
of
measurement
quality
are
reliability
and
validity.
• Reliability:
means
how
repeatable
the
results
are
and
low
reliability
indicates
that
many
random
errors
are
occurring
during
measurement.
• Validity:
means
the
metric
measures
what
it
intended
to
measure.
– Errors
in
validity,
called
systemaQc
errors,
are
usually
experienced
because
not
all
factors
were
taken
into
account
when
the
measurement
was
designed.
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
71. #AgileMetrics
@XBOSoft @philiplew
Metrics Abandoned for Many Reasons
• How many of you have had these
problems?
– That measurement is not valid because…
– That measurement is not reliable because…
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
If no one believes (and therefore takes
improvement actions) your metrics,
then why bother?
72. #AgileMetrics
@XBOSoft @philiplew
Guiding principles about metrics
• Metrics can represent a snapshot
• Metrics can change over time
• Metrics can be simple and still very valuable/
telling
• Metrics can assist us in making decisions today
• Metrics can assist us in predicting the future
• Some metrics are critical for some while useless
for others
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
73. #AgileMetrics
@XBOSoft @philiplew
Metrics can represent a
snapshot
• Not only at a particular moment, like this
week, but for a process over time
Code
quality
Internal
Quality
Tested
in
a
lab
External
Quality
• What
the
end
user
sees
• What
the
end
user
experiences
In-‐Use
Quality
At what stage in the process do you take measurements?
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
74. #AgileMetrics
@XBOSoft @philiplew
Some Metrics – Let’s get started
Schedule
variance
Effort
variance
Cost
variance
Defect
removal
effecQveness
ProducQvity
Defect
aging
CriQcal
Defect
rate
Test
cases
executed
Test
coverage
Pass/Fail
Rate
ROI
AutomaQon
coverage
Value
Provided
%
Defects/
hour
Customer
saQsfacQon
Avg.
Time
to
Fix
Recurrence
Rate
Code
complexity
Test
Density
Defect
Find
Rate
Defect
Arrival
Rate
Require
ments
VolaQlity
Require
ments
Ambiguity
Planned
/Actual
OverQme
Rate
CompleQon
Rate
Velocity
ExecuQon
Rate
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
75. #AgileMetrics
@XBOSoft @philiplew
Some Metrics – Let’s get started
• How many of these do you know/
use?
• What others do you use?
• What are they used for?
• By who?
• How do you evaluate them?
• What is ‘good’, ‘acceptable’?
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
77. #AgileMetrics
@XBOSoft @philiplew
Defects Found/Build
• Common
Usage:
Progressive
Build
Improvement
• One
way
to
measure
the
stability
of
builds
over
Qme
–
and
in
comparison
to
one
another
• Number
of
defects
found
should
decrease
per
build
• Spike
when
builds
contain
many
new
features.
• New
features
will
result
in
higher
bug
counts,
but
over
the
course
of
the
project,
from
build
to
build,
the
number
of
bugs
found
in
each
build
should
go
down.
• Need
baseline.
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
Assumes that your test coverage, methods, and
accuracy do not fluctuate.
78. #AgileMetrics
@XBOSoft @philiplew
Defects found/Module
• Common Usage: Determination of Defect Density
• Discovers which module/features functionality have the
most bugs.
• Benchmarking tool to determine test efficacy and
productivity.
– Used in conjunction with a measurement of how long
each test (or test module) takes >> cost of testing.
– How long it takes to find a defect (average and total)
in each tested area.
– Are you spending many hours testing an area that
never yields bugs.
– Make decisions - where to expend your test effort.
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
79. #AgileMetrics
@XBOSoft @philiplew
Time Between Defects Found
• Common
Usage:
Build
Improvement
• Used
to
determine
whether
a
project
is
progressing
as
it
should.
• As
a
project
gets
closer
to
compleQon
– Time
that
passes
between
defects
found
should
become
greater.
– If
no
change,
or
if
suddenly
the
Qme
between
bugs
found
gets
shorter
as
a
project
nears
its
shipping
date,
cause
for
concern.
• If
the
Qme
between
bugs
found
does
not
increase,
then
the
project
is
not
advancing
quickly
enough.
– Creates
the
risk
of
shipping
the
product
with
a
high
number
of
known
bugs
and/or
with
many
bugs
that
are
not
found
before
it
goes
live.
– Or
miss
the
ship
date
in
order
to
make
sure
the
bugs
are
fixed
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
80. #AgileMetrics
@XBOSoft @philiplew
Priority Defects Found/
• Common
Usage:
TesQng
Efficacy
in
Specified
Timeframe
Used
to
ascertain
if
the
test
team
is
finding
the
high
Severity
and
high
Priority
defects.
• If
the
majority
of
bugs
found
are
nit-‐picky
low
Priority
issues,
then
either
the
release
is
low-‐risk,
or
QA
is
missing
criQcal
issues
that
will
arise
later.
• Important
to
know
which
tests
have
been
executed.
– Are
the
tests
that
are
being
performed
finding
the
issues
that
are
important
to
the
company?
– Is
the
test
team
using
its
Qme
and
resources
as
effecQvely
as
possible?
• Not
every
defect
found
will
be
of
the
highest
priority.
– Tests
run
should
find
issues
that
are
deemed
to
be
important
to
fix.
– Otherwise
effort
is
being
wasted.
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
81. #AgileMetrics
@XBOSoft @philiplew
Mean Time Between P1 Defects
• Common
Usage:
Build
Improvement
/
Shipping
Readiness
• As
soFware/build
progresses
and
nears
shipping
readiness,
the
average
Qme
between
P1
(Priority
=
1)
defects
should
become
greater.
• Any
P1
found
will
be
a
showstopper,
so
it
becomes
important
that
the
product
progress
well
enough
for
P1s
to
stop
appearing.
• When
the
mean
Qme
between
P1s
found
stagnates
or
lessens,
this
should
be
a
warning
sign
that
your
project
is
regressing.
• This
is
one
of
the
clearest
quality
assurance
metrics
that
you
can
track.
It
can
carry
a
great
deal
of
weight
and
impact.
Use
it
wisely.
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
82. #AgileMetrics
@XBOSoft @philiplew
Time to perform tests
• Common
Usage:
TesQng
Efficiency
• Measurement
of
your
team’s
tesQng
efficiency
is
how
long
it
takes
to
perform
selected
tests.
• The
first
Qme
a
test
(or
set
of
tests)
is
executed,
should
be
higher
than
during
subsequent
execuQons.
• As
your
team
becomes
familiar
with
each
test
and
learns
the
nuances,
test
Qme
should
decrease.
• Some
tests
(automated
or
acQvity
dependent)
may
not
vary,
but
the
setup
and
subsequent
result
capture
should
take
less
Qme.
• Can
idenQfy
which
tests
can
be
run
concurrently
or
in
parallel
to
gain
Qme
efficiency.
• Can
use
to
esQmate
tesQng
future
projects
• Used
in
conjuncQon
with
bug
density,
will
show
you
where
you
can
best
spend
your
Qme
tesQng.
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
83. #AgileMetrics
@XBOSoft @philiplew
Bugs found per test
• Common
Usage:
Test
Efficacy
/
Bug
Density
• Evaluate
whether
your
tests
are
targeted
and
thorough
or
not.
• Assuming
your
tesQng
does
not
fluctuate,
this
metric
will
show
you
if
you
are
tesQng
areas
that
are
yielding
bugs.
• Knowing
which
tests
yield
how
many
bugs
–
both
as
a
total
number
and
as
a
percentage
–
allows
you
to
evaluate
your
tesQng.
• When
measured
in
conjuncQon
with
a
“Qme
to
perform
tests”
metric,
you
can
quickly
see
if
you
are
wasQng
your
tesQng
effort
or
leveraging
your
team
effecQvely.
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
84. #AgileMetrics
@XBOSoft @philiplew
Defect Removal Efficiency
• Common
Usage:
TesQng
Efficiency
• DRE
=
E
/
(
E
+
D
)
• Where
E
=
No.
of
Errors
found
before
delivery
of
the
soFware
and
D
=
No.
of
Errors
found
aFer
delivery
of
the
soFware.
• Ideal
value
of
DRE
should
be
1
which
means
no
defects
found
aFer
delivery.
• If
you
score
low
on
DRE
it
means
to
say
you
need
to
re-‐look
at
your
exisQng
process.
• DRE
is
a
indicator
of
the
filtering
ability
of
quality
control
and
quality
assurance
acQvity.
• Encourages
the
team
to
find
as
many
defects
before
they
are
passed
to
the
next
acQvity
stage.
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
85. #AgileMetrics
@XBOSoft @philiplew
Some User Oriented Metrics
• Mean
%me
to
failure
metric
:
the
average
Qme
the
product
runs
before
experiencing
a
crash
– important
for
systems
like
air
traffic
control
(failure
could
be
subsQtuted
or
defined
per
your
objecQve).
Collected
during
help
desk
funcQon.
• Customer
problem
metric
:
measure
of
problems
customers
have
encountered
with
the
product
over
the
total
usage
of
the
product.
– Accounts
for
mulQple
instances
of
the
product
can
be
used
at
the
same
Qme,
which
effecQvely
mulQplies
the
length
of
Qme
the
product
has
been
in
operaQon
by
the
number
of
product
licenses.
Collected
during
help
desk
acQviQes.
• Customer
sa%sfac%on
metric
:survey
asking
customers
to
rate
their
saQsfacQon
with
the
product
and/or
it’s
features
on
a
five-‐point
scale,
usually
done
in
conjuncQon
as
part
of
a
usability
tesQng
effort.
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
86. #AgileMetrics
@XBOSoft @philiplew
QA/Development
Process Metrics
• Defect
arrival
pa@ern
metric
:
combinaQon
of
the
rate
at
which
defects
are
reported
and
the
growth
of
these
rates
through
out
the
process
being
measured.
• Backlog
management
index
metric
:
measures
the
effecQveness
of
the
process
by
comparing
the
number
of
problems
that
arrive
to
the
number
of
fixes
during
a
specified
period.
• Fix
quality
metric
:
measures
the
number
of
defecQve
fixes
compared
to
the
number
of
successful
fixes.
A
fix
is
defecQve
when
either
it
doesn’t
repair
the
problem
reported
or
it
repairs
the
problem
but
introduces
new
defects
into
the
product.
• Fix
response
%me
metric
:
the
Qme
between
when
a
problem
is
reported
and
the
Qme
a
fix
is
issued.
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
87. #AgileMetrics
@XBOSoft @philiplew
More
• Test Coverage = Number of units (KLOC/FP) tested / total
size of the system
• Number of tests per unit size = Number of test cases per
KLOC/FP
• Defects per size = Defects detected / system size
• Cost to locate defect = Cost of testing / the number of
defects located
• Defects detected in testing = Defects detected in testing /
total system defects
• Defects detected in production = Defects detected in
production/system size
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
88. #AgileMetrics
@XBOSoft @philiplew
More and More
• Quality of Testing = No. of defects found during Testing/
(No. of defects found during testing + No of acceptance
defects found after delivery) *100
• System complaints = Number of third party complaints /
number of transactions processed
• Test Planning Productivity = No of Test cases designed /
Actual Effort for Design and Documentation
• Test Execution Productivity = No of Test cycles executed /
Actual Effort for testing
• Test efficiency= (number of tests required / the number of
system errors)
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
89. #AgileMetrics
@XBOSoft @philiplew
More More and More
• Using uninitialized or invalid memory
• Null pointer dereferencing
• Array and buffer overflows
• Division by zero
• Memory or resource leaks
• Dead code
• Code complexity (by module)
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
90. #AgileMetrics
@XBOSoft @philiplew
Even More
• Common metrics include:
– Number of requirements to test
- by function - by priority
– Number of test cases
- created - executed
- passed - failed
– Number of defects
- by severity - by status
Develop a Measurements and Metric Catalog
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
91. #AgileMetrics
@XBOSoft @philiplew
Exercise
Work
in
your
groups
and
list
out:
• The
metrics
you
currently
use
– What
you
use
them
for
– Who
wants
them
– How
you
evaluate
them
– What
decisions
they
make
based
on
them
• The
metrics
you
don’t
currently
use
but
think
you
should
You may find some holes here.
What are the common ingredients missing?
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
93. #AgileMetrics
@XBOSoft @philiplew
Don’t – Forget indicators
• Metrics without indicators cannot be
evaluated
– You collect mounds of data but then what?
– How do you determine what is ‘good’ or ‘bad’?
– Need a way to evaluate the numbers you get
with meaningful indicators that can be used as
benchmarks as you/your process and
organization matures
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
95. #AgileMetrics
@XBOSoft @philiplew
Indicators
• Need
an
elementary
indicator
to
interpret
the
level
of
saQsfacQon
met
with
decision
criteria
with
acceptability
ranges
in
a
percentage
scale:
– 0-‐40
(unsaQsfactory
–red)
means
changes
must
take
place
with
high
priority;
– 40-‐70
(marginal
–yellow)
indicates
a
need
for
improvement
acQons;
– 70-‐100
indicates
a
saQsfactory
level
–green-‐
for
the
analyzed
agribute.
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
96. #AgileMetrics
@XBOSoft @philiplew
Beware of Causality
• The
cause
and
effect
relaQonship
has
three
requirements:
– cause
precedes
effect
in
Qme
or
by
logic.
– shown
cause
and
effect
relaQonship
– direct
constant
relaQonship
• When
the
relaQonship
between
variables
can
be
graphed
and
a
line
of
best
fit
can
be
found
• Be
careful
of
extreme
and
inaccurate
data
values.
• When
defining
a
casual
relaQonship
be
sure
to
watch
for
misguided
interpreted
relaQonships.
CASE
1
Z
causes
changes
in
X
Z
causes
changes
in
Y
X
is
perceived
to
cause
changes
in
Y
or
vice
versa
and
Z
is
overlooked
CASE
2
X
causes
changes
in
Z
Z
causes
changes
in
Y
X
is
perceived
to
cause
changes
in
Y
and
Z
is
overlooked
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
98. #AgileMetrics
@XBOSoft @philiplew
Look at Trends
• Identifying trends over time
– More
– Less
– Stable
• Make informed decisions
– Identify what is happening
– Determine what you want to happen
– Make adjustments
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
99. #AgileMetrics
@XBOSoft @philiplew
Time and Beware of Absolutes
• Don’t use absolute numbers to determine
if a build is ready to ship.
– If they have released a product for
several years metrics can change over
time.
– What reasons can metrics change over
time?
– What particular elements of agile can
cause metrics to change over time?
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
100. #AgileMetrics
@XBOSoft @philiplew
Don’t – Forget about context
• Metrics don’t have consistent context so
they are unreliable – Context needs to be
defined and then maintained for
measurements to be meaningful.
• Difficult in today’s environment with
changing test platforms and other
contextual factors.
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
101. #AgileMetrics
@XBOSoft @philiplew
What contextual factors
could there be?
• Release complexity
• Development methodology
• Software maturity
• Development team maturity and expertise
• Development team and QA integration
• Resources available
• User base
All metrics need to be
normalized for proper
interpretation
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
102. #AgileMetrics
@XBOSoft @philiplew
Exercise
• List out contextual factors in our
organization that are needed to ensure
proper interpretation of any metrics and
measurements
• List out any past mistakes your
organization has made when using and
interpreting metrics
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
104. #AgileMetrics
@XBOSoft @philiplew
Why We Measure?
• Our bosses want us to…
• They want someone to point fingers at
• They need to report to their managers
• They want some basis on which to
evaluate us
• We want to improve our work and improve
software quality
• How do you want to be evaluated?
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
105. #AgileMetrics
@XBOSoft @philiplew
The Metrics Conundrum
• QA and Testing
Language
– Defects
– Execution status
– Test cases
– Pass/fail rates
– DRE…
• Business
Language
– Cost
effecQve
– ROI
– Cost
of
ownership
– Cost
of
poor
quality
– ProducQvity
– Calls
to
help
desk
– Customer
saQsfacQon
– Customer
retenQon
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
106. #AgileMetrics
@XBOSoft @philiplew
Metrics In The Real World
• What measurements do you take in your
organization and why?
• Who uses them and for what?
• Measurement is pie. It takes 2-3 hours to
make and 15 minutes to consume…
• But… many metrics are never reviewed or
analyzed (consumed)
• WHY?
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
107. #AgileMetrics
@XBOSoft @philiplew
The Metric Conundrum (cont.)
• Rarely have the right metrics to show or
quantify value
• Metric collection and reporting are a drag
• QA metrics usually focus only on test
execution
• Test tools don’t have the metrics we want
• Metrics are not connected to anything of
value/meaningful for ________.
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
108. #AgileMetrics
@XBOSoft @philiplew
Poll: How many of you started to
collect metrics but then found it
was too difficult or time
consuming and quit?
Never started:
Started then quit:
Still collecting and reporting:
13%
30%
57%
115. #AgileMetrics
@XBOSoft @philiplew
Organizing Our Thoughts
• Putting metrics into a framework helps us
to categorize and organize
• Helps us to make sure we have all
functions in our organization and process
covered
• Helps us to develop relationships between
different elements in our process
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
116. #AgileMetrics
@XBOSoft @philiplew
Measurement Frameworks
• Many
soFware
metrics
programs
have
failed
because
they
had
poorly
defined,
or
even
non-‐existent
objecQves.
• Basili
developed
a
goal
oriented
approach
(GQM)
to
measurement
according
to
the
following
three
stages:
– Set
goals
specific
to
needs
in
terms
of
purpose,
perspec%ve
and
environment.
– Refine
the
goals
into
quanQfiable
quesQons
that
are
tractable.
– Deduce
the
metrics
and
data
to
be
collected
(and
the
means
for
collecQng
them)
to
answer
the
quesQons.
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
117. #AgileMetrics
@XBOSoft @philiplew
GQM for Software Quality
• Overall
GOAL
is
to
evaluate
the
quality
of
the
soFware
• To
decide
if
the
quality
is
‘good’,
several
key
QUESTIONs
must
be
asked
to
understand
how
to
meet
the
goal
and
determine
quality
• Once
these
quesQons
are
idenQfied,
analyze
each
quesQon
to
determine
what
must
be
measured
(and
an
associated
METRIC
determined)
in
order
to
answer
the
quesQon.
• Generate
those
metrics
that
are
related
to
the
goal.
• Several
measurements
may
be
needed
to
answer
a
single
quesQon
while
a
single
measurement
may
apply
to
more
than
one
quesQon.
• GOAL
provides
the
purpose
for
collec%ng
the
data,
and
the
ques%ons
tell
you
data
needed
to
collect/calculate.
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
119. #AgileMetrics
@XBOSoft @philiplew
Stakeholders and GQM
Goal
1
Goal
2
Goal
3
QuesQon
1
QuesQon
2
QuesQon
3
QuesQon
4
QuesQon
5
Metric
1
Metric
2
Metric
3
Metric
4
Metric
5
Metric
6
Metric
7
QA
Manager
Dev
Mgr
CTO
CEO
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
120. #AgileMetrics
@XBOSoft @philiplew
GQM - Example
• Goals
– Reduce total cost of development
– Reduce total cost of testing-effort
– Reduce the number of new feature bugs
• Questions
– Which areas have the highest re-work rates?
– Which functional areas have the most defects?
– How long does it take to repair defects?
– How complete are my requirements?
– How much time do I spend testing?
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
121. #AgileMetrics
@XBOSoft @philiplew
Exercise
Stakeholder
Workshop
(15
minutes)
1. List out stakeholders
2. Brainstorm stakeholder questions (3-5
per stakeholder)
3. Determine answers and metrics to that
question
4. Group your metrics into categories
5. List out sources of data/information
6. Present and discuss to the group
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
122. #AgileMetrics
@XBOSoft @philiplew
Metric Q&A Worksheet
Category
Ques]on
Metric
Tool
to
Use
Needed
Customer
SaQsfacQon
Are
we
meeQng
schedule?
Schedule
Variance
Project
Management
Tool
None
SoFware
TesQng
Efficiency
SoFware
Quality
What
kinds
of
defects
are
most
common?
Usability
Are
end
users
ge~ng
lost,
understand
our
UI
SoFware
Test
EffecQveness
?
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
126. #AgileMetrics
@XBOSoft @philiplew
Excerpts
from
the
Agile
Manifesto
1. Working
soFware
over
comprehensive
documentaQon
2. Our
highest
priority
is
to
saQsfy
the
customer
through
early
and
conQnuous
delivery
of
valuable
soFware.
3. Deliver
working
soFware
frequently,
from
a
couple
of
weeks
to
a
couple
of
months,
with
a
preference
to
the
shorter
Qmescale.
4. Business
people
and
developers
must
work
together
daily
throughout
the
project.
5. Working
soFware
is
the
primary
measure
of
progress.
6. Agile
processes
promote
sustainable
development.
The
sponsors,
developers,
and
users
should
be
able
to
maintain
a
constant
pace
indefinitely.
7. ConQnuous
agenQon
to
technical
excellence
and
good
design
enhances
agility.
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
128. #AgileMetrics
@XBOSoft @philiplew
What
We
Can
Agree
On
• Want
to
minimize
mistakes
and
doing
things
over
(rework)
• Want
to
uQlize
resources
effecQvely
• Want
to
predict
when
things
could
go
wrong
• Want
to
do
what
we
say
we
will
do,
when
we
say
-‐
Qmelines
• Fix
defects
early
–
later
is
more
expensive
• ProducQon
defects
are
bad
news
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
129. #AgileMetrics
@XBOSoft @philiplew
Agile
–
How
Can
We
Improve?
• Some
say
no
metrics
needed
• Is
velocity
all
you
track?
• What
is
sacrificed
for
velocity?
• How
can
we
know
what
and
how
we
are
doing
or
where
to
improve?
• How
do
we
know
if
we’re
using
resources
effecQvely?
• How
do
we
know
we
are
spending
our
effort/
Qme
in
the
right
place?
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
132. #AgileMetrics
@XBOSoft @philiplew
Poll
• How
do
you
get
(end)
customer-‐user
feedback?
– Surveys
– Defect
metrics
in
producQon
– Server
logging
– Other
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
133. #AgileMetrics
@XBOSoft @philiplew
Matching
Agile
CharacterisQcs
and
ObjecQves
Agile
Characteris]c
Speed
Quality
Customer
and
End
User
Sa]sfac]on
Other
CollaboraQve
x
x
Working
SoFware
x
Adapt
to
changing
requirements
x
x
From the agile
manifesto
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
134. #AgileMetrics
@XBOSoft @philiplew
What
QuesQons
Tell
Me
If
I’m
Reaching
Those
ObjecQves?
Speed
Quality
Owner
ParQcipaQon
Requirements
Working
Product
Planned
versus
unplanned
Schedule
Technical
debt
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
135. #AgileMetrics
@XBOSoft @philiplew
What
QuesQons
Tell
Me
If
I’m
Reaching
Those
ObjecQves?
Speed
Requirements
Did
I
get
them
in
Qme?
Do
I
understand
them?
Did
we
deliver
working
product?
Schedule
Am
I
on
schedule?
Can
I
be
more
producQve?
How
much
wasted
Qme?
Rework?
Unplanned
versus
planned
Did
I
esQmate
well?
Did
I
finish
what
I
said
I
would?
Do
I
make
mistakes
that
cause
more
(unplanned)
work?
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
136. #AgileMetrics
@XBOSoft @philiplew
What
QuesQons
Tell
Me
If
I’m
Reaching
Those
ObjecQves?
Quality
Product
owner
Did
my
end
users
find
defects?
Did
product
owner
give
feedback
regularly?
Are
there
less
defects
than
before?
Requirements
Are
my
requirements
delivered
when
needed?
Did
the
delivered
product
match
the
requirement?
Or
reqt.
wrong?
Do
engineers
have
trouble
understanding
the
requirements?
Technical
Debt
Is
my
soFware
performance
ge~ng
worse
and
worse?
Am
I
prepared
if
my
lead
engineer
leaves?
Do
I
mess
things
up
when
fixing
things?
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
139. #AgileMetrics
@XBOSoft @philiplew
Velocity
• Tracking
our
velocity
over
Qme
tells
us
our
progress
and
capacity
and
helps
us
forecast
based
on
our
capabiliQes
(now
and
in
the
future).
• Measurement
–
Sum
of
all
the
approved
esQmates
of
the
stories
• One
point
in
Qme
measurement
has
ligle
meaning,
so
need
to
track
over
Qme
or
wrt
another
metric
• Metric
– Velocity
(T)-‐(T-‐1)/Velocity
(T)
– Velocity
(T)-‐(T-‐1)/TIME
UNIT
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
143. #AgileMetrics
@XBOSoft @philiplew
Defect
Removal
EffecQveness
• We
don’t
want
the
end
user
to
find
defects
• Any
defect
found
by
the
user
influences
quality
• Metric
–
Defects
found
in
producQon
during
(90
day)
period
post
release
(DFP)/DFP
+
Defects
found
Pre-‐release
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
144. #AgileMetrics
@XBOSoft @philiplew
What
Should
Your
Goal
Be
For
DRE
• EsQmated
(by
Capers
Jones)
that
defect
removal
effecQveness
differs
by
different
levels
of
process
capability
maturity
levels:
– Level
1:
85%
– Level
2:
89%
– Level
3:
91%
– Level
4:
93%
– Level
5:
95%
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
147. #AgileMetrics
@XBOSoft @philiplew
Added
Work
• This
will
help
us
see
if
we
are
making
promises
that
maybe
we
should
not
–
overcommi~ng
and
unsustainable.
• Or
Changes
that
cause
stopping
and
starQng.
• Measurement
–
Sum
of
work
(new
stories
added)
over
and
above
the
original
plan.
• Metric
–
New
stories
added/Original
Stories
Planned
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
150. #AgileMetrics
@XBOSoft @philiplew
Work
Category
AllocaQon
• This
will
help
us
see
where
we
are
spending
our
Qme
and
adjust
our
prioriQes.
• Measurement
–
Sum
of
work
(for
each
category)
Metric
–
Divide
by
total
work
to
get
%
allocaQon
• Categories
could
include:
– Technical
debt
–
refactor
-‐
redo
– Regression
tesQng
– New
feature
development
– Research
– Requirements
review
– Defect
fixing
– Other
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
151. #AgileMetrics
@XBOSoft @philiplew
Is
Effort
in
Alignment?
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
0%
5%
10%
15%
20%
25%
30%
Are we spending
enough time
showing our
stakeholders
what we’ve
done?
Are we fixing
things too much
or making lots of
mistakes?
153. #AgileMetrics
@XBOSoft @philiplew
Work
Sustainability
• “Sustainable”
-‐
We
can’t
sustain
our
effort
if
we
are
conQnuously
pu~ng
in
overQme
• OverQme
is
necessary,
but
needs
to
be
managed
and
causes
mistakes
• Mistakes
manifest
themselves
in
many
ways,
such
as
technical
debt
or
defects
• OverQme
oFen
results
from
poor
work
esQmates
and
directly
proporQonal
to
technical
debt
• Metric
–
Unplanned
OverQme
Work/Total
Work
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
157. #AgileMetrics
@XBOSoft @philiplew
Defect
Rework
Rate
• We
can’t
get
faster
if
we
are
always
fixing
things.
• Depends
on
valid
data
whereby
people
are
logging
real
hours
to
tasks
they
take
on.
• Metric
-‐
Fix
Effort
RaQo
:
Time
to
fix
defects/
Total
effort
expended
(hours)
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
161. #AgileMetrics
@XBOSoft @philiplew
Total
Unexpected
Work
• Measurement
–
Sum
of
work
done
above
the
original
esQmates
for
those
stories
in
the
plan.
• Are
we
underesQmaQng
difficulty?
• Many
reasons:
– Incomplete
user
stories
– Misunderstood
user
stories
– Bigger
and
more
complex
than
I
thought?
– Not
enough
Qme
for
research
– Too
hard…
• Metric
–
(Total
work
done
–
Planned
work)/
Planned
work
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
165. #AgileMetrics
@XBOSoft @philiplew
Defect
in
ProducQon
Avg.
Fix
Time
• If
defects
make
it
to
producQon,
that’s
bad
!
• But
what
is
worse
if
an
important
defect
stays
unfixed.
• Defects
that
take
a
long
Qme
to
fix
represent
either
technical
debt
or
poor
process
• Metric
–
Time
to
fix
Defects
(released
to
producQon-‐
usually
P1
defects
found)
/
Total
P1
Defects
(post
producQon)
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
166. #AgileMetrics
@XBOSoft @philiplew
Don’t
Make
Them
Wait
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
0
10
20
30
40
50
60
1
2
3
4
5
6
7
8
9
#P1
Defects
Found
Average
Fix
Time
(Hrs)
Hey, we’re improving! WooHoo!
What could cause long times to fix defects?
168. #AgileMetrics
@XBOSoft @philiplew
Defect
CategorizaQon
AllocaQon
• Reduce
rework
=
Increase
speed
• Discover
when
defects
are
injected
• Reduce
and
prevent
defects
from
happening
in
the
first
place.
• Drive
down
the
Fix
Effort
RaQo
leading
to
beger
quality
and
resulQng
in
higher
velocity.
• Examine
defect
rates
per
category
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
171. #AgileMetrics
@XBOSoft @philiplew
Poll
• What
actual
improvements
have
you
seen
from
using
Agile?
– Accelerated
release
of
product
– Increase
in
producQvity
– Reduced
risk
– Reduced
cost
– Simplified
Dev
and
Test
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
173. #AgileMetrics
@XBOSoft @philiplew
Working
SoFware
Delivery
Rate
• Does
the
soFware
work
or
not?
• Working
does
not
mean
perfect
• Working
means
can
show
in
order
to
get
feedback
• Metric
-‐
#
story
points
delivered
that
‘work’
/
#
story
points
that
were
to
be
shown
to
the
PO
but
could
not
be
shown
to
work
– Many
variaQons
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
176. #AgileMetrics
@XBOSoft @philiplew
Be
Do
Have
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
Process
• Iterative (sprints)
• Daily standups
• Face to face communication
• Post mortem – end of sprint
• Delivery meeting – end of sprint
• Planning meeting – before sprint
• Self organizing
People
• Communicative
• Collaborative/Cooperative
• Flexible and willing
• Knowledgeable-multi
• Initiative/responsible
• Responsive/adaptive
Results
• Speed
• Quality
Agile
Is
a
Way
of
Thinking
“Be-‐Do-‐Have”
177. #AgileMetrics
@XBOSoft @philiplew
Take
Aways
• Mistakes
people
make
in
(agile)
metrics
and
how
to
avoid
them.
• How
to
examine
root
causes
of
low
velocity.
• How
to
reduce
rework.
• How
to
analyze
your
agile
process
and
determine
meaningful
metrics
to
present
to
management.
• How
to
get
to
velocity
AND
quality,
and
what
to
measure.
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
178. #AgileMetrics
@XBOSoft @philiplew
• What
and
why
• Goals
and
objecQves
Understand
Ask
yourself
quesQons
Evaluate
Measure
to
answer
those
quesQons
and
take
acQon
Improve
SUMMARY
©
2015
XBOSoF,
Inc.-‐
All
Rights
Reserved.
179. #AgileMetrics
@XBOSoft @philiplew
Post your questions on Twitter and I'll answer them @philiplew
Join us to keep updated on all our webinars, reports and whitepapers:
facebook.com/xbosoft
+xbosoft
linkedin.com/company/xbosoft
We post regularly on our blog – check us out!
http://xbosoft.com/software-quality-blog/
Why not download our free Whitepapers, available here:
http://xbosoft.com/knowledge-center/
Thanks
Agile Metrics White
Paper