Presentation by Phil Lew at StarWest 2013.
When implementing software quality metrics, we need to first understand the purpose of the metrics and who will be using them. Will the metric be used to measure people or the process, to illustrate the level of quality in software products, or to drive toward a specific objective? QA managers typically want to deliver productivity metrics to management but management may want to see metrics that describe customer or user satisfaction. Philip Lew believes that software quality metrics without actionable objectives toward increasing customer satisfaction are a waste of time. Learn how to connect each metric with potential actions based on evaluating the metric. Metrics for the sake of information may be helpful but often just end up in spreadsheets of interest to no one. Take home methods to identify metrics that support actionable objectives. Once the metrics and their objectives have been established, learn how to define and use metrics for real improvement.
2. Meet Your Instructor
• Phil Lew
– Telecommunications consultant and network designer
– Architect, Data warehousing product development
– Software product manager, BI product
– COO, large IT services company
– CEO, XBOSoft, software qa and testing services
• Relevant specialties
– Software quality process improvement
– Software quality evaluation and measurement
– Software quality in use / UX design
STARWEST
2013
2
3. Session Spirit and
Expectations
• Interactive
• I won’t read the slides…
• Slides for you as a take-away
• Different than your handouts
– Always thinking of new examples and
ideas, you can write me and I’ll send
these to you
STARWEST
2013
3
4. Understand, Evaluate and
Improve
• If our end goal is improvement in what?
• To improve, we need to evaluate
• In order to evaluate, we must understand what
we are evaluating
• To do this… We need metrics
Can you think of other examples in our lives
where this applies? Where do you use metrics
to evaluate and improve?
STARWEST
2013
4
5. Metrics in real life
Food
Eaten
Weight
Performance
Race
Results
STARWEST
2013
5
• Calories
• Fat
• Carbohydrates
• Protein
• Time of day
• Vitamins
• …
• Blood pressure
• Cholesterol
• Blood glucose
• Red cell count
• White cell count
• Hematocrit
• Hemoglobin
• Body fat %
• …
• Placing
• …
• Effort/Power
• Heart rate/Watts
• Speed
• Time
Intelligence
Finesse
Context
• Training
• Sleep
6. Metrics - Benefits
• Understand how QA, testing, and its processes
and where the problems are
• Evaluate the process and the product in the right
context
• Predict and control process and product qualities
• Reuse successful experiences
– Feed back experience to current and future projects
• Monitor how something is performing
• Analyze the information provided to drive
improvements
STARWEST
2013
6
7. How can measurement help us (YOU)
• Create a organizational memory – baselines of current
practices-situation
• Determine strengths and weaknesses of the current
process and product
– What types of errors are most common?
• Develop reasoning for adopting/refining techniques
– What techniques will minimize the problems?
• Assess the impact of techniques
– Does more/less functional testing reduce defects?
• Evaluate the quality of the process/product
– Are we applying inspections appropriately?
– What is the reliability of the product before/after
delivery?
STARWEST
2013
7
9. Why we need to measure ?
• Our bosses want us to…
• They want someone to point fingers at
• They want to fire some people and save money
• They need to report to their managers
• They
want
some
basis
on
which
to
evaluate
us
and
give
us
a
raise!
• We
need
to
figure
out
a
way
to
do
beHer!
• We
want
to
improve
our
work
and
improve
soKware
quality
STARWEST
2013
9
10. The Metric Conundrum
• QA and Testing
Language
– Defects
– Execution status
– Test cases
– Pass/fail rates
– DRE…
• Business
Language
– Cost
effecRve
– ROI
– Cost
of
ownership
– Cost
of
poor
quality
– ProducRvity
– Calls
to
help
desk
– Customer
saRsfacRon
– Customer
retenRon
STARWEST
2013
10
11. In your organization…?
• What measurements do you take in your
organization and why?
• Who uses them and for what?
STARWEST
2013
11
12. The Metric Reality
• Measurement is like dinner. It takes 2-3 hours to
make dinner, and 15 minutes to consume…
• But… many metrics are never reviewed or
analyzed (consumed)
• WHY?
STARWEST
2013
12
13. The Metric Conundrum (cont.)
• Test leads and test managers rarely have the
right metrics to show or quantify value
• Metric collection and reporting are a drag
• QA metrics usually focus only on test execution
• Test tools don’t have most of the metrics we
want
• Reports generated by QA are only rarely
reviewed
• Metrics are not connected to anything of value/
meaningful for ________.
STARWEST
2013
13
15. Outline
• Choose the right metrics
• Design a measurement framework
• Report what is meaningful
• Use metrics not only to manage testing, but as a
Quality barometer
• Show value
STARWEST
2013
15
16. Guiding principles about
metrics
• Metrics can represent a snapshot
• Metrics can change over time
• Metrics can be simple – don’t have to be
complicated
• Metrics can assist us in making decisions today
• Metrics can assist us in predicting the future
• Some metrics are critical for some while useless
for others
STARWEST
2013
16
17. Metrics can represent a
snapshot
• Not only at a particular moment, like this week,
but for a process
STARWEST
2013
17
Code
quality
Internal
Quality
Tested
in
a
lab
External
Quality
• What
the
end
user
sees
• What
the
end
user
experiences
In-‐Use
Quality
At what stage in the process do you take measurements?
18. Metric Basics
• What is a metric?
• What is a measurement?
• What is an indicator?
STARWEST
2013
18
19. Some Metrics – Let’s get
started
• How many of
these do you
know/use?
• What others do
you use?
• What are they
used for?
• By who?
Schedule
variance
Effort
variance
Cost
variance
Defect
removal
effecRveness
ProducRvity
Defect
aging
CriRcal
Defect
rate
Test
cases
executed
Test
coverage
Pass/Fail
Rate
ROI
AutomaRon
coverage
STARWEST
2013
19
20. Metrics Exercise
Work in teams of 2-3
List the metrics you use, what you use them
for, who wants them, what decisions they
make based on them
STARWEST 2013 20
22. Stakeholders
• Who are they?
• What are their objectives?
• What questions do they have?
• What answers can metrics provide?
STARWEST
2013
22
23. Possible Stakeholders
Stakeholder
Objec0ve
Ques0on
Answer
CIO
CEO
VP
–
QA
–
Development
Director
-‐
Development
Owner
Auditor
Test
Lead
Partner
?
STARWEST
2013
23
24. Metric Q&A Worksheet
Category
Ques0on
Metric
Tool
to
Use
Needed
Customer
SaRsfacRon
Are
we
meeRng
schedule?
Schedule
Variance
Project
Management
Tool
None
SoKware
TesRng
Efficiency
SoKware
Quality
What
kinds
of
defects
are
most
common?
Usability
Are
end
users
gedng
lost,
understand
our
UI
?
?
• QuesRons
are
the
fundamental
building
block
• Answering
quesRons
that
lead
to
decisions
à
VALUE
24
25. Stakeholder Workshop (15
minutes)
• Divide up into teams of 2-3
• List out stakeholders
• Brainstorm stakeholder questions (2 per
stakeholder)
• Determine answers and metrics to that question
• Group your metrics into categories
• Present and discuss to the group
STARWEST
2013
25
26. Organizing our thoughts
• Putting metrics into a framework
• Helps us to make sure we have all functions in
our organization and process covered
• Helps us to develop relationships between
different elements in our process
STARWEST
2013
26
27. Identify Metrics to Answer
Questions
Customer
and
Management
Metrics
OperaRons
OpRmizaRon
Metric
CalculaRon
Departmental
Metrics
Project/Build
Metrics
Planning
Quality
Status
Data
CollecRon
Metric
Identification
Req’ts. &
Questions
Reporting
Efficiency
and
EffecRveness
Schedule
and
Budget
Defect
types
and
quanRRes
Resource
uRlizaRon
Snapshots
and
trends
STARWEST
2013
27
28. Measurement and Reporting
• Measure, build metrics AND develop reports that
answer the questions of your stakeholders
Planning
• Requirements
coverage
%
• Test
case
prioriRes
and
complexity/atomicity
• Requirements
traceability
• Test
ExecuRon
prioriRzaRon
• Requirements
aging
• Resource
and
budget
variance
STARWEST
2013
28
29. Test Measurement and
Reporting
• Measure, build metrics AND develop reports that
answer the questions of your stakeholders
Quality
• Test
pass/fail
rate
• #
open
defects
• #
priority
1
defects
open
• #
priority
1
defects
found
• Re-‐opened
defects
• Defect
density
• Defects
found
in
operaRon
• Defects
found/test
case/script
• Test
execuRon
progress
STARWEST
2013
29
• Customer
saRsfacRon
• Help
desk
calls
30. Test Measurement and
Reporting
• Measure, build metrics AND develop reports that
answer the questions of your stakeholders
OperaRons
• Time
spent
gedng
environment
ready
• Time
to
close
defects
–
aging
• Effort
per
test
case
• TesRng
schedule
variance
• Defect
arrival
and
closure
rate
• Test
schedule
variance
STARWEST
2013
30
31. Test Measurement and
Reporting
• Measure, build metrics AND develop reports that
answer the questions of your stakeholders
OpRmizaRon
• AutomaRon
script
execuRon
Rme
• Regression
Rme
• AutomaRon
coverage
• Resource
efficiency
• Resource
effecRveness
• Test
cycle
Rme
STARWEST
2013
31
32. Test Measurement and
Reporting
• Measure, build metrics AND develop reports that
answer the questions of your stakeholders
Status
• Test
execuRon
%
complete
• Test
case
prioriRes
and
complexity/
atomicity
• Requirements
status/trending
• Test
ExecuRon
EffecRveness
• Over
Rme
(trending)
versus
snapshot
STARWEST
2013
32
33. What’s Missing?
STARWEST
2013
33
Requirem
ents
stability
Internal
Quality
Defects
Automa
Ron
coverage
External
Quality
Customer
retenRon
Help
desk
call
rate
In-‐Use
Quality
Can you think of where other metrics we have
listed out before, and where they would fit?
34. Data and Reporting
OperaRons
OpRmizaRon
Metric
CalculaRon
Customer
and
Management
Metrics
Departmental
Metrics
Project/Build
Metrics
Planning
Quality
Status
Automated
Data
CollecRon
All for nothing unless you have supporting data
Config
mgmt
Rqmt
mgmt
Time
tracking
Test
mgmt
Project
planning
Other?
STARWEST
2013
34
35. Exercise: Develop Your Own
Framework
STARWEST
2013
35
Work backwards or forwards
Work in small groups 2-3
36. Reporting
• Work with stakeholders to report
information so that they can make
decisions
• Metric is meaningless if not understood or
misunderstood
• Develop a report catalog
– Metric library
– Canned reports usually have limited
impact
STARWEST
2013
36
37. Reporting Root Causes and
Trends
• The most important metrics you can report
– Root causes
• Sources of defects
• Sources of mistakes
• Sources of delay or inefficiency
• Sources of improvement
– Trending
• Showing metrics over time, pre-set intervals
• Showing change
• Make sure to differentiate Snapshot versus trend
STARWEST
2013
37
38. Reporting – Show Value
• Answer Questions
– Are we delivering a quality product?
– Are our customers satisfied?
– Are we meeting timelines?
– Did we understand requirements correctly and timely?
– How much is QA costing?
– Are we meeting budget?
– How have we cut costs?
– How have we improved efficiency?
– What are the primary causes of delay?
– What are the causes of poor quality?
– Is quality improving?
– What is the status? – Current versus goal?
– Have we gotten better?
STARWEST
2013
38
39. Show Value – Connect the dots
– Did we understand
requirements correctly and
timely?
– What are the primary causes of
delay?
– What are the causes of poor
quality?
STARWEST
2013
39
Requirem
ents
stability
Internal
Quality
Defects
Automa
Ron
coverage
External
Quality
Customer
retenRon
Help
desk
call
rate
In-‐Use
Quality
QA and Testing Language Business Language
– Is quality improving?
– What is the status? –
Current versus goal?
– Are we delivering a
quality product?
– Are our customers
satisfied?
– Are we meeting
timelines?
– How much is QA costing?
– Are we meeting budget
and schedule?
– How have we improved
efficiency?
– What are the primary
causes of delay?
– What are the causes of
poor quality?
40. Getting Started with
Measurement
• What questions-answers are important to your
organization?
– Develop a model/determine what to measure
– Define improvement objectives
• Based on intuition/available data
• Look at high payoff areas, problem areas
• What data can you collect/Which technique can
you use
– Maybe some elements of the model drop out-
can’t be measured that easily
STARWEST
2013
40
41. Set up an action plan
• Define a metrics improvement plan – What are
your goals?
• Conduct stakeholder workshops – Involve all
• Build your measurement framework with phases
in the process and categories
– Q&A catalog
– Metric catalog
– Report catalog
• Set up the infrastructure and operationalize
• Start collecting and developing benchmarks
STARWEST
2013
41
42. What’s Next
• Choose a project – start small and build up
for early wins
• Organize to support the change
– Committed people
– Recognize the new processes
– Assign roles and resources
STARWEST
2013
42
43. Software Testing Metrics
Take Aways
• Quality is different for each organization
– No one right answer for what metrics
– Different metrics have different meanings for different
people – SET UP FRAMEWORK AND PLAN FOR
YOUR Company
• Management wants metrics
– Know what key questions need to be answered
– REPORTING - Metrics programs often fail due to poor
articulation
• Measurement and improvement is a long term effort, just
like a diet.
STARWEST
2013
43
44. Putting It All Together
Measurement
Framework
Improvement
Decisions
Stakeholders
STARWEST
2013
44
Questions and Answers
45. Thanks
Please fill out an evaluation form
and drop it in the collection basket
located at the back of the room.
www.xbosoft.com
@xbosoft
408-350-0508
Philip Lew
@philiplew
philip.lew@xbosoft.com
Some resources: http://www.xbosoft.com/knowledge_center/
www.facebook.com/xbosoft