Get an impression on how we used GQM to define metrics on how to measure and improve quality and productivity in software development. Maybe our lessons learned might help you in your projects
Dev Dives: Streamline document processing with UiPath Studio Web
Lessons learned from measuring software development processes
1. Software Quality Lab
Markus Unterauer
Consultant and trainer
Lessons Learned from
measuring software development
processes
2. www.software-quality-lab.com | improve your quality
At the end of this session you know …
Agenda
Introduction to process measurement
Experience from practise
Lessons learned from the project
Slide 2
4. www.software-quality-lab.com | improve your quality
Measurement as basis for all improvements
Why measure processes
Slide 4
Define goal and
path to reach it
Discover, where we are
Identify gap, to where
we should be
Find corrective measures
Go next step Goal reached
5. www.software-quality-lab.com | improve your quality
Proper metrics show the way to our goal
Why measure processes
Slide 5
Source:DchicviaFlickr,https://www.flickr.com/photos/agenciamodels/6118171675
6. www.software-quality-lab.com | improve your quality
Boundaries show, when to act
Why measure processes
Slide 6
Upper boundary
Lower boundary
Target value
Measured value
Time
7. www.software-quality-lab.com | improve your quality
Define proper metrics with G-Q-M
GQM for defining metrics
1. Goal
What should be done for whom?
Which aspects do we want to change? What is the context?
e.g. „Improve the quality for all new projects“
2. Question
Which questions guide us to the measurement goal?
e.g. „Are we finding the right bugs?“
3. Metric
How can we answer the questions using metrics?
e.g. „Ratio of bugs found per severity class“
Slide 7
Define goals
Find
questions
Answer
questions
8. www.software-quality-lab.com | improve your quality
Example
GQM for defining metrics
Slide 8
Metric
Question
Goal We want to improve the
quality of our products!
Are we finding
the right bugs?
Ratio of
bugs per severity
No. of bugs
found after release
Do we find the bugs
instead of the users?
Ratio of
bugs found by QA
vs. Users
9. www.software-quality-lab.com | improve your quality
Question: Are we finding the right bugs?
Goal: Improve quality of our products
Ratio of bugs per severity class
Number of bugs found after release
Per severity class
Ratio of recurring bugs
Average number of bugs found per test case
Number of bugs found by test level
Unit tests
Integration tests
System tests
User acceptance tests
Bugs found by type
Coding bug
Bad requirements
UI bug
…
Slide 9
11. www.software-quality-lab.com | improve your quality
Situation at our customer
Experience from practise
Slide 11
http://pixabay.com/en/desert-road-sand-forward-winds-609316/
12. www.software-quality-lab.com | improve your quality
Goals for our customer
Experience from practise
How bad is it really?
What should we do?
Set up an improvement cycle
Slide 12
http://en.wikipedia.org/wiki/PDCA#/media/File:PDCA_Cycle.svg
13. www.software-quality-lab.com | improve your quality
First attempt …
Experience from practise
Our customer wanted to measure …
~120 metrics
in 12 process fields with 21 processes
on 4 levels
from 14 different sources and source system
But …
for more than half of all metrics we could not get values
no one really knew, what to do with the metrics
management did not even look at it
effort for getting numbers was extremly high
Slide 13
14. www.software-quality-lab.com | improve your quality
Some sources of values
Experience from practise
Slide 14
• Different levels of detail
• Errors and missing data
• Bookings on different categories
Time recording
• Each project with different structure
• Not comparable
• Incomplete
Project
planning
• Only for parts of all sourcecode
• Difficult to query and analyze
Source
Control
• Manually built Excel sheets by Project Managers
• Each project with different structure
• Not comparable
Controlling
sheets
• Very high effort
• Ambiguous questions
Questionaires
and interviews
Good at first glance,
but bad in detail …
15. www.software-quality-lab.com | improve your quality
Everything very difficult
Experience from practise
Slide 15
http://pixabay.com/en/squirrel-reading-books-surprise-304021/
16. www.software-quality-lab.com | improve your quality
New approach GQM
Experience from practise
1. Define two goals
Ask management, what they would like to achieve
Define reporting
2. Ask questions
3. Find metrics to answer question
Only automatically measurable ones
Only one per question
4. Define target values and boundaries
Get target values from industry benchmarks
5. Try out and validate
6. Report to management
Slide 16
http://pixabay.com/en/shield-directory-panic-fear-trust-492992/
17. www.software-quality-lab.com | improve your quality
Step 1: Define Goals
Experience from practise
1. We want to improve quality to the customer
2. We want to increase developer productivity
Slide 17
Images from www.pixabay.com
18. www.software-quality-lab.com | improve your quality
Step 2: Ask questions
Experience from practise
1. We want to improve quality to the customer
Do we find bugs instead of the customer?
Are we testing enough?
Are we testing effectively?
2. We want to increase developer productivity
Do our developers spend too much time on bug fixing?
Are we developing fast enough?
Is the quality of our source code high enough?
Slide 18
Images from www.pixabay.com
19. www.software-quality-lab.com | improve your quality
Step 3: Find metrics
Experience from practise
1. We want to improve quality to the customer
Do we find bugs instead of the customer?
• Ratio of bugs found by us to bugs found by customer
Are we testing enough?
• Number of test cases tested per developer day
Are we testing effectively?
• Number of bugs found per tester day
2. We want to increase developer productivity
Do our developers spend too much time on bug fixing?
• Effort-ratio for bug fixing per developer month
Are we developing fast enough?
• Number of changed LoC per developer day
Is the quality of our source code high enough?
• Number of bugs found by customer per developer month
Slide 19
Images from www.pixabay.com
20. www.software-quality-lab.com | improve your quality
Step 4: Define target values and boundaries
Experience from practise
Slide 20
Metric Target Value Current value Conclusion
Changed LoC per
developer day
30 – 40 LoC per day 123 LoC per Day Too high. Results in
poor quality.
Effort-ratio for bug
fixing
25 - 35% 115% Far too high. Low
productivity.
Ratio of bugs found
by customer
1 – 3% of all bugs 68% of all bugs Far too high. Testing
not effective.
… … … …
Target range taken
from literature and
publications
21. www.software-quality-lab.com | improve your quality
Step 5: Try out and validate
Experience from practise
Slide 21
http://pixabay.com/en/calculator-calculation-insurance-385506/
22. www.software-quality-lab.com | improve your quality
Step 6: Reporting
Experience from practise
Slide 22
Trend for past thee
months
Textual comment
from Quality
Manager
23. www.software-quality-lab.com | improve your quality
Document metrics in a metric handbook
Experience from practise
Slide 23
Metric Effort-ratio of bug fixing per developer month
Goal Developers should produce high quality code and spend as much time
for implementing new features as possible
Benefit Fast development of new features
High quality
Formula Ratiobug fixing = Effortbug fixing / Total Effortdevelopment
Method Automatic calculation from timesheets
Datasource Time recording system
Interval Monthly
Responsible Process Manager
25. www.software-quality-lab.com | improve your quality
Tips for starting your quality metric system
Lessons Learned
Define metrics top down, not bottom up
Goal > Question > Metric
Not: „Hey, look how many things we can measure!“
Concentrate on 5 - 10 metrics, e.g. the one presented before
Define lower and upper boundary together with your team
… or measure without boundaries for some iterations and define boundaries then
Start with automatically measurable metrics
When defining metrics, always have in mind, if you have all data necessary in the quality needed
Define in advance, what to do with the metrics
Slide 25
Image:http://pixabay.com/en/bulb-light-lamp-electric-160207/
27. Your Partner in Software Quality and Testing
Software Quality Lab GmbH
www.software-quality-lab.com
Consulting | Service | Academy | Tool Expertise
Markus Unterauer
markus.unterauer@software-quality-lab.com
Notes de l'éditeur
My name is Markus Unterauer
I come from Software quality lab in austria
Out mission is software quality
Quality for us does not start at testing, but with the first line of business goals and requirements
This vision of software quality was also the starting point of this talk
Let‘s have a look at
what you can expect today
And what you should take home
…
Why is requirements quality that important?
QUESTION TO THE AUDIENCE
Who of you has problems with poor or bad requirements? With misunderstandings?
Let‘s take a closer look, to how practical work in the field of requirements quality really looks like today …
ANIMATION
Every improvement starts with defining the goal
…
At „Discover where we are“ and „Identify gap“ we need to measure
ANIMATION
At first I have to decide what to measure
A metric must fulfill some criteria
It must represent the current situation right
It must allow me to derive measures
For each metric we have to define an ideal value
Around this ideal value, there is a range, where things are ok
If our metrics show, that we leave the green zone, we need to act
One approach to find the proper metrics to measure requirements quality is the GQM approach
…
Here we have another example for GQM in requirements quality measurement
We know now
Requirements are important
Often they are done bad
That leads defects, high costs and missed expectations
So, we need better requirements
At the moment it is ok, but first dark clouds on horizon
Storm is coming
First signs are already visible
They tried to start measiring so that they get info, where tehy sould go
without us ;-)
For each metric, we took a look in advance, whether we have the data!
Perform measurement, get values
Reports from existing systems
LoC -> Source-Control
Bug numbers -> Bugtracking tool
We know now
Requirements are important
Often they are done bad
That leads defects, high costs and missed expectations
So, we need better requirements