• A hands-on model for control of product and process quality. • Support of release risk decisions based on defect data. • ODC and Test Matrices applied in different test phases. • Usage of feedback to analyze data and come to actions. • Using project data for a business case for improvement.

1. Controlling Project Performance Using the Project Defect Model 1 March 18, 2005 Ben Linders
Controlling Project Performance
Using the Project Defect Model
PSQT 2005 Conference,
Las Vegas, May 3
Ben Linders
Operational Development & Quality
Ericsson R&D, The Netherlands
ben.linders@ericsson.com, +31 161 24 9885

2. Controlling Project Performance Using the Project Defect Model 2 March 18, 2005 Ben Linders
Overview
• Why a defect model?
• How does it work?
• Experiences from projects
• Conclusions
Measurements for product quality
and process effectiveness

3. Controlling Project Performance Using the Project Defect Model 3 March 18, 2005 Ben Linders
Ericsson, The Netherlands
• Market Unit Northern Europe & Main R&D Design Center
• R&D: Intelligent Networks
– Strategic Product Management
– Product marketing & technical sales support
– Provisioning & total project management
– Development & maintenance
– Customization
– Supply & support
• 1300 employees, of which 350 in R&D
Projects: Quality next to Lead-time and Costs

4. Controlling Project Performance Using the Project Defect Model 4 March 18, 2005 Ben Linders
Purpose Project Defect Model
Why?
– to control quality of the product during development
– and improve development/inspection/test processes
Business Benefit:
➨ Better planning & tracking
➨ Early risks signals
➨ Save time and costs
➨ Happy customers!

5. Controlling Project Performance Using the Project Defect Model 5 March 18, 2005 Ben Linders
History of the Model
• 2001
– Defined, introduced in first project
• 2002
– Used in 2 projects, improved along the way
– First release predictions
• 2003
– Industrialize model/tool
– Used in all (5) major projects
• 2004
– Management decisions based on model
– New applications: Solution/Total projects, defect flows
• 2005
– Extension with Cost of Quality

6. Controlling Project Performance Using the Project Defect Model 6 March 18, 2005 Ben Linders
Modeling Defect Flow
Insertion: Where are defects made? How to prevent?
Detection: Where are defects found? Early/economic removal?

7. Controlling Project Performance Using the Project Defect Model 7 March 18, 2005 Ben Linders
Process View
Process
Inputs and outputs
Influencing factors
Measurement
DefectsInserted
(documentation,
code)
DefectsDetected
(Inspection, test)
(Un)happy customers
Design Process
Competence, skills
Tools, environment
Test Process
Competence, skills
Test Capacity
Tools, environment
Resident Defectsin
Delivered Product
Resident Defectsin
Design Base
Detection Rate
Defect Density
Fault Slip Through
Defect Level
Defect Classification

8. Controlling Project Performance Using the Project Defect Model 8 March 18, 2005 Ben Linders
Planning & Tracking of Quality
• Plan Quality Up Front
– Documents/code (# defects made)
– Inspection & Test effectiveness (% detection rate)
Quality consequence of project approach
• Track Quality during project
– Actual # defects found (inspection/test)
– Estimate remaining defects: to be found / delivered
Quality view of design/test, quicker escalation
• Decide based upon Quality Status
– Toll Gates (go/no go) and Product Release
Product Quality figures

9. Controlling Project Performance Using the Project Defect Model 9 March 18, 2005 Ben Linders
Implementation
• Tool: Excel based defect data base & estimation
• Frequent estimation & analysis/feedback sessions
• Weekly tracking & reporting of product quality
• Includes proven techniques: ODC, requirement coverage, test matrices
Tailored per project, flexible, result oriented
Overall data based on all projects: Planning constants
Quality data, additional to time & costs!

10. Controlling Project Performance Using the Project Defect Model 10 March 18, 2005 Ben Linders
Results
• Data from the projects
• Feedback sessions
• Conclusions
11 projects, of which 2 ongoing
Incremental development, team based
Different size/length: size factor used.
RUP based process

11. Controlling Project Performance Using the Project Defect Model 11 March 18, 2005 Ben Linders
Detection rates projects
Project detection rates Q1 2005 (PSQT Conference)
Proj A Proj B Proj C Proj D Proj E Proj F* Proj G Proj H* Proj J Proj K Proj L Average
Rate 95% 95% 90% 59% 97% 86% 93% 88% 91% 94% 93% 91%
Size 1 4 1 1 5 3 1 4 1 2 3
* Project still ongoing at time of measurement
• Limited variance
– Project D different: Integration of products (no design)
– Range (excluding D) from 86%-97% (projects F and H still ongoing)
• Average detection rate in line with industry figures:
– DACS: Typical software projects 15% slip though (85% detection)
– Jones: Average 85%, most efficient 95%
Analyze/track projects that go below the target performance of 90%

12. Controlling Project Performance Using the Project Defect Model 12 March 18, 2005 Ben Linders
Injection rates phases
Phase injection rates, Q1 2005 (PSQT Conference)
R e quire m e n A rc hite c tureD e s ig n C o de D o c wa re
Rate 7% 18% 12% 58% 4%
• Very elaborated architecture (feasibility phase). Many defects made, but
most of them are found in the architecture reviews.
• Lean design, few defects made.
• Most defects made during coding
“Normal” defect pattern, sufficient focus on defect prevention.

13. Controlling Project Performance Using the Project Defect Model 13 March 18, 2005 Ben Linders
Detection rates phases, averages
Phase detection rates, Q1 2005 (PSQT Conference)
R e quire m e n A rc hite c tureD e s ig n C o de D o c wa re F unc tio n Te S ys te m Te s N e two rk Te A v e ra g e
Det. Rate 56% 64% 51% 36% 70% 56% 49% 23% 51%
• High requirements/architecture/design: effective inspections, good
architecture skills
• Low code detection: improvement program ongoing
• Function & system test: Acceptable rates
• Network test, low rate, but defects that are found would give significant
problems to customers: Good cost/benefit of the test phase
Focus on inspection improvement, capture defects earlier

14. Controlling Project Performance Using the Project Defect Model 14 March 18, 2005 Ben Linders
Detection rates phases, variance
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
R
equirem
entsArchitecture
D
esign
C
ode
D
ocw
areFunction
TestSystem
TestN
etw
ork
Test
Total
Det. Rate
• Large variance, except for docware & total (excl proj D)
Process alignment, standardize & re-use best practices

15. Controlling Project Performance Using the Project Defect Model 15 March 18, 2005 Ben Linders
Feedback sessions
• Frequent, short
• At the workplace
• All data available (Excel)
• Design/test leaders
Show data
ask questions
form conclusions
take needed actions
Feedback sessions enabled earlier conclusions, better acceptance of
results, and quick and focused corrective/preventive actions.
Feedback: Collected data delivered to the
people that have been doing the work, in order
to support their understanding of the situation at
hand and help them to take needed actions

16. Controlling Project Performance Using the Project Defect Model 16 March 18, 2005 Ben Linders
Conclusions
Project Defect Model helps projects to:
– Estimate/track defects: Improve product release quality, save time/cost
– Design/test progress: Better planning, risk management, decisions
Benefits for R&D
– Project portfolio: Dimension project teams/maintenance teams
– Product quality: Less maintenance, satisfied customers
– Employees: More involved, empowered, motivated

17. Controlling Project Performance Using the Project Defect Model 17 March 18, 2005 Ben Linders
Further reading
References
– Managing the software process. Watts Humphrey.
– Metrics and models in Software Quality Engineering. Stephen H. Kan.
Papers (see also PSQT conference paper!)
– Controlling Product Quality During Development with a Defect Model, in
Proceedings ESEPG 2003
– Make what’s counted count, in Better Software magazine march 2004
– Measuring Defects to Control product Quality, in Measure! Knowledge!
Action! The NESMA anniversary book. Oct 2004. ISBN: 90-76258-18-X
– A Proactive Attitude Towards Quality: The Project Defect Model,
Software Quality Professional Dec 2004 (with Hans Sassenburg)
Ben Linders, Ericsson R&D, The Netherlands
ben.linders@ericsson.com, +31 161 24 9885