Last year I presented the Project Defect Model, a defect measurement model piloted in one large project. Meanwhile the model has been used in a multiple of projects, varying from small to large, and from one delivery to multiple increments with various in between deliveries. This presentation shows how the model evolved, the benefits, and what we have learned on defect prevention. To able to use the model in a broad set of projects, a template model was made, including parts that are applicable in most projects in a configurable way. Also a user guide, with industry reference data, and overview presentation has been made. The model was used in several smaller projects with a single team. Main benefit was comprehensive defect information, used in release decisions. Also the model was used in several larger projects, some still ongoing. There the model provides more insight in the defect flows and process performance, supporting early quality risk detection and process improvement. Finally the model is used in several subprojects from one total project, making interdependencies clearer, enabling better planning and more reliable product release decisions. The organization learned a lot while using the model. Initially focus was on earlier defect detection, e.g. function- iso system test and inspection iso test. With the increase of data, more insight is acquired in design and coding activities, providing means for defect prevention. Now that data is available from a larger set of projects, processes can be compared enabling decisions about best practices from one project that can be spread towards future projects. Frequent estimation & feedback sessions with the design & test teams show that it is initially difficult to provide reliable estimates, but defect data that is acquired during the project enables early action taking, and better defect estimates resulting in early release quality predictions.

1. The sequel: Another year using the Project Defect Model 1 May 4, 2004 Ben Linders
The sequel: another year using the Project
Defect Model
ESEPG 2004 Conference,
London, June 14
Ben Linders
Operational Development & Quality
Ericsson R&D, The Netherlands
ben.linders@ericsson.com, +31 161 24 9885

2. The sequel: Another year using the Project Defect Model 2 May 4, 2004 Ben Linders
Overview
• Why a defect model?
• How does it work?
• Experiences from projects
• Conclusions
Measurements for product quality
and process effectiveness

3. The sequel: Another year using the Project Defect Model 3 May 4, 2004 Ben Linders
Ericsson, The Netherlands
• Benelux Market Unit & Main R&D Design Center
• R&D: Intelligent Networks
– Strategic Product Management
– Product marketing & technical sales support
– Provisioning & total project management
– Development & maintenance
– Customization
– Supply & support
• 1300 employees, of which 350 in R&D
Projects: Quality next to Lead-time and Costs

4. The sequel: Another year using the Project Defect Model 4 May 4, 2004 Ben Linders
Purpose Project Defect Model
Why?
– to control quality of the product during development
– and improve development/inspection/test processes
Business Benefit:
Better planning & tracking
Early risks signals
Save time and costs
Happy customers!

5. The sequel: Another year using the Project Defect Model 5 May 4, 2004 Ben Linders
History of the Model
• 2001
– Defined, introduced in first project
• 2002
– Used in 2 projects, improved along the way
– First release predictions
• 2003
– Industrialize model/tool
– First results presented at ESEPG 2003
– Used in all (5) major projects
• 2004
– Target defined (Balanced ScoreCard)
– New applications: Total projects, defect flows

6. The sequel: Another year using the Project Defect Model 6 May 4, 2004 Ben Linders
Modeling Defect Flow
Insertion: Where are defects made? How to prevent?
Detection: Where are defects found? Early/economic removal?

7. The sequel: Another year using the Project Defect Model 7 May 4, 2004 Ben Linders
Planning & Tracking of Quality
• Plan Quality Up Front
– Documents/code (# defects made)
– Inspection & Test effectiveness (% detection rate)
Quality consequence of project decisions
• Track Quality during project
– Actual # defects found (inspection/test)
– Estimate remaining defects: to be found / delivered
'Real-time' prediction of product quality possible
Quality view of design/test progress
Quicker escalation of quality risks

8. The sequel: Another year using the Project Defect Model 8 May 4, 2004 Ben Linders
Implementation
• Tool: Excel based defect data base & estimation
• Frequent estimation & analysis/feedback sessions
• Weekly tracking & reporting of product quality
• Includes proven techniques: ODC, requirement coverage, test matrices
Tailored per project, flexible, result oriented
Overall data based on all projects: Planning constants
Quality data, additional to time & costs!

9. The sequel: Another year using the Project Defect Model 9 May 4, 2004 Ben Linders
Results
• Data from the projects
• Feedback sessions
• Conclusions
7 projects, of which 3 ongoing
Incremental development, team based
Different size/length: size factor used.
RUP based process

10. The sequel: Another year using the Project Defect Model 10 May 4, 2004 Ben Linders
Detection rates projects
• Big projects have a better detection rate:
– More extensive test phases
– Interdependencies/risks between projects clear, quicker actions
– Incremental development, learning from first increments brings benefits
• Average detection rate in line with industry figures:
– DACS: Typical software projects 15% slip though (85% detection)
– Jones: Average 85%, most efficient 95%
Analyze/track projects that go below the target performance of 90%
* Project still ongoing at time of measurement
Project detection rates (inspections & test)
Proj A Proj B Proj C Proj D Proj E* Proj F* Proj G* Average
Rate 95% 95% 90% 59% 94% 86% 89% 90%
Size 1 4 1 1 5 3 1

11. The sequel: Another year using the Project Defect Model 11 May 4, 2004 Ben Linders
Injection rates phases
• Very elaborated architecture (feasibility phase). Many defects made,
most of them are found in the architecture reviews.
• Lean design.
• Most defects made during coding
“Normal” defect pattern, with sufficient focus in all phases on defect
prevention.
Phase injection rates
Requirements Architecture Design Code
Rate 6% 21% 15% 58%

12. The sequel: Another year using the Project Defect Model 12 May 4, 2004 Ben Linders
Detection rates phases
• Lower detection rate in requirements phase: incremental development,
start when only part of requirements is stable
• High architecture/design: effective inspections, good architecture skills
• Lower code detection: one project just starting with code inspections
(when excluded from measurement: 50% code detection rate)
• Function & system test: Acceptable rates
• Network test, low rate, but defects that are found would give major
problems to customers: Good cost/benefit of the test phase
Focus on inspection improvement & test focus, capture defects earlier
Phase detection rates
Requirements Architecture Design Code Function Test System Test Netw ork Test Total
Rate 30% 67% 66% 40% 48% 48% 27% 47%

13. The sequel: Another year using the Project Defect Model 13 May 4, 2004 Ben Linders
Feedback sessions
• Frequent, short
• At the workplace
• All data available (Excel)
• Design/test leaders
Show data
ask questions
form conclusions
take needed actions
Feedback sessions enabled earlier conclusions, better acceptance of
results, and quick and focused corrective/preventive actions.
Feedback: Collected data delivered to the
people that have been doing the work, in order
to support their understanding of the situation at
hand and help them to take needed actions

14. The sequel: Another year using the Project Defect Model 14 May 4, 2004 Ben Linders
Conclusions
Project Defect Model helps projects to:
– Estimate/track defects: Improve product release quality, save time/cost
– Design/test progress: Better planning, risk management, decisions
Benefits for R&D
– Project portfolio: Dimension project teams/maintenance teams
– Product quality: Less maintenance, satisfied customers
– Employees: More involved, empowered, motivated

15. The sequel: Another year using the Project Defect Model 15 May 4, 2004 Ben Linders
Further reading
Papers
– Controlling Product Quality During Development with a Defect Model,
Proceedings ESEPG 2003
– Make what’s counted count, Better Software magazine march 2004
References
– Managing the software process. Watts Humphrey.
– Metrics and models in Software Quality Engineering. Stephen H. Kan.
Ben Linders, Ericsson R&D, The Netherlands
ben.linders@ericsson.com, +31 161 24 9885