Tutorial: Quality Management and Process Improvement with the Team Software Process - João Pascoal Faria (FEUP)

1. Portugal
Quality Management and
Process Improvement with
the Team Software Process
João Pascoal Faria
Assistant Professor
FEUP
2012-07-06

2. Questions
1. Do you have problems/concerns with software quality?
2. Do you have problems/concerns with cost of quality?
3. Do you know your current quality and cost of quality levels?
4. How do they compare with industry benchmarks?
5. How can you improve quality and reduce the cost of quality?
2
2

3. Agenda
Motivation
TSP Performance Results
Factor I – Bottom-Up Performance Improvement
Factor II - Personal Responsibility
Factor III – Feedback Loops for Continual Improvement
Factor IV – Cost-Effective Defect Removal Methods
Factor V - Measurement and Quantitative Methods
Conclusions
3
3

4. Motivation
4
4

5. Top 10 software engineering
trends
Barry Boehm
2011
1. Increasing emphasis on rapid development and
adaptability;
2. Increasing software criticality and need for
assurance;
…
5
5

6. Increasing software criticality
and need for assurance
Barry Boehm
2011
“Although people’s, systems’, and organizations’ dependency
on software is becoming increasingly critical, dependability is
generally not the top priority for software producers.”
6
6

7. Increasing software criticality
and need for assurance
Barry Boehm
2011
“Although people’s, systems’, and organizations’ dependency
on software is becoming increasingly critical, dependability is
generally not the top priority for software producers.”
“(In fact) the IT industry spends the bulk of its resources, both
financial and human, on rapidly bringing products to market.”
7
7

8. Increasing software criticality
and need for assurance
Barry Boehm
2011
“Although people’s, systems’, and organizations’ dependency
on software is becoming increasingly critical, dependability is
generally not the top priority for software producers.”
“(In fact) the IT industry spends the bulk of its resources, both
financial and human, on rapidly bringing products to market.”
“This situation will likely continue until a major software-
induced systems catastrophe similar in impact on world
consciousness to the 9/11 World Trade Center catastrophe
stimulates action toward establishing accountability for
software dependability.”
8
8

9. Why should quality be the top priority?
9
9

10. Why should quality be the top priority?
Increasing dependency on software systems
10
10

11. Why should quality be the top priority?
Increasing dependency on software systems
– 80%+ of functionality of modern aircrafts provided by sw [Humphrey, 2002]
11
11

12. Why should quality be the top priority?
Increasing dependency on software systems
– 80%+ of functionality of modern aircrafts provided by sw [Humphrey, 2002]
Increasing criticality of functions provided by software
12
12

13. Why should quality be the top priority?
Increasing dependency on software systems
– 80%+ of functionality of modern aircrafts provided by sw [Humphrey, 2002]
Increasing criticality of functions provided by software
– e-banking, aircraft control software, medical device software, …
13
13

14. Why should quality be the top priority?
Increasing dependency on software systems
– 80%+ of functionality of modern aircrafts provided by sw [Humphrey, 2002]
Increasing criticality of functions provided by software
– e-banking, aircraft control software, medical device software, …
Software defects are a primary cause of security vulnerabilities
14
14

15. Why should quality be the top priority?
Increasing dependency on software systems
– 80%+ of functionality of modern aircrafts provided by sw [Humphrey, 2002]
Increasing criticality of functions provided by software
– e-banking, aircraft control software, medical device software, …
Software defects are a primary cause of security vulnerabilities
– Buffer overflow, SQL injection, etc.
15
15

16. Why should quality be the top priority?
Increasing dependency on software systems
– 80%+ of functionality of modern aircrafts provided by sw [Humphrey, 2002]
Increasing criticality of functions provided by software
– e-banking, aircraft control software, medical device software, …
Software defects are a primary cause of security vulnerabilities
– Buffer overflow, SQL injection, etc.
Huge economic impact of software errors
16
16

17. Why should quality be the top priority?
Increasing dependency on software systems
– 80%+ of functionality of modern aircrafts provided by sw [Humphrey, 2002]
Increasing criticality of functions provided by software
– e-banking, aircraft control software, medical device software, …
Software defects are a primary cause of security vulnerabilities
– Buffer overflow, SQL injection, etc.
Huge economic impact of software errors
– Direct costs of sw errors represent 0,6% of USA’s GNP [NIST, 2002]
17
17

18. Why should quality be the top priority?
Increasing dependency on software systems
– 80%+ of functionality of modern aircrafts provided by sw [Humphrey, 2002]
Increasing criticality of functions provided by software
– e-banking, aircraft control software, medical device software, …
Software defects are a primary cause of security vulnerabilities
– Buffer overflow, SQL injection, etc.
Huge economic impact of software errors
– Direct costs of sw errors represent 0,6% of USA’s GNP [NIST, 2002]
Quality work saves time and money
18
18

19. Why should quality be the top priority?
Increasing dependency on software systems
– 80%+ of functionality of modern aircrafts provided by sw [Humphrey, 2002]
Increasing criticality of functions provided by software
– e-banking, aircraft control software, medical device software, …
Software defects are a primary cause of security vulnerabilities
– Buffer overflow, SQL injection, etc.
Huge economic impact of software errors
– Direct costs of sw errors represent 0,6% of USA’s GNP [NIST, 2002]
Quality work saves time and money
– Defect prevention and early defect removal reduce rework costs
19
19

20. Quality economics: Cost of Quality (COQ)
20
20

21. Quality economics: Cost of Quality (COQ)
For producers, the goal is to reduce the cost of quality (COQ)
21
21

22. Quality economics: Cost of Quality (COQ)
For producers, the goal is to reduce the cost of quality (COQ)
– Prevention costs
22
22

23. Quality economics: Cost of Quality (COQ)
For producers, the goal is to reduce the cost of quality (COQ)
– Prevention costs
• identifying and resolving defect causes
• formal spec, prototyping, and design work
• process analysis and improvement
23
23

24. Quality economics: Cost of Quality (COQ)
For producers, the goal is to reduce the cost of quality (COQ)
– Prevention costs
• identifying and resolving defect causes
• formal spec, prototyping, and design work
• process analysis and improvement
– Appraisal (defect detection) costs
24
24

25. Quality economics: Cost of Quality (COQ)
For producers, the goal is to reduce the cost of quality (COQ)
– Prevention costs
• identifying and resolving defect causes
• formal spec, prototyping, and design work
• process analysis and improvement
– Appraisal (defect detection) costs
• reviews and inspections
• test development and execution (once)
25
25

26. Quality economics: Cost of Quality (COQ)
For producers, the goal is to reduce the cost of quality (COQ)
– Prevention costs
• identifying and resolving defect causes
• formal spec, prototyping, and design work
• process analysis and improvement
– Appraisal (defect detection) costs
• reviews and inspections
• test development and execution (once)
– Internal failure costs
26
26

27. Quality economics: Cost of Quality (COQ)
For producers, the goal is to reduce the cost of quality (COQ)
– Prevention costs
• identifying and resolving defect causes
• formal spec, prototyping, and design work
• process analysis and improvement
– Appraisal (defect detection) costs
• reviews and inspections
• test development and execution (once)
– Internal failure costs
• repair & rework before delivery
– External failure costs
• repair & rework plus any scrap after delivery
• law suites, loss of customers
27
27

28. Example: Security vulnerabilities
28
28

29. Example: Security vulnerabilities
29
29

30. Example: Security vulnerabilities
What
happened?
30
30

31. Example: Security vulnerabilities
What
happened?
What is the
cause?
31
31

32. Example: Security vulnerabilities
What
happened?
What is the
cause?
5 min
32
32

33. TSP Performance Results
33
33

34. Better product quality (industrial results)
Average Defect Density of Delivered Software
8
7,5
7 6,24
6
4,73
Defecst/KLOC
5
4
3 2,28
2
1,05
1 0,06
0
CMM L1 CMM L2 CMM L3 CMM L4 CMM L5 TSP
TSP impact study
Capers Jones, 2000 2003, 20 projects in
13 organizations,
several maturity levels
34
34

35. Better product quality (training results)
PSP Training results, SEI, 298 developers
Defects
Mean Number of Compile and Test 120
110
100
Defects Per KLOC
90
80
70
60
50
PSP0 PSP1 PSP2 1/3 with 0 defects
40
30
20
10
0
0 1 2 3 4 5 6 7 8 9 10 11
Program Number
35
35

36. Reduced security vulnerabilities (Microsoft)
TSP + secure coding standards,
security design patterns, static
analysis tools, secure code
inspections and reviews
8-person software development
team
Created 30 thousand lines of new
and modified code in 7 months
Source: TSP Secure, Noopur Davis et
all, TSP Symposium 2009, New Orleans
36
36

37. Better, faster, cheaper (Microsoft IT)
TSP
versus
non-TSP
projects
4 releases
with TSP
compared to
previous
releases
without TSP
Source: The TSP Story @ Microsoft IT, TSP Symposium, Phoenix, 2008
37
37

38. Reduced cost of quality (Adobe)
J.Sartain, Senior
Director, Quality,
Adobe 2009
38
38

39. Factor I – Bottom-Up
Performance
Improvement
39
39

40. Build high-performance teams from the
bottom-up
Team
Instance of
high-maturity
Management
Team
practices for Software
teams
Process
Team
Building
Instance of
high-maturity Personal Team
practices for Software Member
individuals Process Skills
http://www.sei.cmu.edu/tsp/
40
40

41. Build high-performance teams from the
bottom-up
Team
Instance of
high-maturity
Management
Team
practices for Software
teams
Process
Team
Building
Instance of
high-maturity Personal Team Process discipline
practices for Software Member Performance measures
individuals Process Skills Estimating & planning skills
Quality management skills
http://www.sei.cmu.edu/tsp/
41
41

42. Build high-performance teams from the
bottom-up
Team
Instance of
high-maturity
Management
Team
practices for Software
teams
Process
Goal setting
Team Role assignment
Building Tailored team process
Detailed balanced plans
Instance of
high-maturity Personal Team Process discipline
practices for Software Member Performance measures
individuals Process Skills Estimating & planning skills
Quality management skills
http://www.sei.cmu.edu/tsp/
42
42

43. Build high-performance teams from the
bottom-up
Team communication
Team Team coordination
Instance of
high-maturity
Management Project tracking
Team
practices for Risk analysis
Software
teams
Process
Goal setting
Team Role assignment
Building Tailored team process
Detailed balanced plans
Instance of
high-maturity Personal Team Process discipline
practices for Software Member Performance measures
individuals Process Skills Estimating & planning skills
Quality management skills
http://www.sei.cmu.edu/tsp/
43
43

44. Build personal performance from the
bottom-up
Personal practices are
introduced stepwise, through
a sequence of small projects
in a training environment Defect & yield
management;
– The objective is to convince design
people by seeing their own practices
performance improving as
they practice Size and
effort
estimation
Team practices are Establish a
measured
introduced in real projects performance
baseline
with the help of a coach
44
44

45. Example: Personal PSP training experience
45
45

46. Example: Personal PSP training experience
Goal: achieve 0 defects in unit testing and near 0% time
estimation error, with the same productivity as before
46
46

47. Example: Personal PSP training experience
Goal: achieve 0 defects in unit testing and near 0% time
estimation error, with the same productivity as before
PSP Fundamentals: Steady reduction of time estimation error, from
70% to 1% in 4 projects/days
47
47

48. Example: Personal PSP training experience
Goal: achieve 0 defects in unit testing and near 0% time
estimation error, with the same productivity as before
PSP Fundamentals: Steady reduction of time estimation error, from
70% to 1% in 4 projects/days
PSP Advanced: Initial decrease of productivity because of the
introduction of design documentation templates
48
48

49. Example: Personal PSP training experience
Goal: achieve 0 defects in unit testing and near 0% time
estimation error, with the same productivity as before
PSP Fundamentals: Steady reduction of time estimation error, from
70% to 1% in 4 projects/days
PSP Advanced: Initial decrease of productivity because of the
introduction of design documentation templates
8th project: compute the minimum difference between two Java
source files, in lines added, delete and modified, ignoring comments
and blank lines (considering that deletions have null cost)
49
49

50. Example: Personal PSP training experience
Goal: achieve 0 defects in unit testing and near 0% time
estimation error, with the same productivity as before
PSP Fundamentals: Steady reduction of time estimation error, from
70% to 1% in 4 projects/days
PSP Advanced: Initial decrease of productivity because of the
introduction of design documentation templates
8th project: compute the minimum difference between two Java
source files, in lines added, delete and modified, ignoring comments
and blank lines (considering that deletions have null cost)
– Almost recovered the initial productivity
50
50

51. Example: Personal PSP training experience
Goal: achieve 0 defects in unit testing and near 0% time
estimation error, with the same productivity as before
PSP Fundamentals: Steady reduction of time estimation error, from
70% to 1% in 4 projects/days
PSP Advanced: Initial decrease of productivity because of the
introduction of design documentation templates
8th project: compute the minimum difference between two Java
source files, in lines added, delete and modified, ignoring comments
and blank lines (considering that deletions have null cost)
– Almost recovered the initial productivity
– 5% effort estimation error
51
51

52. Example: Personal PSP training experience
Goal: achieve 0 defects in unit testing and near 0% time
estimation error, with the same productivity as before
PSP Fundamentals: Steady reduction of time estimation error, from
70% to 1% in 4 projects/days
PSP Advanced: Initial decrease of productivity because of the
introduction of design documentation templates
8th project: compute the minimum difference between two Java
source files, in lines added, delete and modified, ignoring comments
and blank lines (considering that deletions have null cost)
– Almost recovered the initial productivity
– 5% effort estimation error
– 2 defects in unit testing, caused by requirements problems!
52
52

53. Example: Personal PSP training experience
Goal: achieve 0 defects in unit testing and near 0% time
estimation error, with the same productivity as before
PSP Fundamentals: Steady reduction of time estimation error, from
70% to 1% in 4 projects/days
PSP Advanced: Initial decrease of productivity because of the
introduction of design documentation templates
8th project: compute the minimum difference between two Java
source files, in lines added, delete and modified, ignoring comments
and blank lines (considering that deletions have null cost)
– Almost recovered the initial productivity
– 5% effort estimation error
– 2 defects in unit testing, caused by requirements problems!
– Next step: improve requirements analysis!
53
53

54. Factor II – Personal
Responsibility
54
54

55. Personal responsibility: project management
There is only one way to manage knowledge workers:
they must manage themselves [W. Humphrey]
55
55

56. Personal responsibility: project management
There is only one way to manage knowledge workers:
they must manage themselves [W. Humphrey]
Team Leader
Traditional team
The leader plans, directs,
and tracks the team’s
work.
56
56

57. Personal responsibility: project management
There is only one way to manage knowledge workers:
they must manage themselves [W. Humphrey]
Team Leader TSP
Team Leader Coach
Traditional team
The leader plans, directs,
and tracks the team’s
work.
Self-directed team
All team members participate in planning,
managing, and tracking their own work.
57
57

58. Personal responsibility: project management
There is only one way to manage knowledge workers:
they must manage themselves [W. Humphrey]
Team Leader TSP
Team Leader Coach
Planning
Test
Manager
Manager
Process Implementation
Manager Manager
Design
Quality
Traditional team Manager
Manager
The leader plans, directs, Support Costumer Interface
and tracks the team’s Manager Manager
work.
Self-directed team
All team members participate in planning,
managing, and tracking their own work.
58
58

59. Personal responsibility: project management
There is only one way to manage knowledge workers:
they must manage themselves [W. Humphrey]
Team Leader TSP
Team Leader Coach
Planning
Test
Manager
Ownership Manager
Commitment
Motivation Implementation
Process
Performance Manager
Manager
Design
Quality
Traditional team Manager
Manager
The leader plans, directs, Support Costumer Interface
and tracks the team’s Manager Manager
work.
Self-directed team
All team members participate in planning,
managing, and tracking their own work.
59
59

60. Personal responsibility: quality management
The only way to build high-quality products in a cost-effective
way, is by having developers being personally responsible for
the quality of their products.
Since defects can best be managed where they are injected,
developers should
– remove their own defects
– determine the causes of their defects
– learn to prevent those defects
60
60

61. Factor III – Feadback
Loops for Continual
Improvement
61
61

62. Feedback loops: project lifecycle
Business Release planning
and Launch
technical
goals Estimates, plans,
process,
Iteration planning commitment
Re-launch
Developm Status
Lessons, new goals, phase
Development
phase
or cycle
new requirements, new orphase
cycle
risks, etc. or cycle Weekly
team meetings
Phase or cycle Updated status
Retrospective Work products, and plans
status, metrics,
results
Project
Retrospective
62
62

63. Feedback loops: estimating & planning frmwk
Costumer Define
need requirements
Items
PROBE method
Produce conceptual
(PROxy Based Estimating) Tasks
design (PBS)
Estimate Size
size database
Estimate resources Productivity
Costumer
(effort) database
Produce task list, Hist. distrib.,
distribute effort Process def.
Produce Resources Management
schedule available
Product Develop Size, resource, Process Tracking
delivery product schedule data analysis reports
63
63

64. Feedback loops: quality framework
Learn from past errors, because we tend to repeat the same errors!
Development (and defect Scripts,
defect injection) prevention Standards,
phase Awareness
Defect removal Checklists,
phase early defect Test Criteria, …
detection &
removal
Subsequent Defect Process/Data
phases late defect Data analysis
detection &
removal
64
64

65. Feedback loops: coaching (& leadership)
A coach helps improving individual and
team performance through a continuous
coaching/learning cycle
External
Independent voice
2. Conscious 1. Unconscious
Incompetence Incompetence
Guidance Next
and Stage
Feedback
3. Conscious 4. Unconscious
Competence Competence
Practice
65
65

66. Example: Historical defect data analysis
66
66

67. Example: Historical defect data analysis
67
67

68. Example: Historical defect data analysis
Defect Types
10 - Documentation
60 – Checking
80 – Function
40 - Assignment
70 - Data
50 - Interface
30 - Build
100 - Environment
68
68

69. Example: Historical defect data analysis
Defect Types
10 - Documentation
60 – Checking
80 – Function
Most frequent
40 - Assignment
70 - Data
50 - Interface
30 - Build
100 - Environment
69
69

70. Example: Historical defect data analysis
Defect Types
10 - Documentation
60 – Checking
Most frequent Most expensive* 80 – Function
40 - Assignment
70 - Data
50 - Interface
30 - Build
100 - Environment
70
70

71. Example: Historical defect data analysis
Defect Types
10 - Documentation
60 – Checking
Most frequent Most expensive* 80 – Function
40 - Assignment
70 - Data
50 - Interface
30 - Build
100 - Environment
(*) further analyses
showed that they
were mainly injected
in design and
removed in test.
71
71

72. Example: Historical defect data analysis
Defect Types
10 - Documentation
60 – Checking
Most frequent Most expensive* 80 – Function
40 - Assignment
70 - Data
50 - Interface
30 - Build
100 - Environment
(*) further analyses
showed that they
were mainly injected
in design and
removed in test.
Improve design review checklists
Improve design standards and scripts
Produce test spec after design spec and before design review
Minimize documentation and use spell checker
72
72

73. Factor IV - Cost-Effective
Defect Removal Methods
73
73

74. Defect removal methods (V-Model)
Requirements REQ Team System
& ST Spec Inspection Test (ST)
High-Level Integration
HLD Team
Design Test (IT)
Inspection
& IT Spec
Detailed DLD Code Team
Design Personal DLD Team
Inspection Inspection
& UT Spec Review
Code
Unit Test
Code Personal Compile (UT)
Review
74
74

75. Efficiency of defect removal methods
Even experienced developers introduce ~100 defects/KLOC
Such high-number of defects must be removed using the most cost-
effective methods, such as personal reviews and team inspections
(Continue to use unit, integration and system testing)
Minutes to Find and Resolve a Defect by Discovery Activity
Source:
Jim Sartain,
Adobe,
2009
75
75

76. Use checklists derived from historical defect
data
Peter Pronovost
(Dr. Checklist)
“Most influential
people of 2008”
(Time magazin)
http://www.youtub
e.com/watch?v=YK
m8NUmPg58
76
76

77. Use checklists derived from historical defect
data
Make the review more effective
Peter Pronovost
(Dr. Checklist)
“Most influential
people of 2008”
(Time magazin)
http://www.youtub
e.com/watch?v=YK
m8NUmPg58
77
77

78. Use checklists derived from historical defect
data
Make the review more effective
– focus the attention on most frequent problems
Peter Pronovost
(Dr. Checklist)
“Most influential
people of 2008”
(Time magazin)
http://www.youtub
e.com/watch?v=YK
m8NUmPg58
78
78

79. Use checklists derived from historical defect
data
Make the review more effective
– focus the attention on most frequent problems
Make the review more efficient
Peter Pronovost
(Dr. Checklist)
“Most influential
people of 2008”
(Time magazin)
http://www.youtub
e.com/watch?v=YK
m8NUmPg58
79
79

80. Use checklists derived from historical defect
data
Make the review more effective
– focus the attention on most frequent problems
Make the review more efficient
– don’t waste time looking for too rare problems
Peter Pronovost
(Dr. Checklist)
“Most influential
people of 2008”
(Time magazin)
http://www.youtub
e.com/watch?v=YK
m8NUmPg58
80
80

81. Use checklists derived from historical defect
data
Make the review more effective
– focus the attention on most frequent problems
Make the review more efficient
– don’t waste time looking for too rare problems
Reduce the risk of missing critical issues
Peter Pronovost
(Dr. Checklist)
“Most influential
people of 2008”
(Time magazin)
http://www.youtub
e.com/watch?v=YK
m8NUmPg58
81
81

82. Use checklists derived from historical defect
data
Make the review more effective
– focus the attention on most frequent problems
Make the review more efficient
– don’t waste time looking for too rare problems
Reduce the risk of missing critical issues
Peter Pronovost
– even experts benefit from checklists
(Dr. Checklist)
“Most influential
people of 2008”
(Time magazin)
http://www.youtub
e.com/watch?v=YK
m8NUmPg58
82
82

83. Use checklists derived from historical defect
data
Make the review more effective
– focus the attention on most frequent problems
Make the review more efficient
– don’t waste time looking for too rare problems
Reduce the risk of missing critical issues
Peter Pronovost
– even experts benefit from checklists
(Dr. Checklist)
But:
“Most influential
people of 2008”
(Time magazin)
http://www.youtub
e.com/watch?v=YK
m8NUmPg58
83
83

84. Use checklists derived from historical defect
data
Make the review more effective
– focus the attention on most frequent problems
Make the review more efficient
– don’t waste time looking for too rare problems
Reduce the risk of missing critical issues
Peter Pronovost
– even experts benefit from checklists
(Dr. Checklist)
But:
“Most influential
– keep it simple and short people of 2008”
(Time magazin)
http://www.youtub
e.com/watch?v=YK
m8NUmPg58
84
84

85. Use checklists derived from historical defect
data
Make the review more effective
– focus the attention on most frequent problems
Make the review more efficient
– don’t waste time looking for too rare problems
Reduce the risk of missing critical issues
Peter Pronovost
– even experts benefit from checklists
(Dr. Checklist)
But:
“Most influential
– keep it simple and short people of 2008”
– keep it specific for the technology, language, etc. (Time magazin)
http://www.youtub
e.com/watch?v=YK
m8NUmPg58
85
85

86. Use checklists derived from historical defect
data
Make the review more effective
– focus the attention on most frequent problems
Make the review more efficient
– don’t waste time looking for too rare problems
Reduce the risk of missing critical issues
Peter Pronovost
– even experts benefit from checklists
(Dr. Checklist)
But:
“Most influential
– keep it simple and short people of 2008”
– keep it specific for the technology, language, etc. (Time magazin)
– combine with other analysis techniques http://www.youtub
e.com/watch?v=YK
m8NUmPg58
86
86

87. Take enough review time
The review rate (size reviewed per hour) is one of the best
leading indicators of review effectiveness (review yield)
Recommended code review rate: about 200LOC/hour
Revewing too slow or too fast is a waste of time
Also recommended to
review in a quiet environment
take a break between producing and reviewing
87
87

88. Use multiple inspectors to improve yield
(yield = % of defects detected)
88
88
(source: Introduction to the Team Software Process, Watts Humphrey)

89. Estimate defects remaining with the capture-
recapture method
Based on the degree of overlapping between different reviewers
Common defects:
C=A B=4
Total defects:
T A*B/C = 12
Found
Found
by A = 8
by B = 6
Remaining defects:
R A*B/C-(A+B-C) = 2
http://en.wikipedia.org/wiki/Mark_and_recapture
89
89

90. Measure the review process and use data to
improve the reviews
Escaped defects
(collected later)
Defects found
(number, type, description, severity)
Effort Size
(review, rework) (of artifact reviewed)
90
90

91. Measure the review process and use data to
improve the reviews
Escaped defects
(collected later)
Defects found
(number, type, description, severity)
Defect removal rate
(indicator of review
efficiency)
Effort Size
(review, rework) (of artifact reviewed)
91
91

92. Measure the review process and use data to
improve the reviews
Escaped defects
(collected later)
Defects found
(number, type, description, severity)
Defect removal rate Defect density
(indicator of review (indicator of product quality)
efficiency)
Effort Size
(review, rework) (of artifact reviewed)
92
92

93. Measure the review process and use data to
improve the reviews
Escaped defects
(collected later)
Defects found
(number, type, description, severity)
Defect removal rate Defect density
(indicator of review (indicator of product quality)
efficiency)
Effort Size
(review, rework) (of artifact reviewed)
Review rate
(leading indicator of review effectiveness)
93
93

94. Measure the review process and use data to
improve the reviews
Escaped defects
(collected later)
Review yield
(lagging indicator of
review effectiveness)
Defects found
(number, type, description, severity)
Defect removal rate Defect density
(indicator of review (indicator of product quality)
efficiency)
Effort Size
(review, rework) (of artifact reviewed)
Review rate
(leading indicator of review effectiveness)
94
94

95. Factor V - Measurement
and Quantitative Methods
95
95

96. The need for measurement
96
96

97. The need for measurement
In God we
trust, all
others
bring
data.
W. Edwards Deming
97
97

98. The need for measurement
In God we You can't
trust, all manage
others and
bring improve
data. what you
don't
measure.
W. Edwards Deming Tom DeMarco
98
98

99. The need for measurement
In God we You can't When performance is unmeasured or
trust, all manage improperly measured, the results are
others and often disappointing and can even be
bring improve disastrous. Unless your measures cover
data. what you all important aspects, you will likely
don't motivate counterproductive action.
measure.
W. Edwards Deming Tom DeMarco Watts Humphrey
99
99

100. Base measures
In the TSP, 4 core measures are the basis for quantitative
project management,
quality management, and
process improvement
at the personal, team and organization level.
Size Effort Actual and
Planned
Quality Schedule
100
100

101. Defect tracking
Defects are the measure of quality in the TSP.
Any change to an interim or final work product, made to
ensure proper subsequent utilization is considered a defect.
101
101

102. Quality planning: Defect injection and
removal plan
Removed
• Defects injected based on historical defects injected per size unit
• Yields (% of defects removed) based on historical data or benchmarks
102
102

103. Quality tracking: Defect removal profile
The defect removal profile shows
– plan and actual defects removed by phase
– early vs. late defect removal plan
Defects Removed by Phase for Assembly SYSTEM
900.0
800.0
Defects Removed by Phase
700.0
600.0
500.0 Plan
400.0 Actual
300.0
200.0
100.0
0.0
n
st
le
st
n
n
n
e
st
w
w
ti o
tio
tio
Te
tio
od
pi
Te
ie
Te
ie
ec
om
ec
ec
ev
ec
C
ev
n
t
em
sp
ni
io
sp
sp
sp
R
R
C
U
at
In
LD
In
In
In
st
e
gr
od
Sy
EQ
LD
LD
e
D
te
od
C
In
R
H
D
C
d
an
i ld
Bu
Phase
103
103

104. Quality tracking: Defect removal profile
The defect removal profile shows
– plan and actual defects removed by phase
– early vs. late defect removal plan
Defects Removed by Phase for Assembly SYSTEM
900.0
800.0
Defects Removed by Phase
700.0
600.0
500.0 Plan
400.0 Actual
300.0
200.0
100.0 Typical
0.0
software
n
st
le
st
n
n
n
e
st
w
w
ti o
tio
tio
Te
tio
od
pi
Te
ie
Te
ie
ec
project
om
ec
ec
ev
ec
C
ev
n
t
em
sp
ni
io
sp
sp
sp
R
R
C
U
at
In
LD
In
In
In
st
e
gr
od
Sy
EQ
LD
LD
e
D
te
od
C
In
R
H
D
C
d
an
i ld
Bu
Phase
104
104

105. Quality Tracking: Quality profile
It examines Quality Profile for Assembly Common Query Changes (BE)
criteria that
are effective Design Quality
predictors of Design/Code Time 1
Quality Profile for Assembly Common Query Changes (BE)
1
system test
and post- 0.8
release Design/Code Time
0.6
component Design Review 1
Code Review
Quality 0.4 Quality
quality Design Review Time
0.8
Code Review Time
½ design time 0.6 0.2 ½ coding time
0.4 0
Design Review Time Code Review Time
0.2
Plan
0
Actual
Program Quality Code Quality
Unit Test Ddefects/KLOC Compile Defects/KLOC
4 10
Unit Test Ddefects/KLOC Compile Defects/KLOC
105
105

106. Conclusions: TSP Quality principles
106
106

107. Conclusions: TSP Quality principles
To produce quality products, developers must feel personally
responsible for the quality of their products. Superior
products are not produced by accident; developers must
strive to do quality work.
107
107

108. Conclusions: TSP Quality principles
To produce quality products, developers must feel personally
responsible for the quality of their products. Superior
products are not produced by accident; developers must
strive to do quality work.
It costs less to find and fix defects earlier in a process than
later.
108
108

109. Conclusions: TSP Quality principles
To produce quality products, developers must feel personally
responsible for the quality of their products. Superior
products are not produced by accident; developers must
strive to do quality work.
It costs less to find and fix defects earlier in a process than
later.
It is more efficient to prevent defects than to find and fix them.
109
109

110. Conclusions: TSP Quality principles
To produce quality products, developers must feel personally
responsible for the quality of their products. Superior
products are not produced by accident; developers must
strive to do quality work.
It costs less to find and fix defects earlier in a process than
later.
It is more efficient to prevent defects than to find and fix them.
The right way is always the fastest and cheapest way to do a
job.
110
110

111. Conclusions: TSP Quality principles
To produce quality products, developers must feel personally
responsible for the quality of their products. Superior
products are not produced by accident; developers must
strive to do quality work.
It costs less to find and fix defects earlier in a process than
later.
It is more efficient to prevent defects than to find and fix them.
The right way is always the fastest and cheapest way to do a
job.
To maximize productivity, focus on quality first.
111
111

112. Conclusions
The TSP approach for producing high-quality software in a
cost-effective way is based on learning from your errors
and using cost-effective defect removal methods
Data shows that TSP teams produce software with very low
defect density and reduced cost of quality
If you have the need to develop top quality software, you
should take a look at TSP principles and practices
112
112

113. References
Software Assessments, Benchmarks, and Best Practices, Capers Jones,
Addison-Wesley, 2000
Winning with Software, Watts Humphrey, 2002
Introduction to the Team Software Process, Watts S. Humphrey, 2000
TSP: Coaching Development Teams, Watts S. Humphrey, Addison-Wesley,
2006
PSP: A Self-Improvement Process for Software Engineers, Watts S.
Humphrey, 2005
“The Economic Impacts of Inadequate Infrastructure for Software Testing”,
National Institute of Standards and Technology (NIST), 2002
“Inspiring, enabling and driving the Evolution of Quality at Adobe leveraging
the TSP”, Jim Sartain, Senior Director, Quality at Adobe Systems, TSP
Symposium 2009
“Some Future Software Engineering Opportunities and Challenges”, Barry
Boehm, in The Future of Software Engineering, Springer, 2011
113
113

114. Portugal
Thank you!
Contact information:
João Pascoal Faria (*)
Departamento de Engenharia Informática
Faculdade de Engenharia da Universidade do Porto
Rua Dr. Roberto Frias, s/n 4200-465 Porto, PORTUGAL
Email: jpf@fe.up.pt
Phone: +351225082134
Mobile: +351966494914
(*) SEI Qualified PSP Developer, PSP Instructor, TSP Coach