2. ®2016 All Rights Reserved - PPQC 2 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
• Core Concepts
• Iden-fying Sta-s-cal Stability
• Selec-ng Processes and Performance Measures
• Ques-ons and Answers
• Some Common Measures (reference only)
4. ®2016 All Rights Reserved - PPQC 4 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
• Performed Process
o A process that accomplishes the needed work to produce work
products
• Managed Process
o A performed process that is planned and executed in accordance with
policy; employs skilled people having adequate resources to produce
controlled outputs; involves relevant stakeholders; is monitored,
controlled, and reviewed; and is evaluated for adherence to its
process descripLon
• Defined Process
o A managed process that is tailored from the organizaLon’s set of
standard processes according to the organizaLon’s tailoring
guidelines; has a maintained process descripLon; and contributes
process related experiences to the organizaLonal process assets
5. ®2016 All Rights Reserved - PPQC 5 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
• Special Cause of Varia-on
o A cause of a defect that is specific to some transient circumstance and
is not an inherent part of a process
• Common Cause of Varia-on
o The variaLon of a process that exists because of normal and expected
interacLons among components of a process
• Stable Process
o The state in which all special causes of process variaLon have been
removed and prevented from recurring so that only common causes
of process variaLon of the process remain
6. ®2016 All Rights Reserved - PPQC 6 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
• Quality and Process Performance Objec-ves (QPPOs)
o QuanLtaLve objecLves and requirements for product quality, service
quality, and process performance
• Capable Process
o A process that can saLsfy its specified product quality, service quality,
and process-performance objecLves
• Process Performance
o A measure of actual results achieved by following a process. Process
performance is characterized by both process measures (e.g., effort,
cycle Lme, and defect removal efficiency) and product or service
measures (e.g., reliability, defect density, and response Lme)
7. ®2016 All Rights Reserved - PPQC 7 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
• Process Performance Baseline (PPB)
o A documented characterizaLon of process performance, which can
include central tendency and variaLon. A process performance
baseline can be used as a benchmark for comparing actual process
performance against expected process performance
8. ®2016 All Rights Reserved - PPQC 8 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
• Process Performance Model (PPM)
o A descripLon of the relaLonships among the measurable aZributes of
one or more processes or work products that is developed from
historical process performance data and is used to predict future
performance
o One or more of the measureable aZributes represent controllable
inputs Led to a sub-process to enable performance of “what-if”
analyses for planning dynamic re-planning, and problem resoluLon
o PPMs include staLsLcal, probabilisLc, and simulaLon based models
that predict interim or final results by connecLng past performance
with future outcomes
o PPMs model the variaLon of the factors and provide insight into the
expected range and variaLon of predicted results
9. ®2016 All Rights Reserved - PPQC 9 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
• Compose the Defined Process
o Composing the Project’s Defined Process (PDP) involves idenLfying
alternaLves to one or more processes or sub-processes, performing
quanLtaLve analysis of performance and selecLng the alternaLves
that are best able to help the project achieve its QPPOs
• Sta-s-cal Process Control (SPC)
o StaLsLcally based analysis of a process and measures of process
performance, which will idenLfy common and special causes of
variaLon in process performance and maintain process performance
within limits
10. ®2016 All Rights Reserved - PPQC 10 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
• Sta-s-cal Techniques
o Techniques adapted from the field of mathemaLcal staLsLcs used for
acLviLes such as characterizing process performance, understanding
process variaLon, and predicLng outcomes.
o Examples of staLsLcal techniques include sampling techniques,
analysis of variance (ANOVA), chi-squared tests, and process control
charts
• Causal Analysis
o The analysis of outcomes to determine their causes
o Used to address selected issues and diagnose deficiencies in achieving
the project’s QPPOs
12. ®2016 All Rights Reserved - PPQC 12 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
Before an organiza-on can perform High Maturity ac-vi-es, it
must have the following in place:
o Capability to gather and use data at all organizaLonal levels, i.e.,
project members who
• Gather data on their own work
• Understand and use the data in planning and performing their work
o Processes that specify how and when data are collected
o Consistent execuLon of the defined process, where tailoring is
handled in a controlled and disciplined fashion
13. ®2016 All Rights Reserved - PPQC 13 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
• Everything we do in life, including product development and
service delivery, is a series of interconnected processes
• All processes are variable
• People performing the process is the largest source of
varia-on
• Understanding varia-on is the basis for management by fact
and systema-c improvement:
o Understand the past – quan-ta-vely
o Control the present – quan-ta-vely
o Predict the future – quan-ta-vely
15. ®2016 All Rights Reserved - PPQC 15 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
• Real process behavior must be understood before drawing
reasonable conclusions about where performance needs to
be improved
• Ask these ques-ons to find out about real process behavior:
o What is the normal or inherent process variaLon?
o What differenLates inherent from anomalous variaLon?
o What is causing the anomalous variaLon?
• Sta-s-cal Process Control provides the methods and tools
needed to measure and analyze process behavior, draw
conclusions, and drive improvement
16. ®2016 All Rights Reserved - PPQC 16 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
• A summary or characteriza-on of a distribu-on (i.e., a set of
numbers)
• A characteriza-on of a central tendency (e.g., mean,
median, and mode)
• A characteriza-on of
dispersion (e.g., variance,
standard devia-on,
interquar-le range, and
range)
17. ®2016 All Rights Reserved - PPQC 17 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
Statistics (specifically hypothesis testing) enable you to place a
confidence interval on the Central Tendency and Dispersion of the
population and on future samples
A data population consists of the
total possible observations with
which you are concerned but to
which you do not necessarily
have access (X1 thru X15).
X5
X15
X14 X13
X12
X11
X6
X10X8 X7
X9
X3
X2
X1
X4
A sample
is a set of observations selected
from a data population that you
can access.
18. ®2016 All Rights Reserved - PPQC 18 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
• Hypothesis tes-ng uses sta-s-cal analysis as a formal way
of making a comparison and deciding whether or not the
difference is sta-s-cally significant
• Hypothesis tes-ng consists of comparing a null and an
alterna-ve hypothesis:
o The null hypothesis states that the members of the comparison are
staLsLcally equal; there is no difference (a concrete, default posiLon)
o The alternaLve hypothesis states that there is a staLsLcally significant
difference; it is supported when the null hypothesis is rejected
• The conclusion either rejects or fails to reject the null
hypothesis.
19. ®2016 All Rights Reserved - PPQC 19 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
When the p is low, the null must go.
When the p is high, the null must fly.
Note: The p-value is the key output in staLsLcal analysis that students are taught to
idenLfy and use to draw a conclusion regarding the hypothesis test comparison or
regarding the significance of a staLsLcal model
21. ®2016 All Rights Reserved - PPQC 21 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
• “While every process displays varia1on, some processes
display controlled varia1on, while others display
uncontrolled varia1on”
• Rou-ne varia-on:
o A stable paZern of random variaLon over Lme
o Typical observaLon within a common cause system
• A process is out of sta-s-cal control if it is opera-ng in the
presence of assignable causes
o A factor that is not inherent in the process is affecLng its performance
o VariaLon in the process can be “assigned” to a specific cause
• A process under sta-s-cal control is predictable
22. ®2016 All Rights Reserved - PPQC 22 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
• Wheeler’s Empirical Rules:
o 60% ≤ ±1σ ≤ 75%
o 90% ≤ ±2σ ≤ 98%
o 99% ≤ ±3σ ≤ 100%
Wheeler’s Empirical Rules are robust with regard to a remarkable number of distribu-ons
23. ®2016 All Rights Reserved - PPQC 23 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
• SPC is a technique that can be used to tell you how much
variability there is in a process
o How much variability is due to rouLne variaLon
o How much variability is due to some unique event or factor not
inherent in the process (assignable causes)
o A control chart is a tool used in SPC to help do this
• Control charts are generated from two data sets
o PPB data – used to staLsLcally derive a process average, Upper
Control Limit (UCL), and Lower Control Limit (LCL) (±3σ)
o Actual performance data – ploZed like a Run Chart and compared
against baseline data
24. ®2016 All Rights Reserved - PPQC 24 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
• Which of these 20 projects should be acted on?
0
1
2
3
4
5
6
7
8
9
10
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Number of Non-compliances
Project ID
25. ®2016 All Rights Reserved - PPQC 25 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
• Different techniques can be used to understand process
varia-on
o Simple graphs, such as frequency histograms and bar charts
o StaLsLcal formulae, such as standard deviaLon and range
o StaLsLcal process control tools, such as XmR charts and u-charts
o Analysis of variance techniques, such as ANOVA and hypothesis test
27. ®2016 All Rights Reserved - PPQC 27 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
0
1
2
3
4
5
6
7
8
9
10
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Number of Non-compliances
Project ID
Upper Control Limit = 11.7
Mean = 5
Lower Control Limit = 0
28. ®2016 All Rights Reserved - PPQC 28 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
Upper Control Limit
Lower Control Limit
Process Average
ExcepLonal process
variaLon
ExcepLonal
process variaLon
Actual Performance Data from Project XYZ
30. ®2016 All Rights Reserved - PPQC 30 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
• 1 single point outside 3σ
• 2 of 3 successive values on same
side of centerline and outside of
2σ
• 8 successive values on same side
of centerline
31. ®2016 All Rights Reserved - PPQC 31 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
“… in so:ware development it is difficult to use control charts
in the formal SPC manner. It is a formidable task, if not
impossible, to aDempt to define the process capability of a
so:ware development process.”
- Stephen H. Kan (1995, p. 143)
• Cri-cal assump-on for interpre-ng control charts – data are
affected by homogeneous sources of varia-on
o Same populaLon
o Same behavioral implementaLon of the process
o Same common cause system
32. ®2016 All Rights Reserved - PPQC 32 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
“Despite these issues, control charts are useful for so:ware
quality improvement – when they are used in a relaxed manner
… control charts use in so:ware is not in terms of formal
sta1s1cal process control and process capability … used as a
tool for improving consistency and stability … more
appropriately called pseudo-control charts.”
- Stephen H. Kan (1995, p. 144)
33. ®2016 All Rights Reserved - PPQC 33 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
• System development processes rou-nely operate within the
presence of mul-ple common cause systems – PPBs mix natural
varia-on with:
o VariaLon in developer skills
o VariaLon in aZributes of work products (e.g. complexity)
o VariaLon in situaLons
• The voice of the process is a cacophony of many voices from many
behavioral implementa-ons of the process:
o Confounded by variaLon from many different sources
o Control limits are excessively wide – occasionally 10X range
o Out of control events are difficult to detect
o ‘StaLsLcal stability’ may not offer control or predictability
o A single PPB may provide poor characterizaLon
34. ®2016 All Rights Reserved - PPQC 34 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
Experienced Inexperienced
Method &
Support
Tools
No Method
& Support
Tools
Each cell
represents a
unique
common cause
system with
different mean
results
35. ®2016 All Rights Reserved - PPQC 35 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
• In the example, methods and experience levels are:
o Sources of process variaLon that guide the idenLficaLon of common
cause systems
o Factors whose variaLon must be controlled in establishing separate
common cause systems
o Likely to appear as assignable causes of process variaLon if data from
several common cause systems are combined into a single PPB
• Each cell is a different common cause system
o Different PPBs
o Different behavioral implementaLons of the process
• The typical uses of control charts in system development
regularly mix data from mul-ple common cause systems
36. ®2016 All Rights Reserved - PPQC 36 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
• Control charts are appropriate when:
o The ploZed results characterize process performance within a single
common cause system
o Data describe a single behavioral implementaLon of the process
o Process fidelity is high (the process is well-defined and performed in
compliance with how it is defined)
• When condi-ons are appropriate for control charts;
o Process stability is a meaningful concept
o AggregaLng PPBs provides a good basis for understanding
performance
38. ®2016 All Rights Reserved - PPQC 38 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
• Typical ML 4 prac-ces result in plokng aggregated inspec-on results
o Across different developers
o Across different reviewers
o Across components or work products with different aZributes
39. ®2016 All Rights Reserved - PPQC 39 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
• Since people are the greatest source of varia-on in system development
a more Personal Solware Process (PSP)-like approach:
o Treats individuals as separate common cause systems rather than as assignable
causes
o Decreases the width of control limits
o Makes unlikely or unexpected events easier to detect
40. ®2016 All Rights Reserved - PPQC 40 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
“A:er product size, people factors have the strongest influence
in determining the amount of effort required to develop a
so:ware product.”
- Barry Boehm, et. al. (2000)
43. ®2016 All Rights Reserved - PPQC 43 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
• Establish and maintain the organization’s QPPOs,
which are traceable to business objectives
o The QPPOs are derived from the organizaLon’s business objecLves
and will typically be specific to the organizaLon, group, or funcLon
o The QPPOs take into account what is realisLcally achievable based
upon a quanLtaLve understanding (knowledge of variaLon) of the
organizaLon’s historic quality and process performance
44. ®2016 All Rights Reserved - PPQC 44 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
• A simple task that many -mes does not specify enough
informa-on for use at Maturity Level 4 or 5
• QPPOs need to be SMART
o Specific
o Measurable
o Achievable
o RealisLc
o Time-based
45. ®2016 All Rights Reserved - PPQC 45 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
• Select measures and processes that provide insight into the
organiza-on’s ability to meet its QPPOs
o Create/update clear unambiguous operaLonal definiLons for the
selected measures
o Revise and update the set of measures, analyses, and procedures as
warranted
o In usage, be sensiLve to measurement error
o The set of measures may provide coverage of the enLre lifecycle and
be controllable
47. ®2016 All Rights Reserved - PPQC 47 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
• Each project and organiza-on has unique informa-on needs
• When iden-fying goals and objec-ves, consider sources such as
o business and customer needs
o constraints and assumpLons
o project or organizaLon characterisLcs
• domain, technology
• people and skills
• product features and acceptance criteria
• cost, schedule, and milestones
o risks
o past experience of the team
48. ®2016 All Rights Reserved - PPQC 48 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
Enterprise
How do we perform,
compared to others?
Which
improvements
should we fund?
In what training or
tools must we
invest?
Are our customers
sa-sfied?
Process
How consistent is
our performance?
Which processes
require
improvement?
What is the ROI of
improvement
efforts?
How are
improvement efforts
progressing?
Project
How big is this
project?
Is the project plan
feasible?
How is the project
progressing?
When will we be
ready to deliver?
Product
How big is the
applica-on?
Which modules are
have the most
defects?
What is the cost to
maintain?
What are the quality
and growth trends?
Goals
Data
49. ®2016 All Rights Reserved - PPQC 49 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
Term Defini-on
Data A collecLon of actual values assigned to a measure
(example: data for a source lines of code base measure
might be 100, 115, 121).
Measurement A set of operaLons having the object of determining a
value of a measure.
Measurement
process
The process for establishing, planning, performing, and
evaluaLng measurement within a project or organizaLon.
Measure (noun) The rule for assigning a quanLtaLve or categorical value to
one or more enLty/ object/aZribute (e.g. process, project,
resource).
Informa-on
Need
Reason for which data or measures are needed; generally
phrased as “<who> needs to know <what>, in order to <do
what>”
50. ®2016 All Rights Reserved - PPQC 50 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
• Process Compliance
• Process Efficiency
• Process Effec-veness
Process
Perform-
ance
• Milestone CompleLon
• CriLcal Path Performance
• Work Unit Progress
• Incremental Capability
Schedule
and
Progress
• Physical Size and Stability
• FuncLonal Size and Stability
Product
Size and
Stability
Source: Prac=cal So>ware and Systems Measurement
• FuncLonal Correctness
• Supportability – Maintainability
• Efficiency
• Portability
• Usability
• Dependability - Reliability
Product
Quality
• Personnel Effort
• Financial Performance
• Environment and Support
Resources
Resources
and Cost
• Technology Suitability
• Technology VolaLlity
Technology
Effec-ve-
ness
• Customer Feedback
• Customer Support
Customer
Sa-sfac-on
51. ®2016 All Rights Reserved - PPQC 51 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
• Balanced Scorecards provide a mechanism to balance
o internal goals and external goals
o past performance and drivers for future performance
o short term and long term performance
o People, Process, and Technology investment
• Categories generally used
Financial Outcomes
Customer
Sa-sfac-on
Internal Business
Processes
Learning and
Growth
Balanced
Scorecard
52. ®2016 All Rights Reserved - PPQC 52 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
• First, establish and communicate a strategy
o set financial objecLves
o idenLfy key needs of customers and market segments; link those to financial goals
o focus on internal business processes and efficiencies in meeLng customer needs
o invest in learning and growth to excel with the processes
• Within the scorecard
o idenLfy how to see financial goals are met
o set targets for meeLng customer needs
o idenLfy process changes or innovaLons needed
o match learning and improvement objecLves to these
53. ®2016 All Rights Reserved - PPQC 53 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
Financial
Increase
profitability
Increase
revenue
Increase
margins
Customer
Serve more
levels of
business
Use social
networking
services
Internal
Processes
Increase range
of services
Provide social
networking
services
Maintain social
networking
site
Learning
and Growth
Build
capability
Increase social
networking
skills
Business Goals QPPOs
54. ®2016 All Rights Reserved - PPQC 54 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
Area Measure Target
Financial Profit
Revenue Growth
Margin
25% before tax
+30% over last year
+5% over last year
Customer Mean # business levels served
% using social networking
services
+10% over last year
15% this year
Internal
Processes (and
efficiency)
increase range of services
Minimize social networking
downLme
new PMO sales by July
web site up 99.9%
Learning and
Growth
# of PMPs
# of months of total social
networking experience
+3 over last year
+24 over last year
55. ®2016 All Rights Reserved - PPQC 55 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
• In general, process improvement directly affects the lower half of
the Balanced Scorecard
o process development and/or maintenance
o process performance
o learning and growth of process users
o a way of idenLfying and tracing QPPOs
• Changes in these areas can link to
o increased customer saLsfacLon
o increase in customer count or transacLon value
o reducLon in cost to meet customer needs
• These, in turn, translate to financial results
56. ®2016 All Rights Reserved - PPQC 56 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
Financial
Develop new
business
Reduce costs
Customer
Increase
Customer
Contact
Increase
customer
value
Internal
Processes
Increase
process
compliance
Improve
process quality
Improve
product quality
Learning
and Growth
Increase skills
Retain more
employees
Raise
employee
saLsfacLon
Overall Goal: Controlled Growth and Sa-sfied Return
Business Goals QPPOs
57. ®2016 All Rights Reserved - PPQC 57 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
Areas Objec-ves Measures
Financial • Develop New Business
• Reduce Cost
Total revenue
Revenue in new business segment
ProducLvity (LOC/project hr.)
Customer • Increase Customer Contact
• Increase Customer Value
% customers with annual interview
% employees visiLng customers
customer saLsfacLon
% with maintenance contracts
Internal
Processes
• Increase Process Compliance
• Improve Process Quality
• Improve Product Quality
1-5 on key pracLces
% rework
% milestones on schedule
# warranty hours needed
Learning
and Growth
• Increase Skills
• Retain More Employees
• Raise Employee SaLsfacLon
# cerLfied in key areas
% retenLon
employee saLsfacLon score
58. ®2016 All Rights Reserved - PPQC 58 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N GStrategic Goal
supporLng goal
supporLng goal
supporLng goal
supporLng goal
supporLng goal measure
measure
measure
measure
measure performance
performance measure
measure
measure
measure
Corporate Business Goals and
Objec-ves
Quality and Process
Performance Objec-ves
59. ®2016 All Rights Reserved - PPQC 59 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N GImproved Profitability
OpLmize ROI
Improve customer
saLsfacLon
Improve posiLoning
in current market
Minimize cost of
business
Improve operaLons
efficiency
Overhead
rates
Product-
ivity
Customer
SaLsfacLon
Market
Share
Revenue
Corporate Business Goals and
Objec-ves
Minimize
development Lme
Improve review
process to remove
defects early
COQ
# CRs
# defects
Cycle Lme
Quality and Process
Performance Objec-ves
60. ®2016 All Rights Reserved - PPQC 60 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
Improvement Goals QPPO
For small features requested by
customers, provide a response
next year in half the Lme we do
today by using an incremental
release model
• Reduce cycle Lme for a release
by 50%
Reduce negaLve impact of change
management on customer
saLsfacLon
• Cut response Lme on defect
handling by 25%
• Cut response Lme on change
requests by 10% each quarter
Use peer reviews for key parts of
requirements work products on all
projects
• Reduce cost of rework from
requirements defects by 50%
next year
63. ®2016 All Rights Reserved - PPQC 63 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
• Core Base and Derived Measures
o Requirements Measures
o So{ware Size Measures
o Cost and Effort Measures
o Schedule and Progress Measures
o Defect Measures
• Other Key Derived Measures
o Earned Value
o ProducLvity/Unit Costs
• Summary Measures for a Program or Organiza-on
64. ®2016 All Rights Reserved - PPQC 64 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
• How used:
o requirements baselined and tracked throughout project
o provides insight into growth and stability; also used to status progress
• Useful measures:
o number of requirements (growth over Lme)
o number of new requirements, changed or deleted requirements (stability/
volaLlity)
o requirements status (approved, tested, implemented)
• Typical coun-ng issues:
o how to count a “requirement” consistently, since size and difficulty vary
o stated vs. derived requirements
o traceability/lack of requirements tool use
o in maintenance, more likely to see change requests
67. ®2016 All Rights Reserved - PPQC 67 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
70% 73% 73% 76% 79% 80% 81% 82% 82% 83%
8%
10% 10%
13%
13% 15% 14% 13% 13% 12%
4%
6% 6%
6%
6%
5% 4% 4% 4% 2%
18%
11% 11%
5% 2% 0% 1% 1% 1% 2%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
6/3/2009 6/10/2009 6/17/2009 6/24/2009 7/1/2009 7/8/2009 7/15/2009 7/22/2009 7/29/2009 8/5/2009
Passed Failed Blocked Unverified
68. ®2016 All Rights Reserved - PPQC 68 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
• Size is a cri-cal component of measurement and management
o basis for esLmaLng effort, and quanLfying and managing scope
o basis for producLvity measurement (effort to build a unit/size)
o allows normalizaLon and comparison across systems and projects
(defects/size)
• Size Measures
o # of modules, screens/reports
o # of calls
o # of boards
o # of requirements
70. ®2016 All Rights Reserved - PPQC 70 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
0.1
1
10
100
1000
10000
0.1 1 10 100 1000 10000
Staff Months
Source Lines of Code Thousands
Size – Effort
Es-ma-ng Rela-onship
Project Under
ConsideraLon
Average
-1 Standard
DeviaLon
+1 Standard
DeviaLon
71. ®2016 All Rights Reserved - PPQC 71 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
• Effort is es-mated, staffing planned, and actual effort tracked to plan
o overall project level
o by Lme period, by work unit/task, build, phase
• Variance monitored for consistent over or under-run
o underesLmated by size or effort
o unexpectedly complex or defect-prone
• Cost usually derived from effort (plus other costs)
• Useful measures:
o #of labor hrs/FTEs (by event, Lme period)
• Typical coun-ng issues:
o no Lme reporLng tool; tool not used or enforced
o contractors vs. employees; mgt, admin, others not captured
o lack of granularity of plan and actual effort data
0
5
10
15
20
25
30
Q1 Q2 Q3 Q4
Financial Summary
Plan
Actual
73. ®2016 All Rights Reserved - PPQC 73 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
0
1
2
3
4
5
6
7
8
9
10
1-Aug 1-Sep 1-Oct 1-Nov 1-Dec 1-Jan 1-Feb 1-Mar 1-Apr 1-May 1-Jun 1-Jul 1-Aug 1-Sep 1-Oct 1-Nov 1-Dec 1-Jan 1-Feb 1-Mar 1-Apr 1-May
Effort Alloca-on
Subcontractor Actual Internal Actual
74. ®2016 All Rights Reserved - PPQC 74 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
Ac-vity % Plan Hours
Project Management 12% 9654
ConfiguraLon Management 8% 6436
Quality Assurance 3% 2414
Training 4% 3218
Requirements 9% 7241
Design 12% 9654
Code and Unit Test 20% 16090
IntegraLon Test 16% 12872
User Acceptance Test 9% 7241
Maintenance 5% 4023
Customer Support 2% 1609
100% 80450
76. ®2016 All Rights Reserved - PPQC 76 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
• How used:
o examine planned schedule and milestones and actual aZainment of each
o monitor task compleLon at a level of detail to be able to recover from problems
• Useful measures:
o milestone dates (schedule)
o days late
o work unit progress (requirements, design, code, test, etc.)
• # of units planned to be complete, # completed
o schedule accuracy/performance
• Typical Coun-ng Issues:
o plan data not baselined; no CM control
o work not broken down into manageable work units
o work units of different size, complexity, and importance
77. ®2016 All Rights Reserved - PPQC 77 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
Milestone Original Revised Actual
Proposal Complete 14 January 17 January 20 January
Approval to Proceed 1 03 February 03 February 03 February
Reqmts Signed Off 08 April 15 April 15 April
Approval to Proceed 2 02 May 02 May 05 May
Design Signed Off 10 June 14 July 15 July
System Test Complete 17 July 10 August
Beta Test Complete 03 October 07 November
Approval to Proceed 3 14 October 10 November
Release to Customer 10 November 16 December
80. ®2016 All Rights Reserved - PPQC 80 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
• How used:
o idenLfies the number of defects (problem reports) in the so{ware product
o provides insight into test progress and both pre- and post-producLon quality
o provides insight into rework remaining
• Useful measures:
o defect trends (open, closed, severity, age of open, types)
o defect densiLes (by system, subsystem, module)
o defect origins, detecLon effecLveness, and leakage
• Typical Coun-ng Issues:
o defects versus effort; rework not always 1:1
o definiLon of a defect/classificaLon inconsistencies
o cultural/behavioral issues associated with collecLng
81. ®2016 All Rights Reserved - PPQC 81 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
5 5
14
4
8
18
10 10
4
18
8
4
9 9 10
72
66
74
67
60 60
57 57
48
52
45
40
38 39
37
11
6
14 15
18
13
5
12 11 10 10 11
6
8
0
10
20
30
40
50
60
70
80
Number of Defects
Status of Severity 1 Defects
New Open (Including New) Worked off During the Week
Open Last Week =
Closed This Week =
Downgraded =
New This Week =
Upgraded =
Open This Week =
39
8
4
------
27
10
0
------
37
82. ®2016 All Rights Reserved - PPQC 82 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
1017 1028 1034 1039 1048 1063 1077 1078 1096 1097 1111 1118 1129 1143 1147 1165 1179 1191
71 72
108 124 125
379
454 454 484 509 510 522
605
653
726
776 751 782
0
200
400
600
800
1000
1200
1400
Number of Defects
Severity 2 Defects
Severity 2 DRs Total Discovered Severity 2 DRs Total Closed
83. ®2016 All Rights Reserved - PPQC 83 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
0
10
20
30
40
50
60
70
80
90
<1 1 - 2 3 - 4 5 - 8 9 - 18 >18
Number of Problem Reports
Weeks Open
Problem Report Aging
Open Problem Reports
Data as of 15 June 2009
Average = 5.7 weeks
Target < 8 weeks
84. ®2016 All Rights Reserved - PPQC 84 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
4
9
8
10
6
0
2
4
6
8
10
12
# of Severity 1 Defects
Status of 37 Open Severity 1 Defects
Pending
Valida-on
Defect Under
Analysis
Fix Started –
Work in Progress
Fix Made –
Pending Delivery
Fix Delivered –
Pending Test
85. ®2016 All Rights Reserved - PPQC 85 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
0
5
10
15
20
25
30
35
40
A B C D E F
# of Problem Reports
Module ID
Problem Report Status
Open Severity 1 and 2 by Module
Data as of 15 June 2009
86. ®2016 All Rights Reserved - PPQC 86 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
Module
Size
(KSLOC)
Defects
Defect Density
(Defects/KSLOC)
A 44 48 1.1
B 32 60 1.9
C 36 36 1.0
D 28 33 1.2
E 34 42 1.2
F 15 46 3.1
G 9 30 3.3
Total 198 295 1.5
Data as of 15 June 2009
87. ®2016 All Rights Reserved - PPQC 87 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
0
20
40
60
80
100
120
140
160
180
Reqmts Design Code Unit Test Integ Test Sys Test Del + 90
days
Defects/KSLOC
Life Cycle Phase
Defect Detec-on Profile
All Projects
New Process
88. ®2016 All Rights Reserved - PPQC 88 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
Phase
Discovered
Phase Inserted
Reqmts Analysis Design Code Integ. Test Sys Test
Accept.
Test
Ops
Requirements 50
Analysis 65 65
Design 70 80 65
Code & Unit Test 74 83 85 70
Integra-on Test 76 89 94 88 55
System Test 85 93 95 93 70 50
Acceptance Test 94 95 96 94 75 70 70
1 year Opera-ons 98 97 98 96 88 90 90 76
Columns present cumula=ve % of defects removed
89. ®2016 All Rights Reserved - PPQC 89 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
• Core Base and Derived Measures
o Requirements Measures
o So{ware Size Measures
o Cost and Effort Measures
o Schedule and Progress Measures
o Defect Measures
• Other Key Derived Measures
o Earned Value
o ProducLvity/Unit Costs
• Summary Measures for a Program or Organiza-on
90. ®2016 All Rights Reserved - PPQC 90 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
0
20
40
60
80
100
120
140
160
3 6 9 12 15 18 21
Cost ($K)
Months
ACWP
BCWP
Now
Cost Variance
Actual Cost of
Work
Performed
(ACWP)
Budgeted Cost
of Work
Performed
(BCWP)
Cost Variance (CV) is the difference between actual and budgeted costs (BCWP - ACWP)
CPI = .63
(<1 over budget)
91. ®2016 All Rights Reserved - PPQC 91 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
0
20
40
60
80
100
120
140
160
3 6 9 12 15 18 21
Cost ($K)
Months
BCWS
BCWP
Now
Schedule Varianc
Budgeted Cost
of Work
Scheduled
(BCWS)
Budgeted Cost
of Work
Performed
(BCWP)
Schedule Variance (SV) is the difference between budgeted cost for work performed and
work scheduled (BCWP- BCWS)
SPI = .52
(<1 behind
schedule)
92. ®2016 All Rights Reserved - PPQC 92 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
0
20
40
60
80
100
120
140
160
180
Proposal Actuals to Date Re-Plan 1 (2 builds) Re-Plan 2 (3 builds)
SLOCs/Person Month
Build Produc-vity
Build 1
Build 2
Build 3
15 June 2009
93. ®2016 All Rights Reserved - PPQC 93 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
• Core Base and Derived Measures
o Requirements Measures
o So{ware Size Measures
o Cost and Effort Measures
o Schedule and Progress Measures
o Defect Measures
• Other Key Derived Measures
o Earned Value
o ProducLvity/Unit Costs
• Summary Measures for a Program or an Organiza-on
94. ®2016 All Rights Reserved - PPQC 94 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
Performance to Schedule
milestone baseline forecast actual
xxx 1/03 1/03 1/02
yyy 6/16 6/30
zzz 7/04 8/05
Risks
• Descrip-ons
• Ac-on Plans
Overall Status (Red/Yellow/Green): Project Alpha as of April
0
10
20
30
40
1st Qtr 2nd Qtr 3rd Qtr 4th Qtr
Financial Summary
Plan
Actual
0 200 400 600
1st Qtr
2nd Qtr
3rd Qtr
4th Qtr
Func-onality Delivered
Actual
Plan
95. ®2016 All Rights Reserved - PPQC 95 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
0
5
10
15
20
25
30
35
40
45
Monthly Effort (FTE Hours)
Actual
Projected
Budget
0.72 0.72
0.17
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
Cumula-ve Project Costs ($100K)
Projected
Budget
Actual
Cost Situa-on
Current Forecast
Month Total Month Total
DeviaLon more than 30%
DeviaLon more than 15%
MeeLng budget
Milestone
Envision Start
Vision/Scope
Planning Ends
Devel Complete
V&V Compete
Beta Start
Beta Live
GA
Project Closed
Delay more than 60 days
Delay more than 30 days
On Target or Early
0
5
10
15
20
25
30
35
40
45
Monthly Effort (FTE Hours)
Actual
Projected
Budget
0.72 0.72
0.17
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
Cumula-ve Project Costs ($100K)
Projected
Budget
Actual
Schedule Situa-on
Milestone Curr Last
Envision Start
Vision/Scope
Planning Ends
Devel Complete
V&V Compete
Beta Start
Beta Live
GA
Project Closed
Delay more than 60 days
Delay more than 30 days
On Target or Early
96. ®2016 All Rights Reserved - PPQC 96 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
Size
Cost and
Effort
Schedule Quality
Process
Perfor-
mance
Customer
Sa-sfac-
-on
Risk
Quick
Overview
Suppor-ng
Comments
<Project Name> Monthly Status Report
Technology Effec-veness
Size
Cost and Effort
Schedule
Quality and Process Performance
Risk and Customer Sa-sfac-on
Tech.
Effec-ve-
ness
97. ®2016 All Rights Reserved - PPQC 97 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
Suite of indicators for
each area of reporLng
for review by
project managers,
team members,
and business managers
Total Physical Size
-10
0
10
20
30
40
50
60
08-jun
22-jun
06-jul
20-jul
03-aug
17-aug
31-aug
KLOC
EsLmated Actual Milestone Trend Module Size
0 5 10 15 20
Module A
Module B
Module C
KLOC
Current esLmate Actual
Milestone Tracking
19-may 29-may 08-jun 18-jun 28-jun 08-jul
MST1 P
MST1 A
MST2 P
MST2 A
Milestones actual/planned
Milestone Status
Completed
on Lme
20%
Completed Late
10%
Not yet completed
70%
Cost and Schedule Performance
-40 -35
-30 -25 -20 -15
-10 -5 0
5 10
Apr-08
Jul-08
Oct-08
Jan-09
Apr-09
Jul-09
Oct-09
Jan-10
Apr-10
Jul-10
Oct-10
Percent variance Schedule Performance Variance
Cost Performance Variance
Cri-cal Ra-o
0,40
0,60
0,80
1,00
1,20
1,40
Apr-08
Jul-08
Oct-08
Jan-09
Apr-09
Jul-09
Oct-09
Jan-10
Apr-10
Jul-10
Oct-10
CriLcal RaLo
Threshold (red)
Threshold (yellow)
98. ®2016 All Rights Reserved - PPQC 98 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
• Issues in solware projects generally require use of common
core measures to handle
o requirements growth and stability
o monitoring of product size
o management of cost and effort to budget
o schedule and other progress indicators
o monitoring of defect levels, to see trends
• Derived measures provide insight with combina-ons of base
measures
99. ®2016 All Rights Reserved - PPQC 99 4/18/16
PPQC
PROCESS AND PRODUCT
Q U A L I T Y C O N S U LT I N G
Address: Henry Schneider
President/Senior Principal Consultant
Process and Product Quality Consul-ng, LLC
2009 Simond Avenue, Suite B
Aus-n, TX 78723
Phone: 832-628-2486
E-mail: henry@ppqc.net
Web Site: www.ppqc.net