1. Developing Meaningful Metrics:
Driving Action for Results!
Presented to
ASQ St. Petersburg-Tampa-Section 1508
08 February 2010
by
T. M. Kubiak, President
Performance Improvement Solutions
Conquering the Challenge!™ 1 Copyright T. M. Kubiak 1996-2010
Performance Improvement Solutions
2. If you don’t have confidence in
the diagnosis, you won’t have
confidence in the prescription.
— Steven Covey
Conquering the Challenge!™ 2 Copyright T. M. Kubiak 1996-2010
Performance Improvement Solutions
4. Can any of you help me with
metrics? My boss wants a
whole new set by the end of
the week and I don’t even
know where to start! HELP!
Conquering the Challenge!™ 4 Copyright T. M. Kubiak 1996-2010
Performance Improvement Solutions
5. Overview
The Driving Force Characteristics of Measures
Definitions Development Considerations
Why Measure? Principles of Measurement
Barriers to Measurement Where Should You Measure?
Pitfalls to Avoid Goals, Targets, & Benchmarks
What Measures Measure Evolution of a Measure
The “E’s” of Measurement Examples of Measures
Different Views of Measures Some Final Words...
Conquering the Challenge!™ 5 Copyright T. M. Kubiak 1996-2010
Performance Improvement Solutions
6. A Few Definitions, Of Course!
Measure (Metric)
Refers to numerical information that quantify
input, output, and performance of processes,
products, and services
Indicator
Measurement that relates to performance, but
is not a direct or exclusive measure of such
performance
Measure that is a predicator of some more
significant performance
Measure, Metrics, and Indicators! Oh my!
Conquering the Challenge!™ 6 Copyright T. M. Kubiak 1996-2010
Performance Improvement Solutions
7. Why Measure?
If you cannot measure it, you cannot control it.
If you cannot control it, you cannot manage it.
If you cannot manage it, you cannot improve it.
What gets measured and rewarded is what gets
done. Corollary: If the measurements don’t
change, neither do the results.
Measurements allow us to track where we have
been, where we are, and where we are going
Shows how effectively we use our resources
Measurement: The Roadmap to Progress
Conquering the Challenge!™ 7 Copyright T. M. Kubiak 1996-2010
Performance Improvement Solutions
8. More Reasons for Measuring
Focuses attention on factors contributing to
achieving the organization’s mission
Assists in setting goals and monitoring
trends and progress
Provides input for analyzing root cause and
sources of errors
Identifies opportunities for on-going
improvement
Provides a means of knowing whether we’re
winning or losing
Measurement: The Roadmap to Progress
Conquering the Challenge!™ 8 Copyright T. M. Kubiak 1996-2010
Performance Improvement Solutions
9. The Driving Force
Integrated System of Metrics
Customer Operational
Requirements Processes
Lean Six Sigma drives
improvement at the individual
process level.
Corporate Business Business Support
Objectives Strategy Requirements Processes
Corporate Are these processes capable of Supplier
Strategy Processes
meeting business requirements?
Driving Customer Satisfaction and Business Success!
Conquering the Challenge!™ Copyright T. M. Kubiak 1996-2010
Performance Improvement Solutions
10. Barriers to Measurement
Thinking some jobs can’t be measured
Thinking there isn’t enough time to measure
The process of accomplishing a task is not
understood
Fear that the measurement will become a club
Conquering the Challenge!™ 10 Copyright T. M. Kubiak 1996-2010
Performance Improvement Solutions
11. Pitfalls to Avoid!
Using the measure as a carrot or a stick
Reason: the goal becomes to manipulate the number
Conquering the Challenge!™ 11 Copyright T. M. Kubiak 1996-2010
Performance Improvement Solutions
12. Pitfalls to Avoid!
Measuring everything that is possibly helpful
Reason: dilutes the effort
Conquering the Challenge!™ 12 Copyright T. M. Kubiak 1996-2010
Performance Improvement Solutions
13. Pitfalls to Avoid!
Measuring parameters where the organization excels
Reason: reduces the effort to a publicity campaign
Conquering the Challenge!™ 13 Copyright T. M. Kubiak 1996-2010
Performance Improvement Solutions
14. Pitfalls to Avoid!
Requiring precision in the data beyond the
requirements of the decision
Reason: exasperates all involved
Conquering the Challenge!™ 14 Copyright T. M. Kubiak 1996-2010
Performance Improvement Solutions
15. What Measures Measure
Quality (acceptable, good, in-spec, etc.)
Quantity (Number of...)
Time (Cycle time, rates)
Cost
Conquering the Challenge!™ 15 Copyright T. M. Kubiak 1996-2010
Performance Improvement Solutions
16. The “E’s” of Measurement!
Extent (volume-based or deployment)
How much?
Efficiency (productivity-based, usually
expressed as a rate)
How fast?
Effectiveness (customer-based or
objective-based)
How well?
If It Were Only That Easy!
Conquering the Challenge!™ 16 Copyright T. M. Kubiak 1996-2010
Performance Improvement Solutions
17. Different Views of Measures
Organizational level
Operational measures (invisible to the customer)
Predictive measures (in line of sight of the customer)
Process Level
In-process measures (evaluate the method that
creates results)
W IPM W IPM W IPM W EOPM
End-of-process measures (measure the results of a
method)
W IPM W IPM W IPM W EOPM
Conquering the Challenge!™ 17 Copyright T. M. Kubiak 1996-2010
Performance Improvement Solutions
18. Characteristics of Measures
Actionable (So What?) Relevant
Easily measurable Specific
Economical to collect Timely
Focused on processes Traceable
Objective Understandable
If It Ain’t Actionable...
Conquering the Challenge!™ 18 Copyright T. M. Kubiak 1996-2010
Performance Improvement Solutions
19. Development Considerations
How often will the measure be computed and
posted?
What about data stratification/aggregation?
Where will the data come from?
Is the data source reliable?
Can/should the data be normalized?
How will the measure be presented?
Who will see them?
Critical Questions Require Thoughtful Answers
Conquering the Challenge!™ 19 Copyright T. M. Kubiak 1996-2010
Performance Improvement Solutions
20. Operational Definition
Answers: Who, How, What, Where, When
Defines: formulas and terms
Defines: interpretation (e.g., up is good)
Example:
What is a “month?”
Establishes a common language and understanding
Conquering the Challenge!™ 20 Copyright T. M. Kubiak 1996-2010
Performance Improvement Solutions
21. Cash for Clunkers Example Source: WSJ 09-08-05
Issue: 78 cars bumped from the list
Threshold: 18 mpg
Reason: More precise data (4 decimals) caused the
revisions (Note: 0.0001 miles is ½ drop of fuel based on
18 mpg)
Measurement Methods:
Older: Dynamometer, fuel consumption measured,
measured to 4 decimals, numbers rounded
Newer: Tailpipe emissions, measured to 4 decimals
Fix: Old data updated to be “effectively equivalent”
Quote: “Repeatability and accuracy is something we
spend a lot of time on.” EPA scientist
Hey, Joe! Gotta light?
Conquering the Challenge!™ 21 Copyright T. M. Kubiak 1996-2010
Performance Improvement Solutions
22. Cancer Deaths Example Source: WSJ 09-02-09
Does this chart support
the conclusion?
Normalized
•Gaps in age
stratification
•Aggregated across
Stratified by age gender
•Aggregated across
cancer type
Conquering the Challenge!™ 22 Copyright T. M. Kubiak 1996-2010
Performance Improvement Solutions
24. Air Travel Example Source: WSJ 1--01-07
What is your understanding of an “on-time arrival?”
How does plus or minus 15 minutes of the scheduled
arrival time sound?
Is this a good metric or a bad metric?
How many customers are aware of it?
Related metrics (by carrier):
Percentage of flights arriving on-time
Percentage of canceled flights
Mishandled bag reports per 1,000 passengers
Complaint reports per 1,000,000 passengers
Bumped passengers per 10,000
Do key players understand the metrics?
Conquering the Challenge!™ 24 Copyright T. M. Kubiak 1996-2010
Performance Improvement Solutions
25. Principles of Measurement
Measurements must be specific
Measure the outputs of highest value to
the customer
Measures can be applied to all performance
dimensions - external as well as internal
Measure the process as well as the results
Understand the game before you decide
how you’ll keep score
There is no single perfect measure
Extent Effectiveness
Efficiency
Conquering the Challenge!™ 25 Copyright T. M. Kubiak 1996-2010
Performance Improvement Solutions
26. Where Should You Measure?
Early in the process to promote prevention
At the point where a cause-and-effect
relationship can be established
At critical processes
Functional boundaries
Points of convergence
Points of divergence
Conquering the Challenge!™ 26 Copyright T. M. Kubiak 1996-2010
Performance Improvement Solutions
27. Goals, Targets, & Benchmarks
Conquering the Challenge!™ 27 Copyright T. M. Kubiak 1996-2010
Performance Improvement Solutions
28. If you can’t win the race, make
sure the guy ahead of you
breaks the record.
— Steve Prefontaine (1951-
1975), American runner
Conquering the Challenge!™ 28 Copyright T. M. Kubiak 1996-2010
Performance Improvement Solutions
29. Evolution of a Measure
Percent of acceptable computer Effectiveness
reports delivered on time
Number of acceptable computer
reports delivered on time
Efficiency
Number of computer reports
delivered on time
Number of computer reports
delivered Extent
Conquering the Challenge!™ 29 Copyright T. M. Kubiak 1996-2010
Performance Improvement Solutions
30. Examples of Measures
Measure: Trouble calls received/week
The value of this measure is unclear. It is not
customer-focused nor does it measure the
ability of an organization to deal with incoming
trouble calls since no processing of trouble
calls is evident in the metric. Essentially, this
metric measures nothing more than the
volume of incoming trouble calls. The fact that
it is expressed in the form of a rate is
misleading and may fool one into thinking it is
a measure of efficiency.
Type of Measure: Extent
Conquering the Challenge!™ 30 Copyright T. M. Kubiak 1996-2010
Performance Improvement Solutions
31. Examples of Measures
Measure: Job Scheduled
This metric focuses on measuring nothing
more than the volume of jobs scheduled. Other
than that, it provides little insight into the
underlying process of “scheduling jobs.” The
metric is not actionable.
Type of Measure: Extent
Conquering the Challenge!™ 31 Copyright T. M. Kubiak 1996-2010
Performance Improvement Solutions
32. Examples of Measures
Measure: Actual versus planned sales
Measures the planning/forecasting process.
Type of Measure: Effectiveness
Conquering the Challenge!™ 32 Copyright T. M. Kubiak 1996-2010
Performance Improvement Solutions
33. Examples of Measures
Measure: Sales/employee
This metric is highly misleading and often
used for organization to organization
comparisons. It falsely assumes that all
employees generate sales and does not
consider the product being sold or the
associated supporting infrastructure required
to produce and sell the product.
Type of Measure: Efficiency
Conquering the Challenge!™ 33 Copyright T. M. Kubiak 1996-2010
Performance Improvement Solutions
34. Examples of Measures
Measure: Number of training hours/employee
This metric measures the deployment of
training hours to the workforce.
Type of Measure: Extent
Conquering the Challenge!™ 34 Copyright T. M. Kubiak 1996-2010
Performance Improvement Solutions
35. Examples of Measures
Measure: Revenue/sales training hour
This metric measures the impact of training the
sales workforce so long as a clear cause-and-
effect relationship can be established between
training and revenue.
Type of Measure: Effectiveness
Conquering the Challenge!™ 35 Copyright T. M. Kubiak 1996-2010
Performance Improvement Solutions
36. Examples of Measures
Measure: Engineering change orders/drawing
This metric measure the design process from
the point of view of the drawings. However,
note that this metric does not normalize or
adjust for the complexity of a given drawing.
Type of Measure: Effectiveness
Conquering the Challenge!™ 36 Copyright T. M. Kubiak 1996-2010
Performance Improvement Solutions
37. Some Final Words...
Measure, Control, Manage, Improve
Extent, Efficiency, Effectiveness
Performance measurement is
irrelevant without a frame of
reference
Measure twice! But cut once!
Conquering the Challenge!™ 37 Copyright T. M. Kubiak 1996-2010
Performance Improvement Solutions
38. People without information are
unable to change; those with
information are compelled to
change.
Conquering the Challenge!™ 38 Copyright T. M. Kubiak 1996-2010
Performance Improvement Solutions