Metrics & KPIs. Tips To Setup Your Measurement Initiative Right.

www.plays-in-business.com
Metrics & KPIs.
Tips To Setup Your Measurement Initiative Right.
Picture Credit: newsong, Pixabay.com, https://pixabay.com/en/measuring-tape-inches-sewing-953422/
Michael Tarnowski, Plays-In-Business.com

Metric Patterns & Anti-Patterns
Pattern Anti-Pattern
• Define the information needs and goals of
your measurement inintiative first.
• Do not ask for goals or purpose of the
improvements. – All metrics are arbitrary.
• Create a shared understanding of all
stakeholders involved.
• In any event, upper management has their
own views – which are mutually completely
incompatible.
• Derive from the needs and goals
measureable parameters which may provide
relevant data.
• Throw in your metrics right at the beginning.
– Goals and purposes are irrelevant (see 1).
• Metrics come last! – More important is
shared understanding of what and how to
measure.
• Metrics matter. – Do not ask for relevant
reasonable arguments.
• For each measure define upper lower
thresholds and corrective actions.
• Do not define any follow-up action for a
metric status change. – To develop any
potentially helpful action will exceed all your
organizational intellectual capacities.
• Metrics and KPIs are indicators only for
positive or trends in an organization. Ist up to
management to follow the trend or steer
against in a reaonable and controlled way.
• Nobody cares about reasonable follow-up
actions for a metric status change. –
Management is satisfied by immediate and
spontaneous reactions only. – The more the
best, since it shows business agility to react
fast as its best.

• Take the simplest as possible metric. –
Remember, managers must understand
them unambigously. ☺
• Do not thrive for simplicity. – Metrics
stemming from rocket science attest you
being an egg head.
• Present reports and visuals in an easy to
undesrtanding way.
The readers should imidiately be able to
chosse the corrective action needed; not to
ask for help in interpreting the data.
• Only the dashboards must be appealing,
nothing else. – Nobody questions the data if
they are presented in a colorful way (see 1.).
• Management is not by colors or charts;
management is by insights
• Data and information relevance are totally
overestimated (see 1.).
• Don‘t use one measure to satisfy multiple
needs. – Needs are specific, so data must be
specific.
• Satisfy the finance guys only. – KPIs are
always only about money and costs only.
• It depends… ☺ • KPIs about scheduling and timing are
neglectable. – The customer is always willing
to negotiate.
• It depends… ☺ • Totally ignore the KPIs about quality. –
Forget these pale QA guys sitting in the
cellar, in a dark room. They only whine at the
end when nothing can't be rescued anymore.
The finance guys instead, are more
important since they whine right from the
beginning.

• Use less metrics as possible. – Collecting
data takes ist time, reporting takes ist time,
understanding the reports takes ist time…
• Define as much KPIs as possible. – All
people preaching KPI modesty are wimps.
The more your dashboard is cluttered the
best and you will be the Hero.
• Measuring and collecting data are additional
work coming on top. Make the workflow
seamless as possible and automate most as
possible.
• Make the data collecting process for your
employees as cumbersome as possible. –
Your employees will love you when they
waste half of their work time in addition to
only measure useless data nobody cares
about.
• Make the workflow seamless as possible and
automate most as possible.
Each superflous hand-overs might introduce
potentisl errors.
• Introduce as many incompatible KPI tracking
tools as possible. – Your employees will be
happy being waked up from their office naps
and be asked to synchronize tediously
inconsistent values coming from
incompatible data sources

■ “A Key Performance Indicator should immediately
inform the reader how the business is performing
which in turn should suggest what actions need
to be taken.”– http://unilytics.com
■ “A Key Performance Indicator is a calculated metric
showing progress of an established tactic.” – http://unilytics.com
■ “A KPI is something that can be counted and
compared; it provides evidence of the degree
to which an objective is being attained over a
specified time.” – http://intrafocus
■ “A Key Performance Indicator (KPI) is a
measurement with a given purpose.”
■ “Key Performance Indicators (KPIs) are a set of
quantifiable measures that a company or industry
uses to gauge or compare performance in terms
of meeting their strategic and operational goals.”
https://pixabay.com/en/measurement-millimeter-centimeter-1476919/
Definitions of “Key Performance Indicators”

Measurement, Metrics, and Key Performance Indicators
A measurement is a quantitative
observation of one of the following:
■ something relevant to the
decision you have to make;
■ information you have to report
regarding the progress (of
development);
■ effects of progress
improvement.
A metric is a recurring measurement that has informational, diagnostic,
motivational, or predictive power of some kind..
https://pixabay.com/en/diet-calorie-counter-weight-loss-695723/

A Key Performance
Indicator (KPI) is a
measurement with a given
purpose.
David Nicolette 2015: Software Development Metrics.
A Key Performance Indicator should immediately inform the reader how the
business is performing which in turn should suggest what actions need to be taken.
Measurement, Metrics, and Key Performance Indicators
https://pixabay.com/en/diet-calorie-counter-weight-loss-695723/

Key Performance Indicators
■ Key Performance Indicators (KPIs)
are a set of quantifiable measures that
a company or industry uses to gauge
or compare performance in terms of
meeting their strategic and operational
goals.
■ From the business perspective KPIs
have to be directly linked to overall
company goals.
■ From the technical perspective –
data collection – they are
measurements to define and track
specific business goals and objectives.
Graphic adapted from http://unilytics.com/kpi-analysis/
Goal
Measures
Metrics
KPI

Logical Structure Of Decision Making
• Understanding what to measure
begins by appreciating the
logical structure of decision
making.
• It starts by knowing the concepts
and inter-relationship of goals,
strategies, critical success
factors, tactics and KPIs (key
performance indicators).
flickr: https://www.flickr.com/photos/webel/136757257/
A, B, or C ?

https://pixabay.com/de/planung-planen-anpassung-620299/
• A Goal is “what” you want to
accomplish. It is a broad desired
outcome rather than how you will get
there.
• A Strategy is “how” you plan to
achieve the goal. It is a long-term plan
of actions.
• A Critical Success Factor (CSF) is
vital to achieve the strategy. It is
always a measurable amount set over
a defined period of time.
• A Tactic is a short-term action, tasks,
taken to achieve a critical success
factor. For every critical success
factor, there are a number of tactics.

A Key Performance Indicator is a calculated metric showing
progress of an established tactic.

Goal Strategy CSF Tactic KPI
http://unilytics.com/key-performance-indicators-a-measure-of-success/
(Critical Success Factor)

Setting Up Key Performance Indicators
1. Set a Goal
First, the problem needs to be stated.
Goals are only meaningful if they are
well defined and a clear plan of action
is outlined to achieve them.
Goal
KPI

2. Identify Critical Success
Factors to achieve the Goal
Critical Success Factors represent key
performance areas (things that must be done) that
are vital for an organization to accomplish its stated
goals.
Without such factors, an organization has no
roadmap by which to navigate their operations.
Stating CSFs allows management to stay on course
instead of being distracted by daily non-strategic
problems.
The purpose of the CSF is to make sure we are on
track.
As long as we meet the CSF, we should achieve the
stated goal; it’s not guaranteed, but likely.
Goal
KPI

3. Set CSF Targets
Specific targets for Critical Success
Factors (CSFs) need to be achieved
over a set period of time and measured to
make sure the organization is on track.
This informs if the organization is
executing successfully, but because they
are summaries of many activities, they
don’t provide precise insight or direction
into the Actions required to improve the
results.
Goal
Strategy
Critical Success
Factors
Tactic
Measures
Metrics
KPI

4. Identify Specific Actions
Very specific Actions need to be
identified to achieve the CSF of
creating all these new units and in the
required timeframe.
That means that if we are missing the
targets we set for the actions,
everyone knows what needs to be
done differently.
Likewise, we can then identify
activities that are particularly
successful in driving results.
Goal
Strategy
Critical Success
Factors
Tactic
Measures
Metrics
KPI

5. Measure and Track Actions
using Key Performance
Indicators
KPIs are calculated metrics of the Actions
we are taking and monitoring them allows
us to make corrections. The plan is that if
we meet or exceed our KPIs, we will meet
or exceed the CSF and ultimately our goal.
These Actions are considered to be critical
to the success of your organization. KPIs
are important because they directly track the
work being done.
Goal
Strategy
Critical Success
Factors
Tactic
Measures
Metrics
KPI

Key Performance Indicator Types 1/3
Process KPI – measure the
efficiency or productivity of
a business process;
e.g. Cycle Time Of Coding,
Shipping Days Of Order
Input KPI – measure assets
and resources invested in
or used to create business
result; e.g. Project Budget
Output KPI – measure financial an non-financial results of
business; e.g. Revenue, Nmbr Of New Customers Acquired.
https://pixabay.com/en/balance-swing-equality-measurement-2108022/

■ Leading KPI – measure
activities having significant
effect on future performance.
■ Lagging KPI – measure
impact of an event in the past
(mostly failure).
■ Outcome KPI – measure
overall result or impact of
business activity; past
oriented.

■ Qualitative KPI – measure a
descriptive characteristic, an
opinion, a property or a trait:
e.g. Employee Satisfaction
through surveys which gives
a qualitative report.
■ Quantitative KPI – a measurable
characteristic, resulted by
counting, adding, or averaging
numbers.
E.g. Units per man-hour.

Building Blocks Of KPIs
■ Goal – Broad desired outcome. It states “what” we
want to accomplish but not “how.” It states where
you are going rather than how you will get there.
■ Critical Success Factor – Something that is
vital to successfully achieve the Goal.
■ Phase – Stage or state we are addressing. For
SW-Development, it’s one of the phases:
Requirements Elicitation, Design, Coding,
Component and System Testing.
■ Segment – Targeted audience or function being
focused on. For SW-Development, it can be based
on existing customers, plant / production, field, etc..
■ Approach – Short-term plan taken to achieve a
critical success factor. It’s what you do, and for every
critical success factor, there are a number of approaches.
■ Action – Precise activity or business process required to be performed to attain stated goal.
■ Key Performance Indicator – Calculated value that is critical to the business and to attaining
stated goal. Should be expressed as Ratio, Average, Percentage or Rate.
■ Target – Value assigned to Metric or KPI.
http://unilytics.com/not-the-top-10-kpis/
https://pixabay.com/en/target-goal-success-dart-board-1955257/

Specific
Measurable
Assignable
Realistic
Time-related
SMART Target Setting
https://pixabay.com/get/e831b3082ffc073ed1534705fb094491e777e5d104b0154893f9c27caeeab1/mark
eting-online-1427786_1920.jpg

Must have
Should have
Could have
Won’t have
Guidelines to Prioritize Goals, KPIs, …
https://pixabay.com/get/e834b8062bf7013ed1534705fb094491e777e5d104b0154893f9c27ca0e9b2/russia-1199330_1920.jpg

Guidance For KPIs – The 3A
If you are forced to define KPIs and Metrics the 3 A’s are
a guidance:
■ Actionable Metrics – actionable metrics tell the
story of what action needs to be taken to meet your goals.
■ Accessible Metrics – this means metrics that
make sense. A smart metric needs to be accessible to
pretty much everyone: precise enough for the
technical team, visual enough for marketing, and
grounded in real-world behavior for any member of
your team to understand.
■ Auditable Metrics – Auditable metrics mean that ideally any member of your
team could find the source data and reproduce your report.
This means simple report generation.
https://pixabay.com/de/zentimeter-ausr%C3%BCstung-zoll-15656/

KPIs – Monitoring Specific Activities
■ Things like increasing revenue or profit,
or improving customer service are not
performance indicators – they are not
specific enough to tell you what to
change if the numbers are not met.
■ They are instead Results Indicators.
■ They show how we are doing due to
many activities.
KPIs should monitor only very specific activities,
so when the values change you know what to do about it.
https://pixabay.com/de/entscheidung-hilfe-antwort-1548097/

Select KPIs Based On Your Goals
■ There are potentially thousands of
KPIs and various nuances of each
leaving you with an unlimited number.
■ KPIs that work for one organization
may not work for yours. The factors
impacting their performance may or
may not apply in your situation.
■ A well-known and established
approach to define KPIs is the Goal-
Question-Metric method by Basili and
Weiss.
http://unilytics.com/not-the-top-10-kpis/
https://pixabay.com/en/mindmap-brainstorm-idea-innovation-2123973/

How To Define KPIs – Goal Question Metric 1/3
Goal-Question-Metric-Approach (GQM) has been
proposed 1984 by Basili and Weiss in NASA projects.
GQM is based on two assumptions:
• Measurement programs should be “goal-
based” not “metrics-based”, and
• The way both goals and metrics are defined
should be customized to an individual
organization’s needs.

• Goals – They define what the organization
wants to improve
• Questions – They refine each goal to a
more quantifiable way.
• Metrics – They indicate the measurements
required to answer each question.

Image based on V. Basili, G. Caldiera, D. Rombach: “The Goal Question Metric
Paradigm“, 1994
Conceptual Level
Measurement goals involve products,
processes, and/or resources
Quantitative Level
Associated with every question is a set of
data – either subjective or objective – that
helps to provide a quantitative answer.
Operational Level
Question try to characterize the object of
measurement in the context of a qualified
issue from a particular viewpoint
Goal 1 Goal 2
Q1 Q2 Q3 Q4 Q5
M1 M2 M3 M4 M5 M6
Definition
AnalysisandInterpretation

Abuse Metrics – Vanity Metrics

Vanity metric make you feel good, but
they don’t offer clear guidance for what
to do.
-- Eric Ries, The Lean Startup

■ Vanity metrics – data that
are easily manipulated, are
biased toward the short-term,
often paint a rosy picture of
program success, or do not
help campaigners make wise
strategic decisions.
■ Vanity metrics are things that
look great but say nothing about
business results. They are
measurements and calculations
that are designed to be impressive
as opposed to actionable or relevant
to core business goals such as revenue and operating margins.
Example:
Nmbr of Tests performed – can easily be manipulated by running “similar” tests.

Vanity vs. Actionable: What’s the difference?
Vanity metrics: Numbers or
stats that look good on paper, but
don’t really mean anything
important.
vs
Actionable metrics: Stats that
tie to specific and repeatable
tasks you can improve and to the
goals of your business
https://pixabay.com/en/action-art-backlit-doors-indoors-1854117/
Eric Ries 2010: Entrepreneurs: Beware of Vanity Metrics, Harvard Business Review, February 08, 2010,
https://hbr.org/2010/02/entrepreneurs-beware-of-vanity-metrics

Misuse of KPIs 1/4
■ Driving incentives, not learning. “… the
primary function of KPIs is to promote
organizational learning, a mission that will
certainly be undermined when indicators
are used for incentives” – Kevin Kaiser,
INSEAD,
http://www.iedp.com/articles/destroying-value-
and-the-misuse-of-indicators/
■ Missing the forest for the trees. One
common misuse of a KPI is when the KPI,
which is the means, becomes more
important than the end. "The KPI is the
thermometer. It is not the temperature".
KPI is just an indicator of a performance,
not THE performance.
■ Maslow's Hammer. Maslow said “if the only tool you have is a hammer, you tend to treat
everything as if it were a nail".
There's an over-reliance on KPIs as the tool to fix almost everything.
https://www.linkedin.com/pulse/7-most-common-misuses-key-performance-indicators-kpis-lama-a-makarem
https://pixabay.com/en/fear-domination-bully-victim-981384/

Misuse of KPIs 2/4
■ A Sea of metrics. Do we really need so many
KPIs to measure the same results? Having too
many KPIs defeats the meaning of the word
"Key" in KPI and just adds confusion and wastes
time. – “More is less”
■ Over-complication. Do I really need a
complicated KPI to tell me how some things are
obviously doing? Tools are produced to help
finding solutions rather than creating problems.
■ Mixing up the "Cause" with the "Effect". Is a
KPI a cause or an effect? Is a KPI a question or
an answer? KPIs are just the start, they’re not
the end. You can't stop once you find them. You
only start there.

Misuse of KPIs 3/4
■ Not starting with the end in mind. What results
are we really interested in measuring? Are we
measuring the right thing? And if so, are we
using the right KPI?
■ Forgetting that KPIs have owners. Behind
every KPI, there is a team driving it. The people
are more important...take care of them first so
they can take care of their KPIs. KPIs are means
to identify potential improvements and should
result in 'performance improvement' initiatives,
rather than a tool to beat up or penalize
someone.

Misuse of KPIs 4/4
■ Pushing employees too hard towards working for
KPIs (KPI itself becomes a target)
■ Linking incentives to the KPIs and not validating
them later
■ Inventing KPIs on the top and mandating them to
the bottom, skipping the discussion phase
■ Setting the target value for an indicator, but not
discussing an action plan
■ Focusing on the destination, but forgetting about
quality/value aspect. https://pixabay.com/en/fear-domination-bully-victim-981384/

Measuring Can Be Hazardous
In using metrics, be aware of the following potential pitfalls:
■ The KPI inflation – using to many KPIs.
■ The „Pick-a-KPI“ trap – adopting KPIs without
considering context.
…and be aware, the usage using of metrics is additionally
guided by three very important laws:
“When a measure becomes a target, it ceases to be a
good measure.”
– Goodhart's Law
“Measures tend to be corrupted / gamed when used
for target-setting.”
– Campbell's Law
“Monitoring a metric may subtly influence people to
maximize that measure.”
– Hawthorne Effect aka The Observer Effect
https://pixabay.com/de/fehler-mouse-trap-t%C3%B6richt-verletzt-2344150/

Mind The KPI Inflation
■ You should not begin with all the metrics you
might need. – Start with only one or two.
■ Drop a metric when it outlives its usefulness.
■ Don't accumulate a huge set of metrics, lest
you frustrate your teams.
■ Consider an expiration date for each metric
on which to evaluate whether you will
continue to use it.
Define less KPIs as possible
https://pixabay.com/en/measure-scale-double-m-accuracy-629659/

Mind The „Pick-a-KPI“ Trap
■ Don’t fall into the trap of picking your
KPIs because they are the ones used
by others or on someone’s Top 10 KPI
list.
■ KPIs need to be consistent with your
particular needs and always driven by
your unique business goals.
■ Picking from a “list” assumes your
activities are the same as another
company’s for a given goal.
■ They will always be different. https://pixabay.com/en/checklist-clipboard-questionnaire-1622517/
Don’t use KPIs without considering their context

Goodhart's Law
“When a measure
becomes a target,
it ceases to be a
good measure."
- Charles Goodhart, British economist
“Goodhart’s law […] states that when a feature of the economy is picked as an indicator of the economy, then it inexorably
ceases to function as that indicator because people start to game it.”
Mario Biagioli, (12 July 2016). "Watch out for cheats in citation game". Nature. 535 (7611): 201. doi:10.1038/435201a.
PMID 27411599.

Campbell's Law
Once a metric has
been identified as a
primary indicator for
success, its ability to
accurately measure
success tends to be
compromised.
- Donald T. Campbell, American social scientist

Hawthorne Effect
Individuals modify
their behavior in
response to their
awareness of being
observed.
The term was coined in 1958 by Henry A. Landsberger, when analyzing earlier experiments from 1924–32 at the
Hawthorne plant (a Western Electric factory outside Chicago).

Further Reading
■ David Nicolette 2015: Software Development Metrics. Manning Publ., 2015.
■ Christopher W. H. Davis 2015: Agile Metrics in Action. Manning Publ., 2015.
■ Daniel S. Vacanti 2015: Actionable Agile Metrics for Predictability. An Introduction. Leanpub.
■ Encyclopedia of Software Development Life Cycle Metrics,
https://sites.google.com/site/sdlcmetrics/,sdlcmetrics.org, http://www.sdlcmetrics.org/
■ KPIlibrary.com/, http://kpilibrary.com/
■ Victor Basili, David Weiss 1984. A Methodology for Collecting Valid Software Engineering Data.
IEEE Journal Transactions on Software Engineering, Vol. SE-10, No6, Nov. 1984.
https://www.cs.umd.edu/~basili/publications/journals/J23.pdf
■ Victor Basili, Gianluigi Caldiera, Dieter Rombach 1994: Goal Question Metric Paradigm, John Wiley,
1994. https://www.cs.umd.edu/~basili/publications/technical/T89.pdf
■ Rini van Solingen, Ergon Berghout 1999: The Goal Question Metric - A Practical Guide For Quality
Improvement Of Software Development, McGraw Hill, 1999.
https://courses.cs.ut.ee/MTAT.03.243/2015_spring/uploads/Main/GQM_book.pdf
■ Eliyahu Goldratt 1984: The Goal: A Process of Ongoing Improvement. North River Publ., 2014.
■ Eric Ries 2010: Entrepreneurs: Beware of Vanity Metrics, Harvard Business Review, February 08,
2010, https://hbr.org/2010/02/entrepreneurs-beware-of-vanity-metrics
■ Alexander Novkov 2017: Actionable Analytics for Lean Project Management. infoQ Jun 3 2017.
https://www.infoq.com/articles/actionable-analytics-lean

Questions? – Comments?
Twitter: @M_Tarnowski, @PlaysInBusiness
Facebook: http://bit.ly/PiB-FB
LinkedIn: http://bit.ly/MT-LinkdIn
Xing: http://bit.ly/MT-Xing
SlideShare: http://bit.ly/MT-SShare
This document may be further distributed free-of-charge in its original,
complete form only. Please credit Plays-in-Business.com.
Or call me:
+49-172-6915261 • +49-611-2056820
(cell phone/mobile) (regular phone)
Drop me a note:
info@plays-in-business.com

Metrics & KPIs. Tips To Setup Your Measurement Initiative Right.

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

En vedette

En vedette (8)

Similaire à Metrics & KPIs. Tips To Setup Your Measurement Initiative Right.

Similaire à Metrics & KPIs. Tips To Setup Your Measurement Initiative Right. (20)

Plus de Michael Tarnowski

Plus de Michael Tarnowski (20)

Dernier

Dernier (20)

Metrics & KPIs. Tips To Setup Your Measurement Initiative Right.