The Hidden Bias in Customer Metrics

THE HIDDEN BIAS IN CUSTOMER
METRICS
BOB E. HAYES, PHD
BUSINESS OVER BROADWAY

Copyright © 2014 Business Over Broadway · Bob E. Hayes, PhD · bob@businessoverbroadway.com · www.businessoverbroadway.com
Purpose
 Help understand how people interpret two different
metrics:
 Mean Score
 Net Score
 While these two metrics are equivalent (as one goes
up, so does the other), do people see them
differently?

If you know one metric, you know the others
 These metrics are
highly correlated with
each other
 They tell you the same
thing – relatively
speaking
 If you have an NPS of -
30, you know your
mean score will be
about 6.2.
 If you have an NPS of
40, you know your
promoters will be
about 57% and
detractors will be
about 17%.
Net
Promoter
Score®
Mean
% 9-10
(Promoters)
% 0-6
(Detractors)
% 7-8
(Passives)
% 6 or
greater
-100 4.1 -2.7 97.3 5.5 24.9
-90 4.4 -0.4 89.6 10.8 31.0
-80 4.7 2.2 82.2 15.6 36.9
-70 4.9 5.1 75.1 19.8 42.4
-60 5.3 8.3 68.3 23.4 47.8
-50 5.6 11.8 61.8 26.4 52.9
-40 5.9 15.6 55.6 28.8 57.7
-30 6.2 19.7 49.7 30.6 62.3
-20 6.5 24.1 44.1 31.8 66.7
-10 6.8 28.8 38.8 32.4 70.8
0 7.1 33.8 33.8 32.5 74.7
10 7.4 39.0 29.0 31.9 78.4
20 7.7 44.6 24.6 30.8 81.8
30 7.9 50.5 20.5 29.0 84.9
40 8.3 56.7 16.7 26.7 87.9
50 8.6 63.1 13.1 23.8 90.6
60 8.9 69.9 9.9 20.3 93.0
70 9.2 76.9 6.9 16.2 95.3
80 9.5 84.3 4.3 11.5 97.2
90 9.9 91.9 1.9 6.2 98.9
100 10.1 99.9 -0.1 0.3 100.5
Recommend question is on a 0 to 10 scale where 0 = Not at all likely and 10 = Extremely likely; sample to calculate other metrics from NPS
consists of three independent studies on consumer attitudes toward PC manufacturers and wireless service providers.

Summary Metrics Paint Similar Picture
 Each of the 48 data points in the
graph represents a single
company’s summary metric:
mean of recommend and net
score of recommend (NPS)
 Two summary metrics (NPS and
Mean) are strongly correlated (r =
.97).
 Looking at regression equation,
we know comparable Mean and
NPS values: NPS of 0.0
corresponds to a Mean of 7.0.
 NPS of… Equals
Mean of…
y = 0.03x + 7.09
R² = 0.94
0
1
2
3
4
5
6
7
8
9
10
-100 -80 -60 -40 -20 0 20 40 60 80 100
Recommend-MeanScore
Recommend - Net Score (NPS)
Data are from three independent studies about consumer attitudes toward their:
1. PC manufacturer: Survey of 1058 US consumers from 2007. GMI (Global Market Insite, Inc.)
2. Wireless service provider: Survey of 994 US consumers from 2007. GMI (Global Market Insite, Inc.)
3. Wireless service provider: Survey of 5686 global consumers from 2010. Mob4Hire
N = 48 brands, most brands represent different wireless service providers (N = 41) and a handful of
PC manufacturers. Metrics were calculated for brands with 30 or more responses.
-100 4.0
-50 5.5
0 7.0
50 8.5
100 10.0

Study Design
 Participants invited via blog post about the study; post
shared through social media connections, professional
online communities and email list.
 Blog post included hyperlink to web-based data
collection instrument
 CEM professionals were given 5 NPS values (-100, -50, 0, 50 and 100) and 5
comparable Mean values (4.0, 5.5, 7.0, 8.5 and 10. For each metric value, they were
asked to make their best guess about the size of the following customer segments:
 % of respondents who have a rating of 9 or 10 (Promoters / Top Box Score)
 % of respondents who have a rating between 0 and 6, inclusive (Detractors /
Bottom Box Score)
 % of respondents who have a rating of 7 or 8 (Passives / Middle Box Score)
 % of respondents who have a rating of 6 or greater (Satisfied / Top Box Score)

Sample Demographics
B2B
and
B2C
42%
B2C
3%
B2B
55%
Individual
contributor
of CEM
Program
3%
Manager of
CEM
Program
28%
Director of
CEM
Program
9%
Senior
Executive
of CEM
Program
13%
Outside
consultant
22%
Other
25%
Beginner
10%
Average
13%
Proficient
45%
Expert
32%
What is your level of expertise in
your company's CEM Program?What is your current role?
What Best Describes
Your Company?
 Forty-one CX professionals participated in the study.

Sample: Expertise in Metric
Beginner
24%
Average
30%
Proficient
21%
Expert
24%
Beginner
21%
Average
37%
Proficient
17%
Expert
24%
Level of expertise in using NPS
in their CEM program
Level of expertise in using
Mean Scores in their CEM
program
 Wide range of expertise seen for both NPS and Mean
Scores

Estimating % of Promoters From NPS and Mean Values
 People are more accurate at estimating the size of Promoters (i.e., Top Box Score) using the
NPS (except for slight understimation for NPS = 100.
 When given a high Mean Score, people greatly underestimate the percent of Promoters.
Estimated percent of promoters: CEM professionals’ best guess as to the percent of respondents who have
ratings of 9 or 10 (Promoters)
0
10
20
30
40
50
60
70
80
90
100
4.0 5.5 7.0 8.5 10.0
PercentofCustomerswith
aratingof9or10(Promoters)
Mean Value
Actual % Promoters
Estimated % Promoters
Actual percent of promoters: The actual percent of respondents who have a rating of 9 or 10 (Promoters)
Confidence Interval: Sample statistic ± margin of error at a 95% confidence level for estimated percent of Promoters
 CEM professionals were given different NPS and Mean values and
asked to estimate the % of customers who are Promoters.
For estimation using Mean Values, N = 26 to 30.
For estimation using NPS Values, N = 33 to 41.
0
10
20
30
40
50
60
70
80
90
100
-100 -50 0 50 100
aratingof9or10(Promoters)
NPS Value
Actual % Promoters
Estimated % Promoters

Estimating % of Detractors From NPS and Mean Values
 When given a low summary metric, people greatly underestimate the percent of Detractors –
more so for Mean values than for NPS values.
 When given a high summary value, people slightly overestimate the percent of Detractors
Estimated percent of Detractors: CEM professionals’ best guess as to the percent of respondents who have
ratings of 0 - 6 (Detractors)
Actual percent of Detractors: The actual percent of respondents who have a rating of 0 - 6 (Detractors)
Confidence Interval: Sample statistic ± margin of error at a 95% confidence level for estimated percent of Detractors
asked to estimate the % of customers who are Detractors.
-10
0
10
20
30
40
50
60
70
80
90
100
4.0 5.5 7.0 8.5 10.0
aratingof0or6(Detractors)
Mean Value
Actual % Detractors
Estimated % Detractors
-10
0
10
20
30
40
50
60
70
80
90
100
-100.0 -50.0 0.0 50.0 100.0
aratingof0to6(Detractors)
NPS Value
Actual % Detractors
Estimated % Detractors

Estimating % of Passives From NPS and Mean Values
 When given a low summary metric, people greatly underestimate the percent of Detractors –
more so for Mean values than for NPS values.
 When given a high summary value, people slightly overestimate the percent of Detractors
Estimated percent of Passives: CEM professionals’ best guess as to the percent of respondents who have
ratings of 7 - 8 (Passives)
Actual percent of Passives: The actual percent of respondents who have a rating of 7 - 8 (Passives)
Confidence Interval: Sample statistic ± margin of error at a 95% confidence level for estimated percent of Passives
asked to estimate the % of customers who are Passives.
-10
0
10
20
30
40
50
60
70
80
90
100
-100.0 -50.0 0.0 50.0 100.0
aratingof7or8(Passives)
NPS Value
Actual % Passives
Estimated % Passives
-10
0
10
20
30
40
50
60
70
80
90
100
4.0 5.5 7.0 8.5 10.0
aratingof7or8(Passives)
Mean Value
Actual % Passives
Estimated % Passives

 Greater downward bias in predictions when NPS is used.
Estimating % of Positives (6-10) from NPS and Mean Values
 For each NPS value, CEM professionals underestimated the percent of Positive respondents.
 When given a mean, CEM professionals underestimated the percent of Positive respondents
when mean was 5.5 or greater; overestimated percent of Positive respondents when Mean = 4.
Estimated percent of positives: CEM professionals’ best guess as to the percent of respondents who have
ratings of 6 - 10 (Positives)
Actual percent of positives: The actual percent of respondents who have a rating of 6 – 10 (Positives)
Confidence Interval: Sample statistic ± margin of error at a 95% confidence level for estimated percent of Positives
asked to estimate the % of customers who are Positives (6 – 10 rating).
0
10
20
30
40
50
60
70
80
90
100
-100 -50 0 50 100
PercentofRespondentswitha
ratingof6orgreater
NPS Value
Actual % 6 or greater
Estimated % 6 or greater
0
10
20
30
40
50
60
70
80
90
100
4.0 5.5 7.0 8.5 10.0
PercentofRespondentswitha
ratingof6orgreater
Mean Value
Actual % 6 or greater
Estimated % 6 or greater
 Greater downward bias in predictions when NPS is used.

Study Findings
 Some bias in customer metrics
 CX professionals underestimate the size of important customer
segments when only looking at Mean or Net Promoter Scores
 CX professionals are better at estimating the size of
NPS-specific segments based on the Net Promoter
Scores than Mean Scores
 CX professionals underestimate the size of Satisfied
segment based on summary metric on both Mean or
Net Promoter Scores (slightly worse for NPS)

Implications
 Operations
 Correct interpretation of results improves how leaders run their
organizations
 Present both aggregated metric (NPS or Mean) and
segment-specific metric (Top/Bottom Box Score)
 Minimizes misinterpretation of data
 Human Resources
 Data Education / Training on basics of data analysis
 Data literacy is essential to Big Data Success
 Visualization/Analytic Software
 Pick vendors who allow you to summarize your data in different ways

For More Information
Bob E. Hayes, Ph.D.
Email: bob@businessoverbroadway.com
Web: www.businessoverbroadway.com
Blog: www.businessoverbroadway.com/blog
Twitter: www.twitter.com/bobehayes

The Hidden Bias in Customer Metrics

Recommended

Recommended

More Related Content

Similar to The Hidden Bias in Customer Metrics

Similar to The Hidden Bias in Customer Metrics (20)

More from Business Over Broadway

More from Business Over Broadway (16)

Recently uploaded

Recently uploaded (20)

The Hidden Bias in Customer Metrics