A recent newsletter in my ongoing conversation about the perils of using small sample sizes when evaluating individual performance

1. Getting Quality Right from Career Impact, Inc. Page 1 of 3
Greetings!
We're now getting to the heart of the matter concerning the perils of small sample
sizes. This is Chapter 48 in a series of ongoing newsletters about Getting Quality
Right in the call center.
For the past five chapters, we've been focusing on how we can meaningfully determine
the quality performance of a rep, based on a small sample of that person's
performance. It isn't easy. And yet it's vitally important. But...
Nearly every call center in our entire industry does this wrong!
Call center directors everywhere are making faulty judgments about individuals'
performance as a result. We've got to do this differently, even at the cost of learning
something about statistics to do it.
In Chapter 47, I discussed how you can compare any one call with the center's
average using standardized scores known as Z-Scores. But, as I'll demonstrate here,
that method does not translate into evaluating the performance of a rep, based on a
small sample of that reps' calls.
Click here to access our Archives, so you can read from the beginning of this topic,
which began with Chapter 43 of these newsletters.
A Few More Rolls of the Dice
Last week, in our dice analogy, I rolled, first a "5" and then a
"2." And I showed how, using standardized scores, you can
confidently say that the first roll is in the 21st percentile and
the second roll in the 2nd percentile.
But what does that tell us about the underlying distribution of a
larger handful of rolls? As I hinted, "Not much."
So, I'll keep rolling... three more times. Look! I got an 8,
another 8, and a 9. I'll stop at five rolls, because that's about
how many calls per rep many call centers observe on a monthly
basis.
Look at this small sample of 5 rolls. Recall that I rolled a 2,5,8,8,9. What does this tell
me about the average, as compared to the average of the 250 rolls I illustrated in
Chapter 46?
This is the trap that it is so easy to fall into.... I may be tempted to average those five
rolls and put faith in that result. I just did that and it comes to 6.4. But wait!
Be warned! When we make use of statistics such as "averages", "z-scores," and
"standard deviations", we must remember that their meaningfulness is based upon an
http://archive.constantcontact.com/fs063/1101722231059/archive/1102857008708.html 12/8/2009

2. Getting Quality Right from Career Impact, Inc. Page 2 of 3
important assumption. The assumption is that the underlying distribution of the
sample is normally distributed. But, we CAN NOT assume this to be so.
(With dice we actually can make this assumption, because we "know" its underlying
distribution. But we do not know this about reps' calls. People aren't as predictable as
dice.)
We do not know, really, whether the 8,8, and 9 are more representative of this rep
and that, perhaps, the 2 and 5 are just flukes, or whether it is the other way around.
The only way to know with a high degree of certainty is to take a large enough
sample that we can be pretty sure of the normalcy of the underlying distribution. And
that, by the way, means you'll need to observe a lot more calls per rep than
is customary in our industry.
So, here's a less precise, but statistically sound alternative too many more resources.
This method gives you a range within which this person's average performance falls,
rather than one specific number for that average.
T-Test of Independent Means
You can, in fact, make a meaningful, though imprecise judgement about an individual's
performance based on a small sample of observed calls. You do it with a statistic
known as a T-Test, Specifically, a one-sample T-test of independent means.
If you don't want to learn the logic behind the math of all this, just think of a T-test as
a way of applying a penalty of imprecision to the average you are looking for. The
smaller the sample size, the bigger the penalty you pay.
When I apply this statistic to my small sample of 5 rolls of the dice, compared with my
big sample of 250 rolls, here's what I find out.
I find that I can be 95% confident that the "real" average of this small sample of 5
rolls of the dice is somewhere between 2.8 and 10.0. Wow! that's quite a spread.
Would you promote, fire, reward, or chastise someone based on results that can be
pinpointed no more narrowly than this? I hope you would never do so knowingly.
And I hope that this series of conversations has alerted you to the need to stop doing
it until and unless you deal with the inherent imprecision that is built-in when you take
small samples of a person's performance.
Beware of Generalizing from a Small Sample
Beware of generalizing from a sample this small. Remember, 5 rolls of the dice only
give us a 95% confidence interval that the "true" mean is between 2.8 and 10.0
The good news is that even slightly larger samples give you noticeably more precise
output. Here's what I got when I rolled my dice a total of 10 times, then 30, then 50.
Starting with my 5,2,8,8, and 9, I next rolled a 6,6,6,7,and 10.This gave me an
average over 10 rolls of 6.7 with a 95% confidence interval that the true mean is
somewhere between 5.1 and 8.3.
After 30 rolls, I had an average of 6.2 and a 95% confidence interval that the true
mean is somewhere between 5.3 and 7.1.
http://archive.constantcontact.com/fs063/1101722231059/archive/1102857008708.html 12/8/2009

3. Getting Quality Right from Career Impact, Inc. Page 3 of 3
After 50 rolls, I had an average of 6.8 and a 95% confidence interval that the true
mean is somewhere between 6.1 and 7.4.
You can see from this accummulation of data that, with each larger sample size, we
are getting a more and more precise bracketing the "true" average score.
How precise do you want to be? How precise do you need to be? How precise can you
afford to be? These are questions about resource allocation. They are not statistical
matters; they are managerial ones. And they are the ones we will begin to answer
next.
Next time, I'll provide an example from an actual call center instead of dice, and after
that I'll propose several ways of going about approaching the certainty you need to
make meaningful management decisions without exorbitant use of resources.
Thanks for reading,
Cliff Hurst
Career Impact, Inc.
Tel: 207-251-0301 Internationally: +001-207-251-0301
Toll-free: 1-800-813-8105
e-mail: cliff@careerimpact.net
Email Marketing by
http://archive.constantcontact.com/fs063/1101722231059/archive/1102857008708.html 12/8/2009