Pitfalls in Statistical Analysis - Regression to the Mean

1. Healthcare Management
Regression to the Mean
Ninian Peckitt
FRCS FFD RCS FDS RCS FACCS
Oral and Maxillofacial Surgeon / Facial Plastic Surgeon
Adjunct Associate Professor of Engineering Assisted Surgery
Massey University New Zealand

3. Regression to the Mean
His pain got worse, he went to a doctor, and the pain subsided a little.
Therefore, he benefited from the doctor's treatment.
The frequency of accidents on a road fell after a speed camera was installed.
Therefore, the speed camera has improved road safety.

4. Regression to the Mean
Regression toward the Mean
• depends on the way the term is mathematically defined
• if a variable
– is extreme on its first measurement
– it will tend to be closer to the average on a second measurement
• Paradoxical
– if it is extreme on a second measurement
– will tend to have been closer to the average on the first measurement

5. Regression to the Mean
To avoid making wrong inferences,
• regression toward the mean in:
– designing experiments
– interpreting experimental survey,
– and other empirical data
(physical, life, behavioral and social sciences)
• Regression to the Mean is mathematically defined.

6. Regression to the Mean
Regression toward the Mean
…..has also been called
– reversion to the mean
and
– reversion to mediocrity

7. See speaker notes for legend
http://www.socialresearchmethods.net/kb/regrmean.php

8. 100-Item True False variables
• Suppose that all choose randomly on all questions.
• Then, each score would be a realization of
– one of a set of independent and identically distributed random variables,
– with a mean of 50.
• Some score >50 Some score <50 just by chance
• Top scoring 10% and do second test
– on which they again choose randomly
– mean score would again = 50.

9. 100-Item True False variables
• Thus the mean “regresses” all the way back to the mean of the original test.
– No matter what scores on the original test
– the best prediction of the score on the second test is 50
• If there were no luck or random guessing involved with the test questions
– Scoring is the same on the second test as the original test
– there would be no regression toward the mean

10. Realistic Tests
• Most realistic situations fall between these two extremes:
• Test scores as a combination of skill and luck
• Subset scoring above average would be composed of
– those who were skilled and had not especially bad luck
– together with unskilled, but were extremely lucky
On a retest of this subset
– the unskilled unlikely to repeat their luck
– skilled have a second chance to have bad luck
• Hence, those who did well previously are unlikely to do quite as well in the second test.

11. Example – Regression to the Mean
• Students two editions of the same test on two successive days.
• worst performers on the first day will tend to improve their scores on the second day
• best performers on the first day will tend to do worse on the second day.
• Scores are determined in part by
– underlying ability
– in part by chance
• For the first test
– some will be lucky, and score more than their ability
– some will be unlucky and score less than their ability
– Some lucky scores will be lucky again on the second test
– but more scores will be average or below average
– Therefore lucky score (first test) - more likely to have a worse score on the second test
– scores less than the mean on the first test will tend to increase on the second test

12. Francis Galton
Regression towards mediocrity in hereditary stature
• extreme characteristics (e.g., height) in parents - not passed on completely to their offspring
• Characteristics in the offspring regress towards a mediocre point (Mean)
• Galton was able to quantify regression to the mean, and estimate the size of the effect.
• “the average regression of the offspring is a constant fraction of their respective mid-parental deviations”.
• Child / Parents variable difference proportional to its parents' deviation from typical people in the population.
• Parents are each two inches taller than the averages
– Child will be shorter than its parents by some factor
– regression coefficient) times two inches
– Galton estimated this coefficient to be about 2/3:
– height will measure around a mid-point that is two thirds of the parents’ deviation from the population average.

13. Regression to the Mean
Regression towards the Mean
• in all bivariate normal distributions
• any random variation
e.g. height of child vs parents
• if the correlation ≠ 1
• predictions must regress to the mean
• regardless of the underlying mechanisms
• eg. inheritance, race or culture.

14. Importance
• Regression toward the mean is a significant consideration in the design of experiments.
• 1,000 individuals of a similar age scored on risk of a heart attack.
• Statistics could be used to measure the success of an intervention on the 50 who were rated at the
greatest risk.
• The intervention could be a change in diet, exercise, or a drug treatment
• Even if the interventions are worthless
• the test group would be expected to show an improvement on their next physical exam,
– because of regression toward the mean
• The best way to combat this effect is to divide the group randomly into a treatment group that
receives the treatment, and a control group that does not.
• Treatment effective only if the treatment group improves more than the control group.

15. Misunderstandings
Regression toward the Mean - misused
• score >70 on both tests to pass
– Score>70 - strong incentive to study / concentrate
• Score<70 the first time would have no incentive to do well
– might score worse on average the second time.
• Possible movement away from 70
– scores <70 it getting lower
– scores >70 getting higher
• changes between the measurement times may …..
– Augment / Offset / Reverse
……the statistical tendency to regress toward the mean.

16. Caution Errors in Interpretation
Education
• praise for good work
– noticed to do more poorly on the next measure
• Punished for poor work
– were noticed to do better on the next measure.
The educators decided to
• stop praising
• keep punishing on this basis
Error of Concept
• Regression toward the Mean is not based on cause and effect
• Based on random error in a natural distribution around a mean

17. Caution – Errors in Interpretation
• Although individual measurements regress toward the mean
• the second sample - no closer to the mean than the first
• Suppose their tendency is to regress 10% of the way toward the mean of 80,
• Score of 100 the first day is expected to score 98 the second day
• Scored 70 the first day is expected to score 71 the second day
• Those expectations are closer to the mean, on average, than the first day scores
• But the second day scores will vary around their expectations
– some will be higher and some will be lower
– This will make the second set of measurements farther from the mean, on average, than their expectations
– The effect is the exact reverse of regression toward the mean, and exactly offsets it.

18. Caution Errors in Interpretation
So for every individual
• expect 2nd score closer to the mean than 1st score
But for all individuals
• expect average distance from the mean to be the same on both sets of measurements
Regression toward the mean works equally well in both directions
• Highest test score 2nd Day - worse 1st Day.
• Best score 1st Day vs Best Score 2nd Day
• tendency to regress toward the mean going in either direction
• Expect the best scores on both days to be equally far from the mean.

19. Wrong Conclusions
Many phenomena attributed to the wrong causes
when regression to the mean is not taken into account
The Triumph of Mediocrity in Business, Horace Secrist 1933
• mountains of data to prove that:
• Profit Rates of Competitive Businesses tend toward the Average over Time
No such effect:
• the variability of profit rates is almost constant over time
• Description is the Common Regression toward the Mean

20. Wrong Conclusions
• Improvement Scores - Standardized Educational Tests in Massachusetts 1999-2000
• Schools were given improvement goals
• Difference in the average score achieved by students
• Most of the worst-performing schools had met their goals
• Department of Education took as confirmation of the soundness of their policies
• Many “Best Schools” (e.g Brookline High - 18 National Merit Scholarship Finalists)
– were declared to have failed
• Findings - Case of Regression to the Mean

21. Wrong Conclusions
• The psychologist Daniel Kahneman, winner of the 2002 Nobel prize in economics, pointed out that
regression to the mean might explain why rebukes can seem to improve performance, while praise
seems to backfire.[7]
“ I had the most satisfying Eureka experience of my career while attempting to teach flight
instructors that praise is more effective than punishment for promoting skill-learning. When I had
finished my enthusiastic speech, one of the most seasoned instructors in the audience raised his
hand and made his own short speech, which began by conceding that positive reinforcement might
be good for the birds, but went on to deny that it was optimal for flight cadets. He said, “On many
occasions I have praised flight cadets for clean execution of some aerobatic maneuver, and in
general when they try it again, they do worse. On the other hand, I have often screamed at cadets
for bad execution, and in general they do better the next time. So please don’t tell us that
reinforcement works and punishment does not, because the opposite is the case.” This was a
joyous moment, in which I understood an important truth about the world: because we tend to
reward others when they do well and punish them when they do badly, and because there is
regression to the mean, it is part of the human condition that we are statistically punished for
rewarding others and rewarded for punishing them. I immediately arranged a demonstration in
which each participant tossed two coins at a target behind his back, without any feedback. We
measured the distances from the target and could see that those who had done best the first time
had mostly deteriorated on their second try, and vice versa. But I knew that this demonstration
would not undo the effects of lifelong exposure to a perverse contingency.

22. Regression to the Mean - Healthcare
• Regression to the Mean
• widespread statistical phenomenon
• potentially serious implications for Healthcare.
• May result in wrong conclusions
e.g. effect is due to treatment when it is due to chance.
• Ignorance will lead to errors in decision making
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1125994/

23. Regression to the Mean - Healthcare
• The less correlated the two variables…..
……the larger the effect of regression to the Mean
• The more extreme the value from the population mean…
…..the more room there is to Regress to the Mean
• Regression to the Mean occurs when….
– a group is selected with extreme values for one variable
– and another variable is then measured. 1,2
• All Healthcare Professionals need to be aware of Regression to the Mean
– as it has wide ranging effects
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1125994/

24. Regression to the Mean - Healthcare
• Clinicians use diagnostic tests to target and monitor treatment.
• Regression to the Mean can confound this strategy.
• The preliminary test has a high probability of giving an abnormal result through chance
• Initial Treatment may be unnecessary
• Chance effect
– high probability that subsequent measurements will spontaneously regress towards the mean value
• This misleads clinicians and patients into thinking that treatment has been effective when
– the treatment was either not required
– The treatment was ineffective
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1125994/

25. Regression to the Mean - Healthcare
• Regression to the mean affects all aspects of Health Care
• Any intervention aimed at a group or characteristic that is very different from the
average will appear to be successful because of Regression to the Mean
• In clinical practice, the phenomenon can lead to
– misinterpretation of results of tests
– new treatments
– the placebo effect
• Public health interventions are often aimed at sudden increases in disease and
thus vulnerable to the effects of Regression to the Mean
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1125994/

26. Example
See Speaker Notes
Percentage changes in bone mineral density among women treated with alendronate or
raloxifene after 12 and 24 months, showing regression to the mean at 24 months
Women are grouped according to the percentage change at 12 months,
and values for 24 months are percentage change from value at 12 months
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1125994/

27. New Treatments
• Rx New treatments on Patients who are most ill.
– Usually produce a gratifying and sharp response
– Because of Regression to the Mean
• Clinical Trials :
– excluded patients who were resistant to treatment
– treatments outside the licensed conditions are used
– mistaken impression that the new treatment is even better

28. Placebo Effect
• Trials of hormone replacement therapy show a strong placebo effect on menopausal symptoms.6
• This implies that menopausal symptoms are susceptible to placebo treatment.
• Systematic review of placebo vs “open” no treatment - found little evidence for the placebo effect7
• Placebo effect is simply regression to the mean.
• Patients in trials of hormone replacement therapy typically score highly on a symptom index
• Identification of women with relatively extreme menopausal symptoms:
– once treatment starts, improvement will occur in both the placebo and active treatment groups
– because of regression to the mean
– as patients with the worst clinical scores have the biggest placebo effect
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1125994/

29. Public Health
• Public health interventions
• driven by unexpected increases in incidence of disease
• eg response to a sudden rise in traffic incidents
• Because a sudden peak in road crashes is often due to chance
• changes in policy, such as more rigorous enforcement of speed laws
– will reduce crashes because of regression to the mean
• The policy of vaccinating children against meningitis - introduced at a time
of heightened incidence
(see bmj.com)
– The Benefit of a 75%-90% reduction in cases 8 is an overestimate
– Most of the reduction - due to regression to the mean

30. Meningitis
Quarterly figures for laboratory confirmed cases of meningitis C in England and Wales
• Mass vaccination occurred at a time of increased incidence
• So the benefits of immunisation seemed greater

31. Decision Making in Healthcare
“If public health physicians wish to prove their worth, our advice is that
they focus their efforts on a group of problems that are much worse than
the national average or have shown an unexpected increase as there will
usually be an improvement. It is important, however, to focus on a group
of outliers to guarantee an effect because there is a chance that
regression to the mean will not affect the results of a single outlier as it is
a group phenomenon”.
Effect of regression to the mean on decision making in health care
Veronica Morton, research fellow1 and David J Torgerson, director1
BMJ. 2003 May 17; 326(7398): 1083–1084.

32. Healthcare Management
• Regression to the mean can justify league table initiatives for improving poorly performing
hospitals.
• When poor hospitals are helped by allocating them more resources, regression to the mean will
ensure that most will suddenly climb the league table.
• In contrast, hospitals at the top of the league table who are rewarded with increased resources for
their efforts will fall in the table.
• If governments want to justify any initiative, it is better to target those at the bottom of the league
than those at the top.
• For the individual hospital manager, the problem is more complex.
• Those who manage the worst hospitals are likely to see an improvement and thus enhance their
careers
• However, because regression to the mean is a group phenomenon, the improvement is not certain,
and some hospitals will move in the opposite direction.

33. Clinical Audit
• An audit might identify patients that were
operated on by a particular surgical team and
had unexpectedly poor results, such as
increased postoperative infections.
• Implementation of a policy of aggressive
procedures to control infection will again
often seem to work because of regression to
the mean

34. Formula
Percent of Regression to the Mean Formula
Estimation of the percent of regression to the mean
in any given situation
Prm = 100(1 – r)
Prm = the percent of regression to the mean
r = the correlation between the two measures

35. Formula
Consider the following four cases:
if r = 1, there is 0% regression to the mean
– the two variables are perfectly correlated
– and there is no regression to the mean.

36. Formula
Consider the following four cases:
if r = 0.5, there is 50% regression to the mean
the sampled group moves 50% of the distance
from the no-regression point to the mean of the
population

37. Formula
Consider the following four cases:
if r = 0.2, there is 80% regression to the mean
– If the correlation is a small 0.20
– the sample will regress 80% of the distance

38. Formula
Consider the following four cases:
if r = 0, there is 100% regression to the mean
– if there is no correlation between the measures
– the sample will "regress" all the way back to the
population mean

39. Formula
• if r = 0, there is 100% regression to the mean
i.e. zero correlation
• knowing a score on one measure
• gives no information of the likely score on a 2nd measure
• Best guess for performance on 2nd measure…..
• …..will be the mean of that 2nd measure

40. Calculation requirements:
• the mean of the sample on the first measure
• the population mean on both measures
• the correlation between measures
See Speaker Notes
Estmation and Correction of
Regression to the Mean

41. Estmation and Correction of
Regression to the Mean
Assume gain of 15 points (average)
Gain across entire distribution
If r=1 (no Regression to Mean)
• Post Test 30+15 = 45
Correlation .5 = 50% Regression
• Post Test 50% of Means 45 – 65
• Post Test Average of 55
• Pseudo effect of 10 points
See Lecutre Notes

42. Solutions ?
• Understanding the phenomenon
• Whenever possible, Policy should be based on evidence from trials
• The effectiveness of management league tables could be tested by
randomising poorly performing hospitals to new management or
extra resources
• This would tell us which intervention was most effective
• In clinical practice, sequential testing to get an average value, which
most doctors would do for blood pressure, is a solution for some
tests.

43. NICE
NICE Guidelines for Wisdom Teeth Removal
• Political Motivation
• Postive Bias
• Financially Motivated
• Premature Conclusions
• RCTs have not been undertaken !
• RCT Data will take 10 years to collect !
See Speaker Notes

44. 0%
10%
20%
30%
40%
50%
60%
Pain Pericoronitis Caries 8 Caries 7 P/A Path Perio Root
Resorbtion
Cyst Tumour
53%
59%
7%
42%
5% 5% 5%
11%
2%
Wisdom Teeth Removal - Reasons
Pain
Pericoronitis
Caries 8
Caries 7
P/A Path
Perio
Root Resorbtion
Cyst
Tumour

45. Wisdom Teeth - NICE
Increased Risk of Caries
• Van der Linden 1995 (Removal 8’s)
• Caries 8’s (7.1%)
• Caries Adj Molars (42.7%)
• Risk with 8/7 Contact
• “Best Practice vs Negligence?

46. Regression to the Mean
• It is a statistical phenomenon
• It is a group phenomenon
• It happens between any two variables
• It is a relative phenomenon
• You can have regression up or down
• The more extreme the sample group, the greater the regression to the mean
• The less correlated the two variables, the greater the regression to the mean.

47. Thank you for your Attention