Healthcare Management

Regression to the Mean

Ninian PeckittFRCS FFD RCS FDS RCS FACCS

Oral and Maxillofacial Surgeon / Facial Plastic SurgeonAdjunct Associate Professor of Engineering Assisted Surgery

Massey University New Zealand

Regression to the Mean

His pain got worse, he went to a doctor, and the pain subsided a little.

Therefore, he benefited from the doctor's treatment.

The frequency of accidents on a road fell after a speed camera was installed.

Therefore, the speed camera has improved road safety.

Regression to the MeanRegression toward the Mean

• depends on the way the term is mathematically defined

• if a variable

– is extreme on its first measurement– it will tend to be closer to the average on a second measurement

• Paradoxical

– if it is extreme on a second measurement– will tend to have been closer to the average on the first measurement

Regression to the MeanTo avoid making wrong inferences,

• regression toward the mean in:

– designing experiments– interpreting experimental survey, – and other empirical data

(physical, life, behavioral and social sciences)

• Regression to the Mean is mathematically defined.

Regression to the Mean

Regression toward the Mean

…..has also been called

– reversion to the mean

and

– reversion to mediocrity

See speaker notes for legend

http://www.socialresearchmethods.net/kb/regrmean.php

100-Item True False variables• Suppose that all choose randomly on all questions.

• Then, each score would be a realization of

– one of a set of independent and identically distributed random variables, – with a mean of 50.

• Some score >50 Some score <50 just by chance

• Top scoring 10% and do second test

– on which they again choose randomly

– mean score would again = 50.

100-Item True False variables

• Thus the mean “regresses” all the way back to the mean of the original test.

– No matter what scores on the original test

– the best prediction of the score on the second test is 50

• If there were no luck or random guessing involved with the test questions

– Scoring is the same on the second test as the original test

– there would be no regression toward the mean

Realistic Tests• Most realistic situations fall between these two extremes:

• Test scores as a combination of skill and luck

• Subset scoring above average would be composed of

– those who were skilled and had not especially bad luck

– together with unskilled, but were extremely lucky

On a retest of this subset

– the unskilled unlikely to repeat their luck

– skilled have a second chance to have bad luck

• Hence, those who did well previously are unlikely to do quite as well in the second test.

Example – Regression to the Mean• Students two editions of the same test on two successive days.

• worst performers on the first day will tend to improve their scores on the second day

• best performers on the first day will tend to do worse on the second day.

• Scores are determined in part by

– underlying ability

– in part by chance

• For the first test

– some will be lucky, and score more than their ability– some will be unlucky and score less than their ability– Some lucky scores will be lucky again on the second test– but more scores will be average or below average– Therefore lucky score (first test) - more likely to have a worse score on the second test– scores less than the mean on the first test will tend to increase on the second test

Francis GaltonRegression towards mediocrity in hereditary stature

• extreme characteristics (e.g., height) in parents - not passed on completely to their offspring

• Characteristics in the offspring regress towards a mediocre point (Mean)

• Galton was able to quantify regression to the mean, and estimate the size of the effect.

• “the average regression of the offspring is a constant fraction of their respective mid-parental deviations”.

• Child / Parents variable difference proportional to its parents' deviation from typical people in the population.

• Parents are each two inches taller than the averages

– Child will be shorter than its parents by some factor

– regression coefficient) times two inches

– Galton estimated this coefficient to be about 2/3:

– height will measure around a mid-point that is two thirds of the parents’ deviation from the population average.

Regression to the MeanRegression towards the Mean

• in all bivariate normal distributions

• any random variation

e.g. height of child vs parents

• if the correlation ≠ 1

• predictions must regress to the mean

• regardless of the underlying mechanisms

• eg. inheritance, race or culture.

Importance• Regression toward the mean is a significant consideration in the design of experiments.

• 1,000 individuals of a similar age scored on risk of a heart attack.

• Statistics could be used to measure the success of an intervention on the 50 who were rated at the greatest risk.

• The intervention could be a change in diet, exercise, or a drug treatment

• Even if the interventions are worthless

• the test group would be expected to show an improvement on their next physical exam,

– because of regression toward the mean

• The best way to combat this effect is to divide the group randomly into a treatment group that receives the treatment, and a control group that does not.

• Treatment effective only if the treatment group improves more than the control group.

MisunderstandingsRegression toward the Mean - misused• score >70 on both tests to pass

– Score>70 - strong incentive to study / concentrate

• Score<70 the first time would have no incentive to do well

– might score worse on average the second time.

• Possible movement away from 70

– scores <70 it getting lower

– scores >70 getting higher

• changes between the measurement times may …..

– Augment / Offset / Reverse

……the statistical tendency to regress toward the mean.

Caution Errors in InterpretationEducation

• praise for good work – noticed to do more poorly on the next measure

• Punished for poor work – were noticed to do better on the next measure.

The educators decided to

• stop praising

• keep punishing on this basis

Error of Concept

• Regression toward the Mean is not based on cause and effect

• Based on random error in a natural distribution around a mean

Caution – Errors in Interpretation• Although individual measurements regress toward the mean

• the second sample - no closer to the mean than the first

• Suppose their tendency is to regress 10% of the way toward the mean of 80,

• Score of 100 the first day is expected to score 98 the second day

• Scored 70 the first day is expected to score 71 the second day

• Those expectations are closer to the mean, on average, than the first day scores

• But the second day scores will vary around their expectations

– some will be higher and some will be lower

– This will make the second set of measurements farther from the mean, on average, than their expectations

– The effect is the exact reverse of regression toward the mean, and exactly offsets it.

Caution Errors in InterpretationSo for every individual

• expect 2nd score closer to the mean than 1st score

But for all individuals

• expect average distance from the mean to be the same on both sets of measurements

Regression toward the mean works equally well in both directions

• Highest test score 2nd Day - worse 1st Day.

• Best score 1st Day vs Best Score 2nd Day

• tendency to regress toward the mean going in either direction

• Expect the best scores on both days to be equally far from the mean.

Wrong ConclusionsMany phenomena attributed to the wrong causes

when regression to the mean is not taken into account

The Triumph of Mediocrity in Business, Horace Secrist 1933

• mountains of data to prove that:

• Profit Rates of Competitive Businesses tend toward the Average over Time

No such effect:

• the variability of profit rates is almost constant over time

• Description is the Common Regression toward the Mean

Wrong Conclusions• Improvement Scores - Standardized Educational Tests in Massachusetts 1999-2000

• Schools were given improvement goals

• Difference in the average score achieved by students

• Most of the worst-performing schools had met their goals

• Department of Education took as confirmation of the soundness of their policies

• Many “Best Schools” (e.g Brookline High - 18 National Merit Scholarship Finalists)

– were declared to have failed

• Findings - Case of Regression to the Mean

Wrong Conclusions• The psychologist Daniel Kahneman, winner of the 2002 Nobel prize in economics, pointed out that

regression to the mean might explain why rebukes can seem to improve performance, while praise seems to backfire.[7]

“ I had the most satisfying Eureka experience of my career while attempting to teach flight instructors that praise is more effective than punishment for promoting skill-learning. When I had finished my enthusiastic speech, one of the most seasoned instructors in the audience raised his hand and made his own short speech, which began by conceding that positive reinforcement might be good for the birds, but went on to deny that it was optimal for flight cadets. He said, “On many occasions I have praised flight cadets for clean execution of some aerobatic maneuver, and in general when they try it again, they do worse. On the other hand, I have often screamed at cadets for bad execution, and in general they do better the next time. So please don’t tell us that reinforcement works and punishment does not, because the opposite is the case.” This was a joyous moment, in which I understood an important truth about the world: because we tend to reward others when they do well and punish them when they do badly, and because there is regression to the mean, it is part of the human condition that we are statistically punished for rewarding others and rewarded for punishing them. I immediately arranged a demonstration in which each participant tossed two coins at a target behind his back, without any feedback. We measured the distances from the target and could see that those who had done best the first time had mostly deteriorated on their second try, and vice versa. But I knew that this demonstration would not undo the effects of lifelong exposure to a perverse contingency.

Regression to the Mean - Healthcare

• Regression to the Mean

• widespread statistical phenomenon

• potentially serious implications for Healthcare.

• May result in wrong conclusions

e.g. effect is due to treatment when it is due to chance.

• Ignorance will lead to errors in decision making

http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1125994/

Regression to the Mean - Healthcare• The less correlated the two variables…..

……the larger the effect of regression to the Mean

• The more extreme the value from the population mean…

…..the more room there is to Regress to the Mean

• Regression to the Mean occurs when….

– a group is selected with extreme values for one variable

– and another variable is then measured. 1,2

• All Healthcare Professionals need to be aware of Regression to the Mean

– as it has wide ranging effects

http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1125994/

Regression to the Mean - Healthcare• Clinicians use diagnostic tests to target and monitor treatment.

• Regression to the Mean can confound this strategy.

• The preliminary test has a high probability of giving an abnormal result through chance

• Initial Treatment may be unnecessary

• Chance effect

– high probability that subsequent measurements will spontaneously regress towards the mean value

• This misleads clinicians and patients into thinking that treatment has been effective when

– the treatment was either not required

– The treatment was ineffective

http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1125994/

Regression to the Mean - Healthcare

• Regression to the mean affects all aspects of Health Care

• Any intervention aimed at a group or characteristic that is very different from the average will appear to be successful because of Regression to the Mean

• In clinical practice, the phenomenon can lead to

– misinterpretation of results of tests– new treatments– the placebo effect

• Public health interventions are often aimed at sudden increases in disease and thus vulnerable to the effects of Regression to the Mean

http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1125994/

Example

See Speaker Notes

Percentage changes in bone mineral density among women treated with alendronate or raloxifene after 12 and 24 months, showing regression to the mean at 24 months

Women are grouped according to the percentage change at 12 months, and values for 24 months are percentage change from value at 12 months

http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1125994/

New Treatments• Rx New treatments on Patients who are most ill.

– Usually produce a gratifying and sharp response

– Because of Regression to the Mean

• Clinical Trials :

– excluded patients who were resistant to treatment

– treatments outside the licensed conditions are used

– mistaken impression that the new treatment is even better

Placebo Effect• Trials of hormone replacement therapy show a strong placebo effect on menopausal symptoms. 6

• This implies that menopausal symptoms are susceptible to placebo treatment.

• Systematic review of placebo vs “open” no treatment - found little evidence for the placebo effect7

• Placebo effect is simply regression to the mean.

• Patients in trials of hormone replacement therapy typically score highly on a symptom index

• Identification of women with relatively extreme menopausal symptoms:

– once treatment starts, improvement will occur in both the placebo and active treatment groups

– because of regression to the mean

– as patients with the worst clinical scores have the biggest placebo effect

http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1125994/

Public Health• Public health interventions• driven by unexpected increases in incidence of disease• eg response to a sudden rise in traffic incidents• Because a sudden peak in road crashes is often due to chance• changes in policy, such as more rigorous enforcement of speed laws

– will reduce crashes because of regression to the mean

• The policy of vaccinating children against meningitis - introduced at a time of heightened incidence(see bmj.com)

– The Benefit of a 75%-90% reduction in cases 8 is an overestimate

– Most of the reduction - due to regression to the mean

Meningitis

Quarterly figures for laboratory confirmed cases of meningitis C in England and Wales

• Mass vaccination occurred at a time of increased incidence

• So the benefits of immunisation seemed greater

Decision Making in Healthcare

“If public health physicians wish to prove their worth, our advice is that they focus their efforts on a group of problems that are much worse than the national average or have shown an unexpected increase as there will usually be an improvement. It is important, however, to focus on a group of outliers to guarantee an effect because there is a chance that regression to the mean will not affect the results of a single outlier as it is a group phenomenon”.

Effect of regression to the mean on decision making in health careVeronica Morton, research fellow1 and David J Torgerson, director1

BMJ. 2003 May 17; 326(7398): 1083–1084.

Healthcare Management• Regression to the mean can justify league table initiatives for improving poorly performing hospitals.

• When poor hospitals are helped by allocating them more resources, regression to the mean will ensure that most will suddenly climb the league table.

• In contrast, hospitals at the top of the league table who are rewarded with increased resources for their efforts will fall in the table.

• If governments want to justify any initiative, it is better to target those at the bottom of the league than those at the top.

• For the individual hospital manager, the problem is more complex.

• Those who manage the worst hospitals are likely to see an improvement and thus enhance their careers

• However, because regression to the mean is a group phenomenon, the improvement is not certain, and some hospitals will move in the opposite direction.

Clinical Audit

• An audit might identify patients that were operated on by a particular surgical team and had unexpectedly poor results, such as increased postoperative infections.

• Implementation of a policy of aggressive procedures to control infection will again often seem to work because of regression to the mean

Formula

Percent of Regression to the Mean Formula

Estimation of the percent of regression to the mean in any given situation

Prm = 100(1 – r)

Prm = the percent of regression to the mean

r = the correlation between the two measures

Formula

Consider the following four cases:

if r = 1, there is 0% regression to the mean

– the two variables are perfectly correlated

– and there is no regression to the mean.

Formula

Consider the following four cases:

if r = 0.5, there is 50% regression to the mean

the sampled group moves 50% of the distance from the no-regression point to the mean of the population

Formula

Consider the following four cases:

if r = 0.2, there is 80% regression to the mean

– If the correlation is a small 0.20

– the sample will regress 80% of the distance

Formula

Consider the following four cases:

if r = 0, there is 100% regression to the mean

– if there is no correlation between the measures

– the sample will "regress" all the way back to the population mean

Formula

• if r = 0, there is 100% regression to the mean i.e. zero correlation

• knowing a score on one measure

• gives no information of the likely score on a 2nd measure

• Best guess for performance on 2nd measure…..

• …..will be the mean of that 2nd measure

Calculation requirements:

• the mean of the sample on the first measure

• the population mean on both measures

• the correlation between measures

See Speaker Notes

Estmation and Correction of Regression to the Mean

Estmation and Correction of Regression to the Mean

Assume gain of 15 points (average)

Gain across entire distribution

If r=1 (no Regression to Mean)

• Post Test 30+15 = 45

Correlation .5 = 50% Regression

• Post Test 50% of Means 45 – 65• Post Test Average of 55• Pseudo effect of 10 points

See Lecutre Notes

Solutions ?• Understanding the phenomenon

• Whenever possible, Policy should be based on evidence from trials

• The effectiveness of management league tables could be tested by randomising poorly performing hospitals to new management or extra resources

• This would tell us which intervention was most effective

• In clinical practice, sequential testing to get an average value, which most doctors would do for blood pressure, is a solution for some tests.

NICE

NICE Guidelines for Wisdom Teeth Removal

• Political Motivation

• Postive Bias

• Financially Motivated

• Premature Conclusions

• RCTs have not been undertaken !

• RCT Data will take 10 years to collect !

Clinical Excellence?

See Speaker Notes

Pain Pericoronitis Caries 8 Caries 7 P/A Path Perio Root Resorbtion Cyst Tumour0%

10%

20%

30%

40%

50%

60%

53%

59%

7%

42%

5% 5% 5%

11%

2%

Wisdom Teeth Removal - Reasons

%

Wisdom Teeth - NICEIncreased Risk of Caries

• Van der Linden 1995 (Removal 8’s)

• Caries 8’s (7.1%)

• Caries Adj Molars (42.7%)

• Risk with 8/7 Contact

• “Best Practice vs Negligence?

Regression to the Mean

• It is a statistical phenomenon

• It is a group phenomenon

• It happens between any two variables

• It is a relative phenomenon

• You can have regression up or down

• The more extreme the sample group, the greater the regression to the mean

• The less correlated the two variables, the greater the regression to the mean.

Thank you for your Attention