planning, performing, and publishing research with confidence limits a tutorial lecture given at the...
TRANSCRIPT
Planning, Performing, and Publishing Research with Confidence Limits
A tutorial lecture given at the annual meeting of the American College of Sports Medicine, Seattle, June 4 1999.
© Will G HopkinsPhysiology and Physical EducationUniversity of OtagoDunedin NZ
OutlineOutline
Definitions and Mis/interpretationsDefinitions and Mis/interpretations PlanningPlanning
Sample sizeSample size PerformingPerforming
Sample size "on the fly"Sample size "on the fly" PublishingPublishing
Methods, Results, DiscussionMethods, Results, Discussion Meta-analysisMeta-analysis Publishing non-significant outcomesPublishing non-significant outcomes
ConclusionsConclusions Dis/advantagesDis/advantages
Definitions and Mis/interpretationsDefinitions and Mis/interpretations
Confidence limits: DefinitionsConfidence limits: Definitions "Margin of error""Margin of error"
Example: Survey of 1000 votersExample: Survey of 1000 votersDemocrats 43%, Republicans 33%Democrats 43%, Republicans 33%Margin of error is ± 3% (for a result of 50%...)Margin of error is ± 3% (for a result of 50%...)
Likely range of true valueLikely range of true value "Likely" is usually 95%."Likely" is usually 95%. "True value" = population value"True value" = population value
= value if you studied the entire population. = value if you studied the entire population. Example: Survey of 1000 voters Example: Survey of 1000 voters
Democrats 43% (likely range 40 to 46%) Democrats 43% (likely range 40 to 46%) Democrats - Republicans 10% (likely range 5 to 15%) Democrats - Republicans 10% (likely range 5 to 15%)
Example: in a study of 64 subjects, the correlation between Example: in a study of 64 subjects, the correlation between height and weight was 0.68 (likely range 0.52 to 0.79).height and weight was 0.68 (likely range 0.52 to 0.79).
correlation coefficientcorrelation coefficient
observedvalue
observedvalue
0.000.00 0.500.50 11
upperconfidencelimit
upperconfidencelimit
lowerconfidence
limit
lowerconfidence
limit
Confidence Confidence intervalinterval: difference between the upper and : difference between the upper and lower confidence lower confidence limits.limits.
Amazing facts about confidence intervalsAmazing facts about confidence intervals(for normally distributed statistics)(for normally distributed statistics)
To To halvehalve the interval, you have to the interval, you have to quadruplequadruple sample size. sample size.
A A 99% interval99% interval is 1.3 times wider than a 95% interval. is 1.3 times wider than a 95% interval.You need 1.7 times the sample size for the same width.You need 1.7 times the sample size for the same width.
A A 90% interval90% interval is 0.8 of the width of a 95% interval. is 0.8 of the width of a 95% interval.You need 0.7 times the sample size for the same width.You need 0.7 times the sample size for the same width.
How to Derive Confidence LimitsHow to Derive Confidence Limits Find a Find a function(true value, observed value, data)function(true value, observed value, data) with a with a
known probability distribution.known probability distribution. Calculate a Calculate a critical valuecritical value, such that for 2.5% of the time, , such that for 2.5% of the time,
function(true value, observed value, data) < critical value.function(true value, observed value, data) < critical value.
How to Derive Confidence LimitsHow to Derive Confidence Limits Find a Find a function(true value, observed value, data)function(true value, observed value, data) with a with a
known probability distribution.known probability distribution. Calculate a Calculate a critical valuecritical value, such that for 2.5% of the time, , such that for 2.5% of the time,
function(true value, observed value, data) < critical value.function(true value, observed value, data) < critical value.
probabilityprobability
function (e.g. (n-1)s2/2)function (e.g. (n-1)s2/2)
area =0.025area =0.025
critical valuecritical value
probability distributionof function(e.g. 2)
probability distributionof function(e.g. 2)
Rearranging, for 2.5% of the time,Rearranging, for 2.5% of the time,true value > function'(observed value, data, critical value)true value > function'(observed value, data, critical value) = upper confidence limit = upper confidence limit
Rearranging, for 2.5% of the time,Rearranging, for 2.5% of the time,true value > function'(observed value, data, critical value)true value > function'(observed value, data, critical value) = upper confidence limit = upper confidence limit
Mis/interpretation of confidence limitsMis/interpretation of confidence limits Hard to misinterpret confidence limits for simple proportions Hard to misinterpret confidence limits for simple proportions
and correlation coefficients.and correlation coefficients. Easier to misinterpret changes in means.Easier to misinterpret changes in means. Example: The change in blood volume in a study was 0.52 L Example: The change in blood volume in a study was 0.52 L
(likely range 0.12 to 0.92 L).(likely range 0.12 to 0.92 L). For 95% of subjects, the change was/would be between 0.12 and For 95% of subjects, the change was/would be between 0.12 and
0.92 L.0.92 L. The average change in the population would be between 0.12 and The average change in the population would be between 0.12 and
0.92 L.0.92 L. The change for the average subject would be between 0.12 and 0.92 The change for the average subject would be between 0.12 and 0.92
L.L. There may be individual differences in the change.There may be individual differences in the change.
P value: DefinitionP value: Definition The probability of a more extreme absolute value The probability of a more extreme absolute value
than the observed value if the true value was zero than the observed value if the true value was zero or null. or null.
Example: 20 subjects, correlation = 0.25, p = 0.29.Example: 20 subjects, correlation = 0.25, p = 0.29.
probabilityprobability
correlation coefficientcorrelation coefficient
area =p value= 0.29
area =p value= 0.29
no effectno effect
observed effect(r = 0.25)observed effect(r = 0.25)
distribution ofcorrelationsfor no effect and n = 20
distribution ofcorrelationsfor no effect and n = 2000 0.50.5-0.5-0.5
"Statistically Significant": Definitions "Statistically Significant": Definitions P < 0.05P < 0.05 Zero lies outside the confidence interval.Zero lies outside the confidence interval.
Examples: four correlations for samples of size 20.Examples: four correlations for samples of size 20.
0.000.00 0.500.50 11correlation coefficientcorrelation coefficient
-0.50-0.50
rr likely rangelikely range PP
0.700.70 0.37 to 0.870.37 to 0.87 0.0070.007
0.440.44 0.00 to 0.740.00 to 0.74 0.050.05
0.250.25 -0.22 to 0.62-0.22 to 0.62 0.290.29
0.000.00 -0.44 to 0.44-0.44 to 0.44 1.001.00
Incredibly interesting information about statistical Incredibly interesting information about statistical significance and confidence intervalssignificance and confidence intervals
Two independent estimates of a normally distributed statistic Two independent estimates of a normally distributed statistic with equal confidence intervals are significantly different at with equal confidence intervals are significantly different at the 5% level if the overlap of their intervals is less than 0.29 the 5% level if the overlap of their intervals is less than 0.29 (1 - (1 - 2/2) of the length of the interval.2/2) of the length of the interval.
If the intervals are very unequal...If the intervals are very unequal...
p < 0.05p < 0.05 p = 0.05p = 0.05 p > 0.05p > 0.05
p < 0.05p < 0.05 p = 0.05p = 0.05 p > 0.05p > 0.05
Type I and II ErrorsType I and II Errors You could be wrong about significance or lack of it.You could be wrong about significance or lack of it. Type I error = false alarm. Type I error = false alarm.
Rate = 5% for zero real effect.Rate = 5% for zero real effect. Type II error = failed alarm.Type II error = failed alarm.
Traditional acceptable rate = 20% for smallest worthwhile Traditional acceptable rate = 20% for smallest worthwhile effect.effect.
Lots of tests for significance implies more chance of at Lots of tests for significance implies more chance of at least one false alarm: "inflated type I error". least one false alarm: "inflated type I error". Ditto type II error?Ditto type II error? Deal with inflated type I error by reducing the p value.Deal with inflated type I error by reducing the p value. Should we adjust confidence intervals? No.Should we adjust confidence intervals? No.
Mis/interpretation of P > 0.05Mis/interpretation of P > 0.05(for an observed positive effect)(for an observed positive effect) The effect is not publishable.The effect is not publishable. There is no effect.There is no effect. The effect is probably zero or trivial.The effect is probably zero or trivial. There's a reasonable chance the effect is < zero.There's a reasonable chance the effect is < zero.
Mis/interpretation of P < 0.05Mis/interpretation of P < 0.05(for an observed positive effect)(for an observed positive effect) The effect is probably big.The effect is probably big. There's a < 5% chance the effect is zero.There's a < 5% chance the effect is zero. There's a < 2.5% chance the effect is < zero.There's a < 2.5% chance the effect is < zero. There's a high chance the effect is > zero.There's a high chance the effect is > zero. The effect is publishable.The effect is publishable.
Planning ResearchPlanning Research Sample Size via Statistical SignificanceSample Size via Statistical Significance
Sample size must be big enough to be sure you will detect Sample size must be big enough to be sure you will detect the smallest worthwhile effect.the smallest worthwhile effect. To be sureTo be sure: 80% of the time.: 80% of the time. DetectDetect: P < 0.05.: P < 0.05. Smallest worthwhile effectSmallest worthwhile effect: what impacts your subjects : what impacts your subjects
correlation = 0.10 correlation = 0.10 relative risk = 1.2 (or frequency difference = 10%)relative risk = 1.2 (or frequency difference = 10%) difference in means = 0.2 of a between-subject standard deviationdifference in means = 0.2 of a between-subject standard deviation change in means = 0.5 of a within-subject standard deviationchange in means = 0.5 of a within-subject standard deviation
Example: 760 subjects to detect a correlation of 0.10.Example: 760 subjects to detect a correlation of 0.10. Example: 68 subjects to detect a 0.5% change in a crossover study Example: 68 subjects to detect a 0.5% change in a crossover study
when the within-subject variation is 1%.when the within-subject variation is 1%.
But 95% likely range doesn't work properly with But 95% likely range doesn't work properly with traditional sample-size estimation (maybe).traditional sample-size estimation (maybe).Example: Correlation of 0.06, sample size of 760...Example: Correlation of 0.06, sample size of 760... 47.5% + 47.5% (=95%) likely range:47.5% + 47.5% (=95%) likely range:
0.10.100correlation coefficientcorrelation coefficient
-0.1-0.1
Not significant, but Not significant, but could be substantial.could be substantial.Huh?Huh?
0.10.100correlation coefficientcorrelation coefficient
-0.1-0.1
47.5% + 30% likely range:47.5% + 30% likely range:
Not significant, and Not significant, and can't be substantial.can't be substantial.OK!OK!
Sample Size via Confidence LimitsSample Size via Confidence Limits Sample size must be big enough for acceptable Sample size must be big enough for acceptable
precision of the effect.precision of the effect. PrecisionPrecision means 95% confidence limits. means 95% confidence limits. Acceptable Acceptable means any value of the effect within these means any value of the effect within these
limits will not impact your subjects.limits will not impact your subjects. Example: need 380 subjects to delimit a correlation of Example: need 380 subjects to delimit a correlation of
zero.zero.
00 0.100.10correlation coefficientcorrelation coefficient-0.10-0.10
smallest worthwhileeffects
smallest worthwhileeffects confidence
interval forN = 380
confidenceinterval forN = 380
But sample size needed to detect or delimit But sample size needed to detect or delimit smallest effect is overkill for larger effects.smallest effect is overkill for larger effects. Example: confidence limits for correlations of 0.10 and Example: confidence limits for correlations of 0.10 and
0.80 with a sample size of 760...0.80 with a sample size of 760...
0.10.1 0.30.3 0.50.5 0.70.700 0.90.9 11correlation coefficientcorrelation coefficient
-0.1-0.1
So why not start with a smaller sample and do more subjects So why not start with a smaller sample and do more subjects only if necessary?only if necessary?Yes, I call it...Yes, I call it...
Performing ResearchPerforming Research
Sample Size "On the Fly"Sample Size "On the Fly" Start with a small sample; add subjects until you Start with a small sample; add subjects until you
get acceptable precision for the effect.get acceptable precision for the effect. Acceptable precisionAcceptable precision defined as before. defined as before. Need qualitative scale for magnitudes of effects.Need qualitative scale for magnitudes of effects. Example: sample sizes to delimit correlations...Example: sample sizes to delimit correlations...
155155
0.10.1 0.30.3 0.50.5 0.70.700 0.90.9 11
trivial small moderate large
270270350350380380
correlation coefficientcorrelation coefficient-0.1-0.1
nearlyperfect
4646
very large
Problems with sampling on the flyProblems with sampling on the fly
Do not sample until you get statistical significance: the resulting Do not sample until you get statistical significance: the resulting outcomes are biased larger than life.outcomes are biased larger than life.
Sampling until the confidence interval is acceptable produces Sampling until the confidence interval is acceptable produces bias, but it is negligible.bias, but it is negligible.
But researchers will rush into print as soon as they get But researchers will rush into print as soon as they get statistical significance.statistical significance.
And funding agencies prefer to give money onceAnd funding agencies prefer to give money once(but you could give some back!).(but you could give some back!).
And all the big effects have been researched anyway?And all the big effects have been researched anyway?No, not really.No, not really.
Publishing ResearchPublishing Research In the MethodsIn the Methods
"We show the precision of our estimates of outcome "We show the precision of our estimates of outcome statistics as 95% confidence limits (which define the likely statistics as 95% confidence limits (which define the likely range of the true value in the population from which we range of the true value in the population from which we drew our sample)."drew our sample)."
Amazingly useful tips on calculating confidence limitsAmazingly useful tips on calculating confidence limits SimpleSimple differences between meansdifferences between means: stats program. : stats program. Other normally distributed statisticsOther normally distributed statistics: mean and p value.: mean and p value. Relative risksRelative risks: stats program.: stats program. CorrelationsCorrelations: Fisher's z transform.: Fisher's z transform. Standard deviationsStandard deviations and other root mean square variations: and other root mean square variations:
chi-squared distribution.chi-squared distribution.
Coefficients of variationCoefficients of variation: standard deviation of 100x natural log : standard deviation of 100x natural log of the variable. Back transform for CV>5%.of the variable. Back transform for CV>5%.
Use the adjustmentUse the adjustment of Tate and Klett to get shorter intervals of Tate and Klett to get shorter intervals for SDs and CVs from small samples.for SDs and CVs from small samples.
2211coefficient of variation (%)coefficient of variation (%)
00 33
Example:coefficient of variation for 10 subjects in 2 tests
Example:coefficient of variation for 10 subjects in 2 tests
usualusualadjustedadjusted
Ratios of independent standard deviationsRatios of independent standard deviations: F distribution.: F distribution. RR22 (variance explained) (variance explained): convert to a correlation.: convert to a correlation. Use the spreadsheetUse the spreadsheet at at sportsci.org/statssportsci.org/stats for all the above. for all the above. Effect-size (Effect-size (mean/standard deviation)mean/standard deviation): non-central : non-central
F distribution or bootstrapping.F distribution or bootstrapping. Really awful statisticsReally awful statistics: bootstrapping.: bootstrapping.
Bootstrapping (Resampling) for confidence limitsBootstrapping (Resampling) for confidence limits Use for difficult statistics, e.g. for grossly non-normal repeated Use for difficult statistics, e.g. for grossly non-normal repeated
measures with missing values. Here's how...measures with missing values. Here's how...
For a large-enough sample, you can recreate (sort of) the For a large-enough sample, you can recreate (sort of) the population by duplicating the sample endlessly.population by duplicating the sample endlessly.
Draw 1000 samples (of same size as your original) from this Draw 1000 samples (of same size as your original) from this population.population.
Calculate your outcome statistic for each of these samples, Calculate your outcome statistic for each of these samples, rank them, then find the 25th and 975th place-getters. These rank them, then find the 25th and 975th place-getters. These are the confidence limits.are the confidence limits.
ProblemsProblems Painful to generate.Painful to generate. No good for infrequent levels of nominal variables.No good for infrequent levels of nominal variables.
In the ResultsIn the Results In TEXTIn TEXT
Change or difference in meansChange or difference in meansFirst mention:First mention:...0.42 (95% confidence/likely limits/range -0.09 to 0.93) or ...0.42 (95% confidence/likely limits/range -0.09 to 0.93) or ...0.42 (95% confidence/likely limits/range ± 0.51). ...0.42 (95% confidence/likely limits/range ± 0.51). Thereafter:Thereafter:...2.6 (1.4 to 3.8) or 2.6 (± 1.2) etc....2.6 (1.4 to 3.8) or 2.6 (± 1.2) etc.
Correlations, relative risks, odds ratios, standard deviations, ratios of Correlations, relative risks, odds ratios, standard deviations, ratios of standard deviations: can't use ± because the confidence interval is standard deviations: can't use ± because the confidence interval is skewed:skewed:...a correlation of 0.90 (0.67 to 0.97)......a correlation of 0.90 (0.67 to 0.97)......a coefficient of variation of 1.3% (0.9 to 1.9)......a coefficient of variation of 1.3% (0.9 to 1.9)...
In TABLESIn TABLES Confidence intervalsConfidence intervals
rr likely rangelikely range
0.700.70 0.37 to 0.870.37 to 0.870.440.44 0.00 to 0.740.00 to 0.740.250.25 -0.22 to 0.62-0.22 to 0.620.000.00 -0.44 to 0.44-0.44 to 0.44
Variable AVariable AVariable BVariable BVariable CVariable CVariable DVariable D
P valuesP values
rr pp
0.700.70 0.0070.0070.440.44 0.050.050.250.25 0.290.290.000.00 1.001.00
Variable AVariable AVariable BVariable BVariable CVariable CVariable DVariable D
AsterisksAsterisks
rr
0.70**0.70**0.44*0.44*0.250.250.000.00
Variable AVariable AVariable BVariable BVariable CVariable CVariable DVariable D
In FIGURESIn FIGURES
-10 -5 0 5 10Change in power (%)
Bars are 95% likely ranges
Told placebo
Not told
Told carbohydrate
-3
-2
-1
0
1
2
3
4
0 2 4 6 8 10 12 14
live lowtrain low
live hightrain high
live hightrain low
sea level altitude sea level
change in5000-mtime (%)
training time (weeks)
likely range oftrue change
In the DiscussionIn the Discussion Interpret the observed effect and its 95% Interpret the observed effect and its 95%
confidence limits qualitatively. confidence limits qualitatively. Example: you observed a moderate correlation, but the Example: you observed a moderate correlation, but the
true value of the correlation could be anything between true value of the correlation could be anything between trivial and very strong.trivial and very strong.
0.10.1 0.30.3 0.50.5 0.70.700 0.90.9 11
trivial small moderate large
correlation coefficientcorrelation coefficient-0.1-0.1
nearlyperfectvery large
Meta-AnalysisMeta-Analysis Deriving a single estimate and confidence interval for an Deriving a single estimate and confidence interval for an
effect from several studies.effect from several studies. Here's how it works for two:Here's how it works for two:
Study 1Study 1
Study 2Study 2
Study 1+2Study 1+2
Equal Confidence IntervalsEqual Confidence Intervals
Study 1Study 1
Study 2Study 2
Study 1+2Study 1+2
Unequal Confidence IntervalsUnequal Confidence Intervals
Publishing non-significant outcomesPublishing non-significant outcomes Publishing only significant effects from small-scale Publishing only significant effects from small-scale
studies leads to publication bias.studies leads to publication bias. Publishing effects with confidence limits regardless Publishing effects with confidence limits regardless
of magnitude is free of bias.of magnitude is free of bias. Many smaller studies are probably better than a Many smaller studies are probably better than a
few larger ones anyway.few larger ones anyway. So bully the editor into accepting the paper about So bully the editor into accepting the paper about
your seemingly inconclusive small-scale study.your seemingly inconclusive small-scale study.
ConclusionsConclusions Disadvantages of Statistical SignificanceDisadvantages of Statistical Significance
Emphasizes testing of hypotheses.Emphasizes testing of hypotheses. Aim is to Aim is to detectdetect an effect--effects are zero until proven an effect--effects are zero until proven
otherwise. otherwise. Have to understand Type I and II errors.Have to understand Type I and II errors.
Hard to understand; easy to misinterpret.Hard to understand; easy to misinterpret. Have to consider sample size.Have to consider sample size. Focuses on statistically significant effects.Focuses on statistically significant effects.
Advantages of Statistical SignificanceAdvantages of Statistical Significance Familiar.Familiar. All stats programs give p values.All stats programs give p values. Easy to put asterisks in tables and figures.Easy to put asterisks in tables and figures.
Disadvantages of Confidence LimitsDisadvantages of Confidence Limits Unfamiliar.Unfamiliar. Not always available in stats programs.Not always available in stats programs. Cluttersome in tables.Cluttersome in tables. Display in time series can be a challenge.Display in time series can be a challenge.
Advantages of Confidence LimitsAdvantages of Confidence Limits Emphasizes precision of estimation.Emphasizes precision of estimation.
Aim is to Aim is to delimitdelimit an effect--effects are never zero. an effect--effects are never zero. Only one kind of "error".Only one kind of "error".
Meaning is reasonably clear, even to lay readers. Meaning is reasonably clear, even to lay readers. No confusion between significance and magnitude.No confusion between significance and magnitude.
Journals now require them.Journals now require them.