Bigger, better, faster, more! – Sample size calculationSEMINAR SERIES: HOW TO RUIN YOUR CAREFULLY PLANNED STUDY? TIPS FORIMPROVING DATA ANALYSIS – SESSION 6
TOM SMEKENS
TYPE NAME DEPARTMENT IN WINDOW
Would you believe me if I said…
“Patient centered care is not related to health outcomes, based on:
my sample of 8 physicians”
“No! Your sample is not…”
my sample of 5000 orthopedic surgeons”
“No! Your sample is not…”
TYPE NAME DEPARTMENT IN WINDOW
TYPE NAME DEPARTMENT IN WINDOW
4
“Not representative”
Bias Variance
TYPE NAME DEPARTMENT IN WINDOW
“Not representative”
Bias Variance
TYPE NAME DEPARTMENT IN WINDOW
Estimating sampling variance
After the study: standard errors, p-values, confidence intervals…
Before the study: sample size calculation
TYPE NAME DEPARTMENT IN WINDOW
TYPE NAME DEPARTMENT IN WINDOW
Thought process
Hypothesis AnalysisSample
size
TYPE NAME DEPARTMENT IN WINDOW
Sample size goals
Hypothesis testing:Power
Estimation:Precision
TYPE NAME DEPARTMENT IN WINDOW
Sample size goals
Hypothesis testing:Power
Estimation:Precision
TYPE NAME DEPARTMENT IN WINDOW
“A large enough sample tomake the right conclusionin most cases”
Conclusion determined byStatistical significance
“A large enough sample tosufficiently narrow down an estimate”
Precision expressed usingConfidence intervals
Methods of sample size calculation
Derivation
TYPE NAME DEPARTMENT IN WINDOW
Methods of sample size calculation
Derivation Simulation
TYPE NAME DEPARTMENT IN WINDOW
TYPE NAME DEPARTMENT IN WINDOW
TYPE NAME DEPARTMENT IN WINDOW
TYPE NAME DEPARTMENT IN WINDOW
Contents
1. Power
2. Populations
3. Proportions
TYPE NAME DEPARTMENT IN WINDOW
TYPE NAME DEPARTMENT IN WINDOW
1. The preponderance of power
Ingredients (hypothesis testing)
1. Expected value"We want to reduce mean systolic bloodpressure by 5 mmHg"
2. Scale"68% of the population is in a range of 20 mmHgaround the mean"
3. P-value threshold= 5%
4. Power= 80%
TYPE NAME DEPARTMENT IN WINDOW
Ingredients (hypothesis testing)
1. Expected value"We want to reduce mean systolic bloodpressure by 5 mmHg"
2. Scale"68% of the population is in a range of 20 mmHgaround the mean"
3. P-value threshold= 5%
4. Power= 80%
TYPE NAME DEPARTMENT IN WINDOW
If your expectations are true, theprobability of getting a statisticallysignificant result
Ingredients (hypothesis testing)
1. Expected value"We want to reduce mean systolic bloodpressure by 5 mmHg"
2. Scale"68% of the population is in a range of 20 mmHgaround the mean"
3. P-value threshold= 5%
4. Power= 80%
TYPE NAME DEPARTMENT IN WINDOW
If your expectations are true, theprobability of getting a statisticallysignificant result
TYPE NAME DEPARTMENT IN WINDOW
(Bacchetti et al., 2011)
TYPE NAME DEPARTMENT IN WINDOW
(Bacchetti et al., 2011)
80% Power
TYPE NAME DEPARTMENT IN WINDOW
(Bacchetti et al., 2011)
80% Power
Budget
TYPE NAME DEPARTMENT IN WINDOW
(Bacchetti et al., 2011)
80% Power
Budget
Ingredients (hypothesis testing)
1. Expected value"We want to reduce mean systolic blood
pressure by 15 mmHg"
1. Scale"68% of the population is in a range of 20 mmHgaround the mean"
1. P-value threshold= 5%
2. Power= 80%
TYPE NAME DEPARTMENT IN WINDOW
How much power, really???
For a 15 mmHg difference?
80%
For a 5 mmHg difference?
15%
TYPE NAME DEPARTMENT IN WINDOW
Results from an underpowered study
8 mmHg: Not significant...
19 mmHg: Significant!
TYPE NAME DEPARTMENT IN WINDOW
= Filters out plausible, realistic results in favor of fantastical, noisy ones
Ingredients (precision)
1. Scale"68% of the population is in a range of 20 mmHgaround the mean"
1. Confidence level= 95%
2. Margin of error"We will be able to estimate the effect, give or take 2 mmHg"
TYPE NAME DEPARTMENT IN WINDOW
= half the width of the confidence interval
Mentimeter
How often have you based your sample size calculations around precision?
I never calculate any sample sizes...
TYPE NAME DEPARTMENT IN WINDOW
TYPE NAME DEPARTMENT IN WINDOW
TYPE NAME DEPARTMENT IN WINDOW
2. What about population size?
Inference: generalize from a sample to...
A specific group (usually people) in a given place and time
Finite population
A process (usually biomedical) that is repeatedly observed
Infinite population
TYPE NAME DEPARTMENT IN WINDOW
Finite Population Correction
By default: assume infinite population
Optional: account for the population size
-> Smaller p-values, narrower confidence intervals...
Only noticeable if sample is > 1% of the population
TYPE NAME DEPARTMENT IN WINDOW
TYPE NAME DEPARTMENT IN WINDOW
3. Properly prespecify probabilistic properties ofproportion prerogation
Comparing proportions
Instead of mean systolic blood pressure,compare the prevalence of hypertension
1. Expected valueScale
2. P-value threshold = 5%
3. Power = 80%
TYPE NAME DEPARTMENT IN WINDOW
Scale
"Hypertension prevalence is 30%.We want to reduce it to 25%."
=> Test a difference of 5%?
What about 10% to 5%?
TYPE NAME DEPARTMENT IN WINDOW
Comparing proportions: 25% vs 30%
Comparison Value Regression method
Percentage point difference - 0.05 Linear
Risk ratio 0.83 Log binomial
Odds ratio 0.78 Logistic
TYPE NAME DEPARTMENT IN WINDOW
Comparing proportions: 25% vs 30%
Comparison Value Regression method
Percentage point difference - 0.05 Linear
Risk ratio 0.83 Log binomial
Odds ratio 0.78 Logistic
TYPE NAME DEPARTMENT IN WINDOW
Study result: 8% to 11%
Comparison Value Regression method
Percentage point difference - 0.03 Linear
Risk ratio 0.73 Log binomial
Odds ratio 0.70 Logistic
Non-inferiority trials
TYPE NAME DEPARTMENT IN WINDOW
"Not inferior"
(Ellis et al., 2015)
Non-inferiority margin: 4.5 percentage pointsStudy result: 1.7 percentage pointsBut: risk increased by 30%!
TYPE NAME DEPARTMENT IN WINDOW
Sample size calculation with categorical data
Before the study After the study
TYPE NAME DEPARTMENT IN WINDOW
Conclusion
TYPE NAME DEPARTMENT IN WINDOW
Sample size calculation
Essential in planning a quantitative study
But: fraught with bad habits (even from statisticians!)
Future prospects:
Simulation instead of formulas
Precision instead of power
Questions / Comments?
Next seminar is on December 17
Why eating ice cream doesn’t cause summer – Association and causation
Presenter: Jozefien Buyze
These days, it’s commonly known by researchers that an association doesn't necessarily imply causation. Nevertheless, it’s common to still find this mistake even in high impact papers. In this session, we discuss confounding and interaction (also known as effect-modification) and why finding an association may be very valuable in some settings but not sufficient in others.
32
First name Name Tel/Mobile number E-mail address