malmo 30 03-2012

The unity of all science consists alone in the method, not its material.

Pearson K. The grammar of science. London, Black, 1892.

Statistics is the study of uncertainty.

Savage LJ. The foundations of statistics. New York, Wiley, 1954.

The aim of statistics reviewing

Accurate and transparent description of the uncertainty in presented findings.

“Statisticians are experts in handling uncertainty”.

Lindley DV. The philosophy of statistics. The Statistician 2000;49:293-337.

Experi-mental

Study Study design design

Obser-vational

Few ethics concernsBias adjustmentsExternal validityLong follow up

Few sample size restrictions

Ethics concernsRandomization

Controlled conditionsInternal validityShort follow up

Sample size restrictions

Medical research methodology

Experi-mental

Study Study design design

Obser-vational

Internal validity by design (blocking of known risk factors and randomization of

other)

Potential for confounding: none

Internal validity by statistical analysis (confounding adjustment for known and

measured risk factors)

Potential for confounding: massive

Statistical aspects – internal validity

Confounder (or case-mix) adjustmentHow much of the variation in endpoints can be explained by known factors, and how much has unknown causes?

Variation with unknown origin

95%-99% Arthroplasty revision

85%-95% EQ-5D, SF36

70%-80% Coronary heart disease

Risk factors, confounding, and the illusion of statistical control'...it is essential to remember that “statistical control” is nothing more than a highly fallible process filled with judgment calls that often go unnoticed in practice.'

Christenfeld NJS, Sloan RP, Carrol D, Greenland S.Psychosomatic Medicine 2004;66:868–875

Simple modelMultiple model

(or multivariable, but not multivariate)

Linear regression analysis

Stepwise regression

Statistics

We calculated odds ratios by logistic regression analysis, to estimate the relationship between failure of the osteotomy and possible preoperative risk factors. We performed multivariate, stepwise (backward) logistic regression and entered variables with a p-value of ≤ 0.05 into the model.

Unified theory of bias Bias can be reduced to or explained by 3 structures

1. Reverse causation

Outcome precedes exposure measurement or outcome can have effect on exposure. Measurement error or Information bias.

2. Common cause

Confounding by association, confounding by indication.

3. Conditioning on common effects

Collider, selection bias, time varying confounding.

Covariate selectionAdequate Background Knowledge

Confounder identification must be based on understanding of the causal structure linking the variables being studied (treatment and disease).

Condition on the minimal set of variables necessary to remove confounding.

Inadequate Background Knowledge

Remove known instrumental variables, colliders, intermediates (variables with post treatment measurement.

ConfoundingUnder-adjustment

occurs when a confounder is not adjusted for.

Over-adjustment

can occur from adjusting instrumental variables, intermediate variables, colliders, variables caused by outcome.

Confounder

Common cause, i.e., confounder

Confounder L distort the effect of treatment A on disease Y

Always adjust for confounders, unless small data set and confounder has strong association with treatment and week association with outcome

Confounder example

A = treatment1: statin alone0: niacin alone

L = Baseline Cholesterol1: LDL ≥ 160 mg/dL0: LDL < 160 mg/dL

Y = Myocardial infarction1: Yes0: No

Intermediate variableAdjusting for intermediate variable I in a fixed covariate model will remove the effect of treatment A on disease/outcome Y

In a fixed covariate model we do not want to include variables influenced by A or Y

Intermediate exampleA = treatment1: statin alone0: niacin alone

I = Post-treatment Cholesterol1: LDL ≥ 160 mg/dL0: LDL < 160 mg/dL

Y = Myocardial infarction1: Yes0: No

ColliderAdjusting for a collider can produce bias

Conditioning on common effect F without adjustment of U1 or U2 will induce an association between U1 and U2, which will confound the association between A and Y

Collider example

Variables associated with treatment or disease onlyInclusion of variables associated with treatment only can cause bias and imprecision

Variables associated with disease but not treatment (risk factors) can be included in models. They are expected to decrease variance of treatment effect without increasing bias

Including variables associated with disease reduces the chance of missing important confounders

Reality is complicated

http://www.dagitty.net

http://www.dagitty.net

Any claim coming from an observational study is most likely to be wrong

12 randomised trials have tested 52 observational claims (about the effects of vitamine B6, B12, C, D, E, beta carotene, hormone replace- ment therapy, folic acid and selenium).

“They all confirmed no claims in the direction of the observational claim. We repeat that figure: 0 out of 52. To put it in another way, 100% of the observational claims failed to replicate. In fact, five claims (9.6%) are statistically significant in the opposite direction to the observational claim.”

Young S, Karr A. Deming, data and observational studies. Significance, September 2011.

Aetiology Study scope Study scope Treatment

Pre-specified hypotheses

Confirmation

Legislation, regulatory guidelines

Uncertainty intolerance

Hypothesis generation

Exploration

Academic analysis freedom

Uncertainty tolerance

Medical research methodology

Aetiology Study scope Study scope Treatment

Randomized clinical trials

Patient registerstudies

Epidemiologicalstudies

Laboratory experiments

Medical research methodology

Experi-mental

Study Study design design

Obser-vational

Aetiology Study scope Study scope Treatment

Protected type-1 error rate for specified endpoints

Sample size based on the type-2 error rate

Specified type-1 and -2 error uncertainty

(confidence intervals)

No multiplicity consideration for safety endpoints

Multiplicity issuesnot addressed

Sample size not based on type-2 error rate

Bonferroni correction within endpoints

Few type-2 error considerations

Statistical aspects - precision

Experi-mental

Study Study design design

Obser-vational

Aetiology Study scope Study scope Treatment

Experi-mental

Study Study design design

Obser-vational

Drug development

Phase 1Discovery(Phase 0)

Phase 2Phase 3

Phase 4

PMS(Phase 5)

Aetiology Study orientation Study orientation Treatment

Device development

Randomizedperformance

trials

Safetyfollow-up

in registries

Biomechanicsin vitro, etc.

Experi-mental

Study Study design design

Obser-vational

It is impossible to do clinical research so badly that it cannot be published

“There seems to be no study too fragmented, no hypothesis too trivial, no literature citation too biased or too egotistical, no design too warped, no methodology too bungled, no presentation of results too inaccurate, no argument too circular, no conclusions too trifling or too unjustified, and no grammar and syntax too offensive for a paper to end up in print.”

Drummond Rennie 1986 (editor of NEJM and JAMA)

Arthroplasty registry analyses

Crucial issues

- Fulfillment of methodological assumptions (Gaussian distr, homogeneity of variance, proportionality, linearity, etc.)- Confounding adjustment (risk factors, causality, linearity, etc.) - Clinical significance and estimation uncertainty (95%CI).

Should be avoided

- P-value culture- Bonferroni correction- Post-hoc power- Predictions

Thank you for your attention!

Indicators for statistical review- Randomized trials

- Patient registry (safety) studies

- Analyses of knees, hips, elbows... (bilateral observations)

- Pseudo-replicates (esp. in laboratory experiments)

- “No difference” manuscripts

- Stepwise regression

- ???