camarades: bringing evidence to translational medicine study quality and publication bias in...

CAMARADES: Bringing evidence to translational medicine

Study Quality and Publication Bias in Experimental Stroke


Presenter Disclosure Information

FINANCIAL DISCLOSURE:None

Malcolm Macleod PhD FRCPStudy Quality and Publication Bias in Experimental Stroke

UNLABELED/UNAPPROVED USES DISCLOSURE:NoneCONFLICT OF INTEREST DISCLOSURE:President, EuroHYP


1026

1026 interventions in experimental stroke

Tested in experimentsO’Collins et al, 2006


1026603


Tested in focal ischaemiaO’Collins et al, 2006


1026883374


Effective in focal ischaemiaO’Collins et al, 2006


1026883550

97 18


Tested in clinical trialO’Collins et al, 2006


1026883550

97 171 3


Effective in clinical trialO’Collins et al, 2006


Control half dose full dose

Infa

rct V

olum

e

0

50

100

150

200

250

300

10-120 M 10-60 M


What is translational medicine?

From: Institute of Translational Medicine, University of Texas Medical Branch


Animal data in stroke

• There are huge amounts of often confusing data

• Systematic review can help to make sense of it

• If you select extreme bits of the evidence you can “prove” either harm or substantial benefit

• However, if you have a precise and highly significant overall effect, then it is probably real

Hypothermia: a systematic search identified 277 experiments in

3353 animals

Bett

er

Wors

e

Van der Worp et al, 2007


Potential sources of bias in animal studies

• Internal validity

• External validity– Publication bias – Are the models we use good models?

• Co-morbidities

Problem Solution

Selection Bias Randomisation

Performance Bias Allocation Concealment

Detection Bias Blinded outcome assessment

Attrition bias Reporting drop-outs/ ITT analysis


Internal ValidityRandomisation and blinding in studies of hypothermia in experimental stroke

• Infarct Volume– 101 publications– 222 experiments– 3256 animals– Improved outcome by

43.5% (40.1-47.0) Blinded outcome

assessment

Yes No

Effi

cacy

47%39%

Randomisation

47%37%

Effi

cacy

Van der Worp et al, 2007


Internal Validity NXY-059

• Infarct Volume– 11 publications

– 29 experiments

– 408 animals

– Improved outcome by 44% (35-53%)

Macleod et al, 2008


The File Drawer problemPublication bias

0worse better


BetterWorse

Pre

cisi

on

• All outcomes– 29 publications– 109 experiments– 1596 animals– Improved outcome by 31% (27-35%)

External ValidityPublication Bias for FK506

Macleod et al, 2005


Publication bias – G-CSF

England T et al 2009


Publication bias in experimental stroke

• Only 11/525 publications (2.2%) reported no significant treatment effects

• Trim and Fill suggested ~16% (214/1573) of experiments remain unpublished

• Best estimate of magnitude of problem – Observed efficacy 31.3% (29.7-32.8)– Adjusted efficacy 23.8% (22.2-25.5)


Publication bias

Randomisation

Co-morbidity

bias

How much efficacy is left?


There are many other sources of bias (Sacket, 1979)

• Tidying up bias• Repeated peaks bias• Significance bias• Bogus control bias• Hot stuff bias• Selective reporting bias


Current performance against key quality items

RandomisationBlinded Outcome

AssessmentSample Size calculation

Stroke 36% 29% 3%


How does stroke compare?

RandomisationBlinded Outcome

AssessmentSample Size calculation

Stroke 36% 29% 3%

MND 31% 20% <1%

AD 15% 25% 0%

PD 12% 15% 0%

EAE 8% 15% <1%

Glioma 14% 0% 0%


Internal validity in PD models

Blinded outcome assessment Composite quality

Rooke et al, submitted


Estimates of affinity at cannabinoid receptors

McPartland et al 2007


AnimalStudies

Clinical Trial

Systematic Review

AndMeta-analysis


Conclusions

• We could, and should, do a lot better• Other diseases could do much much better• Like any other process, the processes of

translational medicine can be studied, systematically and methodically

camarades: bringing evidence to translational medicine study quality and publication bias in...

Documents

experimental stroke

worsebetter slide

eurohyp slide

selective reporting

experimental stroke

experiments ocollins

potential sources of

sources of bias sacket