Published research - flawed, misleading, nefarious

John HoeyUniversity of Waterloo, 2009

www.slideshare.net/hoeyj

johnhoeymd@gmail.com

Why is research important - and what is it?

Can we believe what we read in journals?

What can be done?

Why is research important - and what is it?

Public health practice Clinical practicePublic policy

RationalEvidence-based i.e. Research

Why is research important - and what is it?

++ research designsRCTsHealth economicsGenomics ...

So far ~ 150 guidelines

2,000 - 3,000 manuscripts /yrLancet - 6,000 to 7,000 /yr

Maintaining the Integrity of the Scientific Record.......why?

•Distorts the scientific corpus

•misleads other investigtors

•Harm to patients and the public

•Physical harm

•Financial harm

It’s also ethically wrong

Nuremberg - Helsinki

“Both authors and publishers have ethical obligations.

In publication of the results of research, the investigators are obliged to preserve the accuracy of the results.

Negative as well as positive results should be published or otherwise publicly available.

Sources of funding, institutional affiliations and any possible conflicts of interest should be declared in the publication.

Reports of experimentation not in accordance with the principles laid down in this Declaration should not be accepted for publication.”

Let’s look at an example

Qualitative EconomicRandomized Clinical Trials Systematic reviews/meta-analysesetc..

RCT

RRx A

Placebo

Outcome

Consort Statement•Poor quality of reporting RCTs.

•Could not be trusted for use in clinical practice

•Could not be combined in systematic reviews

•Identified 22 essential items needed to conduct a RCT.

Ann Intern Med. 2001;134:663-694. www.annals.org David Moher

Research Reporting Guidelines

- Clinical Trials [Consort]•22 essential items/elements

•e.g Randomization

•Item # 8 How was the sequence generated?

•Item # 9 Allocation concealment

•Item # 10 Implementation

Does this really matter?

Miller AB, et al. Canadian National Breast Screening Study: 1. Breast cancer detection and death rates among women aged 40 to 49 years. CMAJ. 1992; 147(10): 1459–1476

Canadian trial of mammogrphy for breast cancer

Begun in about 198050,000 asymptomatic womenRandomly assigned to

mammographyusual care

Followed for 8 years

•Item # 9 Allocation concealment

•Item # 10 Implementation

Randomization in the Canadian National Breast Screening Study: a review for evidence of subversion. Bailar JC, MacMahon B. CMAJ 1997;156:193-9

Harms?

Can science be trusted?Ethical responsibility to study subjects

Today - still unclear if mammography is beneficial - harms to women, costs to society

Schulz et al - Quality of RCTS

Cochrane Pregancy and childbirth data base

33 meta-analyses - 250 trials

Quality of randomization and risk estimates

Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias: dimensions of methogological quality associated with estimates of treatment effects in controlled trials. JAMA 1995;273(5):408-12 (currently cited 1777 times in )

•Randomization

•Concealment of treatment allocation schedule

•Generation of allocation sequences

•Inclusion of all trial subjects in analysis

•Double blinding (subject and investigator)

Of 22 items - one example

Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias: dimensions of methogological quality associated with estimates of treatment effects in controlled trials. JAMA 1995;273(5):408-12

Quality of 250 trials

Concealment

% of trials

Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias: dimensions of methogological quality associated with estimates of treatment effects in controlled trials. JAMA 1995;273(5):408-12

Quality of 250 trials

% Exaggeration of Risk Ratio

% of trials

Study design - RCT

RRx A

Placebo

Outcome

Primary Outcome -Specified in Protocol?or fishing expedition?

QuickTime™ and aPhoto - JPEG decompressor

are needed to see this picture.

Selective publication - Outcome biaspublishing the more interesting (usually

positive) result

Was there an hypothesis? A plan for analysis and reporting

of data?

In an RCT, this is the primary outcome

Selective publication - Outcome bias(publishing the more interesting result)

48 trials1402 outcomes

31% - 59% incompletely reported(40% not reported at

all)

Chan, A.-W. et al. CMAJ 2004;171:735-740

Selective publication - Outcome bias(publishing the more interesting result)

Interpretation: Intensive multitherapy for patients with poorly controlled type 2 diabetes is successful in helping patients meet most of the goals set by a national diabetes association.

However, 6 months after intensive therapy stopped and patients returned

to usual care the benefits had vanished,

However, 6 months after intensive therapy stopped and patients returned to usual care the benefits had vanished.

Kay Dickinson, Reporting and other biases in studies of Neurontin for migraine, psychiatric/bipolar disorders, nociceptive pain, and neuropathic pain.August, 2008

http://dida.library.ucsf.edu/pdf/oxx18r10

Reporting Bias

Some definitions • Positive= Study shows results favourable

to the intersts of the author/sponsor. Usually statistically significant.

•Negative = Study shows results unfavourable to the interest of the the author/sponsor - may or may not be statisticaly significant.

Reporting Biases

Nefarious or Incompetent?•Non-publication of negative or

neutral results

•Selective publication of results - outcome bias

• Multiple publication bias

•Language bias - and publishing in the grey literature

•Time lag bias

•Undeclared conflicts of interest

•Ghost writing Dickinson

images from Wikipedia which also has a nice summary of court proceedings and results

Neurontin(gabapentin)

P Wessely, C Baumgartner, D Klinger, J Kreczi, N … - Cephalalgia, 1987

Bias Example

Publication Final negative primary results not published, only positive preliminary results

Selective outcome reporting

Outcome reported was not primary or secondary outcome

Selective statistical analyses

2 nonrandomized patients assigned to neurotin were include with those randomized

Spin Emphasis on “positive” outcomes

Neurontin for migraine headaches study

16 Citations P Wessely, C Baumgartner, D Klinger, J Kreczi, N … - Cephalalgia, 1987

Who cares? General Principles of Migraine Management: The Changing Role of PreventionE Loder, D Biondi - Headache: The Journal of Head and Face Pain, 2005 - Blackwell Synergy

Preventive treatment of migraine - SD Silberstein - Trends in Pharmacological Sciences, 2006 - Elsevier

Migraine preventionDW Dodick, SD Silberstein - British Medical Journal, 2007 - pn.bmj.com

Neuromodulators for Migraine PreventionR Kaniecki - Headache: The Journal of Head and Face Pain, 2008 - Blackwell Synergy

Practice parameter: Evidence-based guidelines for migraine headache (an evidence-based review)

Stephen D. Silberstein, MD, FACP, for the US Headache Consortium

Neurology 2000;55:754-762

www.flickr.com/photos/dariustwin/3362416244/

the story of Dr. Reckless and the pharmaceutical company - Pfizer

“ Ghost authorship exists when someone has made substantial

contributions to writing amanuscript and this role is not

mentioned in the manuscript itself. WAME considers

ghost authorship dishonest and unacceptable”

www.wame.org/resources/policies

Ultimately not published as a stand alone paper, but incorporated in a paper with other studies.

UK doc. (Dr. Reckless) tries to publish his research

Neuroptin for pain relief59 sites in 6 countries.

Study ends in 1999.

Results negative (no benefit)

By 2002, no manuscript produced.

Dr. Reckless writes to Pfizer

Ghost Writing - the case of Dr. Reckless vs. Pfizer

“I don’t think we should be too hasty with this request,”

• “I agree with your answer. Although I would love to publish SOMETHING about 945- 224,

Donna McVey made it very clear that we should take care not to publish anything that damages neurontin’s marketing success.”

“..We would need to have ‘editorial’ control, but would certainly involve Dr. Reckless in the process, asking for his expert comment.”“

I think that we can limit the potential downsides of the 224 study by delaying the publication for as long as possible and also from where it is published.

More importantly it will be more important to how WE write up the study. We are using a medical agency to put the paper together which we will show to Dr. Reckless.

We are not allowing him to write it up himself.”

So what?

http://dida.library.ucsf.edu/pdf/oxx18r10

In total, I find that there were 43 million off-label prescriptions of Neurontin as a result of the ... promotional activities related to the off-label uses ... that would not have occurred absent the challenged conduct.”

Meredith Rosenthal

what can be done?

Education in researchSkepticism ++

*Conflict of interest*Use Reporting guidelines

Conflict of Interest

1109 Canadian clinicaltrials.gov732 investigators11 ideal practices to mitigate fCOI

- control over design- data collection- analysis- interpretation-write up-authorship-decision to publish-etc.

6% of investigators met all ideal practices

Rochon P, et al. - in peer review

Reporting GuidelinesConsort - RCTs

Prism- Systematic reviewsMoose - Meta-analysesStard - Diagnositic testsGrade - Clinical guidelinesStrobe - Observational studies

etc. (~160) www.equator-network.org

Reporting Guidelines

Protocol Ethics Funding Write up Journal

to sum up•Patient Care, Public Health & Policy •Rationale and Evidence Based

•Much research is•Unreliable, confused, distorted, conflicted

•So read cautiously and skeptically

•Be familiar with good research standards

•Use guidelines

•Teach others about them

Thank-you