Professional faking in treatment studies – the best

prevention is to instruct everybody how to do it

Franz Porzsolt MD, PhDClinical Economics, University of UlmD-89075 Ulm, Germany

www.uniklinik-ulm.de/clinecs

Rio de Janeiro, August 28-30, 2010

I Congresso Basileiro De Pratica Clinica Basada em Evidencias

Professional Faking in Treatment Studies If everybody knows how to do it

The number of false studies will decrease The number of useless treatments will be

reduced The quality of health care will increase The efficiency of healthcare will increase The satisfaction or patients and health care

professionals will increase This course should be mandatory to any

young physician in our countries

Opinions are based on information Information is transferred by

Reading scientific literature Participation in training-programs Participation in quality circles Verbal communication

General remark The most populare mistake: The study

question is not defined as precisely as possible.

Be suspicious if The study protocol had to be chanched

during the course of the study Complicated phrases or explanations are

used which are difficult to understand Unknown „technical terms“ are introduced

to present or discuss the results.

Specific remarks (take 10 min of your time to analyse the validity of your references) Goal of the study

Does this study have a particular goal? How can we find out?

Srat

egy

& ta

ctic

s

The Italian Plan: Iron defense, small presencein midfield, passes to striker.. and…Penalty

Stra

tegi

e &

Tak

tik

The German PlanRadical, ball may reach speed of 297 km/h

Srat

egy

& ta

ctic

s

Srat

egy

& ta

ctic

s

The Brazilian PlanNot as clear as the German plan but not less effective...

Specific remarks (take 10 min of your time to analyse the validity of your references) Goal of the study

Does this study have a particular goal or a hypothesis?

How can we find out? Was the goal of the trial defined prospectively

or retrospectively („sharpshooter“; check plausibility and compare with previous information on that topic; „Beckenbauer-trial“ >> not contaminated with a hypothesis >> „sterile“)

Question of the study Is the study question consistent within the

paper (Title, summary, introduction, discussion) Design

What would be the ideal study design to answer the study question?

Identify the study question and design YOUR OPTIMAL DESIGN to answer this study question

Prepare a graph of the published study design. Compare the published study design with your

„optimal study design“ If these are different, could you think of any

reasons which may explain the difference?

Conduct I Are there any indicators for a retrospective defini-tion

of the inclusion and exclusion criteria? Age [23-87 yrs], laboratory data (BSR) [15-124 mm/hr],

time intervals [0-14 months after surgery] Were the patients randomly allocated to the groups

(was it a RCT)? Compare double publications of same data (randomized

and non-randomized) Was the randomization concealed?

Consider the risk of monocentric trials … Were adequate controls included in the study?

Examples: Immunotherapies, Zelen-Randomization, Waber experiment

TabletsTablets

Placebo ClofibrateCompliance with medication

# ofPatient

5 year mortality

# of patients

5 year mortality

RRR ARR NNT

All patientsTook => 80%Took < 80%

2789 ~2/3 ~1/3

20.9% 15.1% 28.3%

1103 ~2/3 ~1/3

20.0% 15.0% 24.6%

4% 0.9% 111 0% 0.1% 1000 13% 3.7% 27

RRRARRNNT

46.6% 13.2% 8

39.0% 9.6% 10

EFFECT OFCOMPLIANCE ON MORTALITY. NEJM 1980;303(18):1038-1042

Re-interpretation ...

Randomizationaccording to Zelen II

IC

R

Exp Con

yes or no not asked

The „inverted vector“

A B C D E F

Responserate

Experimentals

Controls

Conduct II Were doctors and patients blindet?

Could „deblinding“ be prevented? Were all patients who participated in the study

in-cluded in in the evaluation of the study evaluation?

Which patients have the highest risk to be excluded from a trial?

Was the study closed before the scheduled end-point?

Is the closing rule of a clinical trial comparable to the rules a socker game, or a horse race or a tennis match?

Report / Interpretation Are the recommendations derived from the

treat-ment confined to the investigated population?

Remember the inclusion- and exclusion criteria Compare the conclusions of the study with the in-ex-

criteria Are the results of this study clinically relevant?

Will your patients accept the side effects of the treatment to receive this „lousy“ benefit?Consider patients‘ values and preferences!

Kornmann M, Porzsolt F. Treatment Preferences of Physicians and Lay Persons: Lessons

from a Study Analysing Neoadjuvant Treatment of Rectal Carcinoma. J Clin Oncol 2008;26:2866-4868

Treatment related deathLocal relapse

Distant metastatses

Second cancer

Stool incontinence

Alle

Geh

eilte

Pat

ient

en

Cardiovascular compl.

Survival no difference

Report / Interpretation Is the principle of „perceived safety“ considered“

Is „perceived safety“ ethically justified in this situation (talking about possible cure in an incurable cancer??)

Is the application of the the new principle justified when ethical, medical, epidemiological, and economic aspects are considered?

Further readings Evidence Based Medicine. How to practice

and teach EBM. Churchill Livingstone 3 ed. 2005Portugese translation by ARTMED editors

There will be many more updated books in Portugese language on EBM …

Further thinking For 96% of the population (incluing myself) it is

rather difficult to use „conditional probabilities“

Example of the „GOAT PROBLEM“

Solution afterAberdura

Summary Physicians and managers have to know how

to to identify valid information. The methods to identify ‚at a glance‘ are

easy to adapt. Don‘t try it by yourself – find friends to form

small groups Try to get feedback from experienced people

by joining a Brasilian network of EBM