evaluating medical literature dr. shounak das feb. 28, 2006

Evaluating Evaluating Medical Medical

LiteratureLiterature

Dr. Shounak DasDr. Shounak Das

Feb. 28, 2006Feb. 28, 2006

ObjectivesObjectives

• Gain a basic framework for ways to use the medical literatureGain a basic framework for ways to use the medical literature

• Understand the basic steps behind analyzing any medicalUnderstand the basic steps behind analyzing any medical studystudy

• Have a step-by-step approach to analyze papers specificallyHave a step-by-step approach to analyze papers specifically geared towards therapygeared towards therapy

• Have a resource for further exploring effective use ofHave a resource for further exploring effective use of medical literaturemedical literature

(make sure I sign in so I don’t get in trouble)

(get a free lunch)

(catch up on the latest gossip)

(catch a few zzz’s)

• Show pictures of DanielShow pictures of Daniel(catch some more zzz’s)

OutlineOutline

I. OverviewI. Overview

II. Evaluation of Primary II. Evaluation of Primary StudiesStudies

Assessing TherapiesAssessing Therapies

III.III.User’s Guides to the User’s Guides to the Medical LiteratureMedical Literature

IV.IV. Pictures of DanielPictures of Daniel

I. OverviewI. Overview

I. OverviewI. Overviewa)a) Ways of using the medical Ways of using the medical

literatureliterature

b)b) Problem-solving using the Problem-solving using the medical literaturemedical literature

Ia. Ways of Using the Ia. Ways of Using the Medical LiteratureMedical Literature

• Background and foreground Background and foreground questionsquestions Background questions = background Background questions = background

medical knowledge medical knowledge • ““medical student” learningmedical student” learning• textbookstextbooks

Foreground questions = patient Foreground questions = patient management questionsmanagement questions

• ““clinician” learningclinician” learning• literature searchesliterature searches

Ia. Ways of Using the Ia. Ways of Using the Medical LiteratureMedical Literature

• Browsing and problem-solvingBrowsing and problem-solving Browsing = subscribe, skim, and readBrowsing = subscribe, skim, and read

• <10% of published studies are high quality + <10% of published studies are high quality + clinically relevantclinically relevant

• pile of journal grows until it is too intimidating pile of journal grows until it is too intimidating to deal with and is then thrown out or shelved to deal with and is then thrown out or shelved for display after which the cycle restartsfor display after which the cycle restarts

• consider secondary journals (i.e. ACP Journal consider secondary journals (i.e. ACP Journal Club)Club)

Problem-solving = specific questions Problem-solving = specific questions regarding patient managementregarding patient management

Ib. Problem-Solving Ib. Problem-Solving Using the Medical Using the Medical

LiteratureLiterature

• Frame the question:Frame the question: the populationthe population

• who are the relevant patients?who are the relevant patients? the intervention or exposurethe intervention or exposure

• diagnostic test, drug, surgical interventiondiagnostic test, drug, surgical intervention the outcomethe outcome

• patient-relevant consequences we are patient-relevant consequences we are interested ininterested in

Ib. Problem-Solving Using Ib. Problem-Solving Using the Medical Literaturethe Medical Literature

• Searching for the answer – 4 types of Searching for the answer – 4 types of clinical questions:clinical questions:

1.1. TherapyTherapy• determine the effect of a treatmentdetermine the effect of a treatment

2.2. HarmHarm• ascertain the effect of exposure to a harmful agentascertain the effect of exposure to a harmful agent

3.3. DiagnosisDiagnosis• establish the power of an intervention to establish the power of an intervention to

distinguish those with and without the target distinguish those with and without the target conditioncondition

4.4. PrognosisPrognosis• estimating the future course of a patient’s diseaseestimating the future course of a patient’s disease

Ib. Problem-Solving Using Ib. Problem-Solving Using the Medical Literaturethe Medical Literature

• Sources of evidence:Sources of evidence: PrefilteredPrefiltered

• authors have already accumulated best of authors have already accumulated best of published +/- unpublished evidence (i.e. a published +/- unpublished evidence (i.e. a systematic review)systematic review)

• examples = Best Evidence, Cochrane examples = Best Evidence, Cochrane Library, UpToDate, and Clinical EvidenceLibrary, UpToDate, and Clinical Evidence

UnfilteredUnfiltered• MEDLINE = U.S. National Library of MEDLINE = U.S. National Library of

Medicine database (contains primary Medicine database (contains primary studies and reviews)studies and reviews)

• world wide webworld wide web


Assessing Therapies Assessing Therapies

• When using the medical literature to When using the medical literature to answer a clinical question, approach answer a clinical question, approach the study using three discrete steps* :the study using three discrete steps* :

a)a) Are the results of the study valid?Are the results of the study valid?

b)b) What are the results?What are the results?

c)c) How can I apply these results to patient How can I apply these results to patient care?care?

*these 3 steps are applicable to primary studies assessing harm, diagnosis, and prognosis as well as therapies; they are also applicable for the evaluation of systematic reviews

IIa. Are the Results of the IIa. Are the Results of the Study Valid?Study Valid?

• Assessing validity requires Assessing validity requires answering 2 further questions:answering 2 further questions:

i.i. Did experimental and control Did experimental and control groups begin the study with a groups begin the study with a similar prognosis?similar prognosis?

ii.ii. Did experimental and control Did experimental and control groups retain a similar prognosis groups retain a similar prognosis after the study started?after the study started?

IIai. Did Experimental and Control IIai. Did Experimental and Control Groups Begin the Study with a Groups Begin the Study with a

Similar Prognosis?Similar Prognosis?• To determine whether or not the To determine whether or not the

experimental and control groups began experimental and control groups began the study with similar prognoses, we ask the study with similar prognoses, we ask 4 questions:4 questions:

1.1. Were patients randomized?Were patients randomized?

2.2. Was randomization concealed (blinded or Was randomization concealed (blinded or masked)?masked)?

3.3. Were patients analyzed in the groups to which Were patients analyzed in the groups to which they were randomized?they were randomized?

4.4. Were patients in the treatment and control Were patients in the treatment and control groups similar with respect to known groups similar with respect to known prognostic factors?prognostic factors?


Similar Prognosis?Similar Prognosis?

• Were patients randomized?Were patients randomized? randomization is important so that randomization is important so that

the experimental and control the experimental and control groups are matched for both groups are matched for both known and unknown prognostic known and unknown prognostic factorsfactors


Similar Prognosis?Similar Prognosis?• There are many examples of RCT’s There are many examples of RCT’s

contradicting the findings of contradicting the findings of observational studies, i.e.:observational studies, i.e.:

Clinical QuestionObservational Study Findings RCT Findings


Similar Prognosis?Similar Prognosis?• Was randomization concealed (blinded or masked)?Was randomization concealed (blinded or masked)?

Some years ago, a group of Australian investigators Some years ago, a group of Australian investigators undertook a randomized trial of open vs laparoscopic undertook a randomized trial of open vs laparoscopic appendectomy.appendectomy.

At night, the attending surgeon's presence was required At night, the attending surgeon's presence was required for the laparoscopic procedure but not the open one.for the laparoscopic procedure but not the open one.

When an eligible patient appeared, the residents checked When an eligible patient appeared, the residents checked the attending staff and the lineup for the operating room the attending staff and the lineup for the operating room and, depending on the personality of the attending and, depending on the personality of the attending surgeon and the length of the lineup, held the translucent surgeon and the length of the lineup, held the translucent envelopes containing orders up to the light. As soon as envelopes containing orders up to the light. As soon as they found one that dictated an open procedure, they they found one that dictated an open procedure, they opened that envelope.opened that envelope.

If patients who presented at night were sicker than those If patients who presented at night were sicker than those who presented during the day, the residents' behavior who presented during the day, the residents' behavior would bias the results against the open procedure. would bias the results against the open procedure.


Similar Prognosis?Similar Prognosis?

• Were patients analyzed in the groups to Were patients analyzed in the groups to which they were randomized?which they were randomized?

This principle of attributing all patients to the This principle of attributing all patients to the group to which they were randomized results in group to which they were randomized results in an intention-to-treat analysis, which is analysis of an intention-to-treat analysis, which is analysis of outcomes based on the treatment arm to which outcomes based on the treatment arm to which patients were randomized, rather than which patients were randomized, rather than which treatment they actually received. This strategy treatment they actually received. This strategy preserves the value of randomization: prognostic preserves the value of randomization: prognostic factors that we know about--and those we do not factors that we know about--and those we do not know about -- will be, on average, equally know about -- will be, on average, equally distributed in the two groups; and the effect we distributed in the two groups; and the effect we see will result simply from the treatment see will result simply from the treatment assigned. assigned.


Similar Prognosis?Similar Prognosis?• Let us assume that Let us assume that

the treatment is the treatment is ineffective and that ineffective and that the true underlying the true underlying event rate in both event rate in both treatment and treatment and control patients is control patients is 20%. 20%.

• If the 20 If the 20 nonadherent nonadherent patients are sicker patients are sicker and their event rate and their event rate (60%) is higher, the (60%) is higher, the nonadherent nonadherent patients will suffer patients will suffer 12 of the 20 events 12 of the 20 events destined to occur in destined to occur in the treated patients. the treated patients.

• If one compares only the adherent patients (with an event rate of 8/80, If one compares only the adherent patients (with an event rate of 8/80, or 10%) with the control group (event rate 20/100, or 20%), one will or 10%) with the control group (event rate 20/100, or 20%), one will mistakenly conclude that treatment cuts the event rate in half. mistakenly conclude that treatment cuts the event rate in half.

20 events

20 events

12 events

8 events

8/80 = 10% vs. 20/100 = 20%50% RRR in treatment arm

20/100 = 20% vs. 20/100 = 20%0% RRR in treatment arm


Similar Prognosis?Similar Prognosis?• Were patients in the treatment and Were patients in the treatment and

control groups similar with respect to control groups similar with respect to known prognostic factors?known prognostic factors?

Clinicians should look for a display of Clinicians should look for a display of prognostic features of the treatment and prognostic features of the treatment and control patients at the study's control patients at the study's commencement -- the baseline or entry commencement -- the baseline or entry prognostic features. Although we never prognostic features. Although we never will know whether similarity exists for the will know whether similarity exists for the unknown prognostic factors, we are unknown prognostic factors, we are reassured when the known prognostic reassured when the known prognostic factors are well balanced. factors are well balanced.


• Assessing validity requires Assessing validity requires answering 2 further questions:answering 2 further questions:

i.i. Did experimental and control Did experimental and control groups begin the study with a groups begin the study with a similar prognosis?similar prognosis?

ii.ii. Did experimental and control Did experimental and control groups retain a similar prognosis groups retain a similar prognosis after the study started?after the study started?

IIaii. Did Experimental and Control IIaii. Did Experimental and Control Groups Retain a Similar Prognosis After Groups Retain a Similar Prognosis After

the Study Started?the Study Started?

• To determine whether or not the To determine whether or not the experimental and control groups retain experimental and control groups retain a similar prognosis, we ask 4 questions:a similar prognosis, we ask 4 questions:1.1.Were patients aware of group allocations?Were patients aware of group allocations?

2.2.Were clinicians aware of group allocations?Were clinicians aware of group allocations?

3.3.Were outcome assessors aware of group Were outcome assessors aware of group allocations?allocations?

4.4.Was follow-up complete?Was follow-up complete?



• Were patients aware of group Were patients aware of group allocations?allocations? Patients who take a treatment that they Patients who take a treatment that they

believe is efficacious may feel and perform believe is efficacious may feel and perform better than those who do not, even if the better than those who do not, even if the treatment has no biologic action – the treatment has no biologic action – the placebo effect.placebo effect.

The best way to avoid the placebo effect The best way to avoid the placebo effect skewing the results is to ensure that skewing the results is to ensure that patients are unaware of whether they are patients are unaware of whether they are receiving the experimental treatment. receiving the experimental treatment.



• Were clinicians aware of group allocations?Were clinicians aware of group allocations? If randomization succeeds, treatment and control If randomization succeeds, treatment and control

groups in a study begin with a very similar groups in a study begin with a very similar prognosis. However, randomization provides no prognosis. However, randomization provides no guarantees that the two groups will remain guarantees that the two groups will remain prognostically balanced. Differences in patient prognostically balanced. Differences in patient care other than the intervention (cointerventions) care other than the intervention (cointerventions) under study can bias the results. under study can bias the results.

Effective blinding eliminates the possibility of Effective blinding eliminates the possibility of either conscious or unconscious differential either conscious or unconscious differential administration of effective (co)interventions to administration of effective (co)interventions to treatment and control groups. treatment and control groups.



• Were outcome assessors aware of Were outcome assessors aware of group allocations?group allocations? Unblinded study personnel who are Unblinded study personnel who are

measuring or recording outcomes such measuring or recording outcomes such as physiologic tests, clinical status, or as physiologic tests, clinical status, or quality of life may provide different quality of life may provide different interpretations of marginal findings or interpretations of marginal findings or may offer differential encouragement may offer differential encouragement during performance tests, either one of during performance tests, either one of which can distort results. which can distort results.



• Was follow-up complete?Was follow-up complete? The greater the number of patients who The greater the number of patients who

are lost to follow-up, the more a study's are lost to follow-up, the more a study's validity is potentially compromised. The validity is potentially compromised. The reason is that patients who are lost reason is that patients who are lost often have different prognoses from often have different prognoses from those who are retained. The situation is those who are retained. The situation is completely analogous to the reason for completely analogous to the reason for the necessity for an intention-to-treat the necessity for an intention-to-treat analysis.analysis.


• The final assessment of validity is The final assessment of validity is never a "yes" or "no" decision. never a "yes" or "no" decision. Rather, think of validity as a Rather, think of validity as a continuum ranging from strong continuum ranging from strong studies that are very likely to yield studies that are very likely to yield an accurate estimate of the an accurate estimate of the treatment effect to weak studies treatment effect to weak studies that are very likely to yield a biased that are very likely to yield a biased estimate of effect. estimate of effect.



• When using the medical When using the medical literature to answer a clinical literature to answer a clinical question, approach the study question, approach the study using three discrete steps :using three discrete steps :



c)c) How can I apply these results to How can I apply these results to patient care?patient care?

IIb. What are the Results?IIb. What are the Results?

• Once validity of a study is Once validity of a study is assessed, the next step is to assessed, the next step is to analyze the results. This analyze the results. This involves looking at:involves looking at:

i.i. How large was the treatment How large was the treatment effect?effect?

ii.ii. How precise was the estimate of How precise was the estimate of the treatment effect?the treatment effect?

IIbi. How Large Was the IIbi. How Large Was the Treatment Effect?Treatment Effect?

• Most frequently, randomized clinical trials Most frequently, randomized clinical trials carefully monitor how often patients carefully monitor how often patients experience some adverse event or outcome. experience some adverse event or outcome. Examples of these dichotomous outcomes Examples of these dichotomous outcomes ("yes" or "no" outcomes -- ones that either ("yes" or "no" outcomes -- ones that either happen or do not happen) include cancer happen or do not happen) include cancer recurrence, myocardial infarction, and recurrence, myocardial infarction, and death. Patients either do or do not suffer an death. Patients either do or do not suffer an event, and the article reports the event, and the article reports the proportion of patients who develop such proportion of patients who develop such events. events.


• Even if the outcome is not one of these Even if the outcome is not one of these dichotomous variables, investigators sometimes dichotomous variables, investigators sometimes elect to present the results as if this were the case. elect to present the results as if this were the case. For example, in a study of the use of forced For example, in a study of the use of forced expiratory volume in 1 second (FEV1) in the expiratory volume in 1 second (FEV1) in the assessment of the efficacy of oral corticosteroids in assessment of the efficacy of oral corticosteroids in patients with chronic stable airflow limitation, patients with chronic stable airflow limitation, investigators defined an event as an improvement investigators defined an event as an improvement in FEV1 over baseline of more than 20%. in FEV1 over baseline of more than 20%.

• The investigators' choice of the magnitude of The investigators' choice of the magnitude of change required to designate an improvement as change required to designate an improvement as "important" can affect the apparent effectiveness of "important" can affect the apparent effectiveness of the treatment the treatment


• When clinicians consider When clinicians consider the results of clinical the results of clinical trials, they are trials, they are interested in the interested in the association between a association between a treatment and an treatment and an outcome. outcome.

• Consider three different Consider three different treatments that reduce treatments that reduce mortality administered mortality administered to three different to three different populations.populations.

• Although all three treatments reduce the risk of dying by Although all three treatments reduce the risk of dying by a third, this piece of information is not adequate to fully a third, this piece of information is not adequate to fully capture the impact of treatment. Expressing the capture the impact of treatment. Expressing the strength of the association as a relative risk (RR), a strength of the association as a relative risk (RR), a relative risk reduction (RRR), an absolute risk reduction relative risk reduction (RRR), an absolute risk reduction (ARR) or risk difference (RD), an odds ratio (OR), or a (ARR) or risk difference (RD), an odds ratio (OR), or a number needed to treat (NNT) or number needed to number needed to treat (NNT) or number needed to harm (NNH) conveys a variety of different information.harm (NNH) conveys a variety of different information.


• In this study, the relative risk (RR) = 28% ÷ 45% = In this study, the relative risk (RR) = 28% ÷ 45% = 63%63% the RR tells us the proportion of the original risk (in this case, the the RR tells us the proportion of the original risk (in this case, the

risk of death with sclerotherapy) that is still present when patients risk of death with sclerotherapy) that is still present when patients receive the experimental treatment (in this case, ligation).receive the experimental treatment (in this case, ligation).

• Relative risk reduction (RRR) = (45% - 28%) ÷ 45% Relative risk reduction (RRR) = (45% - 28%) ÷ 45% = = 37%37% the RRR tells us the proportion of baseline risk that is removed by the RRR tells us the proportion of baseline risk that is removed by

the therapythe therapy using nontechnical language, we would say that ligation decreases using nontechnical language, we would say that ligation decreases

the relative risk of death by 37% compared to sclerotherapy. the relative risk of death by 37% compared to sclerotherapy.

(28%)

(45%)


• The absolute risk reduction (ARR) = 45% – 28% = The absolute risk reduction (ARR) = 45% – 28% = 16%16% the ARR tells us what proportion of patients are spared the the ARR tells us what proportion of patients are spared the

adverse outcome if they receive the experimental therapy, adverse outcome if they receive the experimental therapy, rather than the control therapyrather than the control therapy

• The number needed to treat (NNT) = 1/ARR = The number needed to treat (NNT) = 1/ARR = 66 the number of patients one would need to treat to prevent an the number of patients one would need to treat to prevent an

adverse eventadverse event

(28%)

(45%)


• The odds ratio (OR) = 18/46 ÷ 29/36 = The odds ratio (OR) = 18/46 ÷ 29/36 = 0.490.49 the (OR) represents the proportion of patients with the (OR) represents the proportion of patients with

the target event divided by the proportion without the target event divided by the proportion without the target eventthe target event

in most instances in medical investigation, odds in most instances in medical investigation, odds and risks are approximately equal and risks are approximately equal


• Relative Risk and Odds Ratio vs Relative Risk and Odds Ratio vs Absolute Risk Reduction: Why the Absolute Risk Reduction: Why the Fuss?Fuss? distinguishing between OR and RR will distinguishing between OR and RR will

seldom have major importance seldom have major importance we must pay much more attention to we must pay much more attention to

distinguishing between the OR and RR distinguishing between the OR and RR vs the ARR (or its reciprocal the NNT) vs the ARR (or its reciprocal the NNT)


(ARR)

• Forrow and colleagues demonstrated that Forrow and colleagues demonstrated that clinicians were less inclined to treat patients clinicians were less inclined to treat patients after presentation of trial results as the after presentation of trial results as the absolute change in the outcome compared absolute change in the outcome compared with the relative change in the outcome.with the relative change in the outcome.

• The pharmaceutical industry's awareness of The pharmaceutical industry's awareness of this phenomenon may be responsible for their this phenomenon may be responsible for their propensity to present physicians with propensity to present physicians with treatment-associated relative risk reductions. treatment-associated relative risk reductions.

IIb. What are the Results?IIb. What are the Results?

• Once validity of a study is Once validity of a study is assessed, the next step is to assessed, the next step is to analyze the results. This analyze the results. This involves looking at:involves looking at:

i.i. How large was the treatment How large was the treatment effect?effect?

ii.ii. How precise was the estimate of How precise was the estimate of the treatment effect?the treatment effect?

IIbii. How Precise Was the IIbii. How Precise Was the Estimate of the Treatment Estimate of the Treatment

Effect?Effect?• Realistically, the true risk reduction Realistically, the true risk reduction

can never be known. The best we have can never be known. The best we have is the estimate provided by rigorous is the estimate provided by rigorous controlled trials, and the best estimate controlled trials, and the best estimate of the true treatment effect is that of the true treatment effect is that observed in the trial. This estimate is observed in the trial. This estimate is called a point estimate, a single value called a point estimate, a single value calculated from observations of the calculated from observations of the sample that is used to estimate a sample that is used to estimate a population value or parameter.population value or parameter.


Effect?Effect?• Investigators often tell us the Investigators often tell us the

neighborhood within which the true neighborhood within which the true effect likely lies by calculating effect likely lies by calculating confidence intervals.confidence intervals. We usually (though arbitrarily) use the We usually (though arbitrarily) use the

95% confidence interval. 95% confidence interval. You can consider the 95% confidence You can consider the 95% confidence

interval as defining the range that interval as defining the range that includes the true relative risk reduction includes the true relative risk reduction 95% of the time. 95% of the time.


Effect?Effect?

• In this hypothetical situation, 2 different-sized In this hypothetical situation, 2 different-sized studies with the same RRR of 25% have widely studies with the same RRR of 25% have widely varying confidence intervals.varying confidence intervals.

• It is evident that the larger the sample size, the It is evident that the larger the sample size, the narrower the confidence interval. narrower the confidence interval.

This study enrolled 1000 patients

This study enrolled 100 patients

(RRR = 0.2595% confidence intervals 0.09 – 0.41)(RRR = 0.25

95% confidence intervals -0.38 – 0.59)


Effect?Effect?• When is the sample size big enough?When is the sample size big enough?

In a positive study -- a study in which the authors In a positive study -- a study in which the authors conclude that the treatment is effective -- if the conclude that the treatment is effective -- if the lowest RRR that is consistent with the study results lowest RRR that is consistent with the study results is still important (that is, it is large enough for you is still important (that is, it is large enough for you to recommend the treatment to the patient), then to recommend the treatment to the patient), then the investigators have enrolled sufficient patients.the investigators have enrolled sufficient patients.

In a negative study -- in which the authors have In a negative study -- in which the authors have concluded that the experimental treatment is no concluded that the experimental treatment is no better than control therapy -- if the RRR at the better than control therapy -- if the RRR at the upper boundary would, if true, be clinically upper boundary would, if true, be clinically important, the study has failed to exclude an important, the study has failed to exclude an important treatment effect (i.e. has not enrolled important treatment effect (i.e. has not enrolled sufficient patients).sufficient patients).



• When using the medical When using the medical literature to answer a clinical literature to answer a clinical question, approach the study question, approach the study using three discrete steps :using three discrete steps :



c)c) How can I apply these results to How can I apply these results to patient care?patient care?

• Before embarking on therapy based Before embarking on therapy based on a study for a particular patient, ask on a study for a particular patient, ask 3 questions:3 questions:

1.1. Were the study patients similar to the Were the study patients similar to the patient in my practice?patient in my practice?

2.2. Were all clinically important outcomes Were all clinically important outcomes considered?considered?

3.3. Are the likely treatment benefits worth Are the likely treatment benefits worth the potential harm and costs?the potential harm and costs?

IIc. How Can I Apply IIc. How Can I Apply These Results to Patient These Results to Patient

Care?Care?

IIc. How Can I Apply These IIc. How Can I Apply These Results to Patient Care?Results to Patient Care?

• Were the study patients similar to the patient Were the study patients similar to the patient in my practice?in my practice?

If the patient had qualified for enrollment in the If the patient had qualified for enrollment in the study -- that is, if she had met all inclusion criteria study -- that is, if she had met all inclusion criteria and had violated none of the exclusion criteria -- and had violated none of the exclusion criteria -- you can apply the results with considerable you can apply the results with considerable confidence. Even here, however, there is a confidence. Even here, however, there is a limitation. Treatments are not uniformly effective limitation. Treatments are not uniformly effective in every individual. Conventional randomized trials in every individual. Conventional randomized trials estimate average treatment effects. Applying these estimate average treatment effects. Applying these average effects means that the clinician will likely average effects means that the clinician will likely be exposing some patients to the cost and toxicity be exposing some patients to the cost and toxicity of the treatment without benefit. of the treatment without benefit.

A better approach than rigidly applying the study's A better approach than rigidly applying the study's inclusion and exclusion criteria is to ask whether inclusion and exclusion criteria is to ask whether there is some compelling reason why the results there is some compelling reason why the results should not be applied to the patient. should not be applied to the patient.


• Were all clinically important outcomes considered?Were all clinically important outcomes considered? Treatments are indicated when they provide important Treatments are indicated when they provide important

benefits. Demonstrating that a bronchodilator produces benefits. Demonstrating that a bronchodilator produces small increments in forced expired volume in patients small increments in forced expired volume in patients with chronic airflow limitation does not provide a with chronic airflow limitation does not provide a sufficient reason for administering this drug. What is sufficient reason for administering this drug. What is required is evidence that the treatment improves required is evidence that the treatment improves outcomes that are important to patients, such as outcomes that are important to patients, such as reducing shortness of breath during the activities reducing shortness of breath during the activities required for daily living. We can consider forced expired required for daily living. We can consider forced expired volume as a substitute or surrogate outcome. volume as a substitute or surrogate outcome. Investigators choose to substitute these variables for Investigators choose to substitute these variables for those that patients would consider important, usually those that patients would consider important, usually because they would have had to enroll many more because they would have had to enroll many more patients and follow them for far longer periods of time. patients and follow them for far longer periods of time.


• Were all clinically important outcomes Were all clinically important outcomes considered?considered?


• Were all clinically important Were all clinically important outcomes considered?outcomes considered?

Even when investigators report favorable Even when investigators report favorable effects of treatment on one clinically effects of treatment on one clinically important outcome, you must consider important outcome, you must consider whether there may be deleterious effects whether there may be deleterious effects on other outcomes. For example a on other outcomes. For example a cancer chemotherapeutic agent may cancer chemotherapeutic agent may lengthen life but decreases its quality. lengthen life but decreases its quality.

IIc. How Can I Apply IIc. How Can I Apply These Results to Patient These Results to Patient

Care?Care?• Are the likely treatment benefits worth Are the likely treatment benefits worth

the potential harm and costs?the potential harm and costs? If you can apply the study's results to a If you can apply the study's results to a

patient, and its outcomes are important, patient, and its outcomes are important, the next question concerns whether the the next question concerns whether the probable treatment benefits are worth the probable treatment benefits are worth the effort that you and the patient must put effort that you and the patient must put into the enterprise.into the enterprise.

the NNT can help clinicians judge the the NNT can help clinicians judge the degree of benefit and the degree of harm degree of benefit and the degree of harm patients can expect from therapy.patients can expect from therapy.


• Are the likely treatment benefits worth the Are the likely treatment benefits worth the potential harm and costs?potential harm and costs?

As a result of taking aspirin, patients with As a result of taking aspirin, patients with hypertension without known coronary artery disease hypertension without known coronary artery disease can expect a reduction of approximately 15% in can expect a reduction of approximately 15% in their relative risk of cardiovascular related events.their relative risk of cardiovascular related events.

For an otherwise low-risk woman with hypertension For an otherwise low-risk woman with hypertension and a baseline risk of cardiovascular related event and a baseline risk of cardiovascular related event of between 2.5% and 5%, this translates into an of between 2.5% and 5%, this translates into an NNT of approximately 200 during a 5-year period.NNT of approximately 200 during a 5-year period.

For every 161 patients treated with aspirin, one For every 161 patients treated with aspirin, one would experience a major hemorrhage (NNH = would experience a major hemorrhage (NNH = 161). 161).

Thus, in 1000 of these patients, aspirin would be Thus, in 1000 of these patients, aspirin would be responsible for preventing five cardiovascular responsible for preventing five cardiovascular events, but it would also be responsible for causing events, but it would also be responsible for causing approximately six serious bleeding episodes. approximately six serious bleeding episodes.


• Are the likely treatment benefits worth Are the likely treatment benefits worth the potential harm and costs?the potential harm and costs?

Trading off benefit and risk requires an Trading off benefit and risk requires an accurate assessment of medication adverse accurate assessment of medication adverse effects. Although RCTs are the correct effects. Although RCTs are the correct vehicle for reporting commonly occurring vehicle for reporting commonly occurring side effects, reports regularly neglect to side effects, reports regularly neglect to include these outcomes. Clinicians must include these outcomes. Clinicians must often look to other sources of information – often look to other sources of information – often characterized by weaker methodology – often characterized by weaker methodology – to obtain an estimate of the adverse effects of to obtain an estimate of the adverse effects of therapy.therapy.

The preferences or values that determine the The preferences or values that determine the correct choice when weighing benefit and correct choice when weighing benefit and risk are those of the individual patient. risk are those of the individual patient.



• Are the results valid?Are the results valid? Randomized?Randomized? Blinded?Blinded? Intention to treat analysis?Intention to treat analysis? Follow-up complete?Follow-up complete?

• What are the results?What are the results? How large was the treatment effect (ARR/NNT)?How large was the treatment effect (ARR/NNT)? How precise was the estimate (confidence intervals)?How precise was the estimate (confidence intervals)?

• How can I apply these results to patient care?How can I apply these results to patient care? How is my patient different from the study patient?How is my patient different from the study patient? Was the outcome measured clinically significant (surrogate Was the outcome measured clinically significant (surrogate

outcomes), and were all clinically significant outcomes outcomes), and were all clinically significant outcomes considered?considered?

What is the risk of treating vs. the risk of not treating What is the risk of treating vs. the risk of not treating (benefit vs. harm)?(benefit vs. harm)?

SUMMARY

III. User’s Guides to the III. User’s Guides to the Medical LiteratureMedical Literature

• The “User’s Guides to the Medical The “User’s Guides to the Medical Literature” is a valuable resource. Literature” is a valuable resource. Please consult this for analyzing studies Please consult this for analyzing studies on harm, diagnosis, prognosis, and on harm, diagnosis, prognosis, and systematic reviews. The principles are systematic reviews. The principles are the same as presented today, but the the same as presented today, but the steps are different.steps are different.

• There is a direct link to the “User’s There is a direct link to the “User’s Guides” from the JAMA home page – Guides” from the JAMA home page – www.jama.ama-assn.orgwww.jama.ama-assn.org

IV. Pictures of DanielIV. Pictures of Daniel

evaluating medical literature dr. shounak das feb. 28, 2006

Documents