Interpreting the Medical Literature: A real example

Sandra E Moore, MD MSc FAAPAssistant Professor of Clinical Pediatrics

Morehouse School of Medicine

Objectives

• To determine how to critically evaluate the literature

• To determine how to choose an appropriate questions to study

Where do you Start• Start with the patient -- a clinical problem or question arises out of

the care of the patient

• The question - Construct a well built clinical question derived from the case

• The resource - Select the appropriate resource(s) and conduct a search

• The evaluation - Appraise that evidence for its validity (closeness to the truth) and applicability (usefulness in clinical practice)

• The patient - Return to the patient and integrate that evidence with clinical expertise, patient preferences and apply it to practice

• Self-evaluation - Evaluate your performance with this patient

Background questions

• Information about a disease or process that you should already know or can readily access– Who– What– When– Why– Where and– How

Foreground questions

• compares two or more things/groups– Drugs – Treatment modalities– Groups based on exposures– Diagnostic tests or – The harms or benefits of two approaches

PICO format

• P - Patient / Problem• What are the characteristics of the patient or population?• What is the condition or disease you are interested in?

• I - Intervention (or exposure)• What do you want to do with this patient (e.g. treat,

diagnose, observe)?

• C – Comparison (or Control), if any• What is the alternative to the intervention (e.g. placebo,

different drug, surgery)?

• O – Outcome• What are the relevant outcomes (e.g. morbidity, death,

complications)?

PP -ICONS

• P – Problem • P - Patient / Problem• I - Intervention (or exposure)• C - Comparison (or Control), if any• O – Outcome (Disease - orientated Outcomes;

DOEs or Patient Orientated Evidence that Matters; POEMs)

• N - Number of Subjects• S - Statistics

Intervention Studies

• Is the study valid?

• What are the results? and

• Is it applicable to the patient?

Clinical Scenario

• You are in Continuity Clinic• Mother brings a 9 year old daughter in

because of warts on hand. Mother read about the efficacy of duct tape in treating warts and wanted to know your opinion

Background Questions about Warts

– Who?– What?– When?– Why?– Where? and– How?

What is our Question?

• P (problem/patient)• Intervention• Control or comparisons • Outcomes

• children and warts• duct tape• nothing or other

therapies• resolution of warts

Compared to standard treatment, is duct tape effective for the treatment (resolution) of warts in children?

PP –ICONS for Study

• P – Problem • P - Patient / Problem• I - Intervention (or exposure)• C - Comparison (or Control), if any• O – Outcome (Disease orientated Outcomes;

DOEs or Patient Orientated Evidence that Matters;POEMs)

• N - Number of Subjects• S - Statistics

Search in Appropriate Database

• Medline• Pubmed• Cochrane• InfoPOEMS (Patient-

Oriented Evidence that Matters)

• UpToDate• Md Consult• MSM Library

• Focht DR III, et al. The efficacy of duct tape vs cryotherapy in the treatment of verruca vulgaris (the common wart). Arch Pediatr Adolesc Med October 2002;156:971-4.

Is duct tape effective for the treatment (resolution) of warts in children? Focht DR et.al

• Objective To determine if application of duct tape is as effective as cryotherapy in the treatment of common warts.

• Design A prospective, randomized controlled trial with 2 treatment arms for warts in children.

• Patients A total of 61 patients (age range, 3-22 years) were enrolled in the study from October 31, 2000, to July 25, 2001; 51 patients completed the study and were available for

analysis.

• Intervention Patients were randomized using computer-generated codes to receive either cryotherapy (liquid nitrogen applied to each wart for 10 seconds every 2-3 weeks) for a maximum of 6 treatments or duct tape occlusion (applied directly to the wart) for a maximum of 2 months. Patients had their warts measured at baseline and with return visits.

• Main Outcome Measure Complete resolution of the wart being studied.

• Results Of the 51 patients completing the study, 26 (51%) were treated with duct tape, and 25 (49%) were treated with cryotherapy. Twenty-two patients (85%) in the duct tape arm vs 15 patients (60%) enrolled in the cryotherapy arm had complete resolution of their warts (P = .05 by 2 analysis). The majority of warts that responded to either therapy did so within the

first month of treatment.

• Conclusion Duct tape occlusion therapy was significantly more effective than cryotherapy for treatment of the common wart.

Is this a clinical relevant question?

• Warts are common in children• Quick, effective, and inexpensive treatment is

not available ( maybe salicylic acid)• Although warts are medically benign, they are

unsightly and may cause a child to feel self-conscious

• Parents/children often seek medical therapy• Up to 30% resolve in 10 weeks without therapy

Are the results of this therapy (intervention) study valid?

• Was the assignment of patients to treatment randomized?

• Were all the patients who entered the trial properly accounted for at its conclusion?

• Were patients analyzed in the groups to which they were (originally) randomized?

• Where there enough patients (N)?• Where the statistical test appropriate?• Were the results statistically significance?

Was the assignment of patients to treatment randomized?

– Random allocation comes closest to insuring the creation of groups of patients who will be similar in their risk of the events you hope to prevent.

– Randomization balances the groups for prognostic factors (ie. disease severity) which eliminates over-representation of any one characteristic within the study groups.

– Randomization should also be concealed from the clinicians and researchers of the study to help eliminate conscious or unconscious bias.

– Were the patient randomized in the duct tape study?

Number of Subjects (N)• The number of subjects is crucial to whether

accurate statistics can be generated from the data.

• Too few patients in a research study may not be enough to show that a difference actually exists between the intervention and comparison groups (known as the "power" of a study).

• Many studies are published with less than 100 subjects, which is usually inadequate to provide reliable statistics.

• A good rule of thumb is 400 subjects

Did the study have sufficient power?• The power of a test refers to its ability to detect what

it is looking for. The probability of correctly rejecting the null hypothesis (i.e. the probably of finding a difference if it truly exist)– Alpha (): Usually set to be 0.05, although this is somewhat arbitrary.

This is the probability of a type I error, that is the probability of rejecting the null hypothesis given that that the null hypothesis is true. In other words it is the probability of thinking we have found something when it is not really there.

– Power = 1 – Beta (), where beta is the probability of a type II error (acceptance of a false null hypothesis). Typically 80% (0.8) is considered adequate power.

– Typically increase power by increasing sample size– No mention of power, or in duct tape study

Were all the patients who entered the trial properly accounted for at its conclusion?

• All patients who started the trial should be accounted for at the end of the trial. If patients are not accounted for, the validity of the study may be jeopardized.

• A good study will have better than 80% follow-up for their patients.

• Patients may drop out of a study for various reasons. If these patients are not included in the results, they can make the treatment look better than it really is (and vice versa).

• To be sure of a study's conclusions, lost patients should be assigned to the "worst-case" outcomes and the results recalculated. The results are still valid if the recalculations do not change the end results. This is referred to as “Intention to Treat”

Type of Data

Goal Measurement (from Gaussian Population)

Rank, Score, or Measurement (from Non- Gaussian Population)

Binomial (Two Possible Outcomes)

Survival Time

Describe one group

Mean, SD Median, interquartile range

Proportion Kaplan Meier survival curve

Compare one group to a hypothetical value

One-sample t test

Wilcoxon test Chi-square or Binomial test **

Compare two unpaired groups

Unpaired t test Mann-Whitney test Fisher's test (chi-square for large samples)

Log-rank test or Mantel-Haenszel*

Compare two paired groups

Paired t test Wilcoxon test McNemar's test

Conditional proportional hazards regression*

Compare three or more unmatched groups

One-way ANOVA

Kruskal-Wallis test Chi-square test

Cox proportional hazard regression**

Compare three or more matched groups

Repeated-measures ANOVA

Friedman test Cochrane Q** Conditional proportional hazards regression**

Quantify association between two variables

Pearson correlation

Spearman correlation

Contingency coefficients**

Predict value from another measured variable

Simple linear regression or Nonlinear regression

Nonparametric regression**

Simple logistic regression*

Cox proportional hazard regression*

Predict value from several measured or binomial variables

Multiple linear regression* or Multiple nonlinear regression**

Multiple logistic regression*

Cox proportional hazard regression*

Are The Statistics Appropriate?

Statistical Test in our study• Relative risk reduction (RRR): the

percent reduction in events in the treated group compared to the control group event rate.

– Not a good way to compare outcomes– Amplifies small differences and makes

insignificant findings appear significant– Doesn’t reflect the baseline risk of the

outcome event– Can make weak results look good– Popular and will be reported in almost

every journal article– Can mislead you

• RRR would be (85 percent – 60 percent) / (60 percent) x 100 = 42%

• i.e. duct tapes appears to be 42% more effective than cryotherapy in treating warts

• Absolute risk reduction (ARR): the difference in the outcome event rate between the control group and the experimental group.

– A better statistic to evaluate outcome, as it does not amplify small differences, but shows the true difference between the experimental and control interventions

• ARR for the wart study is the outcome event rate (complete resolution of warts) for duct tape (85 percent) minus the outcome event rate for cryotherapy (60 percent) = 25 percent

Statistical Significance

• Are the result statistically significant?– Statistical test p < 0.05– Confidence interval (CI) should not include “0”– Statistical significance DOES NOT mean clinical significance

– Results of Duct tape study had a p value < .05– CI not listed– The calculated 95% CI for this study's reported treatment effect is 1.1

to 48.1, and we can state with 95% confidence that the true treatment effect is somewhere between these 2 values.

– Maximally conservative estimate, including patients lost to follow-up, CI contains 0, and therefore difference not statistically significant

Key terminology for estimating the size of the treatment effect

 Outcome  Risk of outcome

+ -

Treated (Y) a b Y = a/(a + b)

Control (X) c d

X = c/(c + d)

•Relative Risk (RR) is the risk of the outcome in the treated group (Y) compared to the risk in the control group. = Y / X

•Relative Risk Reduction (RRR) is the percent reduction in risk in the treated group (Y) compared to the control group (X) = 1 - Y / X x 100%

•Absolute Risk Reduction (ARR) is the difference in risk between the control group (X) and the treatment group (Y) = X – Y

•Number Needed to Treat (NNT) is the number of patients that must be treated over a given period of time to prevent one adverse outcome = 1 / (X - Y)

Intention to Treat (ITT)• Intention to treat: subjects are analyzed according to the

categories into which they were originally randomized. – Assumes worse case scenario– Benefits of a treatment are more difficult to demonstrate with intention-to-treat

analysis– Helps to mitigate differences by including subjects who are unlikely to have

experienced benefit from the intervention

• Six patients from cryotherapy group and 4 patients from the duct tape group were lost to follow-up (16% of patients).

• Worst case scenario: 6 cryotherapy patients had wart resolution and the 4 duct tape patients had residual wart.

• Wart resolution would then be: duct tape 78% and cryotherapy 68% (95% CI, -17 to 28) – therefore not a statistically significant difference between the two treatments.

Number needed to treat (NNT)

• Number needed to treat (NNT): number of patients who must be treated to prevent one adverse outcome OR the number of patients who must be treated for one patient to benefit

– = 1/ARR. In the case of the duct tape study 1/.25 = 4

– The lower the NNT the better (intervention studies 10 is good, 5 is excellent; for preventive studies 20 is good)

Interpreting our Wart Study• Is this an important clinical question?• Are the patients studied similar to our patient?• Was the intervention acceptable?• Were the outcomes clinically relevant?• Was the assignment of patients to treatment randomized? • Were all the patients who entered the trial properly accounted for at its

conclusion? • Were patients analyzed in the groups to which they were (originally)

randomized? • Where there enough patients (N)?• Are the results statistically significance? • Are the results clinically significant?• Is the NNT appropriate?

Back to the patient

• Would you recommend duct tape to your patient?

Introduction, Background and Methodology No Yes COMMENTSStated the study objective in his/her own words      

Described the significance of the problem addressed by the study

  

  

  

Described how the study objectives and /or hypotheses are relevant to community and/or patient care

     

Described the study design (methodology)      

Described the population studied      

Described the setting in which the study occurred    

Described the statistical analysis plan      

       

RESULTS SECTION      

Able to state major outcome (s) that are reported      

Able to summarize demographic of population studied      

Explained how the major study endpoints are relevant to the population studied

     

Presented the main results of the study with comparison to the original hypotheses.

     

CONCLUSION SECTION  

Described the study conclusions as presented by the authors.

     

Stated their evaluation and clinical significance of study results

     

Identified limitation of the study      

Identified how the results could be applied to his/her patients

     

ADDITIONAL COMMENTS

REFERENCES• The efficacy of duct tape vs cryotherapy in the treatment of verruca vulgaris (the

common wart). Focht DR et al Arch Pediatr Adolesc Med. 2002;156:971-974 • Krejcie RV, Morgan DW. Determining sample size for research activities. Educational

and Psychological Measurement. 1970;30:607-610.• Is Duct Tape Occlusion Therapy as Effective as Cryotherapy for the Treatment of the

Common Wart? Ringold et al Arch Pediatr Adolesc Med. 2002;156(10):975-977.

• Efficacy of Duct Tape vs Placebo in the Treatment of Verruca Vulgaris (Warts) in Primary School Children. de Haen et al. Arch Pediatr Adolesc Med 2006;160:1121-1125

• Interpreting negative results from an underpowered clinical trial: warts and all. Van Cleave et al. Arch Pediatr Adolesc Med 2006;160:1126-1129

• Studies Should Report Estimates of Treatment Effects With Confidence Intervals. Cummings Arch Pediatr Adolesc Med 2007;161:518-519.

•

Online References/Tools

• http://www.med.umich.edu/pediatrics/ebm/Jcguide.htm• http://southmed.usouthal.edu/library/ebmclass/

rotationswinterspring.htm• http://www.hsl.unc.edu/services/tutorials/ebm/

Evidence.htm• http://www.aafp.org/fpm/20050700/37howt.html• http://healthlinks.washington.edu/ebp/pico.html• http://www.aafp.org/fpm/20040500/47asim.html• http://www.jeremymiles.co.uk/misc/power/• http://www.graphpad.com/www/book/Choose.htm