interpreting clinical trial results betty anne schwarz msc, ba, rn ohri clinical research conference...

Interpreting Clinical Trial ResultsBetty Anne Schwarz MSc, BA, RNOHRI Clinical Research Conference Workshop

October 16, 2012

Interpreting Clinical Trial Results

“I have no conflicts of interest related to this presentation”


Workshop Applicability Anyone who is working in clinical trials that is accustomed to

reading published literature (RCTs)


Objectives Review what is considered statistically significant when interpreting

clinical trial results Illustrate how statistically significant is not the same as clinically

significant Interactive workshop will involve participation of the audience

through review of specific trials to determine clinical significance with the tools provided


Outline of Workshop

I. Review statistical features that are routinely expected within published trial results

II. Focus on the merits of considering MCID (minimally clinical important difference)

Look at examples of positive vs. negative trials

III. Break into groups – utilize tools provided in reviewing specific

trials to determine clinical significance

IV. Review and Summarize


Statistics & Clinical Trials Interpreting trials – depends on level of comprehension and

experience

I. Key Statistical Terms Choice of statistical test utilized to analyze data (interval or

categorical, paired vs. unpaired) depends on the data being analyzed

Null hypothesis proposes ~ no difference between study groups with respect to variable's of interest

Ref: (2007) McCluskey & Ghaaliq Lalkhen


I. Key Statistical Terms Unpaired vs. Paired Data ~ compare effects of an intervention within

study samples imperative groups are as similar as possible Randomization – large enough sample size Ensures group differences within the groups cancel out as they may

influence outcome of interest

(weight, age, sex ratio, smoking habit) Study contained independent groups – unpaired statistical tests i.e., comparison of efficacy for two different drugs – HTN trial



I. Statistical vs. clinical significance One should not be confused by another

Scenario Comparison of 2 hypotensive agents: mean arterial pressure after

Rx. with drug A is 2 mm lower < Rx. with drug B. If sample size is not large enough even a small difference between

the 2 groups may be statistically significant with a P <0.05. The clinical significance of a 2 mm Hg drop in mean arterial

pressure is small and therefore not clinically significant



I. “Statistical Significance” and P Values “Not everything that can be counted counts, and not everything that

counts can be counted” Albert Einstein Statistically significant simply means that a result is most likely

caused by something other than chance In other words ~ significant does not mean important Segue into what’s considered clinically significant Interpreting trial results requires a P value (conventional P < 0.05

statistically significant). P ≥ 0.05 not significant, P <0.01 highly significant

Ref: (2004) libdoc.who.int.,(2007) McCluskey & Ghaaliq Lalkhen,


I. Confidence Intervals (CI) Range of sample data measures the probability that a population

parameter will fall between two set values. The confidence interval can take any number of probabilities, with the most common being 95% or 99%

CI should always be presented for the relative risk (RR) and odds ratio (OR)

A RR or OR of 1 ~ no association between the risk factors and the disease under study

If RR > 1, but CI overlaps 1 – the increase in risk is not statistically significant and can attributed to chance

Ref: (2004) libdoc.who.int


I. Important Variables and Concepts Study power ~ Type I and Type II errors After data has been analyzed, the null hypothesis is either accepted or rejected

based on the P value If the null hypothesis is true and the P value (P < 0.05) is obtained; incorrect

inference is drawn that the data collected from the sample groups is different : TYPE I STATISTICAL ERROR

From the data collected conclude null hypothesis is false but the P value obtained is ≥0.05 ~ conclude sample groups are similar but in fact missed a real difference: TYPE II STATISTICAL ERROR


Interpreting Clinical Trial ResultsI. Important Variables and Concepts Interpreting diagnostic tests ~ sensitivity and specificity Comparing to gold standard or reference standard Sensitivity – ability of a test to single out people who have disease. Low sensitivity ~ many false positives Specificity – ability of a test to identify people who do not have a disease (negative). A low sensitivity ~ many false positives Predictive Value (PV) – the frequency in which a positive test actually signifies disease Confound Bias Ref: (2004) libdoc.who.int


Interpreting Clinical Trial ResultsII. MCID (minimally clinical important difference) Interpretation of RCTs emphasis on statistical significance rather than clinical relevance CONSORT (Consolidated Standards of Reporting Trials) statement developed to help authors improve their methods in

reporting trials results with the aid of a checklist and flow diagram Enable readers to comprehend the trial methodology and assess the validity of the results Recently revised based on new evidence and address criticism

Ref: (2001) Chan et al., (2001) Moher et al.



II. MCID (minimally clinical important difference) CONSORT revised – failed to recommend that authors specifically discuss the clinical importance of their results Chan et al., (2001) randomly chose 27 (total 266) published RCTs from 5 major journals over a 1 year time period that were

independently reviewed by 4 reviewers to discern if factors they considered as clinically relevant were stated in the study results




II. MCID (minimally clinical important difference) Specifically ~ primary outcome clearly defined, expected difference between the two groups reported used in the calculation of

the sample size (delta value), whether the results were based on minimal clinically important difference of the intervention, the statistical significance of the results, presentation of confidence intervals pertinent to the study findings and finally the authors overall interpretation of the clinical relevance of the trial results



Interpreting Clinical Trial ResultsII. MCID (minimally clinical important difference) Defined as the smallest treatment effect required that would render a change in the management of a patient taking into account the side

effects, costs and overall inconveniences Key concept to be considered Designing RCT Trial ~ calculate the sample size needed to detect magnitude of difference between the treatment groups that the study

can reliably detect (delta value) Key point ~ for a trial to have a reasonable chance of detecting clinically important effect size, the delta value needs to reflect the MCIDs

for the trial interventions




II. MCID (minimally clinical important difference) Example: individuals without a previous history of MI or CVA; regular use of ASA will reduce the incidence of MI by 0.2% per year (from

a baseline rate of 0.7%/year to 0.5%/year which is a RR reduction of about 25%) Benefit is offset by a concomitant absolute increase in the chance of CVA by 0.02% per year (from 0.30%/year – 0.32%/year, which is

now a relative increase of about 10%/year) and GI bleed of about 1% per year (from about 1%/year – 2%/year) Thoughts? Ref: (2001) Chan et al.,



II. MCID (minimally clinical important difference) Weigh the benefits and disadvantages of taking ASA in this clinical situation; expert panel recommended not to take ASA in the

prevention of MI as there was insufficient evidence to overcome the increased incidence of CVA, GI bleed in this low risk group. Therefore; the efficacy of ASA in this particular scenario was insufficient to either meet or exceed its MCID

Ref: (2001) Chan et al.,



II. MCID (minimally clinical important difference) Compare the actual study results ~ include point estimates and accompanying confidence intervals with the MCID values will render the

clinical importance of the study results Point estimate: can be either a single number or a range of scores. Point estimates are not usually as informative as confidence intervals. Their importance lies in the fact that many statistical formulas are

based on them




II. MCID (minimally clinical important difference) If the MCID estimate is less than the value of the lower limit of the 95% CI ~ results are statistically significant and most likely clinically

significant If the MCID value is greater than the upper limit of the 95% CI, the results are very likely to be less clinically significant If the results provided for the MCID are somewhere in the middle of the CI – the clinical importance is less clear




Methods December 1, 1998 – November 30, 1999 of 266 articles randomly chosen - 27 RCTS were selected from 5 peer reviewed journals Utilized standard data collection sheets to evaluate key features when interpreting the trial results and whether the results were

clinically significant (MCID) Disagreements were resolved collaboratively




Methods (hand-out)

1. Primary Outcome – clearly defined in the methods section and is also defined within the calculation of a sample size No sample size reported – remainder of the article was reviewed for an explicitly stated primary outcome No primary outcome – the outcome that was deemed of greatest clinical importance was chosen



Interpreting Clinical Trial ResultsMethods

2. Reporting of a Sample Size Delta Value Magnitude of difference in outcomes between treatment groups that the trial was attempting to detect Recorded as either the author’s interpretation of a MCID for the various study intervention, and whether or not the value was in absolute or relative terms

3. Statistical significance of results for the primary outcome P value < 0.05% Ref: (2001) Chan et al.,



4. Confidence Intervals Primary outcome – whether the 95% CI surrounded the point estimate for the efficacy of a particular trial intervention

5. Authors overall interpretation of clinical importance found from their trial results If the results were discussed – clinical relevance explicitly noted whereas; indirect reference to the results was considered implicitly reported




Levels of Justification Rating scale used to grade the level of strength that was developed to justify the reviewers overall interpretation of the clinical importance from the trial results obtained Level 1: the clinical importance of the primary study result in relation to the work done to determine the MCID Level 4: no accompanying justification



Methodological attributes important in the interpretation of study results from a clinical

perspective

Section

Methods

Attribute

• Explicit primary outcome stated

• Expected magnitude of difference “delta value” stated

• Expected magnitude of difference identified explicitly as the MCID

• Delta/MCID value reported as an absolute value


Methodological attributes important in the interpretation of study results from a clinical

perspective

Section

Results

Discussion


Attribute

• Statistical significance of primary outcome reported

• Confidence intervals for primary outcome reported

• Clinical importance of primary outcome discussed

• Discussion explicit or implicit• Level of justification• Appropriate clinical

interpretation of results

Interpreting Clinical Trial ResultsWorkshop Articles:

1. GISSI-Prevenzione Investigators. Dietary Supplementation with n-3 polyunsaturated fatty acids and vitamin E after myocardial infarction: results of the GISSI-Prevenzione trial. Lancet 1999;335 (9177):447-55.

2. Cleare AJ., Heap, E, Malhi GS, Wessely, S, O’Keane V, Miell J. Low-dose hydrocortisone in chronic fatigue syndrome. A randomized crossover trial. Lancet 1999;353:455-58.

Ref: (2001) Chan et al., (1999) Cleare et al.,



1. GISSI-Prevenzione Investigators. Dietary Supplementation with n-3 polyunsaturated fats acids and vitamin E after myocardial infarction: results of the GISSI-Prevenzione trial. Lancet 1999;335 (9177):447-55. Chan et al., (2001) found that 26% studies did not report a sample size or a delta value If reported – reported in relative instead of absolute terms Likewise the delta value did not appear to reflect the MCID of the intervention




1. GISSI-Prevenzione Trial Assessed the effect of dietary supplementation with polyunsaturated fats demonstrated an absolute decrease of 1.3% (95% CI 0.1%-2.6%) in the primary outcome (combined end point of death, nonfatal MI and

stroke) In the sample size calculation; delta value was a 4% absolute difference between groups over 3.5 year period Therefore: the efficacy of the intervention found by the study was significantly smaller than the sample size delta




1. GISSI-Prevenzione Trial If we assume that this value represents the MCID of the intervention, the magnitude of the intervention's effect is definitely not clinically important. Authors reported “a clinically important and statistically significant benefit”, their estimation of the MCID of the intervention is not reflected in the sample size delta

Ref: (2001) Chan et al., (2002) Hing et al.,



2. Cleare AJ., Heap, E, Malhi GS, Wessely, S, O’Keane V, Miell J. Low-dose hydrocortisone in chronic fatigue syndrome. A randomized crossover trial. Lancet 1999;353:455-58. Assessed the effect of low-dose hydrocortisone on chronic fatigue disorder, a 9 point reduction on a fatigue scale was deemed to be clinically important. However; the delta value for the sample size was reported as a 4 point reduction on the same scale




2. Hydrocortisone in chronic fatigue study Therefore; the authors reported that the study result, a statistically significant 4.5 point reduction on the fatigue scale when comparing the intervention group with the control group , was not clinically important.



Interpreting Clinical Trial ResultsInterpreting Study Results Study results are regarded as statistically significant (P < 0.5) when values indicating the lower limit of the 95% confidence interval (CI) are greater than the null effect Judging clinical relevance can be difficult Clinical relevance can take 4 forms, pending the relationship of the MCID on the intervention to the point estimate (the best single value of the efficacy of the intervention that has been derived from the study

results) and the 95% CI surrounding it Ref: (2002) Hing et al.,


Interpreting Clinical Trial ResultsInterpreting Study Results Definite: when the MCID is smaller that the lower limit of the 95% CI Probable: when the MCID is greater than the lower limit of the 95% CI, but smaller that the point estimate of the efficacy of the intervention Possible: when the MCID is less than the upper limit of the 95% CI, but greater than the point estimate of the efficacy of the intervention Definitely not: when the MCID is greater than the upper limit of the 95% CI

Ref: (2002) Hing et al.,


Interpreting Clinical Trial ResultsStudy Results of Definite Clinical Importance Randomized placebo controlled trial that assessed the efficacy of the addition of spironlactone to the medication regime of patients with severe heart failure. Results demonstrated that the study intervention reduced the chance of death by an absolute value of 11.4% (95% CI, 6.7% to 16.1%). The delta value calculated for the sample size was an absolute 6.5% reduction in mortality. If the delta value is assumed to be the MCID, the results of this study can be interpreted as both statistically significant and clinically important Ref: (2002) Hing et al.,



Interpreting Clinical Trial ResultsLimitations of MCID Methodological limitation: uncertainty in the choice of MCID MCID values will vary from person to person depending on values/perspectives Without widespread used of sample size deltas that accurately reflect the MCID is it appropriate to use the sample size deltas as benchmarks for determining the clinical relevance? However……….. Compare the relationship of possible MCID values to the point estimate of the efficacy of the intervention and it’s CI – this will allow determining the level of clinical significance



Interpreting Clinical Trial ResultsSummary Formal statistical methods of analyzing and reporting clinical trial results are routinely performed and expected within the literature Interpreting and reporting trial results from the perspective of clinical relevance is not always done with the same emphasis Creates imbalance as clinically significant trials are often considered clinically significant and those that are statistically insignificant are also clinically unimportant Encourage evaluate both when interpreting clinical trial results with the tools reviewed today (MCID)


References1. Chan, K. B. J. Y., Man-Son-Hung, M., Molnar, F. J. & Laupacis, A. (2001) 'How well is the clinical importance of study results reported? As assessment of randomized controlled trials', CMAJ, 165, (9), pp. 1197-1202.2. Man-Son-Hing, M., Laupacis, A., O’Rourke, K., Molnar, F. J., Mahon, J., Chan, K. B. Y. & Wells, G. (2002) ‘Determination of the Clinical Importance of Study Results’, JGIM, (17) pp. 469-476.3. McCluskey, A. & Ghaaliq Lalkhen, A. (2007) ‘Statistical IV: Interpreting the results of statistical tests’, Continuing Education in Anesthesia Care & Pain, [Online]. DOI: 10.1093/biaceaccp/mkm042ical, Volume 7, (6) Accessed: September 16, 20124. Moher, D., Schulz, K. F. & Altman, D. G. (2001) ‘The CONSORT statement revised recommendations for improving the quality of reports of parallel group randomized trials’, BMC Medical Research Methodology, [Online]. Available from: http://www.biomedcentral.com/content/pdf/1471-2288-1-2.pdfAccessed: September 17, 2012.5. Moher, D., Dulberg,, C. S. & Wells, G. (1994) ‘Statistical Power, Sample Size, and Their Reporting in Randomized Clinical Trials’, JAMA, 272 (2) pp. 122-124.6. Library Doctor WHO International (1994) ‘Chapter 9 Interpreting research results’, Available from: http://whqlibdoc.who.int/emro/2004/9290213639_chap9.pdf (Accessed: September 21, 2012).

http://www.biomedcentral.com/content/pdf/1471-2288-1-2.pdf

http://whqlibdoc.who.int/emro/2004/9290213639_chap9.pdf

Thank you!

interpreting clinical trial results betty anne schwarz msc, ba, rn ohri clinical research conference...

Documents