Should FDA, EMEA, Health Canada, CONSORT accept Last-Observation-Carried-Forward Analyses? A Systematic Review of Dementia Drug RCTs.

Dr. Frank MolnarAssociate Professor of Medicine, University of Ottawa, Canada

Staff, The Ottawa Hospital Division of Geriatric Medicine

Medical Director, The Ottawa Hospital Geriatric Day Hospital (focus on Dementia)

Co-chair, Champlain Dementia Network ( www.champlaindementianetwork.org )

Medical Director, Regional Geriatric Program of Eastern Ontario ( www.rgpeo.com )

Conflict of Interest Disclosure Frank Molnar MSc, MDCM, FRCPC

Has no real or apparent

conflicts of interest to report.

Background – the basics of group-to-group comparisons in RCTs The effectiveness of medications is tested via RCTs that

follow response to new treatments and comparators over time.

It is almost inevitable that some participants will drop out of such studies before trial completion.

if analyses only include participants who remain in the trial, then study power is lost, benefit of randomization is lost and trials risk generating erroneous conclusions.

Consequently, intention-to-treat (ITT) - the inclusion of all patients in the analysis according to the group determined at randomization - has become the accepted standard for the analysis of controlled clinical trials.

The strength of ITT is that it not only preserves power but it also promotes balance between treatment groups for both known and unknown confounders, thereby preserving the benefits of randomization.

Background - dropouts In order for ITT approaches to analyze all

patients randomized, including dropouts, several methods to estimate missing data have been developed

Due to its simplicity (and possibly due to tradition and/or lack of motivation / impetus to change), a commonly employed technique to impute missing data is Last-Observation-

Carried-Forward (LOCF), also known as

‘endpoint analysis’

Background - LOCF Last-Observation-Carried-Forward replaces

subjects’ missing outcomes with the last available measurement and requires that two basic conditions be met:

that the subjects’ responses would have been constant (i.e. stable) from the last observed value (i.e. point of dropout) to the endpoint of the trial rather than declining or improving further

that missing values are ‘missing completely at random’ or MCAR (i.e. that the probability of dropout is not related in any way to variables such as disease severity, symptoms, group assignment or drug side-effects).

The Concern in Dementia RCTs

LOCF ignores trajectory - whether the subject was improving or getting worse at the time of dropout.

LOCF freezes outcome values at the last observation, thereby creating an apparent stabilization of disease, symptoms and/or function in dropouts (problematic in progressive conditions)

Copyright ©2008 Canadian Medical Association or its licensors

Molnar, F. J. et al. CMAJ 2008;179:751-753

Figure 1: Potential impact of last-observation-carried-forward analysis in longitudinal randomized controlled trials in chronic progressive diseases

Differential LOCF bias if have greater or earlier dropouts in treatment grouptreatment group than control group (Effect measured by LOCF [c – d] > True effect [a – b] resulting in exaggerated positive effect - biased in favor of treatment)

a = True treatment valuesb = True control valuesc = Treatment values altered by LOCFd = Control values altered by LOCF

Sco

re o

n o

ut c

ome

vari

abl e

(c

ogn

itiv

e or

fu

nct

i on

al s

cale

)

100

0

1 2 3 4 5 6 7 8 9 10

Time (visit #)

c

a

b

d

} LOCF bias

} LOCF bias

Differential LOCF bias if have greater or earlier dropouts in control groupcontrol group than treatment group (Effect measured by LOCF [c – d] < True effect [a – b] resulting in underestimate of effectiveness - biased against treatment )

a = True treatment valuesb = True control valuesc = Treatment values altered by LOCFd = Control values altered by LOCF

Sco

re o

n o

ut c

ome

vari

abl e

(c

ogn

itiv

e or

fu

nct

i on

al s

cale

)

100

0

1 2 3 4 5 6 7 8 9 10

Time (visit #)

c

a

b

d

} LOCF bias

} LOCF bias

Systematic Review Methods; Identification of Studies An electronic literature search of Medline and the

Cochrane Register of Controlled Trials from January 1984 (the year of publication of the McKhann criteria for Alzheimer disease) to February 2008

This electronic search was supplemented by hand searching of the reference lists of selected articles, meta-analyses and review articles.

The full text of RCT reports were then independently reviewed by two certified specialists in Geriatric Medicine with clinical expertise in dementia, with formal research methodological training.

Methods; Eligibility Criteria Double-blinded, randomized controlled trials of

cholinesterase inhibitors (donepezil, Aricept, rivastigmine, Exelon, galantamine, Reminyl) and N-methyl-D-aspartate (NMDA) receptor antagonists (Namenda, Ebixa, memantine) examining progressive symptoms (e.g. cognition, function) in Alzheimer disease, Vascular dementia, Mixed dementia or Mild Cognitive Impairment employing DSM or NINCDS-ADRDA criteria for Alzheimer disease or NINDS-AIREN criteria for Vascular Dementia were eligible for this review.

The systematic review was restricted to studies with full trial reports published in English language peer-reviewed journals.

Results; Selection of articles Of the 1146 articles identified by the search

strategies, 191 papers (including RCT reports, non-randomized trial reports, commentaries, systematic reviews and meta-analyses) were selected for full text and reference section review. Of these, 57 RCT reports met the eligibility criteria for systematic review

Results; Trial Characteristics of 57 studies 45 Alzheimer disease

21 donepezil 11 rivastigmine [1donepezil-rivastigmine] 7 galantamine, 6 memantine),

8 vascular dementia or mixed dementia 3 donepezil 3 galantamine 2 memantine

4 MCI 2 donepezil 1 rivastigmine 1 galantamine).

Results; Reporting of Dropouts Cholinesterase inhibitor studies were more likely to

demonstrate greater dropout in the treatment group (73% of studies) compared to memantine studies (25% of studies). When cholinesterase inhibitor studies were combined,

there was a higher dropout rate in the treatment group (23.3%) than in the control group (16.8%). LOCF would bias in favor of treatment

When memantine trials were combined the opposite pattern was noted with fewer dropouts in the treatment group (14.6%) than in the control group (18.5%). LOCF would bias against treatment

Results; Types of non-ITT Analyses 35 employed Observed Case analysis (OCA)

Randomized patients with at least one evaluation while on study medication at designated assessment times – analyze data at that point but do not carry forward

Fully Evaluable Population analysis (FEP) Described as completing study with a preset level of compliance [%

compliance with meds, number of visits attended] – the compliance criteria may vary from study to study.

Treatment Per Protocol analysis (TPP) Similar to FEP as meet a preset level of compliance [% compliance

with meds, number of visits attended] that varies from study to study but do not necessarily have to complete study

Completer analysis (CA) Analyze all patients that complete trial regardless of compliance

Results; Types of ITT Analyses 12 (21%) of the 57 studies did not identify the

type of ITT analysis performed or performed only non-ITT analysis.

Of the remaining 45 studies that did perform an identifiable form of ITT analysis, 42 (93%) employed LOCF.

35 of the trials that performed ITT analysis (78 %) relied on LOCF as the only form of ITT analysis.

Results; Types of ITT Analyses Of the 45 studies that employed LOCF, only 8 reported performing

another type of ITT analysis to confirm the results of LOCF analyses. 3 of these 8 studies did not comment on the results of

the alternate non- LOCF ITT analysis. Of the 5 studies that commented on the results of the

alternate ITT analysis, 4 did not report the values of the outcomes calculated by the alternate ITT analysis but did indicate that the direction of results was unchanged.

It is uncertain if the values of the point estimates of the outcomes were similar when the alternate ITT analyses were performed.

Results; Types of ITT Analyses Only 1 study verified the values of the

point estimates of outcomes measuring drug efficacy generated by LOCF with point estimates of an alternate form of ITT analysis. The study verified the values of 3 positive

outcomes (out of hundreds published)

Results; Contraindications to the use of LOCF Of the 35 studies employing LOCF as the only form of ITT analysis, 23

(66 %) explicitly demonstrated contraindications (factors that could introduce bias) to the use of LOCF that would exaggerate effectiveness..

Greater dropout rate in treatment group Earlier dropouts in treatment group More rapid progressors in treatment dropout group

66% is likely an underestimate It was unclear if the remaining 12 studies were free of contraindications as

most studies failed to report adequate data regarding the timing and severity of disease of dropouts to permit a precise estimate of the number of contraindications to the use of Last-Observation-Carried-Forward.

Reasonable Concerns

It is highly unlikely that, given the high prevalence of conditions promoting ‘LOCF bias’ uncovered in this study, the point estimates of some of the hundreds of outcomes generated in the reviewed trials that used LOCF have not been altered in some way.

The question is likely not ‘has bias been introduced’, but rather How many outcomes have been biased ? To what degree have they been biased ?

Why keep promoting LOCF? LOCF provides no benefits over other forms of ITT analysis while

introducing unnecessary risk of bias in progressive disease states. Unless we have an interest in exaggerating the effectiveness of the

drugs, why should we continue to use a technique such as LOCF if it provides no benefit and unnecessarily introduces bias?

LOCF is unnecessary – other ITT approaches that carry less risk of bias are readily available and have been used

time response relationship for ADAScog/11 analyzed using generalized linear modeling

mixed effects modeling mixed models repeated measures (MMRM)

Sensitivity analyses of ITT analyses should be done to explore if one form is introducing bias.

Petersen RC, Thomas RG, Grundman M et al. Vitamin E and Donepezil for the Treatment of Mild Cognitive Impairment. NEJM 2005; 352: 2379 – 2388.

Barrier to optimal clinical care

The use of LOCF creates unnecessary risk of generating biased results that will tend to favor more toxic therapies (e.g. cholinesterase inhibitors) over less toxic therapies (e.g. memantine). Drug Toxicity →increase in measured efficacy When using LOCF, toxicity becomes a positive attribute

of a drug.

This effect may create a barrier to the funding and use of current (memantine) and future less toxic therapies.

FINAL CONCLUSION: AnalysisFINAL CONCLUSION: Analysis remains the remains the weak weak linklink in proving efficacy / effectiveness in dementia drug in proving efficacy / effectiveness in dementia drug RCTs (an unnecessary vulnerability that will prevent RCTs (an unnecessary vulnerability that will prevent increased use of these medications).increased use of these medications).