Effectiveness: Overview of Current Approaches and Emerging Trial Designs

Doug Taylor, PhDDirector of Biostatistics

Family Health International

Fleming and Richardson (2004). Some Design Issues in Trials of Microbicides for the Prevention of HIV Infection. JID.

Trussell and Dominik (2005). Will Microbicide Trials Yield Unbiased Estimates of Microbicide Efficacy? Contraception.

Pocock and Abdalla (1998). The Hope and the Hazards of Using Compliance Data in Randomized Controlled Trials. Statistics in Medicine, 17(3): 303-317.

Extra References

• Efficacy vs. Effectiveness

• Basic Design of Effectiveness Trials

• Choice of Control Group

• Study Populations

• Strength of Evidence

• Power, Study Size and Adherence

• Future Directions

OUTLINE

Efficacy vs. Effectiveness

Efficacy: reduction in risk of acquiring HIV when the microbicide is used as intended for

• Every act during the study (coitally dependent)

• Every day during the study (daily product)

• Following any other specified regimen (twice-daily, etc.)

Efficacy vs. Effectiveness

Effectiveness: reduction in risk, recognizing that

• Participants don’t use microbicide for all acts/days

• Microbicide may be withdrawn for AE or pregnancy

• Condom use not independent of microbicide use

• Possibility of infection due to other exposure routes

Basic Design

Phase 2b or 3 randomized, controlled trial powered to make a conclusive statement about effectiveness

Chance of concluding effectiveness depends on….

• What we mean by “conclusive statement”

• The efficacy of the microbicide

• Adherence to product use

• How much information (events) observed

Basic Design

• HIV-negative participants randomized to either the microbicide or control

• Study staff and participants blind to treatment assignment (where possible)

• Intensive counseling on HIV risk-reduction and use of product; condom promotion; treatment of STIs

• Monitoring of AEs

• Independent Data Monitoring Committee (IDMC)

• Testing for HIV at regular intervals for a year or more of follow-up

• Primary Outcome: estimated time to HIV infection

• Primary Analysis: compare the rate of infection between treatment groups using Intent-to-Treat principle

note on ITT: all participants/outcomes included, regardless of whether they continue to use product (e.g., even if withdrawn due to pregnancy)

Basic Design

Choice of Control Group

Placebo

• Ideally an ‘inert’ microbicide

• Blinding possible: risk-taking behaviors may be comparable between groups

Arguments against relying on a placebo control

• Placebo may not be inert

• Fail to capture effect of condom migration or other behavior changes that occur in real world

Choice of Control Group

Condom-only

• Attempts to evaluate the effect of adding a microbicide to real world settings

• Risk-taking behaviors (including condom use) will almost certainly differ between groups

Arguments against relying on a condom-only arm

• Differences in behaviors may overwhelm any effect of microbicide on HIV

• Clinical trials are not the real world

Study Populations

Minimal Requirements

• Risk of HIV exposure

• Minimal exposure to HIV via routes not protected by microbicide

• Willingness to use product

Study Populations

Participants recruited from

• General, sexually active populations

• STI clinics

• Discordant couples

• Sex workers

Refers to how confident we are that an apparent effect observed in a clinical trial is real and not due to chance

Related to concepts of

• Type I error (‘α level’) of a statistical hypothesis test when planning a trial, and

• Observed p-value or confidence interval for relative risk when interpreting trial results

Strength of Evidence

Typical trials are designed to have no more than a 2.5% chance of concluding treatment is effective, if in reality there is no effect (type I error)

e.g. “the one-sided p-value for test of effect must be less than 0.025,” or “the upper 95% confidence bound for Relative Risk of HIV must be less than 1.0”

A study designed to meet this requirement can be thought of as targeting the strength of evidence of one trial

Strength of Evidence - One Trial

RR2.01.00.7 3.00.3

0.40 0.99 95% CI

Estimated RR

p-value < 0.025 (one-sided)

Strength of Evidence of Single Trial

• FDA has traditionally required two trials, each demonstrating that there is a protective effective at the one-sided α=0.025 level

• Trials could be conducted sequentially or in parallel

• Protects against a spurious result

• Helps to ensure that product will be effective in different settings/populations

Strength of Evidence - Two Trials

RR2.01.00.7 3.00.3

0.40 0.94 95% CI (p-value < .025)

RR2

RR1

0.20 0.80 95% CI (p-value < .025)

Strength of evidence - Two Trials

Could be unethical to conduct second trial. A single microbicide study could suffice if

• Well conducted

• Multi-site

• Large strength of evidence (p-value < 0.005, equivalent to 1.5-2 trials)

• Results consistent across sites and within important subgroups (e.g. age)

• Strong safety data

Strength of Evidence

RR2.01.00.7 3.00.3

EstimatedRR

0.60 0.99 99.9% CI

p-value < 0.005 (one-sided)

Single Study with Larger Strength of Evidence

• Achieving the strength of evidence of more than one trial could require 5000-10000+ participants, 50+ million dollars

• Difficult to commit those types of resources without plans for stopping trial early if product appears unlikely to achieve desired effect

→ futility analysis

Resource Management

Phase 3 study designed to have 90% chance of detecting a 40% reduction in risk with α=0.005

Need to observe ~240 events

Half-way through the trial (120 events), the IDMC finds that the estimated treatment effect is zero

Even if the product is truly 40% effective (and the interim result is simply really bad luck), the chance of observing final p-value < 0.005 is only 20%

(but was interim result due to poor adherence?!)

Futility Analysis - example

Alternatively, could consider a “2b screening trial” (HPTN/MTN 035; MTN 003 VOICE):

• One-third the size of the single Phase 3 trial

• Clear rules for when/how to proceed to second trial

Resource Management

Power

No. ofEvents

Effectiveness

EfficacyAdherence Follow-upIncidence

Strength of evidence (alpha level)

Number enrolled

• In treatment trials, every participant may have an outcome measure (e.g. viral load, CD4 count) and contribute to power to detect an effect of treatment

• In a prevention trial, very few participants have the outcome (HIV) and so very few contribute directly to the precision of the estimate of effectiveness

• Prevention trials are “event driven”

Number of Events and Power

Events Required to Achieve 90% Power(Strength of Evidence of One Trial, one-sided α =0.025)

Estimated Study Size for 90% Power(one-sided α=.025)

When designing a study we state an effectiveness level that would be important to detect

For example, we might design a study to have 90% power to detect a 40% effectiveness level, assuming participants use it for 80% of acts/days

Such a study would require 160 events to achieve strength of evidence of one trial

Adherence and Power

Effect of Non-Adherence on Power (160 Event trial)

Challenges

• Poor adherence will doom an otherwise efficacious microbicide

• Condom use and risk-reduction counseling may significantly reduce incidence of HIV in control arm

• Do not know the infection status of partners

• Do not know whether microbicide was used for acts with exposure to HIV, or the route of exposure for individual acts leading to infection

• N9: Heterogeneity of effect across sites and studies; the same is possible for other products

Future Directions - Adherence

• Target something closer to efficacy by monitoring of adherence (IPM)

• Better identify participants who will use the product

• Develop products or delivery systems (e.g. rings) that people will use

• Develop products that take adherence largely out of the hands of participants (e.g. injectables) or that participants are less likely to forget (e.g. rings)

Future Directions – Design & Analysis

• Still have to perform the study

• Adaptive designs for screening out products, re-estimating power, futility and safety analyses

• Bayesian methods for combining phase 2 and phase 3 evidence

• Non-inferiority trials (when we have an effective microbicide)

Discussion

Questions?