statistical & design considerations for non-inferiority trials andrew nunn mrc clinical trials...

Statistical & Design Considerations for

Non-inferiority trials

Andrew NunnMRC Clinical Trials Unit

London

TB Forum December 2005 2

Outline

• What is a non-inferiority trial?• How do they differ from superiority

trials?• What do the regulators say?• How large do the trials need to be?• How should the trials be conducted and

analysed?


What is a non-inferiority trial?

• How is it different from an equivalence trial?• Does non-significant imply non-inferior?• Does non-inferior imply non-significant?• Are non-inferiority trials always larger than

superiority trials?• Can a failed superiority trial be turned into a

non-inferiority trial?• Why do we need these trials in TB?


A little bit of history - 35yrs ago

• Study R, the first East African/BMRC trial of short course chemotherapy could be regarded as a non-inferiority trial.

• 2STH/16TH worked well under strict trial conditions.

• The main objective was to see if a six month regimen was at least as good as the standard treatment, better - would be a bonus.

S = streptomycin, T = thiacetazone, H = isoniazid

Study R – 30 month relapse free rates

0-35%

5%-5%

A possible δ

Confidence intervals for difference from control 2STH/16TH regimen

6SHR

6SHZ

6SHT

6SH

No difference

6

A point to remember

• “It is never correct to claim that treatments have no effect or that there is no difference in the effects of treatments. It is impossible to prove … that two treatments have the same effect. There will always be uncertainty surrounding estimates of treatment effects, and a small difference can never be excluded.”

Alderson P, Chalmers I. BMJ 2003:326:1691-8.


Why non-inferiority for new TB drugs?

• Under a wide variety of trial conditions the gold standard 2EHRZ/4HR regimen is at least 95% effective.– Nomads in the Algerian Sahara– Recently published IUATLD study in Africa and Asian

centres.

• We will be very unlikely to better it.– we would require a total of 2600 evaluable patients

to demonstrate a reduction from 5% to 2.5% relapses


Our goal• Our goal is to reduce treatment duration to

a maximum of 4 months and preferably less.

• How much are we prepared to pay, if anything, for such a reduction?

– must the new regimen be as good as the standard?

– would we be satisfied with a regimen that was almost as good?

– if so, how good is almost?


EMEA quote

• “If no degree of possible inferiority of the test [new regimen] to the reference [control] is acceptable, then the development of products with equal efficacy to a comparator by means of non-inferiority trials would become impossible.”

EMEA /CPMP /EWP /2158 /99


Does non-significant = non-inferior?

• No! definitely not.• Common sense will tell us that a non-

significant result from an under-powered study is, in the extreme case of little value.

• BUT, non-inferior does not necessarily mean non-significant!


How do we do it?

• We need a null hypothesis.• The situation is the reverse of what is

required in a superiority design.• For superiority

– H0 is there is no difference.

• For non-inferiority– H0 is there is a difference.

• The alternative hypothesis is also reversed.


Equivalence & non-inferiorityWhat’s the difference?


Determining equivalence

• First step in establishing equivalence - define ‘limits of equivalence’ (± δ)

• Having conducted the trial, calculate the 95% confidence intervals for the difference between the control and the new treatment

• If the confidence interval is entirely within ± δ then equivalence is established


Non-inferiority

• Equivalence requires that the difference control - new intervention is both > -δ and < δ, the new treatment mustbe neither worse nor better than the control by a fixed amount.

• In contrast to equivalence with non-inferiority we are only interested in determining whether new treatment is no worse by an amount δ.


Non-inferiority

0-δ No difference

The 95% CI for the difference between the control and the intervention are all > -δ, i.e. non-inferiority demonstrated.


Non-inferiority

0-δ No difference

The lower 95% CI for the difference between the control and the intervention are all > -δ, i.e. non-inferiority demonstrated.

The lower 95% CI is < -δ, non-inferiority has not been demonstrated.


Non-inferiority and superiority

0-δ No difference

The 95% CI for the difference between the control and the intervention are all >-δ, i.e. non-inferiority demonstrated.

In this case both non-inferiority and superiority have been demonstrated


Non-inferiority and inferiority

0-δ No difference

The 95% CI for the difference between the control and the intervention are all >-δ, i.e. non-inferiority demonstrated.

In this case both non-inferiority and superiority have been demonstrated

In this case both non-inferiority and inferiority have been demonstrated


Choosing δ

• The value of δ must be chosen before the trial begins.

• It’s value will depend on clinical, statistical and possibly regulatory considerations.


Example: 2NN Study

• van Leth, Phanuphak et al (Lancet 2004), a study of first-line antiretroviral therapy in HIV

• Main comparison between nevirapine twice daily and efavirenz (plus stavudine and lamivudine) in terms of ‘treatment failure’ (based on virology, disease progression, therapy change)

• Primary objective was to establish the non-inferiority of nevirapine twice daily (δ =10%)


Example: 2NN Study

• Confidence intervals for failure rates (E-2NN)– All data (-12.8%, 0.9%)– Only those starting med. (-14.6%, -0.8%)– Concurrently randomised (-11.9%, 3.4%)

• Non of these intervals are completely above δ value of -10%; one interval also excludes zero


Example: 2NN Study

• BUT, the authors concluded:‘Antiviral therapy with nevirapine or efavirenz showed similar efficacy, so triple-drug regimens with either … are valid for first-line treatment’

Lancet 2004, 363:1253-63


Does it matter?

• A non-inferiority trial can demonstrate significant benefit from the new treatment - (cf Study A).

• But is it possible to have non-inferiority and a significantly worse outcome in the new treatment?

• Yes! provided δ is acceptable to clinicians.– if N is large enough any difference can be

shown to be significant!


Adverse effects

• Assessment of adverse effects is particularly important in equivalence trials.

• It is not enough to prove non-inferiority in terms of efficacy.

• A new treatment must be as safe, or safer, than the old one.


Choosing δ

• On 27th July 2005 the European Medicines Agency (EMEA) issued a new European “Guideline on the choice of the non-inferiority margin”

• This guideline comes into effect in January 2006.



Quote from EMEA document

“The lower limit of the confidence interval [of the difference between the new regimen and the control] ... represents a lower bound and is usually interpreted as the degree of inferiority to the reference that can be excluded based on the data presented…..



Quote from EMEA document

“Of course this is not an actual lower bound and the magnitude of inferiority could be greater. However it is generally considered that the chance of the true difference being worse than that suggested by this bound is acceptably small.”



General EMEA recommendations

• If possible three study arms should be included, test, reference and placebo - allows validation of the non-inferiority margin.

• The margin should be such there is assurance that the test arm has a clinically relevant effect.

• The primary focus is the relative effect of the test arm and the reference arm.

• The choice of the margin should be justified in the protocol

• The choice of the margin should be independent of power considerations.


Design consideration

• It is important to ensure that the design of equivalence trials, including definitions of a favourable response, should be as similar as possible to earlier trials assessing the control regimen.


Internal validity

• In a superiority trial there is a strong incentive to ensure high quality of conduct.

• In contrast in an non-inferiority trial the conclusion of non-inferiority could be reached because of poor discriminatory power.

• In a TB trial this could occur if follow-up rates were poor and/or there was failure in the lab to detect all relapses.


But

• If there are already many treatments being used interchangeably for the disease under consideration a possible approach might be to consider the information available from all of them. From this a delta may be constructed which summarises the information known about the relative efficacy of these products, and the new trial can be designed to provide a similar level of knowledge of the relative efficacy of the new product.


Accepting a larger δ

• “In the situation where the test product is anticipated to have a safety advantage over the reference it is likely that a larger delta could be justified as some loss of efficacy might be accepted in exchange for the safety benefits”


Is there a case for a larger δ if treatment can be

shortened?

• “It may be possible to justify a wider non-inferiority margin for efficacy if the product has an advantage in some other aspect of its profile. This margin should not, however, be so wide that superiority to placebo is left in doubt”


How large a δ would you accept?

• If treatment could be shortened from 6 to 4 months would an increase in the failure/relapse rate from 5% to 10% be acceptable?


How large a δ would you accept?

• If treatment could be shortened from 6 to 4 months would an increase in the failure/relapse rate from 5% to 10% be acceptable?

- provided that the failures and relapses could be satisfactorily retreated.


FDA position

• FDA (as described in FDA’s 1992 Points to Consider document) originally used a ‘step function’:

Cure Rate δ 90% 10%

80 - 89 % 15%

< 80 % 20%

• A more flexible approach has since been adopted

37

Example : Pediatric Meningitis TrialInvestigational Drug vs. Active Control

Sponsor’s FDA

Proposal Proposal

Projected response rate 80% 80%

Delta 15% 10%

Evaluable total sample size 224 504

Projected % evaluable 70% 70%

Total to be enrolled 320 720

Projected enrollment time 2-4 years 4-6 years

FDA proposed study considered not to be feasible

Note: = 5%, power = 80%

38

What confidence level?

• Traditionally we use 95% confidence in superiority trials (thanks to RA Fisher!)

• Guidelines for pharmacokinetic equivalence have traditionally used 90% CI.

• In regulatory situations the choice is based on level of risk regulators are prepared to accept.

• Could be appropriate to use 90%, 95% or even 99%.

• Need for flexibility.


Calculating power - an example

• Given the expected range of, say 3-6% relapse rates in the control, 2EHRZ/4HR regimen.

• What study size would we require for a range of δ?

40

N per arm, one sided Confidence Interval (P=0.1), 90% power; 3% -6% relapse rate

0

100

200

300

400

500

600

700

800

900

10 9 8 7 6 5 4 3

Delta (% )

N p

er

arm

3

4

5

6

41

N per arm one sided Confidence Interval (P=0.05) 90% power, 3% -6% relapse rates

0

200

400

600

800

1000

1200

1400

10 9 8 7 6 5 4 3

Delta (% )

N p

er a

rm

3

4

5

65% relapse, δ = 10%, 100 per arm

42

N per arm one sided Confidence I nterval (P=0.05) 90% power, 3% -6% relapse rates

0

200

400

600

800

1000

1200

1400

10 9 8 7 6 5 4 3

Delta (% )

N p

er a

rm

3

4

5

6




But ...

• These power calculations do not allow for additional numbers required for a Per Protocol analysis, or patients excluded because they do not have TB, or because they have MDR disease.

• Neither do they allow for losses to follow-up.


How should we analyse non-inferiority trials?

• Superiority trials are analysed by ITT because it is the most conservative and least likely to be biased.

• ITT analysis of non-inferiority trials is not conservative - there is a bias towards no difference.

• PP biased since not all randomised patients included.

• It is recommended that non-inferiority trials should be analysed by both ITT and per protocol (PP).

45

Defining ITT and PP

• Definitions vary. – For ITT some definitions exclude patients who either

do not have confirmed diagnosis, or who never received treatment.

– PP includes all receiving full course of treatment with no major protocol violations.

• What definitions are appropriate for TB trials?• CPMP: ‘similar conclusions from both the ITT

and PP are required in a non-inferiority trial.’• ‘Sample size computations should ensure

sufficient numbers in the PP population’.

CPMP: Committee on Proprietary Medical Products (2000)


CAVE!

• Drop-outs from the two regimens need to be carefully evaluated.

• Suppose patients not responding dropped out early from one treatment arm, or

• Possibly because of differential withdrawal rate for adverse events -

• This would suggest there may be important differences between the treatments.


Interim analyses

• Do we need them?• Probably not if it is to consider stopping

early for strong evidence of non-inferiority.

• Such evidence would support a case for the possible superiority of the new treatment to the control - a strong incentive to keep on.


Conclusions 1• A major concern among regulators in many NI trials

is that the efficacy of the control is not well established.

• This is NOT the case with the control regimen 2EHRZ/4HR. One advantage of no new drugs for 40 years!!

• In the event of establishing a 4 month regimen to be non-inferior it would be unwise to use that regimen as the control in the next NI trial - biocreep.

Biocreep - slightly inferior treatment becomes the control for next generation of NI trials


Conclusions 2

• NI trials must be conducted with rigour• The value of δ needs to be determined

before the start of the trial and should take into account both clinical and statistical considerations.

• Both the value of δ and other aspects of design need to be discussed with regulators

• Non-inferiority needs to be demonstrated not only for efficacy but also for safety.


Regulatory Guidance

• ICH E9 ‘Note for Guidance on Statistical Principles for Clinical Trials’, September 1998

• ICH E10 ‘Note for Guidance on Choice of Control Group’, July 2000

• CPMP ‘Note for Guidance on the Investigation of Bioavailability and Bioequivalence’, July 2001

• CPMP ‘Points to Consider on Switching between Superiority and Non-Inferiority’, July 2000

• CHMP ‘Guideline on the Choice of the Non-Inferiority Margin’, July 2005


Selected references

• D’Agostino RB, Massaro JM et al: Non-inferiority trials: design concepts and issues - the encounters of academic consultants in statistics. Statist Med 2003; 22: 169-186.

• Altman DG, Bland JM: Absence of evidence is not evidence of absence. BMJ 1995; 311:485.

• Blackwelder WC: Current issues in equivalence trials. J Dent Res 2004; 83:C113-115.

• Jones B, Jarvis P et al: Trials to assess equivalence: the importance of rigorous methods. BMJ 1996; 313:36-9.

Study R – 30 month relapse free rates

0-35%

5%-5%

A possible δ

Confidence intervals for difference from control 2STH/16TH regimen

6SHR

6SHZ

6SHT

6SH

No difference

statistical & design considerations for non-inferiority trials andrew nunn mrc clinical trials...

Documents

nonsignificant result

superiority trials

isoniazidtb forum

equivalence trial

relapsestb forum

failed superiority trial

new tb drugs

degree of possible inferiority