Some thoughts on benefit to risk methods applied

to the monitoring of clinical trials

Frank W Rockhold, PhD

Senior Vice President, Global Clinical Safety and Pharmacovigilance, GlaxoSmithKline

Adjunct Professor of Biostatistics and Bioinformatics, Duke University School of Medicine

Affiliate Professor of Biostatistics, Medical College of Virginia

Duke-Industry Statistics Symposium

Durham, NC October 23, 2015

Outline

Objective: Introduce concept of integrating efficacy and safety information in a

structured and systematic way in an ongoing IDMC setting to aid in decision

making.

Evaluating Benefit to Risk – Up front characterization of B/R to aid in decision making during the trial

– Patient Input on B/R

– Graphical and other methods used to display BR data

– Consideration of the lag between risk and benefit on the context of monitoring

– Standard exercise in regulatory reporting especially for marketed drugs

Potential issues specific to interim looks

Hypothetical example

Summary and Discussion

What do we mean by Benefit and Risk?

Benefit: what we want a treatment to do for patients and what is important

about the outcomes

–Clinically relevant outcomes or biomarkers / surrogates that are considered

favorable effects and rationale for choosing them

– Intensity, duration, and uncertainty of effects

Risk (Harm) : the potential consequence to the patient and how to manage the

events when they occur

–Clinically relevant outcomes or biomarkers / surrogates that are considered

unfavorable effects

– Severity, duration, predictability, “monitorability,” and reversibility of effects

Benefit-Risk Balance: how the favorable effects compare to

the unfavorable effects

“Asymmetry of Benefit: Risk Evaluation“*

3 *O’Neill, Drug Information Journal 2008 42: 235

Benefits may be the hardest to quantify in the end - an RCT

endpoint may not really show the patient benefit

Priority Depends on

Treatment Goals Benefits Risks

Intensity How good are they? How severe are they?

Time

How soon do they

happen?

How soon do they

happen?

How long do they

last?

How long do they

last?

Probability

Do they only happen

for some people?

Who gets them?

Can they be avoided?

If no, can they be

managed?

4

What do we mean by “good”? Are we

measuring outcomes that matter? Why

do they matter, and to whom?

Safety is hard to specify but

ironically easier to quantify.

A value tree can help frame your thinking

*from PhRMA BRAT Framework; EMA has successfully field tested a similar “effects tree”

** uncertain relationship to treatment

Identified Benefit

or Risk Category

Identified benefit/

risk Outcome

Potential Outcome

or B/R category**

Rapid Onset

Pain-free Response Reduced Pain

Sustained Response

Reduced Sensitivity to Sound and Light Reduced Sensitivity

Reduction in Functional Disability

Other Reduction in Nausea or Vomiting

Headache Relief

Transient Triptans Sensations

Individual Risks "Chest-related" Adverse Events

Central Nervous System Adverse Events

Benefits

Risks

Benefit-Risk Balance

How are you

defining and

measuring

benefits and

risks?

What are the key

benefits and risks?

Example Value Tree for Triptans* as Migraine Treatment

5

Explaining your conclusions from your analysis

Does benefit outweigh risk?

How does your evidence support this conclusion*?

Consider the following:

How much uncertainty is there around the favorable and unfavorable

effects?

Is the benefit durable?

Does the risk increase or decrease over time?

Do some patients experience more benefit and/or more risk than

others?

–Are there subgroup differences? (e.g., age, sex, ethnicity, organ function,

disease severity, or genetic polymorphism)

–Do those patients at higher/lower risk experience higher or lower benefits?

What are the implications for the patient?

6

Summary Measures e.g

–Relative Risk (RR): RR = PC / PT

–Odds Ratio (OR): OR = PC (1 - PT) / PT (1 - PC)

–Absolute Risk Reduction (ARR): ARR = PC – PT

–Number needed to treat/harm (NNT/NNH): 1/ARR

Consistent measures needed to truly evaluate B/R

PT = Probability of an event (benefit or harm) on treatment, PC the same for control

Measures of Effect

7

Example of Interval Plot to Display

Risk Differences*

Reduction in

-38

-8

8

-10

10

23

25

34

61

88

-150 -100 -50 0 50 100 150

“Chest-related” AEs

CNS AEs

Transient triptans sensations

Sustained response

Headache relief

Rapid onset

Reduced sensitivity to sound and light

Pain free response

Reduction in functional disability

Reduction in nausea or vomiting

Excess Number of Events (per 1,000 patients)

Favors comparator Favors study drug

Reduction in

Reduction in

Efficacy 95% CI

Safety 95% CI

Mean

-

Risk Difference (per 1,000 patients)

*Created by Bennett Levitan et al., PhRMA Benefit Risk Action Team (BRAT)

Can be used for acute,

dichotomous endpoints

and ranked in order of

importance

8

A straightforward benefit-risk case: Rotavirus (RV)

http://www.cdc.gov/vaccinesafety/vaccines/rotavsb.html#3; http://www.cdc.gov/rotavirus/about/photos.html

Causes gastroenteritis (inflammation of the stomach and intestines)

– severe watery diarrhea, often with vomiting, fever, and abdominal pain

– in babies and young children, can lead to dehydration (loss of body fluids)

– leading cause of severe diarrhea in infants and young children worldwide

– globally, causes about half a million deaths each year in children <5 years of age

Transmission electron micrograph of intact

double-shelled rotavirus particles, Public

Health Image Library (PHIL) ID#178

Risk Context: Intussusceptions (IS)

http://www.mayoclinic.com/health/intussusception/DS00798

Serious disorder in which part of the intestine — either the small intestine or

colon — slides into another part of the intestine, described as "telescoping“

Often blocks the intestine, preventing food or fluid from passing through; also

cuts off the blood supply to the part of the intestine that is affected

Most common cause of intestinal obstruction in children; most cases in children

have no demonstrable cause

There’s a clear B-R case for Rotavirus Vaccination in

Latin America

Received 8 September 2011; accepted 9 January 2012. Correspondence: Rishi Desai, MD, MPH, Division of Viral Diseases, Centers for Disease Control and Prevention, 1600 Clifton

Rd, NE, MS-A34, Atlanta, GA, 30333 (rdesai1@cdc.gov). Clinical Infectious Diseases Published by Oxford University Press on behalf of the Infectious Diseases Society of America

2012.DOI: 10.1093/cid/cis191

Benefit-Risk Ratios

Hospitalization 841:1 (90% CI, 479:1 to 1142:1)

Death 395:1 (90% CI, 207:1 to 526:1)

What about something more challenging, like MS ?

http://www.cdc.gov/vaccinesafety/Vaccines/multiplesclerosis_and_hep_b.html; http://phil.cdc.gov/phil/details.asp A photomicrograph of the lumbar cord revealing histopathology indicating previous demyelination and inflammation, CDC/Dr. Karp, Emory University

Disease of the central nervous system characterized by the destruction of the myelin

sheath surrounding neurons: myelin = light-colored substance that coats and protects

nerve cells much like the insulation shielding electrical cables in a computer

Progressive and usually fluctuating disease with exacerbations (patients feeling worse)

and remissions (patients feeling better) over many decades; attacks can vary in location,

severity, and duration; may include muscles, elimination, eyes, sensation, thought,

hearing, swallowing, speech, sexual function, or stamina

Cause of MS is unknown. The most widely held hypothesis is that MS occurs in patients

with a genetic susceptibility and is triggered by certain environmental factors

3 times more common in women than men,

with diagnosis usually made in young adult

In many patients with MS, permanent disability

and even death can occur

Progressive Multifocal Leukoencephalopathy

http://www.ninds.nih.gov/disorders/pml/pml.htm; http://www.ninds.nih.gov/news_and_events/news_articles/news_article_JCVirus_Gene_Major.htm

Disease of the white matter of the brain: nerve fibers that handle communications across

the central nervous system; white matter gets its name from myelin

Caused by a viral infection that targets cells that make myelin--the material that insulates

nerve cells (neurons).

Polyomavirus JC (often called JC virus) is carried by a majority of people and is harmless

except among those with lowered immune defenses

Symptoms of PML are diverse, since they are related to the location and amount of

damage in the brain, and may evolve over the course of several weeks to months.

– Most prominent symptoms are clumsiness; progressive weakness; and visual, speech, and

sometimes personality changes

– Progression of deficits leads to life-threatening disability and (frequently) death

– Diagnosis of PML can be made following brain biopsy or by combining observations of a progressive

course of the disease, consistent white matter lesions visible on an (MRI) scan, and the detection of

the JC virus in spinal fluid.

In general, PML has a mortality rate of 30-50 percent in the first few

months following diagnosis but depends on the severity of the

underlying disease and treatment received. Those who survive PML

can be left with severe neurological disabilities.

Treatment

Convenience

Benefits

Risks

Infection

Reproductive

Toxicity

Liver Toxicity

Neurological

Other

Relapse

Disability Progression

Reactivation of serious herpes

viral infections

PML

Congenital abnormalities

Transaminases elevation

Seizures

Infusion/injection reactions

Hypersensitivity

reactions

Flu-like reactions

Note that as the weight for a relapse is for

a value function with the measure scale

with a range from 0 to 2, then actual

weight of a single relapse is half that

shown here.

Basler Biometric Section - September 2012 | IMI PROTECT - Tysabri Case Study – our

thanks to Richard Nixon, Novartis

The value tree provides categories for

outcomes

Weights

PML is 10x worse

than disease

progression

Although not part of BRAT, weights can

be added to the framework process

Weights and Values – importance within and between

outcomes Incremental Benefit-Risk of Tysabri v. Placebo

Basler Biometric Section - September 2012 | IMI PROTECT - Tysabri Case Study – our thanks to Richard Nixon, Novartis

Display and Interpret Key Benefit-Risk Metrics –

Quantitative Information, Qualitative Judgment

Interval Plot

Basler Biometric Section - September 2012 | IMI PROTECT - Tysabri Case Study – our thanks to Richard Nixon, Novartis

Hypothetical Example where mortality rate is an

indicator of benefit and harm

0

0.05

0.1

0.15

0.2

0.25

Time 1 Time 2 Final

Trt Death Dx

Ctl Death Dx

Trt Death AR

Ctl Death AR

Odds Ratio for Benefit and Harm

0

0.5

1

1.5

2

2.5

3

3.5

Interim 1 Interim 2 Final

OR Death Dx

OR Death AR

Absolute Risk Difference for benefit and harm

(deaths per 1000 patients)

-80

-70

-60

-50

-40

-30

-20

-10

0

10

20

30

AR Deaths Dx per1000

AR Deaths AR per1000

Difference in ARD between treatment and control

-10

0

10

20

30

40

50

60

Net Reduction in Deaths per 1000

Net Reduction inDeaths per 1000

Summary

Benefit to Risk is an Art, a Science and a Tool

–Well accepted and increasingly used in NDA’s with multiple methods emerging

–Easily scalable to combine across trials

–The BR context matters: Disease state and Intervention v. prevention dictate

different patient messages

–Patient input critical to success

–Formal weighting of benefits and risks remain important philosophical issues

with no immediate tactical solutions (each person has their own prior)

Communication of interim and final results: It isn’t a useful BR analysis if no one

understands it (Regulator, Payer, Physician, Patient)

Applying this to interim monitoring seems obvious but has not been described to

my knowledge

Aside from aiding in decisions making will allow final assessment of B/R to

readily line up with interim reviews

21

Backup

22

Benefit and Risk Over Time, for Each Patient

Active

From Jonathon Norton, FDA. A Longitudinal Model and Graphic for Benefit-risk Analysis, with Case Study. DIJ 2011.

Each row is a patient

• Sorted from most withdrawal time to most time on benefit

• Can see whether the same patients experience both benefit and risk

23

Benefit-Risk begins with context – Why is the patient

being treated?

Disease Prevention •safe

•minimally invasive

•durable effect

Acute life-saving •effective

• immediate

•manageable AEs

Chronic treatment •durable

•minimal AEs

•convenience for

desired activity level

Curative •targeted

•durable

•manageable AEs

Chronic disease delay •durable

•manageable AEs

•convenience for desired

activity level

End of Life •minimal AEs

•based on desire for

hospital, hospice, or home

care 24

Naming and Measuring Benefits and Risks

Clinically relevant outcomes or biomarkers / surrogates that are considered

benefits (favorable effects) and risks (unfavorable effects)

Uncertainty around favorable and unfavorable effects

Rationale for how they were identified and selected

How well surrogates predict the benefit/risk outcome

Intensity or severity, timing, and duration of benefit and risk

“Monitorability” and reversibility of risks

Any ranking or weighting that was applied

25

Tysabri Case Study - Background

Tysabri (natalizumab) was approved in 2004 by the FDA for the

treatment of relapsing remitting multiple sclerosis (RRMS).

In 2005 the drug was suspended because of an associated incidence

of progressive multifocal leukoencephalopathy (PML), a rare

neurological disorder.

In 2006 it was re-introduced due to patient demand, but with strict risk

minimization measures.

In 2009, due to occurrence of further PML in monotherapy post

marketing, CHMP reassessed the PML risk of Tysabri and confirmed

the current approval.

Basler Biometric Section - September 2012 | IMI PROTECT - Tysabri Case Study – our thanks to Richard Nixon, Novartis

Summary of B/R for Rotavirus vaccine by Desai, et al.

We found that essentially for every 1 child who dies of vaccine-associated

intussusceptions, we would save about 400 children from death from rotavirus,

and for every 1 child who is hospitalized for intussusception because of the

vaccine, we would prevent about 841 children from being hospitalized for

rotavirus,"

By the investigators' calculations, rotavirus vaccine would avert 144,746

hospitalizations for rotavirus gastroenteritis and 4124 rotavirus deaths if a

hypothetical cohort of 9.5 million infants in 14 Latin American countries was

vaccinated. At the same time, it could potentially cause 172 excess

intussusception hospitalizations (1.8/100,000 vaccinated infants) and 10 excess

deaths (0.11/100,000 vaccinated infants). These figures result in risk-to-benefit

ratios of 1:841 for hospitalization and 1:395 for death in favor of vaccination.

Safety reported throughout clinical

development

Annual review and evaluation of pertinent safety information collected during

the reporting period related to a drug under investigation, whether or not it is

marketed

– examines whether in accord with previous knowledge of the investigational drug’s

safety

– describes new safety issues that could have an impact on the protection of clinical trial

subjects

– summarizes current understanding and management of identified and potential risks

– provides an update on the status of the clinical investigation/development program and

study results

Assures that sponsors are adequately monitoring and evaluating the evolving

safety profile of the investigational drug

Includes relevant information from post-marketing studies

DSURs: Development safety update reports

Presentation title 28

http://www.ich.org/fileadmin/Public_Web_Site/ICH_Products/Guidelines/Efficacy/E2F/Step4/E2F_Step_4.pdf

Template Created Internally for Statisticians

Our thanks to GengQian Cai and Andrew Miskell for this software template

Benefit 1

Benefit 2

Benefit 3

Benefit 4

Benefit 5

Risk 1

Risk 2

Risk 3

Risk 4

Risk 5

Risk6

0.69 (0.53, 0.89)

0.64 (0.51, 0.81)

0.97 (0.75, 1.25)

0.73 (0.55, 0.97)

0.71 (0.39, 1.27)

0.87 (0.54, 1.4)

0.86 (0.47, 1.6)

0.96 (0.29, 4.25)

1.96 (0.24, 16.3)

1.45 (0.85, 2.53

1.07 (1, 1.1.4)

Ratio (95% CI) Active v. Placebo

Favors Active Favors Placebo

29

Graph by Bob Schriver, Mike Durante, Mike Williams, Marilyn Metcalf

Adding Benefit to a Standard Safety Graph

Can be used for acute, dichotomous endpoints and ranked in order

of importance. Could also compare subgroup responses.

Benefit

Risk 1

Risk 2

Risk 3

Risk 4

Risk 5

Risk 6

Risk 7

30

Graphing displays benefit and risk together as

appropriate for the data, e.g., in an interval plot *

*based on PhRMA BRAT Framework forest plot 31

Changing views of BR over the life of

the medicine

32

Discovery Clinical study design

What will the

drug do for

the patient?

Do we need

to design

outcomes

studies and

what should

they be?

When do we need

to update

information to

regulators and the

public?

What should be

emphasized in the

risk(-benefit)

communications?

Regulatory

submissions Post-marketing

surveillance

Launch

What should be

in the

benefit/risk

section of the

dossier?

BR proposition is at its maximum at

Candidate Selection; perception of

maximum benefit and low risk

Over time, less benefit observed or

measured incompletely, e.g., surrogates,

while accumulating more data on risk

Perceived Benefit

Perceived Risk

Patient health and public health

Population v. individual benefit

(prophylaxis)

When is vaccination a choice and

when is it a necessity? (school

vaccinations, Ebola epidemic)

Good for one or good for all? (risks

and burdens for the larger populace

when patients are more at risk and

need more health care that they

don’t want)

How do I know that a prophylactic

worked? Do I need to know?

(screening, preventive medicines)

Presentation title 33

Regulators are working at this level

Looking at information on multiple

drugs, including clinical trial results,

spontaneous reports, and

sometimes observational data to

understand how new interventions

may affect morbidity, mortality,

incidence, prevalence…

34

…as well as

gathering

information at

patient meetings

and through

advisory groups.

Gathering data in clinical trials from

what they hope is a representative

sample…

Pharma and Biologics Companies are working at this level

35

…and projecting back on

the likely impact the

medicine will have on a

broader group of patients

…as well as

interviewing

patients, e.g., in

focus groups…

A patient deciding on a treatment for themselves

Will this help me or hurt

me? Which one am I?

36

?

NNT/NNH Not recommended for statistical analysis and inference given its

properties. All statistical procedures (estimation, hypotheses

testing, trial design) are more natural and transparent for

Absolute Risk Reduction.

The name NNT/NNH may encourage the idea that it is a precise

number, but it has probabilistic content. If presenting NNT,

setting, time period, outcome, and baseline risk of patients for

whom the NNT is thought to be applicable should be

considered.

Might not be correct to draw conclusions at the level of

individual patients based on NNT calculations.

A clear distinction should be made between data analysis and

subsequent risk communication. NNT may be considered as a

way of presenting results, not as a tool for statistical

computations.

from Quartey G et al. internal technical review document based on Rockhold F. and Fedorov V. Pitfalls of Number Needed to Treat

(NNT) as a Measure of Comparative Benefit or Risk. Internal GSK presentation. Citing Bender R., Calculating confidence intervals for

the NNT, 2001, Controlled Clinical Trials 22, pp.102-110 and Lesaffre E. and Pledger G., A note on the NNT.,1999, CCT, 20, pp. 439

- 447 37

Weighting

Like any analysis, weighting is a structured way to capture thinking, not

an answer in and of itself.

Weighting of Benefits and Risks is difficult and controversial

Methods for collecting weights exist and can be improved.

Methods for communications among groups who offer different weights

will be the more important contribution.

38