Introduction Benefit-risk Discussion References

Automated decision support for evidence-basedbenefit-risk decision making

Gert van Valkenhoef

Department of Epidemiology, University Medical Center Groningen (NL),Faculty of Economics and Business, University of Groningen (NL)

Escher meeting, 2 December 2011Amersfoort, The Netherlands

Introduction Benefit-risk Discussion References

Escher 3.2 Goals

Develop a drug information system:

Effective knowledge access and management

Answer drug efficacy and safety questions

in an efficient, transparent and accountable waywithin and across compoundsfor a broad audience (including regulators)

Improve consistency in regulatory decision making

Based on systematic review and meta-analysis

Introduction Benefit-risk Discussion References

Problem 1: effective knowledge access

Review of existing systems:

Evidence-based decision making time-consuming/error-prone

No comprehensive source of trial information existsTrial information is insufficiently structuredLack of systems to use the data

Missed opportunities to introduce more structure

Trial registration, regulatory submission and systematic review

It is unclear how the information should be structured

Prototypes should be developed now, to discover this

[Tervonen et al., 2010, van Valkenhoef et al., 2011b]

Introduction Benefit-risk Discussion References

Problem 2: transparent decision making

Regulatory benefit-risk decision making:

Based on pivotal evidence from trials

But not using formal decision modeling

Lack of (the use of) formal methods:

Hides reasoning supporting decision

Causes lack of transparency / traceability

Raises doubt about consistency

Causes of informal process:

Lack of suitable models/tools

Weighting criteria potentially controversial

Introduction Benefit-risk Discussion References

PhD overview

The ADDIS system & database

Automating network meta-analysis

Benefit-risk modeling

Introduction Benefit-risk Discussion References

ADDIS: Aggregate Data Drug Information System

Assisted evidence synthesis and benefit-risk assessment

Based on a database of clinical trials

Focussed on aggregated data

ADDIS is our (partial) solution [van Valkenhoef et al., 2011e]

Database is assumed available (or delivered by company)

Gathering the data is still a major hurdle!

But the ADDIS system / data model is a first step

Introduction Benefit-risk Discussion References

Evidence synthesis

If > 2 studies provide information on the same outcome

This information must be combined (synthesis)

Implemented through meta-analysis

Decisions involving > 2 alternatives

Require synthesis of networks of trials

Network meta-analysis

Existing technique, requiring manual modeling

Automated for use in ADDIS [van Valkenhoef et al., 2011a,c]

Introduction Benefit-risk Discussion References

Benefit-risk modeling in ADDIS

Available methods:

BRAT framework

‘Lynd & O’Brien’ model

SMAA (MCDA) based method

Based directly on the evidence from:

Single trials

Evidence synthesis

Introduction Benefit-risk Discussion References

BRAT Framework

Provides a number of steps to perform

Define decision context, identify outcomes, identify datasources, ... (similar to MCDA)

Defines useful visual summaries

But only applies to pair-wise comparisons

Does not define how to make trade-off decisions

Left open in frameworkADDIS models fill the gap

[Coplan et al., 2011]

Introduction Benefit-risk Discussion References

BRAT framework (ADDIS)

Introduction Benefit-risk Discussion References

BRAT framework (ADDIS)

Introduction Benefit-risk Discussion References

Quantitative benefit-risk modeling

Based on clinical trials or (network) meta-analysis

Making trade-offs explicit

But allowing for imprecise/vague/incomplete preferences

Making uncertainty explicit

[Tervonen et al., 2011, van Valkenhoef et al., 2011d]

Introduction Benefit-risk Discussion References

A simple stochastic model

The ‘Lynd & O’Brien’ model:

Based on cost-effectiveness analysis techniques

Compares 2 alternatives

On 2 criteria (benefit vs. risk)

Sample (∆B,∆R) values from a joint distribution

Plot them on a plane

Count how many points are below the threshold µ

[Lynd and O’Brien, 2004]

Introduction Benefit-risk Discussion References

Lynd & O’Brien example: set up (ADDIS)

Introduction Benefit-risk Discussion References

Lynd & O’Brien example: set up (ADDIS)

Introduction Benefit-risk Discussion References

Lynd & O’Brien example: set up (ADDIS)

Introduction Benefit-risk Discussion References

Lynd & O’Brien example: results (ADDIS)

Introduction Benefit-risk Discussion References

Benefit-risk plane

+Benefit A+Benefit B

+R

isk

A+

Ris

kB

µB better

A better

p = aa+b

count b

count a

Introduction Benefit-risk Discussion References

Benefit-risk plane

+Benefit A+Benefit B

+R

isk

A+

Ris

kB

µ

B better

A better

p = aa+b

count b

count a

Introduction Benefit-risk Discussion References

Benefit-risk plane

+Benefit A+Benefit B

+R

isk

A+

Ris

kB

µ

B better

A better

p = aa+b

count b

count a

Introduction Benefit-risk Discussion References

Benefit-risk plane

+Benefit A+Benefit B

+R

isk

A+

Ris

kB

Trade-off

Trade-off

µ

B better

A better

p = aa+b

count b

count a

Introduction Benefit-risk Discussion References

Benefit-risk plane

+Benefit A+Benefit B

+R

isk

A+

Ris

kB

µThe acceptability threshold.

We are willing to ‘pay’ µ

units risk to get 1 unit of

benefit.

B better

A better

p = aa+b

count b

count a

Introduction Benefit-risk Discussion References

Benefit-risk plane

+Benefit A+Benefit B

+R

isk

A+

Ris

kB

µThe acceptability threshold.

We are willing to ‘pay’ µ

units risk to get 1 unit of

benefit.

B better

A better

p = aa+b

count b

count a

Introduction Benefit-risk Discussion References

Benefit-risk plane

+Benefit A+Benefit B

+R

isk

A+

Ris

kB

µB better

A better

p = aa+b

count b

count a

Introduction Benefit-risk Discussion References

Example: acceptability curve (ADDIS)

Introduction Benefit-risk Discussion References

SMAA BR analysis

The Lynd & O’Brien model is limited to 2x2 problems.

Stochastic Multi-criteria Acceptability Analysis (SMAA)allows m × n problems:

m alternativesevaluated on n criteriaperformance of alternative i on criterion j : Ci,j ∼ f (ci,j)

[Tervonen et al., 2011, van Valkenhoef et al., 2011d]

Introduction Benefit-risk Discussion References

SMAA BR analysis

SMAA models for benefit-risk:

Can be based on a single trial

Or (network) meta-analysis

And is implemented in ADDIS

Introduction Benefit-risk Discussion References

SMAA example (ADDIS)

SMAA modelbased on networkmeta-analysis.

Introduction Benefit-risk Discussion References

SMAA example (ADDIS)

Measurements (input distributions).

Introduction Benefit-risk Discussion References

SMAA example (ADDIS)

Model without preference information.

Introduction Benefit-risk Discussion References

SMAA example (ADDIS)

Model without preference information.

Introduction Benefit-risk Discussion References

SMAA example (ADDIS)

Preferences for severe depression.

Introduction Benefit-risk Discussion References

SMAA example (ADDIS)

Severe depression results.

Introduction Benefit-risk Discussion References

SMAA example (ADDIS)

Preferences for mild depression.

Introduction Benefit-risk Discussion References

SMAA example (ADDIS)

Mild depression results.

Introduction Benefit-risk Discussion References

Relevance: EMA BR methodology project

Approach/method Relevance to regulators UsefulnessProbabilistic simulation Can illuminate the risk/benefit trade-off when uncertainty is a major

feature of a regulatory decision.Medium

Bayesian statistics Can integrate evidence and its uncertainty, both pre- and post-approval, with multiple criteria in decision models.

High

MCDA Multi-criteria decision analysis extends decision theory to accommo-date multiple, conflicting objectives. Provides common units of valuefor both benefits and risks.

High

Table: MTC/SMAA integrates 2 of 3 quantitative approaches rated’High’ on usefulness, and 1 rated ’Medium’.

Introduction Benefit-risk Discussion References

Summary

ADDIS (Aggregate Data Drug Information System)

Decision support system

Evidence based regulatory decision making

ADDIS incorporates

statistical methods

decision modeling

in an automated framework

Introduction Benefit-risk Discussion References

Conclusions

ADDIS can enable

formal, explicit and transparent BR assessment

while taking uncertainty into account

The implemented decision modeling allows

imprecise or (wholly or partially) missing preferences

may enable a decision without the need for exact weights

Making preference elicitation

less time consuming

more feasible in groups

less controversial (unless controversial trade-off pivotal)

Introduction Benefit-risk Discussion References

Key lessons & future work

Key lessons

Decision support system based on trials data feasible

Requires semantically structured data sets

Statistical analysis should be tailored to decision

Requires semantically structured data sets

But structured data is hardly available

Future work

Beyond proof-of-concept

Further statistical & decision modelingEven better data modelingValorization / product development

Make structured repository of clinical trials data (?!)

Escher is project T6-202 of the Dutch Top Institute Pharma

Partners in Escher:

Introduction Benefit-risk Discussion References

P. M. Coplan, R. A. Noel, B. S. Levitan, J. Ferguson, and F. Mussen.Development of a framework for enhancing the transparency, reproducibilityand communication of the benefit-risk balance of medicines. ClinicalPharmacology and Therapeutics, 89(2):312–315, 2011. doi:10.1038/clpt.2010.291.

L. D. Lynd and B. J. O’Brien. Advances in risk-benefit evaluation usingprobabilistic simulation methods: an application to the prophylaxis of deepvein thrombosis. Journal of Clinical Epidemiology, 57(8):795–803, 2004.doi: 10.1016/j.jclinepi.2003.12.012.

T. Tervonen, B. de Brock, P. de Graeff, and H. Hillege. Current status andfuture perspectives on drug information systems. In Proceedings of the 18thEuropean Conference on Information Systems, ECIS 2010, Pretoria, SouthAfrica, 2010.

T. Tervonen, G. van Valkenhoef, E. Buskens, H. L. Hillege, and D. Postmus. Astochastic multi-criteria model for evidence-based decision making in drugbenefit-risk analysis. Statistics in Medicine, 30(12):1419–1428, 2011. doi:10.1002/sim.4194.

Introduction Benefit-risk Discussion References

G. van Valkenhoef, , G. Lu, B. de Brock, H. Hillege, A. E. Ades, and N. J.Welton. Automating network meta-analysis. Submitted manuscript, 2011a.

G. van Valkenhoef, T. Tervonen, B. de Brock, and H. Hillege. Deficiencies inthe transfer and availability of clinical evidence in drug development andregulation. Submitted manuscript, 2011b.

G. van Valkenhoef, T. Tervonen, B. de Brock, and H. Hillege. Algorithmicparametrization of mixed treatment comparisons. Statistics and Computing,2011c. doi: 10.1007/s11222-011-9281-9. (in press).

G. van Valkenhoef, T. Tervonen, J. Zhao, B. de Brock, H. L. Hillege, andD. Postmus. Multi-criteria benefit-risk assessment using networkmeta-analysis. Journal of Clinical Epidemiology, 2011d. doi:10.1016/j.jclinepi.2011.09.005. (in press).

G. van Valkenhoef, T. Tervonen, T. Zwinkels, B. de Brock, and H. Hillege.ADDIS: a decision support system for evidence-based medicine. Submittedmanuscript, 2011e.