Modelling in IQWiG's Cost-Benefit Assessment

Uwe Siebert, MD, MPH, MSc, ScD1,2

Professor of Public Health (UMIT)Assoc. Professor of Radiology (Harvard Univ.)

1 Chair, Dept. of Public Health, Medical Decision Making and HTA,UMIT - University of Health Sciences, Medical Informatics, and

Technology, Hall/Innsbruck, Austria2 Director, Cardiovascular Research Program, MGH Institute for Technology Assessment, Harvard Medical School, Boston, MA, USA

2

Overview

1. General introduction to clinical and economic decision-analytic modeling

2. The role of decision-analytic modeling in IQWiG's cost-benefit assessments

3. Challenges

What is a model?

4

Definition Model IThe International Society for Pharmacoeconomics and Outcomes Research (ISPOR) Task Force on Good Research Practices – Modeling Studies:

"[...] an analytic methodology that accounts for events over time and across populations, that is based on data drawn from primary and/or secondary sources, and whose purpose is to estimate the effects of an intervention on valued health consequences and costs."

[Weinstein et al., Value in Health 2003]

5

Definition Model IIU.S. National Research Council:

Simulation model = "[...] a replicable, objective sequence of computations used for generating estimates of quantities of concern [...]"

[National Research Council, 1991]

6

Definition Model IIIBuxton and colleagues defined models in scientific disciplines:

"Models [...] are a way of representing the complexity of the real world in a more simple and comprehensible form"

[Buxton, Health Economics. 1997]

7

Purposes of Modeling in Economic Evaluation

[Siebert, Eur J Health Econom 2003]

8

Purposes of Modeling in Economic Evaluation

1. Combining evidence from short-term clinical trials and long-term epidemiological studies

2. Assessment of screening programs and diagnostic procedures

3. Extrapolating efficacy beyond the time horizon of a clinical trial

4. Generalizing from efficacy to routine effectiveness

5. Transferring the evidence from one health care system or country to another

6. Resource allocation7. Considering patient

preferences8. Informing decisions in the

absence of clinical trial data

9. Fine-tune technologies10. Value-of-information

analysis11. Health policy models and

national projections12. Health technology

assessment[Siebert, Eur J Health Econom 2003]

Decision-Analytic Modeling

10

Goals of Decision Analysis (DA)

I. Selecting the "optimal strategy" after balancing medical benefits, risks and costs of different alternatives

II. The process of decision analysis makes structure, elements, and parameters of the decision problem explicit and transparent; accessible for discussion

11

Definition DA

• Decision analysis is a systematic, explicit and quantitative approach for decision making under uncertainty

• Synonyms / similar terms: – decision-analytic modeling, decision modeling,

mathematical modeling, pharmacoeconomic modeling, simulation

12

Clinical DA

Natural course of disease

Therap./diag. efficacy

Life expectancy

To treat or not to treat?To test or not to test?To screen or not to screen?Which treatment, test, or screening strategy?

Decision

Ethics

Policy

Risk, Side effects

Quality of life

13

Economic DA

Natural course of disease

Therap./diag. efficacy

Life expectancy

Costs

To treat or not to treat?To test or not to test?To screen or not to screen?Which treatment, test, or screening strategy?

Decision

Ethics

Policy

Risk, Side effects

Quality of life

14

Examples for Decision Problems• Diagnostic tests

– sensitivity vs. specificity• Treatment

– efficacy vs. side effects• Life expectancy vs. quality of life

– e.g., chemotherapy in advanced cancer• Costs vs. health affects

– societal willingness-to-pay• Risk preference

– risk-averse vs. risk-seeking• further ...

Tradeoffs

15

The decision must be made

16

The Decision-Analytic Framework• Precise research question• Target population• Alternatives (strategies)• Perspective• Time horizon• Health states, events, other elements• Outcomes: e.g. response, clinical status, major

adverse events, life expectancy, QALYs, costs, cost-effectiveness

• Model type: e.g. decision tree, Markov model, DES• Analysis: cohort simulation, microsimulation• Basecase and sensitivity analysis

17

Model Types

Decision analysis uses probabilistic models to analyze primary and/or secondary data

• Decision trees• Markov models• Discrete event simulation• Transmission models• Agent-based models• Further …

18

Response

No ResponseTreat

Response

No ResponseWait and see

Response

No ResponseNo Treat

Patient or Population

p1

1-p1

p2

1-p2

p3

1-p3

Decision Tree

19

Response

No ResponseTreat

Response

No ResponseWait and see

Response

No ResponseNo Treat

Patient or Population

p1

1-p1

p2

1-p2

p3

1-p3

ER LE QALY C

ER LE QALY C

ER LE QALY C

ER LE QALY C

ER LE QALY C

ER LE QALY C

Consequences

Decision Tree

20

Markov Model: Bubble Diagram

Well Disease

Death

Chronic Disease

21

Markov Model: Bubble Diagram

Well Disease

Death

Chronic Disease

22

Markov Model: Bubble Diagram

Well Disease

Death

p = 1.00

p = 0.65 p = 0.60p = 0.30

p = 0.05 p = 0.40

Chronic Disease

23

Markov Model: Bubble Diagram

Well Disease

Death

p = 1.00

p = 0.65 p = 0.60p = 0.30

p = 0.05 p = 0.40

U=1 U=0.6

U=0

Chronic Disease

24

Markov Model: Bubble Diagram

Well Disease

Death

p = 1.00

p = 0.65 p = 0.60p = 0.30

p = 0.05 p = 0.40

U=1 U=0.6

U=0

Costs€2000

Chronic Disease

25

Markov Trace

Well Disease DeadStart

Cumulative Costs, LE, QALYs

26

Markov Trace

Well Disease Dead

0.650.30 0.05

0.60 1.000.40

after 1 year after 1 year:Cost, LE, QALYs

Well Disease DeadStart

Cumulative Costs, LE, QALYs

27

Markov Trace

Well Disease Dead

0.650.30 0.05

0.60 1.000.40

after 1 year after 1 year:Cost, LE, QALYs

Well Disease DeadStart

Well Disease Dead

etc.

after 2 years

0.650.30 0.05

0.60 1.000.40

After 2 years:Costs, LE, QALYs

Cumulative Costs, LE, QALYs

28

Comparing Different Strategies

Clinical and economic weighting

Without Intervention

WellU=1

DiseaseU=0.6

DeathU=0

0.65 0.60

1.00

0.30

0.05 0.40

Without Intervention

WellU=1

DiseaseU=0.6

DeathU=0

0.65 0.60

1.00

0.30

0.05 0.40

WellU=1

DiseaseU=0.6

DeathU=0

0.65 0.60

1.00

0.30

0.05 0.40

With Intervention

WellU=1

Disease U=0.6

DeathU=0

0.75 0.60

1.00

0.10

0.15 0.40

Intervention costs: 20.000 €With Intervention

WellU=1

Disease U=0.6

DeathU=0

0.75 0.60

1.00

0.10

0.15 0.40

With Intervention

WellU=1

Disease U=0.6

DeathU=0

0.75 0.60

1.00

0.10

0.15 0.40

WellU=1

Disease U=0.6

DeathU=0

0.75 0.60

1.00

0.10

0.15 0.40

Intervention costs: 20.000 €

29

Markov-Trace

Without interventionWithout

preventionCycle z

WellU1=1Pz(G)

DiseaseU2=0,6Pz(K)

DeathU3=0Pz(T)

Cycle-Effect.

(QALY)

Cumulat.Effect.

(QALY)

Cycle-Cost(€)

Cumulat.Cost(€)

0 1.000 0.000 0.000

1 0.650 0.300 0.050 0.830 1.830 600 600

2 0.423 0.375 0.203 0.648 1.478 750 1350

3 0.275 0.352 0.374 0.486 1.963 704 2054

... ... ... ... ... ... ... ...

19 0.000 0.001 0.998 0.001 3.141 3 4281

20 0.000 0.000 1.000 0.000 3.141 0 4281

Sum 2.857 2.140 16.003 3.141 4281

30

Cumulat. Effectiveness(QALY)

Cumulat. Cost (€)

Without intervention 3.141 4.281

With intervention 3.582 21.983

Increments (differences) 0.441 17.703

Incremental cost-utility ratio = €17703/0.441 QALY = €40099/QALY

Comparing Different Strategies

31

Discrete Event Simulation (DES)

• Entities (Patients)

• Attributes (Characteristics of patients)

• Events, Processes (Relapse, hosp., surgery)

• Resources (Staff, facilities, devices)

• Variables (not tied to entity, e.g. discount rate)

• Time (Start and end of simulation, warm up)

32

Model Taxonomy

[Brennan et al., 2006]

33

Decision-analytic models are an aid

-- not a complete procedure --

to decision makers

Modelling in IQWiG's Cost-Benefit Assessment

35

Rationale of Modeling Costs

Economic data from RCTs not sufficient to inform decisions– study environment modifies practice– not transferable– no information on long-term consequences– may not be available at all

⇒ Modeling economic outcomes is an essential component of comprehensive economic evaluations

36

Rationale of Modeling Costs

Intervention Clinical Outcome

Long-term Health

Resources Costs

37

Purpose of Economic Modeling in IQWiG

• Assess efficiency (Wirtschaftlichkeit) of a technology and provide firm basis for decisions on reasonableness of reimbursement and pricing

• a) provide value for x-axis of EF– for y-axis, use health outcome from IQWiG's clinical

benefit assessment (EbM-based)– health outcomes must be modeled in IQWiG's

economic models to generate valid cost estimates• b) estimate budget impact

– see BIA

38

Specific Purposes of Modeling in the IQWiG Framework

• y-axis:– estimate complete economic consequences– model beyond trial time horizon– combine different data sources– perform indirect comparisons in absence of head-to-head trials– project cost over time, budget impact

• x-axis:– usually no modeling, use of empirical clinical benefit measure– combining benefit and harm, transferring benefit to cardinal

scale, or linking valid surrogates to patient-relevant outcomes may involve modeling

• General:– Assess different scenarios, uncertainty, value of information &

suggest further research priorities

39

IQWiG's Modeling Requirements1. Full transparence

– structure, input parameters, assumptions2. Sufficient depth

– adequate representation of disease, intervention effects and costs

3. Flexibility– multiple assumptions, scenarios, settings

4. Ability to assess uncertainty– structural and parameter uncertainty

5. Validity– rigorous peer review

6. Specificity for German health care context– demographics, epidemiology, practice patterns, costs

40

Modeling Process

Model development Validation Analysis Reporting

41

10 Steps1. Define precise research question2. Develop graphical model representation3. Collect input parameters4. Define functional relationships5. Choose modeling technique6. Program model7. Validate model8. Perform analysis9. Report results10. Critically discuss and interpret results

42

1. Define precise research question (PICO)

• Population & subgroups• Intervention of interest• Comparators• Outcomes (cost categories)

43

2. Develop graphical model representation

• relevant components (patient characteristics, natural history of disease, intervention, outcomes)

• links (causal and statistical associations)• analytic structure• benefit measure as pre-defined by IQWiG• present concept to expert panel (clinicians,

methodologists)

44

3. Collect input parameters• systematic search for published and other

available data, different sources:– RCTs for treatment effect– observational cohort studies for natural history of disease

and long-term mortality– pharmacoepidemiologic studies for safety– cross-sectional studies and registries for routine resource

use and costs (cost data from RCTs have low relevance)• quality assessment• transformations, adaptations, standardizations,

deflation• additional targeted data acquisition (e.g., from

registries, new studies)

45

4. Define functional relationships

• functional relation: outcome = f(determinants)• determinants: pat. characteristics, intervention,

compliance, time, setting, etc.• different types:

– stratified data– prediction scores– regression functions (linear, logistic, Cox, mixed, two-

step, etc.)– others

CAVE: causality, may need to use causal epidemiologic methods (g-estimation, MSM)

46

5. Choose modeling techniqueIQWiG has no a-priori preference for modeling

technique, choice depends on research question

• Decision tree• Markov model• Discrete event simulation (DES) • Transmission model• Agent-based model (ABM)• others

Currently, DES and ABM underused in health economics, powerful tools

47

6. Program model

• no suggested software• complete documentation• electronic version of the model

48

7. Validate model• face validity: expert panel, peer-review• technical verification: debugging• internal validation: check with input parameters• external validation: check with independent data• cross-validation: compare with other models

(structure/parameters/results)

CAVE: heterogeneous quality of published models!Expert panel: clinical, epidemiologic, statistical,

decision-analytic, public health, economic, etc.

49

8. Perform analysis

• relevant scenarios• subgroup analyses• uncertainty analysis (1-way, multi-way, PSA)

50

9. Report results

• clear and complete description of framework, methods, model, and rationale for choice of technique (no standard format)

• description of assumptions and input data with sources/references

• validation techniques and results• basecase, subgroups, uncertainty

assessment/sensitivity analyses• electronic version of the model (protection of IPR)

51

10. Critically discuss and interpret results

• interpret EF and describe cost-effectiveness (EF zone)

• discuss budget impact • limitations• potential bias: direction and magnitude• uncertainty assessment• further research recommendations, value of

further information

52

IQWiG's Mandatory Features"Minimum set of requirements"

a) Model framework– perspective– time horizon– duration of treatment– discounting

b) Model quality– appropriate depth– justification of modeling technique

CAVE: Minimum set of requirements = necessary (but not sufficient) for good model quality

53

a) Model Framework• Economic study type

– cost-effectiveness study using EF approach• Perspective

– community of German citizens insured by the SHI– costs to statutory health care system, includes cost for

patient such as out-of-pocket costs, etc.• Time horizon

– sufficient to cover extent of illness, capture all costs affected by technology

– for many chronic diseases: lifetime• Duration of treatment

– as in trial vs. German practice• Discounting

– discount costs, sensitivity analysis on discount rate

54

b) Model Quality

• Appropriate depth– all aspects of disease impacting costs (& health)– benefits and harms– heterogeneity– German practice patterns– real world (e.g., reduced compliance)– short-term and long-term costs– different subgroups, settings, regions, scenarios

• Justification of modeling technique

Challenges

56

Challenges• Several legal and regulatory "givens" restrict modeling options• Modeling approaches differ across agencies/countries need

international modeling reference case?• Predictability for manufacturers?• Simplicity vs. validity• Modeling restricted to economic evaluation and only economic

outcomes• Need experience with "German approach", then refine IQWiG's

modeling guidelines• Policy impact unclear (rationale decision vs. political/market-

driven decisions)• Ethical considerations (utilitarism vs. equity principles)• Need for experienced modelers, modeling education

57

Decision-analytic models are an aid

-- not a complete procedure --

to decision makers

58www.smdm.org

Want to hear more about modeling?