Advancing Health Economics, Services, Policy and Ethics

Health Technology Assessment, Multi-Criteria Decision Analysis (MCDA) and resource allocation Structuring complex evidence and values

Stuart Peacock Canadian Centre for Applied Research in Cancer Control (ARCC) University of British Columbia British Columbia Cancer Agency

• Multi-Criteria Decision Analysis (MCDA) – Criteria, Weights, Aggregation

• MCDA, Program Budgeting and Marginal Analysis (PBMA) and priority setting/resource allocation

• Thoughts on a decision analysis perspective

Overview

2

• Some early thinkers: Roy (1968); Zeleny (1974); Keeney and Raiffa (1976)

• We only have a decision problem in front of us once we have to deal with at least two (conflicting) criteria, or else we may have a problem of getting the right information

• Decision Theory ≈ body of knowledge on which MCDA is founded (including utility theory)

• Decision Analysis ≈ Decision Theory put into practice

3

Early constructs in decision analysis

• Around the late sixties / early seventies: a

general dissatisfaction with decision aiding techniques that relied on a single figure of merit (e.g. B/C analysis, NPV, IRR, etc)

• Practical difficulties with using Cost-effectiveness analysis

4

Beginnings

5

Challenges with current decision-making processes

• Humans are quite bad at making complex, unaided decisions • There is a temptation to think that honesty and common sense will

suffice • Individuals respond to complex challenges by using intuition and/or

personal experience to find the easiest solution • Groups can devolve into entrenched positions resistant to

compromise • Groups make better judgments than average individual members in

analysis and evaluation tasks • Never underestimate the power of stupid people in large groups

6

Challenges to complex decision-making

7

Evolving decision-making processes

• Explicitly define problems and objectives • List and describe alternatives for achieving objectives • Define criteria (often called attributes or performance indicators) to

measure performance of alternatives • Design and execute studies to collect data to evaluate decision criteria • Populate a decision matrix of alternatives versus decision criteria • Elicit appropriate weightings for criteria • Synthesize criteria, assign weights to rank alternatives, communicate

results with stakeholders • Decision-makers make decisions with stakeholder input and guided by

MCDA results

8

Iterative steps of MCDA (Yoe 2002)

9

Iterative steps of MCDA (Yoe 2002)

10

Elements of the MCDA Decision Matrix

The Decision matrix represents the outcomes for a set of alternatives and a set of evaluation criteria

11

Elements of MCDA

• Column: decision makers, their preferences and evaluation criteria in hierarchical structure – Preferences mean weights of evaluation criteria, vary

according to decision alternatives – A criterion: standard of judgment or a rule to test the

desirability of alternative decisions; it includes objectives and attributes

– An objective is a statement about the desired state of the system

– An objective requires (a function of) one or more attributes

• Row: decision alternatives – There is a set of possible outcomes for each decision

alternative – State of nature/state of environment = uncontrollable

factors – Nature – unpredictability of the decision making

environment, i.e. state of economy, a weather condition, an action of a competitor

– Each state is independent to other state and cannot be manipulated by decision-maker

– If one state of nature is considered, only one decision outcome is associated with a given alternative

12

Elements of MCDA

• Column and Row – Outcomes depend on the set of attributes for evaluating

alternatives – Intersection of row and column = an outcome associated

with a particular alternative and attribute – One outcome is for one state of nature considered – A number of outcomes is for more than one state of

nature – Decision outcomes for each row are in attribute levels

measuring degree of achievement or performance of a decision alternative

– Set of outcomes should be ordered so that the best alternative can be identified

13

Elements of MCDA

• Sensitivity analysis – How the alternatives are changed/affected when the

inputs (health data, preferences etc.) are varied – It helps to learn how the decision elements interact to

determine the most preferred alternative, and which element is the source of disagreement among interest groups

– The output information can be used to vary the inputs/problem formulation step to achieve satisfied result

– This step can be used as checked point. If ‘yes’, goes for recommendation. if ‘no’, start over with varied input

14

Elements of MCDA

• 1970s/80s – debate in economics and ethics literature about relevant criteria for making resource allocation decisions in health care – focus on clinical and cost-effectiveness

• 1990s/2000s – emergence of health technology assessment bodies; growing recognition that other criteria are important, relating to equity, acceptability, burden, sustainability etc.

• 2010s – growing interest in decision analytic methods for considering multiple criteria – driven primarily by the NICE in the UK and moves to Value Based Pricing

• Multi Criteria Decision Analysis (MCDA) is a methodology designed to help decision-makers when making complex choices – first developed in the 1960s/70s

Making complex decisions in health

15

• A practical economic method for priority setting used in >100 studies

• Based on economic principles for priority setting – need to consider opportunity costs

– need for marginal analysis of costs and benefits

– existence of a fixed budget - some services must be contracted if others are to be expanded

• Unlike economic evaluation, PBMA considers the budget constraint

• Is compatible with sustainability and deficit financing

Priority setting and PBMA

16

• “If we had $100 000 more to spend, which services would we expand?”

• “If we had $100 000 less to spend, which services would we contract?”

• That is “What is the extra benefit we could get by shifting $100 000 from service A to service B?”

• Allocate resources to services with higher benefits per dollar spent

• Benefit based on consideration of health outcomes and other decision-making criteria

Basic PBMA principles

17

Define aim and scope

Form “Advisory

Panel”

Establish program budget

Develop decision criteria

Identify and rank options

Decisions and rationale

Decision review process

Evaluate and improve

PBMA

Continuous quality improvement loop

18

Define aim and scope

Form “Advisory

Panel”

Establish program budget

Develop decision criteria

Identify and rank options

Decisions and rationale

Decision review process

Evaluate and improve

PBMA

Economic evidence and MCDA

19

• Some relevant priority setting questions: – what objectives would decision-makers choose to pursue

(what are relevant decision-making criteria)? – How important are different criteria? – How should health care interventions be evaluated or

‘scored’ against those criteria? – How should scores be combined in a model to reflect the

overall performance of different interventions?

• MCDA is a ‘tool’ designed to help decision-makers make such complex choices

MCDA and priority setting

20

1. Problem structuring: generating a set of alternatives and a set of criteria against which the alternatives are to be evaluated and compared

2. Model building: constructing some form of model which represents decision-makers’ objectives and their value judgements

Key methodological considerations: – methods used to describe decision-makers preferences

and elicit importance weights for decision-making criteria – type of aggregation model used to combine criteria scores

Two main stages in MCDA

21

A simple MCDA Model

∑=

=n

iijij swWBS

1

i = 1,…,n criteria wi = criteria weights j represents alternatives sij = scores for alternatives for different criteria WBS = Weighted Benefit Score

22

• New literature review – searched PUBMED, Condit, grey literature since 2001 – hand searches of references and contacting key authors – 14 priority setting applications using MCDA – 21 priority setting and decision analysis methods papers – much of grey literature ‘not accessible’ or embargoed – MCDA examples go back to the early 90s and maybe

earlier

MCDA in priority setting

23

• Often analytical approaches offer little guidance to decision makers – Focus on a single criteria (e.g., effectiveness, or cost-

effectiveness)

• If more then one criteria is in play, approaches don’t inform how to integrate these – Decision makers need to make choices when considering

multiple criteria simultaneously

• All criteria relevant to the policy context need to be taken into account

24

Baltussen and Niessen (2006)

• What are the broader strategic priorities of the system/ organization under study?

• Out of the strategy flows a set of objectives, which can be seen as a bridge between the criteria and the strategy

• Criteria serve to operationalize the objectives and in this are rooted in the strategic context

25

Starting point

• A coherent criteria set is: (Roy, 1985) – Exhaustive (nothing important left out) – Consistent (no secret preferences) – Non-redundant (no double counting)

• Effective criteria are: (Yoe, 2002) – Directional (maximum, minimum or optimum) – Concise (smallest number of measures) – Complete (no significant impact left out) – Clear (understandable to others)

• Criteria are often correlated but can still be acceptable • Criteria should be tested throughout the decision process

26

Requirements for Decision Criteria

MCDA methods: decision criteria

• 52 different criteria listed in 14 studies • Most common:

Criteria Frequency

Accessibility 10

Reducing inequalities 10

Effectiveness 8

Alignment with strategic plan/policies 7

Value for money 7

Affordability 7

Integration with other programs 7

27

MCDA methods: criteria domains

Domain Frequency

Acceptability 4

Affordability 7

Alignment with policy 10

Availability of alternatives 1

Burden of disease 11

Effectiveness 18

Equity 26

Feasibility 14

Partnerships 10

Prevention 3

Public/patient centredness 16

Quality of evidence 5

Quality of intervention 5

Research and development 2

Value for money 18

28

Criteria and cancer control

• Criteria developed by the National Cancer Control Initiative (Australia) - Health gain – Disability Adjusted Life Years (DALYs) - (Vertical) equity – reducing inequalities in mortality rates in

vulnerable populations - Size of the health burden – disease burden in DALYs and

cost to the health system - Acceptability and feasibility - Level of Evidence

29

Criteria and cancer control

• Criteria developed with BC cancer control decision-makers - Health gain – Quality Adjusted Life Years (QALYs) - Resource impact – budget constraints - Resource impact – resource constraints - Availability/advisability of alternatives

30

NICE (England and Wales) – Clinical effectiveness – Cost-effectiveness – Feasibility and impact – Equity and equality – Acceptability and appropriateness

PBAC (Australia) – Gap in current coverage – Cost-effectiveness – Effectiveness and safety – Community ‘need’

Criteria used by national HTA Agencies

MCDA methods: criteria weights

• 4 studies did not report weights • 7 studies used allocation of points (direct rating) • 2 studies used a combination of ratio estimation and

direct rating • 1 study used indifference methods (DCEs) • Previous studies have also used swing weights

(hybrid of indifference method) • No studies have used gambles – all choices riskless

32

Weighting the criteria

33

• Ranking method • the criteria are simply ranked in perceived order of

importance by decision-makers: c1 > c2 > c3 > … > ci. The method assumes that the weights are non-negative and sum to 1.

Weighting the criteria

34

• Examples of rating methods: • Point allocation approach is based on allocating points ranging

from 0 to 100, where 0 indicates that the criterion can be ignored, and 100 represents the situation where only one criterion need to be considered.

• Ratio estimation is a modification of the point allocation method. A score of 100 is assigned to the most important criterion and proportionally smaller weights are given to criteria lower in the order. The score assigned for the least important attribute is used to calculate the ratio

MCDA methods: aggregation rule

• Where the aggregation rule was presented almost all applications (9) used an additive functional form

• 3 did not state the functional form used • 1 used an exponential function • 1 used a variant of the multiplicative function • Choice of functional form was rarely justified

35

• Additive Difference Independence (ADI) assumption – ADI requires that the relative strength of preference between

attributes does not depend on levels of other attributes – i.e. amount of utility derived from changing one attribute is same

regardless of all other attributes

– Assessing scaling constants: wi are utilities for each attribute best, with other attributes set at their worst level; constants sum to unity

36

∑=

====3

1

*3

02

013

03

*2

012

03

02

*11 1),,(,),,(,),,(

iiwandxxxuwxxxuwxxxuw

)()()(),,( 333222111321 xuwxuwxuwxxxu ++=

• Multiplicative Difference Independence (MDI) assumption – ADI relaxed so that interdependencies between pairs, triplets etc. of

attributes are allowed restrictive as have only a single interaction parameter, k, i.e. x1 depends as much on x2, as x2 does on x3 etc

– Assessing scaling constants: wi scaling constants as for additive model; k interaction constant found by solving:

37

)()()(

)()()()()()()()()(),,(

3322113212

332232331131221121

333222111321

xuxuxuwwwkxuxuwwkxuxuwwkxuxuwwk

xuwxuwxuwxxxu

+

+++++=

∑∑∏===

<>><<−+=+3

1

3

1

3

1

10101]1[1i

ii

ii

i wifkandwifkwherekwk

• Multilinear Difference Independence (MLDI) assumption – MDI relaxed so that wide range of interaction parameters are allowed

– Assessing scaling constants: wi scaling constants as for additive model; additional interaction constants found by solving:

38

)()()()()()()()()(

)()()(),,(

332211123

332223331113221112

333222111321

xuxuxuwxuxuwxuxuwxuxuw

xuwxuwxuwxxxu

++++

++=

321231312*3

*2

*1123

32*3

*2

0123

31*3

02

*113

2103

*2

*112

),,(

),,(

),,(

),,(

wwwwwwxxxuwwwxxxuw

wwxxxuwwwxxxuw

−−−−−−=

−−=

−−=

−−=

• Primary aim of MCDA is to develop models of decision-maker objectives and their value trade-offs so that alternatives under consideration can be compared with each other in a consistent and transparent manner

• Process is often more important than the numbers • Value focussed thinking and values clarification • MCDA practice suggests preferences are constructed as

part of the decision-making process, not endowed • Consistent with deliberative-analytic methods

What’s useful about MCDA?

39

What are some of the challenges?

• Some methodological challenges - who’s criteria count – society, patients, decision-

makers etc. - methods used to elicit and describe decision-makers

preferences, including the relationship between objectives and criteria

- methods used to elicit importance weights for decision-making criteria

- type of aggregation model used to combine criteria scores

40

• Criteria will reflect values of different players • Society at large

– Public involvement but some challenges

• Health care system/ organization – Dept. of Health, Clinical groups, Board of Directors

• Individual decision-makers – Easiest but may be limiting

• Surveys, focus groups, business plans

Whose Criteria Count?

41

• MCDA does not: – Decide which criteria to include – Decide what weight to place on each criterion – Replace decision-making

• Decision making in health care is value based – MCDA output can inform the final decision but should not

be the final decision – Consensus model in decision making; use MCDA results at

starting point for discussion

42

Value judgements

• Economics has often focussed on prescriptive behavioural rules, based on utility maximisation and game theory

• Psychology has sought to explain actual individual behaviour, and why it can deviate from prescriptive rules

• Decision analysis – including MCDA - tries to combine prescriptiveness with practicality (to a greater or lesser extent)

• All share common heritage from von Neumann and Morgenstern

Common ground

43

A d v a n c i n g H e a l t h E c o n o m i c s , S e r v i c e s , Po l i c y a n d E t h i c s

Acknowledgements

ARCC is funded by the Canadian Cancer Society

Ian Crowmell and Craig Mitton are co-authors on the MCDA and priority setting study

Email: speacock@bccrc.ca

ARCC website: www.cc-arcc.ca