The uptake of value of information methods

Solutions found and challenges to come

Alan BrennanDirector of Operational ResearchScHARR

Aim of this session

• Discuss the roles of VoI analysis • Demonstrate recent technical progress • Growth in interest in and use of VoI

methods• Discuss challenges for further uptake• Some recommendations

• Personal Views !

What is Value of Information?

• Given …. a choice between strategies, a decision rule for adoption of strategies,some uncertainty VoI analysis tells us…› How valuable more information would be

to reduce uncertainty, to help us choose › Which uncertain parameters are most crucial› How valuable a sample of size n=10 would be

compared to n=100, or n=1000 ….

Roles of VoI

Probabilistic Sensitivity Analysis Without VoI

• C-E plane and CEAC• i.e. describes how uncertain we are

-£2,000

-£1,500

-£1,000

-£500

£0

£500

£1,000

£1,500

£2,000

-1.4 -1.2 -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6

Inc QALY

Inc

Co

st

Cost Effectiveness Acceptability of T1 versus T0

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

£- £20,000 £40,000 £60,000 £80,000 £100,000 £120,000 £140,000

Threshold (MAICER)

Pro

ba

bil

ity

Co

st

Eff

ec

tiv

e

T1

Probabilistic Sensitivity Analysis With VoI adds (1)

All

(tr

ial,

util

+du

rati

ons)

Tri

al +

Uti

lity

Tri

al

EVSI n=50£-£200£400£600£800

£1,000£1,200£1,400

EVSI n=50 2 level EVPI

EVI for Parameters: research options

• Which parameters or groups of parameters are important

• i.e. the causes - why we are uncertain

Probabilistic Sensitivity Analysis With VoI adds (2)

EVSI :- 2 Level Algorithm Results(1,000 outer x 1,000 inner simulations)

£0

£200

£400

£600

£800

£1,000

£1,200

£1,400

0 50 100 150 200 250

Sample Size (n)

EV

SI (

£)

% response/ utility trial +duration obsn'l study

% response and utility trial

duration of responseobservational study

utility observational study

% response trial

• Value of different samples and research design combinations

• i.e. how best to resolve the uncertainty

Probabilistic Sensitivity Analysis - VoI difficulty myth

• “It is complex”

• “It takes a long time”

• “Methods are not developed”

• “People can’t understand it”

Probabilistic Sensitivity Analysis - VoI truth

• If you have done a probabilistic sensitivity analysis i.e.

•Then you are 90% to 95% there

-£2,000

-£1,500

-£1,000

-£500

£0

£500

£1,000

£1,500

£2,000

-1.4 -1.2 -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6

Inc QALY

Inc

Co

st

Cost Effectiveness Acceptability of T1 versus T0

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

£- £20,000 £40,000 £60,000 £80,000 £100,000 £120,000 £140,000

Threshold (MAICER)

Pro

bab

ility

Co

st E

ffec

tive

T1

Prioritising and Planning HTA

• Recommendations from systematic review of the use of modelling in planning and prioritising trials (HTA 2003; Vol 7: number 23)

• Results of Pilot Study show ..• Beneficial in refining research design and

quantifying uncertainty• choice of timing and topic are vital • who should do the analysis is important

Industry Roles – Societal Perspective

• Undertaking probabilistic sensitivity analysis and VoI to be ahead of re-imbursement authorities

• VoI to identify likelihood of cost-effectiveness and priorities for further data collection› Very early Discovery phase work› Phase II results available – now design phase III› Phase III results available – what else do we

need to make the economic case?

Industry Roles – Commercial Perspective

• Approaches still developing (and commercial in confidence)

• Define Net benefit not as λ * QALY – cost but rather as commercial

sales / profit Or • model linking sales to likelihood of re-

imbursement or extent of cost-effectiveness

Growth in UptakeA personal view

• 1996 – Claxton and Posnett – “what are all these squiggles, it will never catch on”

• 1999 – doing our review, realising the conceptual validity and practicability of VoI

• 2000 – some applications but people are unsure on methods

• 2001 – IHEA York / MDM San Diego 2 or 3 speeches (all UK)

Growth in UptakeA personal view

• 2002 – CHEBS Focus fortnight with ourselves Karl Claxton and Tony Ades – methods sorted

• Nice Appraisals beginning to use VoI• 2002 – MDM Baltimore

-7 people UK and Canada – lots of interest • 2003 – MDM Chicago

-15 to 20 people (US, Canada, Netherlands, UK) even more interest

Technical Problems Recently Solved

Technical Problems Recently Solved:EVPI

• Correct method for EVPI = 2 level simulation

EVPI =

• 1 level simulation EVPI works if ….(a) the net benefit functions are linear

functions of the -i for all of the decisions d and all of the possible values of the parameter set of interest i, and (b) if i and -i are independent. “

)NB(d,max|)NB(d,max EEEd

id ii

Technical Problems Recently Solved:EVSI

• Correct Method for EVSI = 2 level simulation (Bayesian update given simulated collected data)

• EVSI easy with conjugate distributions (i.e. retain same functional form when additional data is collected and synthesised)› Normal, Beta, Gamma, Lognormal

• EVSI more complex without conjugacy

)NB(d,max|,max EXdNBEEEVSI

di

dX i

Technical Problems Recently Solved: Shortcuts

• Complex models can be ‘emulated’ e.g. Gaussian Processes, which also offer quicker EVPI and EVSI for 1 variable calculation functionality

• Laplace approximation can help shortcut to a 1 level simulation for EVSI calculations (poster)

• The EVSI curve has a common but not universal shape – exponential of square root of n

• Valuing additonal data in survival trials (poster)

)NB(d,max|,max EXdNBEEEVSI

di

dX i

Technical Problems Recently Solved: Software

• EXCEL for models• R / Splus advantages

› More sampling functionality› More optimisation functionality› Faster running times› Easier code

• WinBUGS for complex posterior distributions – interlinkage with R and Splus

Key Challenges: Uptake for Sensitivity Analysis - AIDA

• Awareness Interest Desire Action

• Publications / Conferences / Seminars• Training

York / Oxford Advanced Modelling CourseCHEBS characterising uncertainty and analysing outputs courses (2004)Bristol WinBUGS course (2004)

• Network of experts• Web resources

Key Challenges: Uptake in trial design

• Engaging with traditional trial designers• Clinically significant difference is a

proxy for the decision rule• Need to work together to compare

results and approaches on pilot trials • particularly ones with Economic and

Quality of life data to be collected

Key Challenges: Meeting Criticisms

1. How do you know uncertainty is properly characterised i.e. you might be uncertain about the uncertainty

2. You need to multiply by the number of people affected by the decision - How long does the technology / decision last – 2 years, 5 years, 10 years? An issue even without doing EVI

3. Information has value beyond the jurisdiction

Key Challenges: Technical

• Correlation, Correlation, Correlation

• Integrated VoI and evidence synthesis quickly

• Reversible decisions

• Deciding to wait – options pricing

Conclusions

• Significant recent progress, interest and growth

• Key challenges› Speeding up and teaching standard routines to

do the calculations› Working through when exactly VoI is likely to

be most valuable› Engaging sceptics to collaborate

• Now is the time ….

Conclusions