value of information for complex economic models jeremy oakley department of probability and...
Post on 21-Dec-2015
216 views
TRANSCRIPT
![Page 1: Value of Information for Complex Economic Models Jeremy Oakley Department of Probability and Statistics, University of Sheffield. Paper available from](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d6d5503460f94a4d93f/html5/thumbnails/1.jpg)
Value of Information for Complex Economic Models
Jeremy Oakley
Department of Probability and Statistics,University of Sheffield.
Paper available from www.sheffield.ac.uk/chebs/papers.html
![Page 2: Value of Information for Complex Economic Models Jeremy Oakley Department of Probability and Statistics, University of Sheffield. Paper available from](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d6d5503460f94a4d93f/html5/thumbnails/2.jpg)
Outline
1. Motivation
2. Expected value of perfect information (EVPI)
3. Emulators and Gaussian processes
4. Illustration: GERD model
![Page 3: Value of Information for Complex Economic Models Jeremy Oakley Department of Probability and Statistics, University of Sheffield. Paper available from](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d6d5503460f94a4d93f/html5/thumbnails/3.jpg)
1) Introduction
• An economic model is to be used to predict the cost-effectiveness of a particular treatment(s).
• The economic model will require the specification of various input parameters. Values of some or all of these are uncertain.
• This implies the output of the model, the cost-effectiveness of the treatment is also uncertain.
![Page 4: Value of Information for Complex Economic Models Jeremy Oakley Department of Probability and Statistics, University of Sheffield. Paper available from](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d6d5503460f94a4d93f/html5/thumbnails/4.jpg)
Introduction
• We wish to identify which input parameters are the most influential in driving this output uncertainty.
• Should we learn more about these parameters before making a decision?
![Page 5: Value of Information for Complex Economic Models Jeremy Oakley Department of Probability and Statistics, University of Sheffield. Paper available from](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d6d5503460f94a4d93f/html5/thumbnails/5.jpg)
Introduction
• A measure of importance for an input variable have been proposed, based on the expected value of perfect information (EVPI) (Felli and Hazen, 1998, Claxton 1999).
• Computing the values of these measures is conventionally done using Monte Carlo techniques. These invariably require a very large numbers of runs of the economic model.
![Page 6: Value of Information for Complex Economic Models Jeremy Oakley Department of Probability and Statistics, University of Sheffield. Paper available from](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d6d5503460f94a4d93f/html5/thumbnails/6.jpg)
Introduction
• For computationally expensive models, this can be completely impractical.
• We present an efficient alternative to Monte Carlo, in terms of the number of model runs required.
![Page 7: Value of Information for Complex Economic Models Jeremy Oakley Department of Probability and Statistics, University of Sheffield. Paper available from](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d6d5503460f94a4d93f/html5/thumbnails/7.jpg)
2) EVPI
• We work with net benefit: the monetary value or utility of a treatment is
K x efficacy – cost with K the monetary value of a unit
increase in efficacy. • The net benefit of any treatment option
will be a function of the parameters in the economic model.
![Page 8: Value of Information for Complex Economic Models Jeremy Oakley Department of Probability and Statistics, University of Sheffield. Paper available from](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d6d5503460f94a4d93f/html5/thumbnails/8.jpg)
EVPI
• Denote the net benefit of treatment option t given model parameters X to be
NB (t , X )• Given X, the economic model returns
NB (t , X ) for each t . • The ‘true’ values of the model
parameters X are uncertain.
![Page 9: Value of Information for Complex Economic Models Jeremy Oakley Department of Probability and Statistics, University of Sheffield. Paper available from](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d6d5503460f94a4d93f/html5/thumbnails/9.jpg)
EVPI
• The baseline decision is to choose t with the largest expected net benefit:
NB* = maxt EX {NB (t , X )}
• The decision maker will have utility NB* if they choose the best treatment now with no additional information.
![Page 10: Value of Information for Complex Economic Models Jeremy Oakley Department of Probability and Statistics, University of Sheffield. Paper available from](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d6d5503460f94a4d93f/html5/thumbnails/10.jpg)
EVPI
• Now suppose the decision-maker chooses to learn the value of all the uncertain input variables X before choosing a treatment.
• They would then choose the treatment with the highest net benefit conditional on X, i.e., they would consider
maxt {NB (t , X )}
![Page 11: Value of Information for Complex Economic Models Jeremy Oakley Department of Probability and Statistics, University of Sheffield. Paper available from](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d6d5503460f94a4d93f/html5/thumbnails/11.jpg)
EVPI
• Before actually observing X, they will expect to achieve a net benefit of
EX [maxt {NB (t , X )}]
• The expected value of this course of action is the expected gain in net benefit over the baseline decision:
EX [maxt {NB (t , X )}] – NB*.
• This is the (global) EVPI.
![Page 12: Value of Information for Complex Economic Models Jeremy Oakley Department of Probability and Statistics, University of Sheffield. Paper available from](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d6d5503460f94a4d93f/html5/thumbnails/12.jpg)
Partial EVPI
• Now suppose the decision-maker chooses to learn the value of a single uncertain input variable Y , an element of X before making a decision.
• They would then choose the treatment with the highest net benefit conditional on Y , i.e., they would consider
maxt EX | Y {NB (t , X )}
![Page 13: Value of Information for Complex Economic Models Jeremy Oakley Department of Probability and Statistics, University of Sheffield. Paper available from](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d6d5503460f94a4d93f/html5/thumbnails/13.jpg)
Partial EVPI
• The expected value of learning Y before Y
is actually observed is then:EY [maxt EX |Y {NB (t , X )}] – NB *
• This is the partial expected value of perfect information (partial EVPI) for Y .
• The partial EVPI is zero if the decision-maker would choose the same treatment for any (plausible) value of Y .
![Page 14: Value of Information for Complex Economic Models Jeremy Oakley Department of Probability and Statistics, University of Sheffield. Paper available from](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d6d5503460f94a4d93f/html5/thumbnails/14.jpg)
Computing partial EVPIs
• We need to evaluateEY [maxt EX |Y {NB (t , X )}]
for each element Y in X.• The outer expectation EY is a one-
dimensional integral, and can be evaluated using numerical integration.
• The term maxt EX |Y is the maximum of (several) higher-dimensional integrals. This requires a large Monte Carlo sample to be evaluated.
![Page 15: Value of Information for Complex Economic Models Jeremy Oakley Department of Probability and Statistics, University of Sheffield. Paper available from](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d6d5503460f94a4d93f/html5/thumbnails/15.jpg)
Patient Simulation Models
• Computing partial EVPIs for computationally cheap models, while not trivial, is relatively straightforward.
• However, for one class of models, patient simulation models, a sensitivity analysis using Monte Carlo methods will be out of reach for the model user.
![Page 16: Value of Information for Complex Economic Models Jeremy Oakley Department of Probability and Statistics, University of Sheffield. Paper available from](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d6d5503460f94a4d93f/html5/thumbnails/16.jpg)
Patient Simulation Models
• An example is given in Kanis et al (2002) for modelling osteoporosis:
• For an osteoporosis patient, a bone fracture significantly increases the risk of a subsequent fracture.
• Residential status of a patient needs to be tracked, in order that costs are not double-counted.
![Page 17: Value of Information for Complex Economic Models Jeremy Oakley Department of Probability and Statistics, University of Sheffield. Paper available from](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d6d5503460f94a4d93f/html5/thumbnails/17.jpg)
Patient Simulation Models
• Progress is to be modelled over a 10 year period. Including the approptiate features in the model necessitates a patient simulation approach.
• The net benefit for a given set of input parameters is obtained by sampling events for a large number of patients.
• The model takes over an hour for a single run at one set of input parameters.
![Page 18: Value of Information for Complex Economic Models Jeremy Oakley Department of Probability and Statistics, University of Sheffield. Paper available from](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d6d5503460f94a4d93f/html5/thumbnails/18.jpg)
Patient Simulation Models
• For a model with 20 uncertain input variables, computing the partial EVPI reliably using Monte Carlo for each input variable would require a possible minimum of 500,000 model runs.
• At one hour for each run, this would take 57 years!
• Something more efficient is needed…
![Page 19: Value of Information for Complex Economic Models Jeremy Oakley Department of Probability and Statistics, University of Sheffield. Paper available from](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d6d5503460f94a4d93f/html5/thumbnails/19.jpg)
3) Emulators
• For each treatment option t, and given values for the input parameters X = x, the economic model returns NB (t , x )
• We think of the model as a collection of functions
NB (t , x ) = ft (x)
• Partial EVPIs can be computed more efficiently by exploiting the `smoothness’ of each ft (x)
![Page 20: Value of Information for Complex Economic Models Jeremy Oakley Department of Probability and Statistics, University of Sheffield. Paper available from](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d6d5503460f94a4d93f/html5/thumbnails/20.jpg)
Emulators
• We can compute partial EVPIs more efficiently through the use of an emulator.
• An emulator is a statistical model of the original economic model which can then be used as a fast approximation to the model itself.
• An approach used by Sacks et al (1989) for dealing with computationally expensive computer models.
![Page 21: Value of Information for Complex Economic Models Jeremy Oakley Department of Probability and Statistics, University of Sheffield. Paper available from](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d6d5503460f94a4d93f/html5/thumbnails/21.jpg)
Gaussian processes
• Any regression technique can be used. We employ a nonparametric regression technique based on Gaussian processes (O’Hagan, 1978).
• The gaussian process model for the function ft (x) is non-parametric; the only assumption made about ft (x) is that it is a continuous function.
![Page 22: Value of Information for Complex Economic Models Jeremy Oakley Department of Probability and Statistics, University of Sheffield. Paper available from](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d6d5503460f94a4d93f/html5/thumbnails/22.jpg)
Gaussian processes
• In the Gaussian process model, ft (x) is thought of as an unknown function, and uncertainty about ft (x) is described by a normal distribution.
• Correlation between ft (x1) and ft (x2) is modelled parametrically as a function of ||x1-x2||
![Page 23: Value of Information for Complex Economic Models Jeremy Oakley Department of Probability and Statistics, University of Sheffield. Paper available from](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d6d5503460f94a4d93f/html5/thumbnails/23.jpg)
Gaussian processes
• The partial EVPI for input variable Y is given by
EY [maxt EX |Y {NB (t , X )}] – NB *
• We need to evaluate EX |Y {NB (t , X )} for each t at various values of Y.
• Denote G (X |Y) to be the distribution of X given Y. Then
EX |Y {NB (t , X )} = ft (x) dG (x |y )
![Page 24: Value of Information for Complex Economic Models Jeremy Oakley Department of Probability and Statistics, University of Sheffield. Paper available from](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d6d5503460f94a4d93f/html5/thumbnails/24.jpg)
Gaussian processes
• We can use Bayesian quadrature (O’Hagan, 1993) to rapidly speed up the computation:
• Under the Gaussian process model for ft
(x), ft (x) dG (x |y )
has a normal distribution, and can be evaluated (almost) instantaneously.
• This reduces the number of model runs required from 100,000s to 100s.
![Page 25: Value of Information for Complex Economic Models Jeremy Oakley Department of Probability and Statistics, University of Sheffield. Paper available from](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d6d5503460f94a4d93f/html5/thumbnails/25.jpg)
4) Example: GERD model
• The GERD model, presented in O’Brien et al (1999) predicts the cost-effectiveness of a range of treatment strategies for gastroesophageal reflux disease.
• Various uncertain inputs in the model related to treatment efficacies, resource uses by patients.
• Model outputs mean number of weeks free of GERD symptoms, and mean cost of treatment for a particular strategy.
![Page 26: Value of Information for Complex Economic Models Jeremy Oakley Department of Probability and Statistics, University of Sheffield. Paper available from](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d6d5503460f94a4d93f/html5/thumbnails/26.jpg)
Example: GERD model
• We consider a choice between three treatment strategies:› Acute treatment with proton pump inhibitors
(PPIs) for 8 weeks, then continuous maintenance treatment with PPIs at the same dose.
› Acute treatment with PPIs for 8 weeks, then continuous maintenance treatment with hydrogen receptor antagonists (H2RAs).
› Acute treatment with PPIs for 8 weeks, then continuous maintenance treatment with PPIs at the a lower dose.
![Page 27: Value of Information for Complex Economic Models Jeremy Oakley Department of Probability and Statistics, University of Sheffield. Paper available from](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d6d5503460f94a4d93f/html5/thumbnails/27.jpg)
Example: GERD model
• There are 23 uncertain input variables. • Distributions for uncertain inputs detailed in
Briggs et al (2002).• We estimate the partial EVPI for each input
variable, based on 600 runs of the GERD model.
• We assume a value of $250 for each week free of GERD symptoms. (It is straightforward to repeat our analysis for alternative values).
![Page 28: Value of Information for Complex Economic Models Jeremy Oakley Department of Probability and Statistics, University of Sheffield. Paper available from](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d6d5503460f94a4d93f/html5/thumbnails/28.jpg)
Example: GERD model
![Page 29: Value of Information for Complex Economic Models Jeremy Oakley Department of Probability and Statistics, University of Sheffield. Paper available from](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d6d5503460f94a4d93f/html5/thumbnails/29.jpg)
Conclusions.
• The use of the Gaussian process emulator allows partial EVPIs to be computed considerably more efficiently.
• Sensitivity analysis feasible for computationally expensive models.
• Can also be extended to value of sample information calculations.