decision and cost-effectiveness analysis: understanding sensitivity analysis training in clinical...
TRANSCRIPT
Decision and Cost-Effectiveness Analysis: Understanding Sensitivity Analysis
Training in Clinical ResearchDCEA Lecture 5
UCSF Dept. of Epidemiology & Biostatistics
Jose Luis Burgos, MD, MPH, AAHIVS
March 4, 2009
Project Map
• Think through your research question• Sketch your analysis • Collect data for your model• Adjust model• Run-test your model• Conduct Sensitivity Analysis • Write it up.
Objectives
To understand the purposes of sensitivity analysis.
To understand techniques used for sensitivity analysis.
Why do sensitivity analyses?
• All CEAs have substantial uncertaintyAll CEAs have substantial uncertainty
• Sensitivity analyses deal with that Sensitivity analyses deal with that uncertainty systematicallyuncertainty systematically
• Convince audience results are ‘robust’ – Convince audience results are ‘robust’ – qualitativequalitative findings don’t findings don’t change with small changes in inputschange with small changes in inputs
Sensitivity analysis
Prior lectures reviewed how inputs are determined, Prior lectures reviewed how inputs are determined, plus a few simple sensitivity/threshold analyses.plus a few simple sensitivity/threshold analyses.
This lecture will cover four topics:This lecture will cover four topics:
1. Types of uncertainty 1. Types of uncertainty
2. Deterministic sensitivity analyses (one-way, 2. Deterministic sensitivity analyses (one-way,
multi-way, scenario) multi-way, scenario)
3. Probabilistic sensitivity analysis (Monte Carlo)3. Probabilistic sensitivity analysis (Monte Carlo)
4. Uses of sensitivity analysis4. Uses of sensitivity analysis
Types of uncertainty
1.1. Parameter Uncertainty Parameter Uncertainty • What’s are the correct input values?What’s are the correct input values?
2.2.Methodological uncertaintyMethodological uncertainty
a)a) Model Structure Model Structure
• How values are combined How values are combined
or modeledor modeled
b)b) Model Process Model Process
a)a) Implicit decisions made by the analyst such as Implicit decisions made by the analyst such as
viewpoints or effects consideredviewpoints or effects considered
Addressing Methodological Uncertainty
• Sensitivity Analysis– Scenario analysis with different models to
combine costs and estimate effects
• Statistical Analysis – Where multiple parameter sets available, can test
the fit of different models
Addressing Model Process Uncertainty• Standardized CE analysis – difficult and different panels give
different recommendations, but key common components are (from Drummond): 1. The background of the question2. The viewpoint for the analysis3. The reason for selecting a particular form of analysis4. The population to which the analysis applies5. The comparators being assessed6. The source of the medical evidence and its quality 7. The range of costs considered and their measurement8. The measure of benefit in the economic study (e.g. LY gained, QALYs
gained) 9. The methods for adjusting costs and benefits according to their
timing10. The methods dealing with uncertainty 11. The incremental analysis of costs and benefits12. The overall results of the study and its limitations
Deterministic Sensitivity AnalysesHow does assigning specific different values to inputs change
output?
One-way (‘univariate’): Vary 1 input at a time Multi-way (‘multivariate’): Vary 2+ inputs at a time Scenario (variant of multi-way): Tests set of relevant
conditions. Threshold analysis (one-way or multi-way): Input values
beyond which cost-effectiveness achieved (or lost).
Univariate Sensitivity Analysis
• Examine robustness of ICERs to changes in a single parameter: – ‘best’, ‘high’ and ‘low’ estimates (but experts
consistently underestimate true variability)– Value +/- 1 SD– 95% CI (based on observed or assumed distribution)– ‘clinical meaningful range– Extremes– Threshold analysis
One-way SA – Aneurysm managementOne-way SA – Aneurysm management
Sensitivity analysis (1-way),aneurysm management CUA
$0
$100,000
$200,000
$300,000
$400,000
0 0.005 0.01 0.015 0.02 0.025
Annual risk of rupture
$ / Q
ALY
, clip
ping
Base case estimate
One-way SA
Muennig, 2008
13
One-way SA – LTBI programOne-way SA – LTBI program
Peginterferon Model Inputs: Estimated “base case” and rangePeginterferon Model Inputs: Estimated “base case” and range
European Journal of Gastroenterology & Hepatology. 2007;19:631-638.
LTBI Model Inputs: Estimated “base case” and rangeLTBI Model Inputs: Estimated “base case” and range
Int Jour Tuberc Lung Dis 2008
Univariate Sensitivity Analyses: Univariate Sensitivity Analyses: Base case and range of outcomes for 1,000 IDUsBase case and range of outcomes for 1,000 IDUs
Burgos JL, Kahn JG, et al. Int Jour Tuberc Lung Dis 2008
Automating one-way SAs: Automating one-way SAs: Male circumcision for HIV prevention in South AfricaMale circumcision for HIV prevention in South Africa
$100
$200
$300
$400
Co
st p
er H
IV c
ase
aver
ted
Percentiles of the variables
Protective ef fect
Cost per male circumcision
Multiplier due to epidemic ef fects
Proportion HIV-uninfected
Disinhibition impact on protective ef fect
Frequency of short-term adverse events (outpatient)
Kahn JG, et al. PLoS Med 3(12): e517. doi:10.1371/journal.pmed.0030517
Male circumcision for HIV prevention in South AfricaMale circumcision for HIV prevention in South AfricaConsidering HAART cost avertedConsidering HAART cost averted
Kahn JG, et al. PLoS Med 3(12): e517. doi:10.1371/journal.pmed.0030517
Tornado Diagram
Two-way SA:CE of Empowerment program
Figure 1. Sensitivity analysis: cost per HIV infection averted by reduction in risk behavior, 5 years
$0
$10,000
$20,000
$30,000
$40,000
0% 10% 20% 30% 40% 50%
Reduction in risk behavior
Cos
t per
HIV
infe
ctio
n av
erte
d
Steady state
Pre-steady state
Two-way sensitivity analysis for changes in HIV risk and average cost for managing active TB cases
Burgos JL, Kahn JG, et al. Int Jour Tuberc Lung Dis 2008
Three-Way SA
Kahn JG, et al. PLoS Med 3(12): e517. doi:10.1371/journal.pmed.0030517
Scenario SA
Burgos JL, Kahn JG, et al. Int Jour Tuberc Lung Dis 2008
Low HIV prevalence setting: 25% in CSWs; 16.4% in clients
Low HIV/AIDS treatment cost: $1,433 Med. HIV/AIDS treatment cost: $2,507 High HIV/AIDS treatment cost: $3,582
Low FC cost: $0.33
Medium FC cost:
$0.66
High FC cost: $1.32
Low FC cost: $0.33
Medium FC cost:
$0.66
High FC cost: $1.32
Low FC cost: $0.33
Medium FC cost:
$0.66
High FC cost: $1.32
($3,864) ($1,824)$2,076 [$509]
($7,426) ($5,386) ($1,486) ($10,989) ($8,949) ($5,059)
Medium HIV prevalence setting: 50.3% in CSWs; 33.0% in clients
Low HIV/AIDS treatment cost: $1,433 Med. HIV/AIDS treatment cost: $2,507 High HIV/AIDS treatment cost: $3,582
Low FC cost: $0.33
Medium FC cost:
$0.66
High FC cost: $1.32
Low FC cost: $0.33
Medium FC cost:
$0.66
High FC cost: $1.32
Low FC cost: $0.33
Medium FC cost:
$0.66
High FC cost: $1.32
($6,023) ($3,983) ($83) ($11,203 ($9,163) ($5,263) ($16,386) ($14,346) ($10,446)
Multivariate SA on female condom promotion: Net costs by HIV prevalence and key cost inputs for 1,000 CSWs
Threshold Analysis: NVP for prevention of vertical transmission of HIV in sub-Saharan Africa
Input values needed for $50/DALY
15% HIV prevalence
30% HIV prevalence
Regimen efficacy (47%)18.0% 10.6%
VCT cost ($7.30) $18.50 $36.00
HIV transmission (25.1%) 9.6% 5.6%
HIV prevalence for $50/DALY
4.5%
Threshold Analysis: NVP $ for prevention of vertical transmission of HIV in sub-Saharan Africa
Marseille E, Kahn JG, Saba J. AIDS 1998; 12:939-948
Clipping, asympt, <10 mm, SAH hx Coiling, asympt, <10 mm, SAH hx Rupture risk/yr 0.0050 Rupture risk/yr 0.0050 RR rupture w/ surgery 0 RR rupture w/ surgery 0.1 Surgical mortality 0.023 Surgical mortality 0.004 Surg morb (disability) 0.075 Surg morb (disability) 0.037 Cost of surgery 25,150 $ Cost of surgery 20,660 $
Δ QALYs -0.58 Δ QALYs 0.40
Δ $ $34,324 Δ $ $22,492
Scenario Analyses Aneurysm management CUA
$ / QALY Dominated Strategy
$ / QALY $56,230
Addressing Parameter Uncertainty:Multivariate Sensitivity Analysis
Types of Multivariate Sensitivity Analysis:
• Repeat bivariate• Maximize / minimize CE ratio for different
parameter combinations• Scenario analysis • Monte Carlo simulations under different
assumed distributions for parameters (probabilistic sensitivity analysis’ Gold)
Probabilistic sensitivity analysis
What is it?
What is it good for?
The problem with deterministic SAs
No estimate of the probability of achieving a particular outcome
(Probabilistic SAs are the remedy)
Probabilistic sensitivity analyses
Value •Returns the likelihood of attaining particular outcome or outcome range.• Everything known about each input expressed all at once.•Particularly valuable when many inputs important.
DrawbackNeed to know, or be able to make decent estimates of,
the underlying probability distribution.
From empirical data to PD (1)
# of clients # of Subjects1 to 19 320 to 39 1540 to 59 3060 to 79 2580 to 99 12
100 to119 7120 to 139 5140 to 159 2
≥160 1
Frequency distribution of # of clients reported by 101 FSWs
Variable | Obs Mean Std. Err. SD [95% Conf. Interval]-------------+---------------------------------------------------------------# of Clients | 100 67 3.22 32 60.6 - 73.4
From empirical data to PD (2)
Graphical Representation of the # of clients reported by 101 FSWs
# of clients # of Subjects1 to 19 0.0320 to 39 0.1540 to 59 0.360 to 79 0.2580 to99 0.12
100 to119 0.07120 to 139 0.05140 to 159 0.02
≥160 0.01
Probability distribution of # of clients reported by 100 FSWs
From empirical data to PD (3)
From empirical data to PD (4)Graphical Representation of the Prob dist. of clients reported by 101 FSWs
From empirical data to PD (5)Probability Distribution of # of clients among 10,000 FSWs
Variable | Obs Mean Std. Err. SD [95% Conf. Interval]-------------+----------------------------------------------------------------------------------- # clients | 10,000 68 0.36 33 66.7 - 67.7
Triangular Distribution
Muennig, 2008
Published distributions
Henson SJ. Estimating costs of acute gastrointestinal Illness in BC. Int Jou Food Microb. 2008; 127: 43-45
Frequency Chart
%
Mean = 85.7%.000
.007
.014
.021
.029
0
14.25
28.5
42.75
57
79.5% 82.6% 85.7% 88.7% 91.8%
2,000 Trials 23 Outliers
Forecast: Percent reduction in mortality
Monte Carlo simulation output
Crystal Ball output
$226-$504
SA For QALYs Gained
Treeage output
Other uses of sensitivity analysis
(the inner teachings)(the inner teachings)• Planning the analysis
• Debugging the model
• Documenting relationships between inputs and outputs
• Identifying thresholds
• Influencing policyInfluencing policy
Other uses: Planning the analysis
• Program software to permit SAs on likely SA variables.
• SA curves provide a check on integrity of model.
• Identify candidates for more data collection early.
Other uses: Debugging the model
Tricks of the trade
• One-ways best because simple and intuitive. • Plug in extreme values. • Separate diagnosis of numerator from denominator.
• Break outputs down further if necessary (intervention versus control arms).
Other uses: Documenting relationships between inputs and outputs
Distinguish between ‘bugs’ and insights.
Examples of insights: • Slowing disease progression can increase costs.
• Higher disease prevalence can mean lower benefits.
• Benefits decrease with age - competing mortality risks.
Unexpected dynamic uncovered Unexpected dynamic uncovered by SA: Female condoms studyby SA: Female condoms study
Figure 2: HIV cases averted per 1,000 CSWs by HIV prevalence in CSWs and partners
0.0
1.0
2.0
3.0
4.0
5.0
6.0
7.0
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
HIV prevalence in CSWs (Prevalence in clients = 67% of CSW value)
HIV
cas
es a
vert
ed
Marseille et. al. Soc Sci Med. 2001 Jan;52(1):135-48.
Other uses: Identify thresholds – Influence Policy
Preventing HIV vertical transmission in sub-Saharan Africa
• Cost of ARVs to prevent vertical transmission.
• Universal versus targeted provision of NVP.
Cost per DALY of HIVNET 012 NVP regimen as function of HIV
seroprevalence and type of counseling/testing regimen
0
5
10
15
20
25
30
35
40
45
50
0% 5% 10% 15% 20% 25% 30%
HIV-1 seroprevalence
Cos
t per
DA
LY (U
S$)
UniversalTargeted
Pegynterferon CE acceptability curve
European Journal of Gastroenterology & Hepatology. 2007;19:631-638.
C-E Acceptability Curve (QALYs)WTP
$0 0.4$50 0.4$100 0.5$150 0.5$200 0.6$250 0.6$300 0.7$350 0.7$400 0.8$450 0.8$500 0.8
WTP
$550 0.8$600 0.9$650 0.9$700 0.9$750 0.9$800 0.9$850 0.9$900 0.9$950 0.9$1,000 0.9
50
Summary
• SA is a set of techniques for the explicit management of uncertainty.
• Essential part of establishing key findings.
• Indispensable for convincing your audience that your results are technically sound and policy-relevant.
Practicum