che seminar 20 november 2013

Improving health worldwide

www.lshtm.ac.uk

Developing appropriate methods

for handling missing data in

health economic evaluation

Manuel Gomes

CHE seminar, University of York

November 20, 2013

MRC Early Career Felloswship in Economics of Health

PROMs Nils Gutacker

Chris Bojke

Andrew Street

Others Rita Faria

David Epstein

Acknowledgments

Improving health worldwide www.lshtm.ac.uk

• Missing data problem in health economic evaluation

• Alternative methods for addressing missing data

• Why multiple imputation?

• Framework for sensitivity analysis

• Illustrate the methods in a re-analysis of PROMs for comparing provider performance

Overview


• Cost-effectiveness analyses are prone to missing data: – Patients lost to follow-up

– Incomplete resource use or quality-of-life questionnaires:

• Individual non-response

• Item non-response

• Key concern is that patients with missing information tend to be systematically different from those with complete data

• Concerns face economic evaluations based on a single-study and those that synthesise data from several sources in decision models

• Most published studies fail to address missing data (Noble et al 2012)

Missing data in economic evaluation

• Missing completely at random (MCAR) – Reasons for missing data are independent of both observed and unobserved

factors

E.g. Resource use questionnaires were lost

• Missing at random (MAR) – Prob of missingness is unrelated to unobserved values, given the observed data:

any systematic differences between missing and observed values can be explained by differences in observed factors.

E.g. Older people may be less likely to return their QoL questionnaire.

• Missing not at random (MNAR) – Conditional on the observed data, the likelihood of non-response is still related

to unobserved values.

E.g. Patients in poor health may be less likely to return their EQ-5D questionnaires because they are depressed.

Reasons for missing data

• Missing endpoints (e.g. costs and QALYs) are made up of multiple individual components with potentially distinct missingness patterns

• The probability distribution of missing data may differ by treatment group and endpoint

• Probability of observing one endpoint (e.g. costs) may depend on the level of the other endpoint (e.g. QoL)

• Distribution of costs and QALYs are often non-Normal

• Model for the missing data must recognise the structure of the data (e.g. hierarchical) and be compatible with model for the endpoints

Key considerations when dealing with

missing data in CEA


Alternative methods for dealing with missing data in CEA

Why multiple imputation?

• Complete-case analysis

• Available-case analysis

• Last value carried forward

• Mean Imputation

All assume MCAR

– very unlikely, particularly for patient-reported outcomes

– typically leads to biased estimates

Unprincipled methods (MCAR)

Regression (single) imputation

• Some sort of regression model is used to predict missing values conditional on the observed data (MAR).

• Missing observations are replaced by the predicted values

• Analysis model is then applied to the complete dataset

• Do not recognise that the ‘imputed’ values are estimated rather than known (uncertainty is underestimated)

Principled methods (MAR)

Likelihood-based approaches

• Maximum likelihood does not fill in the missing values (!)

• ML uses all observed data to search for the parameters that maximise the likelihood

• ‘Borrows’ information from observed data to estimate parameters for the incomplete variables

• Variables associated with missingness are often beyond those included in the analysis model (e.g. post-randomisation variables)

• With missing covariates may require more complex algorithms (E.g. Expectation-Maximisation).


Inverse probability weighting (IPW)

• Complete cases are weighted by the inverse probability of being observed

• Tend to be less efficient than, say ML methods, because uses only a subset of all available information.

• Although efficiency can be improved (e.g. augmented IPW within a SUR framework), implementation can be challenging in complex analysis models

• Limited use when covariates are missing


Full-Bayesian models

• Model for the missing data and analysis model are estimated simultaneously (typically using MCMC methods)

• May be advantageous when prior evidence is available to inform either model

• With flat priors, this method approximates multiple imputation

• Relatively complex to implement when covariates are missing

• May face convergence issues


Multiple Imputation

• Each missing value is replaced by a set of plausible values from the conditional distributional of the missing data given the observed

Key advantages when compared with previous methods:

– Imputation model is estimated separately from the analysis model (e.g. allows for the inclusion of ‘auxiliary variables’)

– Recognises the uncertainty associated with the missing data and the estimation of the imputed values

– Can handle missingness in both outcomes and covariates

– Provides a flexible framework for conducting sensitivity analyses


Multiple Imputation

Key requirements for valid inference:

– Imputation model must be compatible with analysis model: • E.g. Bivariate hierarchical model

– Imputation model is correctly specified

• E.g. Non-linear relationships between follow up and baseline QALY

– Imputed values drawn from multivariate Normal distribution

• Normalising transformations (e.g. skewed costs)

• Latent Normal models (e.g. ordinal EQ-5D components)


• Joint imputation model for missing costs and outcomes

• Missingness predictors (X) allowed to differ by endpoint

• Joint model recognises missing costs may be associated with outcomes, and vice-versa

• Can impute individual cost and outcome components (multivariate)

May be insufficient in more complex settings:

– Hierarchical studies (e.g. multicentre studies, meta-analysis)

– Non-randomised studies, where correctly specifying the imputation model can be challenging

Standard MI in CEA

𝑐𝑖 = 𝛽0𝑐 + 𝛽1

𝑐𝑋𝑖 + 휀𝑖𝑐

𝑒𝑖 = 𝛽0𝑒 + 𝛽1

𝑒𝑋𝑖 + 휀𝑖𝑒

휀𝑖𝑐

휀𝑖𝑒 ~𝐵𝑉𝑁

00

𝜎𝑐2 𝜌𝜎𝑐𝜎𝑒

𝜎𝑒2

• Multilevel MI recognises hierarchical data structures (probability of missing data may be more similar within than across centres)

• Imputes from multivariate hierarchical Normal (Gomes et al 2013):

• Random effects MI found to perform better than fixed effects MI (Diaz-Ordaz et al 2013)

• R and Stata code available

Multilevel MI

𝑐𝑖𝑗 = 𝛽0𝑐 + 𝛽1

𝑐𝑋𝑖𝑗 + 𝛽2𝑐𝑍𝑗 + 𝑢𝑗

𝑐 + 휀𝑖𝑗𝑐

𝑒𝑖𝑗 = 𝛽0𝑒 + 𝛽1

𝑒𝑋𝑖𝑗 + 𝛽2𝑒𝑍𝑗 + 𝑢𝑗

𝑒 + 휀𝑖𝑗𝑒

휀𝑖𝑗𝑐

휀𝑖𝑗𝑒 ~𝐵𝑉𝑁

00

,𝜎𝑐2 𝜌𝜎𝑐𝜎𝑒

𝜎𝑒2

𝑢𝑗𝑐

𝑢𝑗𝑒 ~𝐵𝑉𝑁

00

,𝜏𝑐2 𝜙𝜏𝑐𝜏𝑒

𝜏𝑒2


MULTIPLE IMPUTATION FRAMEWORK AND SENSITIVITY ANALYSIS

‘Robust’ MI (Daniel and Kenward 2012)

• Robust MI is based on the concept of ‘double robustness’ which aims to reduce reliance on correct specification of imputation model

• ‘Doubly robust’ estimators remain unbiased even if imputation model is misspecified, given the probability of missingness is correctly specified

• Analysts can be more confident about the correct specification of the latter (typically some sort of logistic model)

Sensitivity Analysis 1: model

specification

IMPLEMENTATION

1 – First, estimate the model for missing data, given the observed data.

2 – Obtain the predicted probabilities, say 𝜋 𝑖𝑗, of observing the

endpoint

3 – Estimate your imputation model, by including the inverse of the

probabilities obtained in step 2 (𝜋 𝑖𝑗−1) as an additional predictor.

4 – Combine the multiple estimates using Rubin’s rules as usual.

Remarks:

- estimates of the variance are not doubly robust

- extreme weights

Sensitivity Analysis 1: model

specification

Framing the problem

• True data mechanism is unknown given the data at hand

• Assessing whether CEA inference is sensitive to alternative assumptions about the reasons for the missing data is required

• Sensitivity parameters can be formulated:

1. In terms of missing data selection mechanism (departures from MAR), or

2. According to differences between distribution of observed and unobserved data

• Probability distribution for these parameters should ideally be informed by external evidence (e.g. expert elicitation)

Sensitivity analysis 2: Departures

from MAR

1. Selection models

• Requires the specification of a model that explicitly recognises the MNAR mechanism:

logit {𝑃 𝑅𝑖𝑗 = 1 𝑊𝑖𝑗 , 𝑍𝑗 } = 𝜂0 + 𝜂1𝑊𝑖𝑗 + 𝜂2𝑍𝑗 + 𝜑𝑗 + 𝜹𝒀𝒊𝒋

• Selection model is jointly estimated with analysis model, typically using MCMC methods in a Bayesian framework (Mason et al 2012)

• Prior distributions for 𝜹 are best informed by expert opinion


from MAR

Selection models approximated by importance weighting

• Avoids having to estimate 𝜹.

Implementation (Carpenter et al 2007)

• Impute missing data under MAR

• Results are combined by computing a weighted average instead of standard Rubin’s rules

• For a plausible 𝛿, imputations judged to have a more plausible MNAR mechanism are given a relatively higher weight.


from MAR

𝑤𝑚 =𝑤 𝑚

𝑤 𝑚𝑚𝑖=1

𝑤 𝑚 = exp −𝛿𝑌𝑖𝑗𝑚

𝑛1

𝑖=1

2. Pattern-mixture models (Carpenter and Kenward 2013)

i) Approach starts by imputing the missing data under MAR

ii) Then, it is assumed the distribution of unobserved values differs from that of observed values by, say 𝜃

iii) Using the draws from 1 and probability distribution of 𝜃, impute missing 𝑌𝑖𝑗 using model above.

(Implementation code for R and Stata under way)


from MAR

𝑌𝑖𝑗 = 𝛽0 + 𝛽1𝑋𝑖𝑗 + 𝛽2𝑍𝑗 + 𝑢𝑗 + 휀𝑖𝑗 𝑖𝑓 𝑌𝑖𝑗 𝑖𝑠 𝑜𝑏𝑠𝑒𝑟𝑣𝑒𝑑

𝑌𝑖𝑗= (𝛽0+𝜽) + 𝛽1𝑋𝑖𝑗 + 𝛽2𝑍𝑗 + 𝑢𝑗 + 휀𝑖𝑗 𝑖𝑓 𝑌𝑖𝑗 𝑖𝑠 𝑢𝑛𝑜𝑏𝑠𝑒𝑟𝑣𝑒𝑑

Choice of method

• Choice of approach for conducting sensitivity analysis will be dependent on the type of inference of interest.

• Example: Study comparing a new therapy for glycaemia control with standard medical management.

Are we interested in?

• Making inference for the population (patients with diabetes), given that they attended clinic visits as they should (de jure question)

• Making inference for the population, taking into account those whose participation may be erratic (de facto question)


from MAR

Patient-reported outcome measures (PROMs)

• Routinely collected in the English NHS and used to compare hospital performance

• Prone to missing data:

– Providers may fail to administer a questionnaire at hospital admission

– Patients may refuse to participate or fail to return post-operative questionnaire

• Over 140,000 patients undergoing hip replacement in 2010-2012

• PROMs questionnaires linked to HES

• Post-operative EQ-5D was missing for 54% patients

Case study

Different forms of incomplete data in PROMs

Investigating the reasons for missing

data

38% 46%

12% 14%

35% 26%

15% 14%

0

10000

20000

30000

40000

50000

60000

70000

80000

2010-2011 2011-2012

Nu

mb

er

of

ep

iso

de

s

Year

Not linked: missing NHSnumber or insufficientmatch

Not linked: patientrefusal or questionnairesnot administered

Linked: incompletequestionnaires

Linked: completequestionnaires


data

0.5 0.6 0.7 0.8 0.9 1 1.1 1.2 1.3 1.4 1.5

Age 55-65

Age 65-75

Age >75

Male

Ethnicity (white)

Provider type (private)

Most Deprived

2nd quintile

3rd quintile

4th quintile

Previous admitted to hospital

Charlson Index (>1)

Length-of-stay (per extra day in hospital)

Waiting time (over 3 months)

Odds ratio of missingness

Negative association Positive association


data

0.2

.4.6

.81

Ad

just

ed p

ost

-op

era

tive E

Q-5

D

0 .5 1 1.5 2Standardised response rate

Association between provider-specific outcome and response rate

Analysis model

Assessment of relative provider performance

• Provider-specific outcomes (𝑦 2𝑗) were estimated using the NHS risk adjustment

method (Nuttall et al 2013):

𝑦2𝑖𝑗 = α + 𝑥𝑖𝑗′ 𝛽 + 𝑦1𝑖𝑗𝛾 + 𝑢𝑗 + 휀𝑖𝑗, 휀𝑖𝑗~𝑁(0, 𝜎𝜀)

𝑦 2𝑗 = 𝜌𝑗𝑦 2, 𝜌 𝑗 =1

𝑛

𝑦2𝑖𝑗

𝑦 2𝑖𝑗

𝑛𝑖=1

• To help identify potential outliers we followed the methodology recommended by Department of Health (funnel plots)

Imputation model

Multivariate multilevel ordered probit model (Goldstein et al 2009)

Let ℎ𝑘,𝑖𝑗 be the kth component (k=1,…,5) of the EQ-5D, with M ordinal categories, m=1,…,3,

and 𝑦𝑘,𝑖𝑗∗ its latent (unobserved) Normal response defined as:

Imputed values can then be drawn from the multivariate multilevel Normal distribution:

ℎ𝑘,𝑖𝑗 =

1 𝑖𝑓 𝑦𝑘,𝑖𝑗∗ ≤ 𝜅1

2 𝑖𝑓 𝜅1 < 𝑦𝑘,𝑖𝑗∗ ≤ 𝜅2

3 𝑖𝑓 𝑦𝑘,𝑖𝑗∗ > 𝜅2

𝑦𝑘,𝑖𝑗∗ = 𝜹0

𝑘 + 𝑤𝑖𝑗′ 𝜹1

𝑘 + 𝑧𝑗′𝜹2

𝑘 + 𝝊𝑗𝑘 + 𝝂𝑖𝑗

𝑘

Where 𝝂𝑖𝑗𝑘 ~𝑀𝑉𝑁(0, Ω𝜈) and 𝝊𝑗

𝑘~𝑀𝑉𝑁 0, Ω𝜐 , and Ω is the 𝑘 × 𝑘 covariance matrix

We conducted 50 imputations and 5000 MCMC, with each set of imputed values drawn from the posterior distribution every 100th iteration of the MCMC chain

Funnel plots & provider performance

Funnel plots and performance status

0.50

0.60

0.70

0.80

0.90

1.00

0 20 40 60 80 100

Pro

vid

er-

spe

cifi

c o

utc

om

es

Volume

99.8% 95% Target

Positive outlier: Alarm

Negative outlier: Alarm

Positive outlier: Alert

Negative outlier: Alert

MI, funnel plots & provider

performance

Funnel plots’ horizontal and vertical effects after MI

Complete-case analysis

Funnel plot of provider-specific outcomes for complete cases

.6.7

.8.9

1

Pro

vid

er-

specifi

c E

Q-5

D

0 500 1000 1500Volume

Upper 95% CI Lower 95% CI

Upper 99.8% CI Lower 99.8% CI

Adjusted post-operative EQ-5D

22 negative outliers: alarm

CCA vs Multiple Imputation

01

23

45

6

Den

sity

0 .2 .4 .6 .8 1Adjusted post-operative EQ-5D

CCA After MI

Kernel densities for adjusted outcomes according to CCA and MI

Multiple Imputation

.6.7

.8.9

1

Pro

vid

er-

sp

ecific

EQ

-5D

0 1000 2000 3000Volume



Adjusted post-operative EQ-5D

9 negative outliers: alarm

Funnel plot of provider-specific outcomes after multiple imputation

Multiple Imputation

Changes in the performance status

.6.7

.8.9

1

Pro

vid

er-

sp

ecific

EQ

-5D

0 1000 2000 3000Volume



Adjusted post-operative EQ-5D Lost positive alarm status

Moved away from negative alarm status

Sensitivity analysis

Provider performance according to alternative methods for missing data

Post-operative EQ-5D

Control

limits

Upper @99.8%

(positive

alarm)

Upper @95%

(positive

alert)

In control Lower @95%

(negative alert)

Lower @99.8%

(negative

alarm)

CCA

(MCAR)

9 (3%) 26 (9%) 203

(71%)

25 (9%) 22 (8%)

Multilevel MI

(MAR)

8 (3%) 16 (5%) 254

(83%)

17 (6%) 9 (3%)

Robust MI

(MAR)

6 (2%) 23 (8%) 251

(82%)

14 (5%) 10 (3%)

MNAR 7 (2%) 21 (7%) 253

(83%)

11 (4%) 12 (4%)

• CEA context poses specific challenges to methods for addressing missing data

• Unprincipled methods unlikely to be plausible in practice

• Under MAR, MI framework is flexible and naturally extends to sensitivity analysis

• PROMs illustrates how MI can help address specific challenges

Future work

• Simulation work to compare relative merits of alternative methods in complex circumstances

• Project website (R and Stata code, guindance, etc.)

Summary


• Carpenter, J. & Kenward, M. 2013. Multiple Imputation and its Application, Chichester, UK., Wiley. • Carpenter, J. R., Kenward, M. G. & White, I. R. 2007. Sensitivity analysis after multiple imputation

under missing at random: a weighting approach. Stat Methods Med Res, 16, 259-75. • Daniel, R. M. & Kenward, M. G. 2012. A method for increasing the robustness of multiple

imputation. Computational Statistics & Data Analysis, 56, 1624-1643. • Diaz-Ordaz, K. Gomes, M. Grieve, R. and Kenward, M. 2013. Random effects versus fixed effects

multiple imputation for handling clustered missing data. For submission to Computational Statistics & Data Analysis

• Goldstein, H., Carpenter, J., Kenward, M. G. & Levin, K. A. 2009. Multilevel models with multivariate mixed response types. Statistical Modelling, 9, 173-197.

• Gomes, M. Diaz-Ordaz, K. Grieve, R. and Kenward, M. 2013. Multiple imputation methods for handling missing data in CEA that use data from hierarchical studies: an application to cluster randomized trials. Med Decis Making, 33: 1051-1063.

• Mason A, Richardson S, Plewis I, Best N. Strategy for modelling nonrandom missing data mechanisms in observational studies using Bayesian methods. J Off Stat. 2012;28(2):279–302.

• Noble, S. M., Hollingworth, W. & Tilling, K. 2012. Missing data in trial-based cost-effectiveness analysis: the current state of play. Health Econ, 21, 187-200.

• Nuttall, D., Parkin, D. & Devlin, N. 2013. Inter-Provider Comparison of Patient-Reported Outcomes: Developing an Adjustment to Account for Differences in Patient Case Mix. Health Econ.

References


che seminar 20 november 2013

Technology

observed data

missing covariates

missing information

missing observations

missing values conditional

overview missing data

missing data noble et

cea missing endpoints