part3

The Department of Health Policy

Comparative Effectiveness Research (CER) Methods

Patrick Richard, PhDAdjunct Assistant Professor of Health Economics

GWU Health Policy and Economics Program


Outline

Major limitations of Randomized Clinical Trials (RCTs)

Advantages and challenges of observational studies Sample selection bias or confounding

Methods to address limitations of observational studies Propensity Score (PS) Instrumental Variables (IV)

2


Instrumental Variables (IV)

IV techniques have the potential to address the problem of omitted or residual bias due to unobserved confounding in CER

A carefully chosen observed variable that should be: (1) Correlated with treatment (2) But uncorrelated with the error term in the

outcome equation

3



IV is relatively easily to implement in CER if the outcome equation is linear such as test scores, biometric measurements, or nonzero spending amounts The IV is constructed by estimating a logistic

regression (or probit) model to predict the probability of treatment for each observation

Substitute the predicted probability for the treatment variable in the outcome equation

4



Challenging if outcome is non-linear such as mortality, hospitalization, heart attacks, strokes, recurrence of cancer, or other binary or count variables

Use the approach developed by Newey et al. (1999) and others Estimate a predicted value for the error term and

include it explicitly in the outcome equation (2 stage residual inclusion-2SRI)

5



Both linear and non-linear IV models are two-step procedures estimated simultaneously, not sequentially Sequential estimations produce incorrect standard

errors Linear or non-linear, the following issues still apply:

The strength of instruments Overidentifying restrictions Local average treatment effects (LATE)

6


IV-Strength of the instrument

An IV should be strongly related to treatment because weak instruments present several problems: Magnify any potential bias (Bound et al. 1995) Lead to substantial inconsistency in the IV

estimator Yield highly variable estimates, which make it

difficult to detect small effects, even in very large studies

7


IV-Strength of the instrument Staiger and Stock (1997) suggest that a joint test of

the hypothesis that all coefficients on the instruments equal zero should generate an F-statistic of 10 or more

As a rule of thumb, F statistics less than 10 are thought to be problematic

Models with stronger instruments (those that ‘‘move’’ more people) produce more generalizable results

8


IV- Overidentifying restrictions tests Used to test for the correlation between

instruments and the error term in outcome equation (Davidson and MacKinnon 1993)—The so called “exclusion criterion” Can only be used if more than one IV indicator No test is possible if there is only one IV indicator

Hence, the model is described as ‘‘exactly identified’’

9


Interpretation of the IV results

The Wald estimator yields the average effect of treatment among the “compliers”:

Patients who would always take their assigned treatment Take active therapy if assigned to it Take placebo if assigned placebo

In other words, only patients whose treatment status is influenced by the IV-Local Average Treatment Effect (LATE)

10


IV-Good practice

Justify Need for and Role of IV in the Study IV methods are inefficient and should not be used

as a primary analysis unless there is strong evidence of unmeasured confounding

Discuss why substantial unmeasured confounding is expected

11


IV-Good practice Describe Theoretical/Empirical Basis for the Choice

of IV A good IV should have a theoretical motivation

Why it is expected to influence treatment, but unrelated to outcome?

Is it supported by empirical evidence? For example, do patients chose hospitals without

knowledge of their formulary? Thus, formulary status may be effectively randomly

assigned

12


IV-Good practice Report the first-stage F statistic and the partial R2

attributable to the inclusion of the IV F-statistic >10 is desirable The partial R2 is the proportion of the variance

explained by the addition of the IV to the model Discuss issues related to interpretation of the

estimator The IV effect only generalizes to patients whose

treatment status depends on the instrument

13


Examples of IVs

Preference-based: Defined at the level of the geographic region, hospital, dialysis center, or individual physician

Local variations in physicians’ practice patterns Institutional factors such as formulary design

differences, program eligibility rules, implementation timelines, or provider network policy changes

14


PS & IV- An example

Stukel et al. (2007) used four different methods to assess the effects of cardiac catheterization on elderly patients hospitalized for acute myocardial infarction

Methodological concern: patients in poorer health were less likely to receive invasive care, potentially making the effects of treatment look better than they actually were

15


PS& IV-An Example

The authors compared multivariate risk adjustment, propensity score risk adjustment, propensity score matching, and instrumental variables results

IV: Regional cardiac catheterization rate The results were substantially different depending

upon method

16


PS & IV –An Example

Multivariable risk adjustment, propensity score risk adjustment, and propensity score matching show reductions in mortality risk between 46 and 49 percentage points

IV estimates show 16 percentage points reductions in mortality risk, comparable to estimates from RCTs, which ranged from 8 to 21 points

Evidence of sample selection bias in observational data

17


Conclusions

Very difficult to find “good” instruments Heterogeneity issues

In a recent report submitted to the President and Congress, the Federal Coordinating Council on Comparative Effectiveness Research states: “Clinicians and patients need to know not only that a treatment works on average but also which interventions work best for specific types of patients (e.g. the elderly, racial and ethnic minorities)” (FCC Report, June 30, 2009).

18

part3

Health & Medicine

iv indicator

instrumental variables

error term

ps

outcome equation

treatment

effects

outcome