Understanding causal pathways within health systems policy evaluation through mediation

analysis: an application to payment for performance (P4P) in Tanzania

Laura Anselmi, Peter Binyaruka, Masuma Mamdani

Josephine Borghi

Payment for Performance: a health systems perspective

A workshop for scientists and practitioners

Dar es Salaam, 26 November 2015

Rationale

• Programme evaluation has mainly focused on measuring the impact on outcomes, with little attention to the causal pathways

• The relevance of undertaking process evaluation to be integrated with outcome evaluation is increasingly recognised

• Process evaluation is particularly relevant for complex interventions: It increases confidence in the plausibility of outcome effects

It increases the external validity of the evaluation

• Process evaluations have been carried out, but without formal assessment of causal pathways

Causal mediation analysis

• A causal mechanism is a process through which a programme or intervention influences an outcome

• It can be identified by specifying intermediate outcomes or variables(mediators) that are on the causal pathway between intervention and outcome

• Causal mediation analysis has been employed to test change pathways within the evaluation of public health programmes

• Mediators have been limited to individual level indicators, psychological or physical

• Health system mediators which are relevant to the evaluation of health systems or services interventions have not been considered

Program Outcome

Mediator 1 P4P Indirect

effect

P4P Measured effect = P4P Direct Effect + P4P Indirect effect 1 + P4P Indirect Effect 2

Confounder

Mediator 2 P4P indirect

effect

Sequential ignorability b):Given treatment status and pre-treatment confounders the mediators are ignorable (no confounders affecting both mediators and outcome)

Programme Measured effect

Confounder

Confounder

Causal mediation analysisSequential ignorability a):

Given pre-treatment confounders the treatment is assigned

independently of potential outcomes and mediators

P4P direct effect

P4P Programme in Tanzania

• P4P Scheme introduced in 2011 by the MoH in the Pwani region

• Target on Maternal and Child Health Care outcomes

• Outcome evaluation:• 8.2% increase in coverage of institutional deliveries (ID)

• 6.5% increase in delivery in public health facilities

• 10.3% increase in the uptake of two doses of anti-malarial (IPT) during pregnancy

• The effect of P4P on a number of governance, financing and human resources factors has been identified

• Data:• Health facility survey: 150 HFs (75 P4P + 75 control)

• 1-2 interviews with health workers per HF

• 1,500 exit interviews

• 3,000 household survey

• Baseline: Jan-March 2012, Endline: February 2013

P4P theory of change in a Causal Mediation Analyses Framework

P4P Outcome

P4P indirect effect through Governance

P4P indirect effect through Human

Resources

P4P indirect effect through Financing

P4P direct effect

Methods

• Step1: Estimating the impact of P4P on outcomes (DiD)

P4Pt indicator of P4P districtδt time indicatorXijt women socio-economic characteristicsγj HF fixed effects

• Step 2: Identifying effect of P4P on potential mediators (DiD)

• Step 3: Identifying direct and indirect causal effects (DiD)

𝛽13 P4P direct effect

𝛽12 X 𝛽4

3 P4P indirect effect through mediator M

𝑌𝑖𝑗𝑡 = 𝛽03 + 𝛽1

3(𝑃4𝑃𝑗 × 𝛿𝑡) + 𝛽23𝛿𝑡 + 𝛽3

3𝑋𝑖𝑗𝑡 + 𝛽43𝑀𝑖𝑗𝑡 + 𝛾𝑗 + 휀𝑖𝑗𝑡

3

𝑌𝑖𝑗𝑡 = 𝛽01 + 𝛽1

1(𝑃4𝑃𝑗 × 𝛿𝑡) + 𝛽21𝛿𝑡 + 𝛽3

1𝑋𝑖𝑗𝑡 + 𝛾𝑗 + 휀𝑖𝑗𝑡1

𝑀𝑖𝑗𝑡 = 𝛽02 + 𝛽1

2(𝑃4𝑃𝑗 × 𝛿𝑡) + 𝛽22𝛿𝑡 + 𝛽3

2𝑋𝑖𝑗𝑡 + 𝛾𝑗 + 휀𝑖𝑗𝑡2

Results: Potential mediators (Step 2)Potential mediators Effect of P4P (% change)

Financing

Proportion of women who paid for delivery in a HF (1) -8.0**

Proportion of women who paid for delivery in a public HF (1) -7.5***BS

Service delivery disrupted due to broken equipment last 90days -149**

Drug stock-out index-general (0-1 index) -17.2***BS

Medical supplies stock-out index (0-1 index) -14.8***BS

Oxytocin injection stock-out last 90days -36.2***BS

Ergometrine injection stock-out last 90days -26.1**

Drugs at delivery stock-out index (0-1 index) -27.0***BS

Mean all financing indicators(0-1 index) -8.3**

Factor analysis weighted score (2) -60.0***BS

Governance

Max time from external supervision: 90 days ago -18.0**

Dist/Regional supervision provided positive feed-back 23.8**

Dist/Regional supervision provided negative feed-back 28.2**

Dist/Regional supervision delivered supply -19.3**

Dist/Regional checked records 1.5**

Dist/Regional observed consultation 0.8**

Human resources

Mean patient satisfaction with interpersonal care (0-1 scale) (1) 6.7***BS

Mean kindness ranks for HW at delivery (0-1 scale) (1) 10.3***BS

* p<0.10, ** p<0.05, *** p<0.01 , BS: Significant at 5% level with Bonferroni adjusted p-value for multiple outcomes: Bonferroni adjusted p-value

Financing 0.0047, Governance 0.0017, Human resources 0.0414, (1) Out of all women delivering in a HF in same catchment area (2) equipment,

vaccines, drugs, medical supply

Results: P4P direct and indirect effect (Step 3)

• Facility based delivery

P4P total effect: +8.2%

P4P indirect effect through mean of all financing indicators: +1 %

P4P indirect effect through reduction in stock-out of oxytocine: +1.8 %

P4P direct effect: +7.2% or +6.4%

• Delivery in public health facility

P4P total effect: +6.5 %

P4P indirect effect through reduction in stock-out of oxytocine: +1.9 %

P4P direct effect: +4.6%

• Uptake of two doses of IPT during pregnancy

P4P total effect: +10.3 %

P4P indirect effect through reduction in last supervision being 90 days ago: +1.5 %

P4P direct effect: +8.8%

Sensitivity analysis

• Semiparametric mediation analysis to quantify the sensitivity of results to the assumption of no confounders affecting mediator and outcome

• Analysis carried out at the HF level

• Estimate a logit model for binary mediators

• Multiple hypothesis testing adjustment of p-values for families of mediators

• DiD with district fixed effects

Summary: Indirect effects of P4P

• P4P significantly affects a number of financing, governance and human resources factors which could potentially mediate its effect on maternal care outcomes

• The effect of P4P on the reduction of oxytocine injection stock-out mediates the effect of P4P on institutional deliveries (22%) and deliveries in a public health facility (30%)

• The effect of P4P on the frequency of supervisions mediates 15% of the effect of P4P on the uptake of at least two doses of IPT during pregnancy

Some reflections on mediation analysis

• Mediation analysis rarely applied using difference in difference

• Quasi–experimental setting + Difference-in-Difference provide confidence that the assumption of no pre-treatment confounders is satisfied

• But how plausible are the assumptions of no confounders between mediator and outcome?

• Data availability limits testing pre-trends for mediators

• Possible differences in results according to the level of the analysis

• Little analysis of the role of individual level factors or other moderating factors

• Possibly simplified description of the causal chain

Conclusions

• Mediation analysis is helpful to quantify causal direct and indirect effects and the relative relevance of change pathways

• It requires assumptions to identify causality and these can not be tested formally

• Quantify the P4P indirect effects helps in thinking about relative cost-effectiveness compared to alternative interventions

Thank you!

Acknowledgements to the whole “P4P team”