Effects of Payment for Performance on

Knowledge, Practice and the Know-Do Gap

Evidence from Pwani,

Tanzania

Josephine Borghi

24th NovemberRBF – a health systems perspective. White Sands Hotel, Dar es Salaam.

Rationale

• P4P is expected to improve quality and service coverage through changed health worker behaviour

• Improvements in health worker knowledge and practice is necessary for optimal health gain

• Knowledge may increase through substitution of health workers; greater investment in training

• Practice (or application of knowledge) is likely to increase through a desire to meet targets, and improved resource availability

• In Rwanda found limited effect on knowledge and improvements in practice, especially among those with higher knowledge levels.

• Aim: examine P4P effects on knowledge, practice and

the gap in Tanzania using data from our evaluation in

Pwani

P4P in Tanzania

Aim: A pilot introduced in 2011 focusing on MCH service coverage to inform a national programme

Location: Pwani region of Tanzania

Implementers: MOHSW and CHAI

Funder: Government of Norway

Scheme Design

Facility level targets:– ANC: IPT2; % HIV+ women on ART – Institutional delivery rate – % of newborns with OPV0 in first 2 weeks– % infants with Penta 3; measles vaccine– % of PNC visit w/n 7 days – CYP – HMIS reports correctly filled and submitted on

time + use of partogramsDistrict – regional level targets:

– % of maternal/perinatal deaths audited on time – % of facilities with stock outs

Study Design

• Design: Controlled before and after study design

– 7 intervention districts

– 4 neighbouring control districts

– Comparable poverty, literacy, rate of institutional deliveries, IMR, pop. per health facility, no. of children < 1 yr

• Timing:

-Baseline in January-February 2012

-Endline in March-April 2013 (13 months)

7 P4P districts 4 districts with no P4P

150 health facilities, 75 in each arm incl.

6 hospitals16 health centres53 dispensaries

1 facility survey at each facility

20 interviews with women who delivered

in past 12 months, from the catchment area of

each facility

Only include facilities eligible for first cycle payment

1-2 health workers at each

facility

Measurement:

Knowledge

• Used a clinical “vignette”: a hypothetical patient case, in this case, a woman attending her first antenatal visit.

• Derived from the World Bank Impact Evaluation Toolkit

• Presented to health workers who regularly provide ANC

• 45 items from the antenatal clinical guidelines were covered in the vignettes, with items corresponding to four dimensions: – patient medical history

– physical examinations

– laboratory investigations

– drugs prescriptions.

• Measure scores for each dimension and in total: number of items mentioned by the total number of items.

Measurement: Practice

• Procedures performed by the provider on patients (adherence to protocol).

• Household interviews with women attending ANC during their current or last pregnancy living within the catchment area of facilities where health workers were surveyed

• A total of 18 items regarding ANC services, 11 match the 45 items in the health workers survey

• Dimensions of care: – physical examinations

– laboratory investigations

– drugs prescription.

• We constructed an additional dimension relating to client counselling and educational services.

Measurement: Gap

• The knowledge–practice gap measures provider efficiency to translate knowledge into actual ANC practice.

• Defined as the difference between the knowledge and practice share of clinical guidelines for an ANC visit.

• Women linked to a given facility by its catchment area are matched to the health workers’ responses for that same facility.

• Take average value for knowledge measure at facility level where more than one health worker was surveyed per facility.

• Measured for 11 items across:– physical examinations

– laboratory investigations

– drugs prescribed.

Analysis

• Used a difference-in-difference identification strategy:

• 𝐾ℎ𝑗𝑡 = 𝛽0 + 𝛽1(𝑃4𝑃𝑗 × 𝛿𝑡) + 𝛽2𝛿𝑡 + 𝛽3𝑍ℎ𝑗𝑡 + 𝛾𝑗 + 휀ℎ𝑗𝑡• 𝑌𝑖𝑗𝑡 = 𝛽0 + 𝛽1(𝑃4𝑃𝑗 × 𝛿𝑡) + 𝛽2𝛿𝑡 + 𝛽3𝑍𝑗𝑡 + 𝛽4𝑋𝑖𝑗𝑡 + 𝛾𝑗 + 휀𝑖𝑗𝑡

• 𝐾ℎ𝑗𝑡 knowlegde share of ANC guidelines by provider h at health facility j in period t

• 𝑌𝑖𝑗𝑡 is the practice share or the gap

• 𝑃4𝑃𝑗 is a dummy variable taking the value of 0 for comparison facilities and 1 for intervention facilities;

• 𝛿𝑡 is a year fixed effects dummy taking the value of 0 at baseline and 1 at endline;

• 𝑍ℎ𝑗𝑡 health worker-level characteristics expected to drive programme outcomes

• 𝑋𝑖𝑗𝑡 are household level characteristics expected to drive programme outcomes

• 𝛾𝑗 is a facility fixed effects to control for facility-level time invariant characteristics; and 휀ℎ𝑗𝑡 is a random error term

• Assumption: pre-trends in outcomes are parallel

Impact on ANC Knowledge

Variables Baseline Impact

P4P Control Diff % effect of P4P

Knowledge shares for each dimension

Medical history taking (% of items known) 26 items

20.2 34.1 -13.9*** 12.5***

Physical examinations (%) 10 items

25.5 42.7 -17.2*** 11.9***

Lab investigations (%) 7 items 26.7 48.4 -21.6*** 18.5***

Drug prescriptions (%) 2 items 63.8 87.1 -23.3*** 16.2***

Total items known (%) 45 items 24.3 40.6 -16.3*** 13.4***

Total items known – gap (%) 11 items

42.0 64.6 -22.7***17.6***

Impact on ANC Practice

Variables Baseline Impact

P4P Control Diff % effect of P4P

Practice shares for each dimension

Client Counselling (% of items done) 7 items

78.6 71.8 6.8*** -3.2*

Physical examinations (%) 6 items 89.4 87.7 1.7 0.2

Lab investigations (%) 3 items 86.8 83.2 3.6* 2.2

Drug prescriptions (%) 2 items 71.4 73.6 -2.2 7.1**

Total items done (%) 18 items 82.6 79.1 3.5*** 0.0

Total items done – gap (%) 11 items

85.3 83.9 1.52.0*

Impact on ANC Know-Do Gap

Variables Baseline Impact

P4P Control Diff % effect of P4P

Gap shares for each dimension

Physical examinations (%) 6 items -54.4 -34.7 -19.7*** 9.2*

Lab investigations (%) 3 items -43.1 -6 -28.0*** 18.6***

Drug prescriptions (%) 2 items -2.5 14.9 -17.5*** 2.8

Total items (%) 11 items -39.1 -17.9 -21.2*** 10.1*

Conclusions

• P4P significantly improved health worker knowledge across all dimensions

• Plausible?• District managers shifted efficient workers to help those struggling to meet

targets.

• Opportunities to upgrade skills with training increased as a result of P4P from health worker survey.

• Knowledge indication of ‘intended behaviour’ – may be more responsive to P4P in the short term than practice

• P4P improves practice in relation to incentivised components of care: drug prescriptions but no evidence of other improvements in adherence to care guidelines

• As knowledge increases and practice generally doesn’t –

the inefficiency gap increases

Limitations

• Imbalance in baseline knowledge between intervention and control

• Unable to assess whether trends in knowledge and practice were parallel prior to P4P; trends in ANC coverage were

• Gap analysis:– Tools not originally intended to pursue this gap analysis – and only a limited

number of items could be compared in this way

• Consider the one-two health workers interviewed as representative of ‘practice’ at a given facility which may not be the case

• Assume households went to their nearest facility for ANC

• Concern that practice found to exceed knowledge at baseline (negative gap)

Acknowledgements

• Josephine Borghi – LSHTM

• Paola Vargas – LSHTM/OPM

• Peter Binyaruka - LSHTM

• Powell-Jackson T - LSHTM

• Patouillard E - LSHTM

• Torsvik G – CMI

• Mayumana I – IHI

• Masuma Mamdani - IHI

• Lange S - CMI

• Maestad O - CMI