Jennifer BryceInstitute for International Programs

The Johns Hopkins University

Institute for International Programs

ASADI VAccra, November 2009

Outline – What have we learned?

1. From the evaluation of the ACCELERATING CHILD

SURVIVAL AND DEVELOPMENT (ACSD) Program?

2. From prospective evaluations of the CATALYTIC

INITIATIVE TO SAVE ONE MIIILION LIVES (CI) to date?

3. The way forward

ACSD, 2002-2005

11 countries in Africa

Support from CIDA and other partners

Implemented through UNICEF

Aim: To reduce mortality among children less than 5 years of age

Strategy: Accelerate coverage with three packages of high-impact interventions, with a special focus on community-based delivery

Mali

ChadNiger

Nigeria

Cameroon

Central AfricanRepublic

Congo - Democratic Republic

Congo

SenegalCape Verde

Gabon

Equatorial Guinea

Sao Tome &Principe

GambiaGuinea Bissau

Guinea

Sierra Leone

Liberia

Burkina Faso

Ghana

TogoBenin

High Impact Package

EPI + Expansion

Accelerated Child Survival Accelerated Child Survival and and DevelopmentDevelopment

CIDA CIDA funded projectfunded project

Côte d’Ivoire

Mauritania

ACSD Program:Intervention packages

EPI+ Vaccinations Vitamin A supplementation ITNs for U5s & pregnant women De-worming

IMCI+ Facility IMCI Community case management

(CCM) of childhood illnesses Diarrhea: oral rehydration therapy

(ORT) Malaria: based on current policy Pneumonia: referral to facility

Promotion of timely initiation of breastfeeding, exclusive breastfeeding to 6 months, timely complementary feeding

Promotion of household consumption of iodized salt

ANC+– Malaria prevention in pregnant

women (IPTp)– Tetanus Toxoid– Iron/folic acid supplementation – Vitamin A post-partum– PMTCT

The retrospective independent evaluation of ACSD

High-impact districts in Benin, Ghana, Mali

Standard indicators

Existing DHS/MICS with oversampling

National comparison areas

Documentation of program implementation & contextual factors

No cost component

Stepwise design

Key: Bars represent districts in the following order: Builsa, Bawku East, Kasena-Nankana, Bolgatanga, Bawku West, Bongo

ACSD Implementation: GHANA

Coverage for EPI+ interventionsbefore and after ACSD, in HIDs

Benin

51

63

10

6

49

60

61

26

Measles

DPT

Vitamin A

ITNs

Ghana Mali

*

*

*

*

*

*

*

*

*

*

*Change was significant at p < 0.05.

Increases in coverage across the board in Ghana and Mali; Benin achieved increases for vitamin A and ITNs.

3

Before ACSD

After ACSD

Key

Coverage for IMCI+ interventionsbefore and after ACSD, in HIDs

Benin Ghana Mali

*

*

*

*

*

*

*

*Change was significant at p < 0.05.

No coverage gains, and some significant losses, in sick child care. Exclusive breastfeeding increased in Ghana, declined in Mali.

Before ACSD

After ACSD

Key

Coverage for ANC+ interventionsbefore and after ACSD, in HIDs

Benin

71

0

44

76

5

64

7

55

74

38

3+ antenatal care visits

IPTp with SP

Tetanus Toxoid

Skilled attendant at delivery

Postnatal vit A

Ghana Mali

*Change was significant at p ≤ 0.05.

*

*

*

*

*

*

*

*

*

*

** Measured level was 28%, but country team reported this was incorrect as IPTp had not been implemented in 2001.

Ghana and Mali improved care for childbearing women; delivery of TT and postnatal vit A benefited from EPI system in Mali.

*

**

Before ACSD

After ACSD

Key

Under-five mortality in the ACSD HIDs

19% (p=0.10)

Under-five mortality in the ACSD HIDs and national comparison areas

Declines in U5M in ACSD focus districts, but not greater than national comparison areas.

141

107

260

145

248

123

86

197

109

172

0

40

80

120

160

200

240

280

Benin Ghana Mali

Un

de

r-fiv

e m

ort

alit

y (p

er 1

00

0 li

ve b

irth

s)

Comparison area Comparison area

Jul 1999-Jun

Jan 2004-Dec 2006

Jul 1998-Dec 2001

Jul 1998-Dec 2001

Jul 2003-Dec 2006

Jan 2004-Jul 2007

141

107

260

145

248

123

86

197

109

172

0

40

80

120

160

200

240

280

Benin Ghana Mali

Un

de

r-fiv

e m

ort

alit

y (p

er 1

00

0 li

ve b

irth

s)

Comparison area Comparison area

Jul 1999-Jun

Jan 2004-Dec 2006

Jul 1998-Dec 2001

Jul 1998-Dec 2001

Jul 2003-Dec 2006

Jan 2004-Jul 2007

No changes in child nutritional status

attributable to ACSD.

Did ACSD implementation contribute to reducing inequities?

Yes, in Mali, where socioeconomic and urban/rural inequities decreased more in the ACSD HIDs than in the comparison area.

Baseline sample sizes too small to support analysis of equity trends in Benin or Ghana.

Socioeconomic inequalities, showing breakdown by wealth quintiles of ANC 3+ coverage in ACSD “high-impact” zones and the comparison area, Mali, 2006-7.

3+ antenatal visits

0%

20%

40%

60%

80%

100%

Poorest 2nd 3rd 4th Richest

Cov

erag

e (%

)

HID (before) Comp (before) HID (after) Comp (after)

Conclusions & implications

1. Intervention coverage CAN be accelerated if there is adequate funding & human resources.

2. Acceleration of mortality declines require:

a) Focus on interventions that have a large and rapid impact on major causes of child death

b) Sufficient time to fully implement approach and for coverage to translate into declines in mortality

c) Reasonable expectations, given level of resources

► Work for closer match between program resources & cause of death

► Be realistic about what can be accomplished

► Level of funding matters

Conclusions & implications

3. Policy barriers prevented key ACSD interventions directed at pneumonia and malaria from being fully implemented.

4. Breakdowns in commodities and gaps in funding stall progress toward impact.

5. More attention and operations research needed on incentives and supports for community-based workers

► Work for policy reform as first step, where needed

► Pay attention to health systems supports such as commodities, supervision, & incentives

Contributors & acknowledgements

Contributors

Jennifer Bryce

Kate Gilroy

Elizabeth Hazel

Gareth Jones

Robert Black

Cesar Victora

AcknowledgementsMinistries of Health, National Statistics Offices, UNICEF country staff, Collaborators in documentation

UNICEF regional and global staff Genevieve Begkoyian, Mark Young, Sam Bickel

Technical consultants Trevor Croft, Macro International

UNICEF leadership For their commitment to learning and change

EVALUATING THE CATALYTIC INITIATIVETO SAVE A MILLION LIVES

Part 2

Independent Evaluation of the MNCH Rapid Scale-Up

Overall objective: Provide “proof of concept” that proven interventions can be scaled up rapidly to reduce newborn and child mortality.

Supported by: BMGF Implementing partners: Governments and

UNICEF, WHO, UNFPA

“Real-time” Mortality Monitoring (RMM)

Overall objective: To monitor changes in under-five mortality in real-time.

Countries: Burkina Faso, Malawi, Mozambique

Countries: Ghana, Malawi, Mali, Mozambique

Two Linked EvaluationsThe Catalytic Initiative

Supported by: CIDA Implementing partner:

Governments and UNICEF

Process of evaluation design

Progress: MalawiIn-country partners: Centre for Social Research and National Statistics Office

Implementation Features of accelerated approach:

Government-paid CHWs trained to deliver CCM for pneumonia, malaria, diarrhea (including zinc)

Strengthening district health management

Implementation status:

In 10 intervention districts, 5-15% of CHWs trained by June 2009

Evaluation (full) Mortality monitored by:

Having CHWs report vital events

Calibrating facility deaths against community deaths

Two rapid survey approaches

Full documentation of program & contextual factors

Quality of care assessments at 1st-level facilities and for CHWs

Costs & equity tracked 12 districts: 6 “accelerated” and 6

routine National platform approach under

discussion

Progress: MozambiqueIn-country partner: Eduardo Mondlane University

Implementation Features of accelerated approach:

Increased access to quality child health care in facilities

Quality of immunization services improved

Long-lasting insecticide-treated nets (ITNs) distributed and used

Vitamin A supplementation

Breastfeeding promotion

BMGF funds used to fill gaps in maternal health

Implementation status:

CI planned to be implemented in 33 districts each year from 2008 to 2012

Evaluation (full)Mortality monitoring by calibrating facility to community deaths.

Stepped-wedge design based on scheduled cohorts for Rapid Scale Up

Documentation of baseline health & nutrition, inputs & contextual factors and coverage for each cohort using national evaluation platform approach

Will support dose (program intensity) – response (coverage & modeled impact) analyses

Progress: Burkina FasoIn-Country Partner: Institut Supérieur des Sciences de la Population (ISSP)

Implementation Features of accelerated approach:

Volunteer community health workers

providing:

• CCM for diarrhea (ORT + zinc) and malaria (ACT) in 9 districts

• CCM for pneumonia in 2 districts

Strengthened district planning and

supervision

Implementation status: Materials ready; cascade training

of all CHWs in 9 districts to be completed before end 2009

Evaluation (no RMM)National platform approach in 9 intervention and 2-3 comparison districts

New “LiST” survey to collect district-level estimates of coverage for proven MNCH interventions

Modeled mortality using LiST

Analysis using pre-/post-intervention with comparison and/or dose response

*If targets fully achieved at adequate service quality.

Lesson 1:Existing plans include high-impact interventions…

…but if feasibility and speed are issues,just 4 or 5 interventions can achieve ≥ 20% reduction in U5M by 2015

Malawi Burkina Faso GhanaNumber of interventions included in national plan

18 13 20

Number to ≥ 20% reduction in U5M4 5 5

Interventions (current and target coverage levels)Pneumonia treatment with antibiotics

(29; 67)

(30; 50)

(33; 60)

Diarrhea treatment with ORS and zinc (55; 85)

(41; 60)

(42; 60)

Malaria prevention with insecticide-treated nets

(23; 69)

(10; 70)

(40; 55)

Malaria treatment with ACTs (27; 69)

(48; 57)

(65; 70)

Vitamin A supplementation (67; 90)

Improved sanitation (18; 70)

Pre-publication results; not for citation or distribution

Lesson 2:Implementation takes time

Implementation status of functional village health clinics with CHW trained in CCM, Malawi CI districts, June 2009(18 months after project start-up)

*1 trained CHW per village health clinic

…especially when policy reform is needed.

In Mali, the MoH scheduled a “forum” to decide on CCM for childhood pneumonia & malaria.

July 2008

November2009

November2008

February2009

March2009

Original date

(cancelled)Planned

(cancelled) Planned

(cancelled)

Forum held; agreed

“YES” on CCM

Months 4 3 1 7 months+ + =

Discussions about how to

implement are still under way

in a 3-year CI project

Lesson 3:“Virgin” comparison areas do not exist

Mozambique

Simultaneous implementation of multiple programsSeparate, uncoordinated, inefficient evaluations, if any

Lesson 4: There are no shortcuts for mortality measurement (at least not yet)

Capturing a 25% difference-in-difference for rates of child mortality in a two-year period requires a survey of ≈ 12,300 households in each group*

Promises of measuring declines in 1 year using survival analysis or other techniques still require these prohibitively large sample sizes, plus detailed info on age of death

CI work on “real-time” mortality monitoring will assess the validity of alternative methods, but in first trials require validation against a gold standard

*based on Malawi; sample sizes will increase as mortality rates decrease, e.g. in Ghana

RMM Options by countryOptions for RMM methods

Data collection at community level Deaths recorded

in facilities vs. com-munity survey

Vital registra-

tion program

New methods for using surveys

Vital events

reporting at Child Health Days

Paid Gov’t health

workers

Unpaid Gov’t health

workers

Lay volun-teers

Ghana Malawi Mali Mozambique

Contributors

In-country partners

Agbessi Amouzou, Abdullah

Baqui, Robert Black, Jennifer

Bryce, Kate Gilroy, Elizabeth

Hazel, Gareth Jones, Marjorie

Opuni, Jeremy Schiefen, Cesar

Victora, Damian Walker

IIP-JHU

Burkina Faso: ISSP, INSP

Ghana: Noguchi Institute, University of Ghana

Malawi: NSO, CSR, Department of Economics, University of Malawi

Mali: CREDOS

Mozambique: Eduardo Mondlane University

THE WAY FORWARD:NATIONAL EVALUATION PLATFORMS (NEPS)

Part 3

What is a national evaluation platform (NEP)?

District-level databases covering the entire country

Containing standard information on: Inputs (partners, programs, budget allocations, infrastructure) Processes/outputs (DHMT plans, ongoing training,

supervision, campaigns, community participation, financing

schemes such as conditional cash transfers) Outcomes (availability of commodities, quality of care

measures, human resources, coverage) Impact (mortality, nutritional status) Contextual factors (demographics, poverty, migration)

Permits national-level evaluations of multiple simultaneous programs

NEPs: A common evaluation framework

Common principles (with IHP+, Countdown, etc.)

Standard indicators Broad acceptance

NEPs: Sound evaluation principles

In-country evaluation counterparts Local expertise, able to provide continuing evaluation research

support to the MOH Continuity of inputs from evaluation team; cross-country

network of investigators

Linked “independence” Investigators not involved in implementation of MNCH activities Regular exchange with in-country implementation team Ongoing activity; not one-off approach

Attribution by approach Documentation of all contributions Comparison of accelerated approach with “routine” approach

What types of questions can an NEP answer?

? Are programs being deployed where need is greatest?

? Is implementation strong enough to have an impact?

? Did programs increase coverage?

? Was coverage associated with impact?

? How equitable are the programs?

? How much did programs cost?

How can the MOH and partnersuse the platform?

To learn from well-performing districts and guide those doing less well

? Which approaches or combinations are contributing to rapid scale-up?

? Are some districts more efficient than others? Why?

? Are changes in epidemiology (e.g., due to IRS) reflected in reallocation of resources in district plans?

Why should you consider a national platform approach (or not) ?

AdvantagesAdapted to current reality of

multiple simultaneous programs/interventions and partners

Flexible design allows for changes in implementation

Can be used to evaluate multiple programs (child survival, HIV, malaria, maternal health, etc.)

Supports country ownership and capacity building

LimitationsObservational design (but no

other alternative may be possible)

Cost, particularly due to large size of surveys (!But cheaper than many standalone surveys!)

Requires transparency and collaboration by multiple programs and agencies

Further details at www.jhsph.edu/iip and

www.cherg.org

Thank you