SIMULTANEOUSLY DESIGNING PROJECTS

AND IMPACT EVALUATIONS: THE

BENEFITS AND CHALLENGES

EXAMPLES FROM NIGERIA, ZAMBIA, AND ZIMBABWE

EXAMPLE FROM RECENT HISTORY: PROGRESA (OPORTUNIDADES) •  In 1998, the Mexican government adopted a new social protection program: Progresa (a conditional cash transfer)

•  provides a small amount of cash assistance to the poor while simultaneously making progress towards goals of improving children’s health and education

•  Policy design team had substantial concerns about program design and performance

•  most social programs do not survive government changeover • Policy makers felt evidence assessed by international experts would help program survive. • Program survived after change in government as evaluation evidence was too politically costly to ignore

•  even though new government campaigned on ending the program, new head of state was convinced by evidence and decided to honor his pledge by renaming it: Oportunidades

EXAMPLE FROM RECENT HISTORY: PROGRESA (OPORTUNIDADES) The team has attributed the following benefits to the simultaneous design of the program and the IE: 1.  Political hardiness– PROGRESA is a pillar of Mexican SP policy. 2.  Increased and diverse funding sources–donors were attracted to

the program due to the clean, non-partisan evidence of its impact. 3.  Significant expansion-- not only has PROGRESA continued over the

years, it has expanded vastly from the initial pilot phase. 4.  Explicit codification of requirement for independently-run

evaluation of all Mexican social programs; CONEVAL established. 5.  Impact on how donors such as DfID and WB program and evaluate

support

  Conclusion – policy makers anticipating benefits of program were able to design a robust evaluation that led to increased political support

CHIEF PURPOSES OF IE 1.  Establish whether the program is having a significant

effect on targeted outcomes.

2.  Carefully-designed IE can tell you why something works or doesn’t work.

3.  Inform policy makers of program performance and gaps, including in gaps implementation for improvement.

4.  Most reliable way to achieve this is consider IE design / simultaneously with program design

5.  But there are many paths to the same destination, and we will explore at least three stories, their challenges, and the lessons learned

FIVE “GUIDING PRINCIPLES” TO ENHANCE IE POLICY RELEVANCE Generalizable to scale – typically a need to replicate successful interventions on a wider scale than study context Effectiveness studies with operational orientation – evaluations of programs implemented within existing system capabilities Regular feed-back to implementers – process monitoring validates the program model and identifies areas that need improvement Include a cost focus – to inform policy choices and trade-offs Local ownership – core involvement of government and local investigators is critical both for effective implementation as well as policy adoption All of these principles much easier to attain when IE and program are jointly designed and planned

IE DESIGN TO SATISFY THE FIVE PRINCIPLES: RBF IN ZAMBIA Wide population coverage of key MCH services remains a challenge in Zambia – MoH wants to increase provision of services while simultaneously strengthening health system

After extensive discussion, results based financing (RBF) reform takes the form of fee-for-selected-services paid to facilities with:

• quality-of-care assessments, enhanced supervision and monitoring, devolved autonomy to clinic

•  incentivized activities focus on MCH services

ZAMBIA, WHAT DOES THE IE LOOK LIKE Q: What is the causal impact of the RBF on service provision and population health indicators of interest?

Intervention is a government sponsored system-level reform with a central role for district supervision and monitoring

Treatment arm: Facilities and patients residing in districts that introduce the RBF program

Control: facilities and households in matched “comparison” districts

A difference-in-difference estimator between matched facilities in treatment and control will estimate program impact

BROAD GEOGRAPHIC COVERAGE FACILITATES EXTERNAL VALIDITY In Zambia, MoH with strong desire to pilot RBF in every province. For policy learning, select districts “representative” for that province, based on district health systems capacity and service coverage. 27 out of 62 total districts in study

ZAMBIA, HOW DID THE TWO SIDES WORK TOGETHER Operational and IE team were integrated from the very beginning – the first design discussions with government involved both teams

Repeated consultations with RBF experts, IE experts, and government lead to design of RBF pilot as well as created constituency for IE

As a result, IE extended to answer additional questions •  what is the cost-effectiveness of offering higher incentive levels

•  randomly selected subset of facilities offered additional 25% bonus

• what is cost-effective audit rate in external verification •  facilities presented with randomized likelihood of audit:

100%, 30%, or 10%

ZIMBABWE, AN OVERVIEW OF THE RBF AND THE IE Zimbabwe created a quality adjusted fee-for-service RBF pilot with a focus on improving priority MCH outcomes

Evaluation of RBF grouped comparable pairs of districts and randomly selected one of each pair to receive the RBF pilot and the other to serve as a comparison

Pilot districts stratified by province, so every Provincial Health Management Team gains experience in RBF, as well as involved in the IE planning

Strong constituency for IE within Ministry – creation of a Technical Evaluation Working Group to advise on all steps of IE and to monitor progress

BROAD GEOGRAPHIC SCOPE IN ZIMBABWE FACILITATES EXTERNAL VALIDITY AND INVOLVEMENT OF ALL PMDS IN IE

GOVERNMENTAL IE WORKING GROUP ADVISES ON ALL DATA ACTIVITIES Survey data regularly collected from multiple perspectives:

•  Household •  Facility and health worker •  Community •  District and provincial management

Routine administrative data also critical due to scope and frequency of observation Qualitative assessment with complementary focus on complex subjective qualia: understanding, motivation, and innovative actions Collection, integration, and analysis of multiple data sources (both continuous and sporadic) underscores benefit of locally based investigators and government counterparts

NIGERIA: AN OVERVIEW OF RBF PROGRAM

•  Three states purposively selected:

Adamawa, Nasarawa and Ondo.

•  Selected health facilities in local government (LGAs) within each state receive one of the two interventions: PBF and DFF. LGAs are randomized to receive PBF or DFF.

•  Local and state governments receive bonus payments tied to performance

•  Control LGAs are randomly selected from matched states for each project state

NIGERIA: ADVANTAGES OF DESIGNING THE PROJECT AND IE SIMULTANEOUSLY

Simultaneous design of the project interventions and IE required careful definition of interventions and empirical approaches of measuring impacts. •  Simultaneous development of the interventions and IE

ensured that the distribution of LGAs into PBF and DFF were random so that impact of the interventions can be measured retrospectively.

•  Joint definition of interventions and components. •  The adaptation of the questionnaires were guided by the

definitions of interventions as well as the impacts to be measured.

•  Joint development of routine monitoring tools (quality checklist, counter verification, mhealth questionnaires etc) with the intention of setting up a platform where all data, including HMIS, could be triangulated in the future

NIGERIA: DEFINING THE INTERVENTIONS, PBF AND DFF

NIGERIA: AN OVERVIEW OF IE DESIGN

The IE design of Nigeria will be able to address the following: •  Evaluate conditionality: Impacts of PBF package versus DFF •  Evaluate DFF: Impacts of DFF package versus ‘business as

usual’ •  Evaluate PBF: PBF package versus ‘business as usual’

NIGERIA: HOW IE AND OPERATIONAL TEAMS WORKED TOGETHER TO INTEGRATE ACTIVITIES AND MAXIMIZE LEARNING •  Random selection of LGAs and facilities were

possible without significant political pressure •  Helped in the revision and adaptation of

questionnaires: various aspects of interventions, measures of impacts and intermediate outcomes, capturing potential spill-over effects.

•  Baseline survey will provide some preliminary indications of potential effects of interventions (comparing pre-pilot areas to matched areas), even though baseline information will not allow pre-post, baseline-midline comparisons.