Why Evaluate?

Evaluating the Impact of Projects and Programs, Beijing, China

April 10-14. 2006Shahid Khandker

World Bank Institute

Rationale for evaluation

Evaluation

Helps identify and measure the resultsHelps identify the causal link between intervention and results Provides a systematic and objective assessment of program impactsHelps determine if interventions are relevant and cost-effective Promotes accountability, evidence-based policymaking, and learning.

When to evaluate?

Evaluation can only be done after a certain time and requires thorough investigationIt is a snapshot against some benchmark at a point in time of intervention

What to evaluate?

Operational versus impact evaluationOperational evaluation (OE) measures cost efficiency of program implementation but does not establish causality.Impact evaluation (IE) measures economic efficiency in delivering services and seeks causality. Not all programs should be subject to evaluation: Exception:.. A program of strategic importanceContributes to knowledge gapsInnovative design

How to evaluate?

Impact evaluation hinges on determining what would have happened if the program had not existed.Good practice involves a comparison of outcome before and after intervention with those with and without interventionThe problem is finding a counterfactual – various methods are availableFindings sensitive to the method; alternative methods desirable

Country experiences:

Policymakers ask for feedbacks to determine whether to upscale a program

Micro-finance program in Bangladesh Policymakers required to determine what works and how does it work?

Thailand village fundIndonesia urban poverty projectSocial development fund of Egypt

Government needs to determine the innovative nature of the program design?

Integrated versus sectoral intervention: The case of social fund in Egypt

Country experiences (continued):

Government needs to focus on possible consequences of any policy change Subsidized education policy for girls in Bangladesh and Pakistan TTeacher Incentive structure in KenyaGovernment needs to determine if an economy wide change caused any negative effect: Impact of financial crisis in East AsiaImpact of China’s accession to WTO

Chinese experience • The Grazing Ban component in the China’s

Loess Plateau Watershed Rehabilitation Project:• Initially by a few village committees in the project

area;• Concerns: poor people’s capability of adapting the

change, enforcement, etc• Impact Evaluation: • Free grazing is one of the major cause of soil erosion on

the plateau; • Grazing ban led to pen-feeding and the introduction of

improved breeds of livestock; • Together with other measures, the soil erosion started to

improve;• Farmer themselves get the incentive to enforce the ban.• Eventually it was adopted on a countrywide basis by

many counties widely beyond the project counties.

China’s Poverty Reduction

Nearly 400 million people were lifted out of poverty from 1981 to 2001 (World Bank 2005b, using $1/day poverty line), representing a decline of poverty incidence from 53% in 1981 to 8% in 2001(Ravallion, Chen 2004). But the progress has been highly uneven across time and space (Wang 2005). What are the driving factors? Many studies have been conducted:

• An econometric study showed 46.9% of the total output increase in 1978-85 (one of the fastest growing period) can be attributed to the Household Responsibility System (Lin 1992).

• Ravallion and Chen (2004) found that nearly half of the poverty reduction that happened before 1985 was the result of the rural reform, which led to a clearly defined property right, more equitable access to land, and less distortion in agri prices.

• …

Results: Government’s focus on Rural Income Growth.

China’s Southwest Poverty Reduction Project

Impact analysis shows:

Had a significantly favorable impact

on many dimensions

What worked and how?

Impact Evaluation

•The labor mobility component—benefits the extremely poor (human capital, remittance, etc)

•Extensive participation of poor households in all phases—fundamental

•Emphasis on capacity building, flexibility

•Etc.

Key lessons of the project like new approaches to labor mobility, village deve. Planning, poverty monitoring, etc. have become part of China’s national poverty reduction policy and have been extended to poor counties throughout the country

Monitoring versus Impact Evaluation

Policymakers do not need an impact assessment of all programs and policiesWhat policymakers need most often is better design and effective delivery of program inputs, which can be done with a better monitoring system in place

What does monitoring do?

Monitoring is a tool that provides regular information on:

1.how things are working2.how a project is being implemented3.how project progresses relative to targets

What does monitoring do? (continued)

Rationale for Monitoring: Monitoring holds implementers accountable for delivery of inputsProvides basis for corrective actionProvides assessment of continued relevanceWhile monitoring looks at progress relative to targets and assumes there is causality, evaluation seeks causality.

Components of a M&E Strategy

Outcome-based monitoring such as poverty monitoring

Complemented with systematic impact evaluation

Creating a feedback processBuilding capacity for monitoring

and impact evaluationPromoting participation