Political economy of aid evaluation: How to build a sustainable and effective

movementLant Pritchett

(with Salimah Samji)Harvard Kennedy School

Feb. 5, 2009

Outline of the presentation

• How technocratic approaches can get it completely wrong—or “how I got my lunch eaten” (and that by people with no academic credentials)

• MeE: An alternative to an exclusive focus on “Big E” evaluation that might actually work

• (if there is time) What the poor do actually say…

My Indonesia Story

• Crises of 1997/98, currency crisis, economic meltdown, Soeharto resignation, poverty rising rapidly

• July 1998 budget of new government allows (insists on) fiscal space for “Safety Net” programs

• August 1998 I arrive in Indonesia working for the World Bank and end up in charge of a cumulative 1.2 billion dollar loan to support the design and finance crisis mitigation safety net programs (aside: Total loans of Grameen Bank in 2002 ≈200 million)

Safety Net Programs in Indonesia, 1998-2000

Benefit Incidence of the Indonesian Crisis Safety Net Programs

Benefits relative to the poorestQuintile of income

I II III IV V

11

Uniform transfer(dollar to each person)

Only to the poor—“Perfect” targeting

Actual performance Of the Safety Net Programs

Budget for 2000/2001

• Economy was stagnant• Real wages, and hence poverty, was recovering• How much of the “safety net” programs should

remain• Nearly all of them got axed (scholarships, pro-

poor health cards, employment creation)—in part because they were attacked for not being sufficiently targeted to the “poor” as our own studies documented “leakage” to the “non-poor”

The subsidy to Kerosene

• At the same time, there was an effort to eliminate the massive subsidy to fuel, including a subsidy to kerosene.

• Huge hue and cry, protests in the street, political opposition

• The subsidy to kerosene was claimed to be “pro-poor” and hence untouchable

• The kerosene (bigger than many safety net programs) was spared

The facts about benefit incidence

All of the safety netprograms

Kerosene subsidy—4th

Quintile got 2.5 times what Poorest quintile did

The Truth—Which Sets You Free

• The kerosene subsidy created a price differential with nearby Singapore and Malaysia

• A small group of generals skimmed off a substantial amount of kerosene production and shipped it to Singapore, making millions of dollars a year

• The protests were orchestrated (the going rate in the market for protestors was 2000 rp/day (plus lunch if all day, a snack if only afternoon))

• The newspaper editorials were similarly purchased (they were almost as cheap)

Kerosene Subsidy, round II

• Design a “compensatory” targeted safety scheme to cushion the “shock” to the poor of the kerosene price rise

• Launch the price rise and the safety net program simultaneously in the next round of budget discussions—takes “the poor” off the table

• The program itself is self-liquidating (as the magnitude of transfer is based on the change n the price of fuel)

• But the purpose of the program was not about the poor and the impact of the kerosene price (which was trivially small) but about a two step political gambit—get the fiscal savings from cutting the subsidy in the first round, taking the poor off the table, then get the program eliminated the next--which only a dozen or so people in the country understood.

What is “evaluation” about—finding out the most “cost-effective” policy/program designs

Program objectives,Technocratically conceived

Design parameter

Optimal design

Budget availableto the program

Technocratic naiveté:Budget assumedThe same for all designs

Rangel vs. Gingrich:Simple story about targeting (with a rigorous

model) • Technocrat assuming fixed budget tries to maximize benefit to the poor

• At any given budget increasing targeting increases well-being of the poor

• But the true budget is downward sloped (more targeting means less budget)

• The naïve pro-poor technocrat produces the worst outcome for the poor –full targeting of zero budget

What is “evaluation” about—finding out the most “cost-effective” policy/program designs

—or inform actual policy?

Program objectives,Technocratically conceived,Per dollar

Design parameter

Budget availableto the program

DesignA

DesignB

Rigorous evaluation demonstratesdesign B is much more cost-effective than design A,but both are above the “threshold”—what is the “policy” implication?

Threshold

And if design B would leadto program elimination (zero budget)?Same “policy” iimplication? Move to B?

“Evaluation” as a Development Initiative

• So far as deeply unscientific as any other advocacy group—claims with no evidence of any kind

• Maybe right, maybe wrong, but certainly methodologically incoherent

• Success cases? (Colombia? Great paper, dead program)

Well-being of the poor

Resources toevaluation

Claims about impactof evaluation requirea model of how additional knowledge affects realized policies

?

What do the poor say?

“Is this information you are gathering from us just to help you write your report or can you really be helpful to us?”

Woman in South Sudan to me, two weeks ago

Its all about MeE—Adding little ‘e’ evaluation to the standard mix

Monitoring:The gathering of evidence to show what progress has been made in the implementation of programs. Focuses on inputs and (sometimes) outputs.

Big E--Evaluation:Measuring changes in outcomes and evaluating the impact of specific interventions on those outcomes. Focuses on “with and without” interventions (needs “control” group) and identifies causal impacts.

Little E—evaluationUses within project design variations to identify differentials in the efficacy of the project on inputs and outputs for real time feedback into project/program implementation

Complementary roles for M&E

Monitoring

• Routine collection of information

• Tracking implementation progress (actual vs. target)

• Measuring efficiency

“Is the project doing things right ?”

Evaluation

• Ex-post assessment of effectiveness and impact

• Confirming project expectations (unintended results)

• Measuring impacts

“Is the project doing the right things?”

Introducing “e” to the mix

• “e” lies in between M and E

• Analyzing existing information (baseline data, monitoring data)

• Drawing intermediate lessons

• Serves as a feed-back loop into project design

• Don’t always have to do Impact Evaluation

The problem in pictures

Lost opportunity: No timely “e” to help the project!!

Lets begin with the project time line

T-1 T+1T T+2 T+5

Pre-appraisal Project effectiveness

Project closure

Lots of “M” – passing raw data unto God for whatever use …

Findings of “E” come too late to

be of much assistance to implementers

Ideally …

T T+1

T+2

If your M&E system doesn’t provide you with timely decision making information – what is the point for the

implementers?

Implement design A

and B, collect “M” + analyze the M (“e”)

Point of

decision making:

to scale up

A or B?

Scale up A

Scale up B

Implement design A-1

and a-2, collect “M” + analyze the M (“e”) Point of

decision making:

to scale up

A-1 or A-2?

Or Similarly with B-1 and B-2

So … where do you begin?• Clear objectives of the project (what is the

problem?)

• Clear idea of how you will achieve the objectives (causal chain or storyline)

• Outcome focused:– Answer the question: What visible changes in

behavior can be expected among end users as a result of the project, thus validating the causal chain/ theory of change?

Design the “M, e and E” Plan• What?

– Type of information and data to be consolidated

• How? – Procedures and approaches including methods for data

collection and analysis

• Why? – How the collected data will support monitoring and project

management

• When?– Frequency of data collection and reporting

• Who? – Focal points, resource persons and responsibilities

• No collection of data on a “no program” group required—the comparisons are “within program/project” variants

• Project implementers feel part of the process, see benefits to themselves and their objectives

• Big E evaluation often cannot distinguish causes of failure—many projects simply fail to be implementer

• Big E evaluation can explore only a tiny part of the design space (even with 5 design parameters, 2 options each, with complementarities the dimensionality blows up)

Advantages of little e over Big E evaluation

• Clear relationship to other sub-movements within development (see next set of slides)

• Is within the capability of nearly all implementing organizations– M is always feasible– Big E requires substantial expertise to produce reliable, replicable

(publishable?) results– Little e can be done by project management units

• Using M data for little e makes the M unit high stakes and keeps the data real and relevant (otherwise MIS systems drift out of date, people lose interest).

• Requires conscious consideration of design parameters, keeps open alternatives

Advantages of little e over Big E evaluation (con’t)

“Evaluation” as a innovation/movement/advocacy

position to improve “development”Successful Movements

• Clearly articulated vision• Politically feasible

coalition• “Career” trajectories• Patina of “normal

science”

…but can be ineffective

• Insularity, not open to question fundamental premises

• Lock-in of movement specific “human capital” politically defensive

• Takes too long to shift if proves ineffective

How does evaluation fit in “development”

• “Development” is a coalition of narrower sub-movements both objective specific (e.g. education, health, gender, environment) and instrument specific (e.g. micro-credit, irrigation)

• Help to make “successful” movements also effective

• Eventually weed out the successful but ineffective sub-movements (but this is hard and unlikely to be the result of Big E evaluation)