the development impact of impact evaluation - … story/japan/pdf...the development impact of impact...
Post on 12-Apr-2018
216 Views
Preview:
TRANSCRIPT
$
RESEARCH
The development
impact of impact
evaluation
Arianna Legovini
Development Impact Evaluation
World Bank, Tokyo, December 5, 2014
2
Get the delivery right
Subsidy for malaria treatment $350 million
Sealing packages saved 16,600 under five +
2,200 adult lives in 5 years
Problem 83% reduction in stockouts
3
Source: Vledder, M.; Friedman, J; Yadav, P., and Sjoblom M.
Get people to do it Source: Goldstein, Thirumurthy, and Zivin
HIV treatment $ 2.1 billion
Reminders increase
adherence to treatment by 35%
Problem IE scalable solution
5
does IE make a difference?
1. Impact of impact evaluation on World Bank project implementation
– Rigorous econometric analysis
2. Theory of how IE influences policy decisions
– Monitoring indicators
3. Institutional citations of impact evaluation
– Descriptive trends in Bank documents
4. Is the cost worth the benefit?
– Basic cost-benefit analysis
7
It does!
…when the IE starts
IE increases disbursements
IE helps projects stick to plans
9
Does IE affect project implementation? 1000 World Bank projects - 101 impact evaluations
(2005-2011)
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28
IE=1 & IE_quarter==0 IE=1 & IE_quarter==1 IE=0
Disbursements increase after IE starts
10
IE increases disbursements (proportion of total commitments)
statistically significant at 1% level
14.2
19.4
0
5
10
15
20
25
Cumulative quarterly disbursement without IE Cumulative quarterly disbursement with IE
without IE with IE
37%
11
Plan to actual disbursement gap: IE decreases delays in implementation
12
-0.15
-0.1
-0.05
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
IE=1 & IE_quarter==1 IE=0
without IE
with IE
Why? Clear roadmap
Clearer roadmap
What
To whom When
Where
14
Why? Better capacity
Better capacity
Research team
Full time field coordinator
Training
More and better data
15
dime results indicators
88%
74%
75%
36%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Client participated in IE workshop(s)
Baseline results discussed with client
IE results discussed with client
Training provided for data analysis
Capacity Building
16
dime workshops increase quality of proposals
FCS window, July 2014 - double blind external evaluation
dime results indicators
84%
85%
48%
50%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%
High-quality baseline survey
High-quality follow-up survey(s)
Improved administrative data
Availability of Data as a Public Good
Quality of Data
19
Why? More effort into implementation
Motivation to perform
Goal: this is what we are
trying to achieve
Control: this is how we want
to do it
…and BTW somebody
measures (and cares)
20
Why? Better decisions 21
dime results indicators
36%
23%
30%
0% 5% 10% 15% 20% 25% 30% 35% 40%
Baseline informed policydesign/implementation
Adopted delivery or behavioral mechanismbased on IE results
IE results were used to motivate scale-up/scale-down of policy at national level
Quality of Policy Decisions
22
DOES ANYONE CARE? TRENDS IN INSTITUTIONAL CITATIONS OF IE
23
Source: WB Archives, author’s calculations Note: 1. IE citations include “IE” and “Impact Evaluation” 2. The statistics include 158 types of documents at WB, such as Country Assistance Strategy Document, Project Appraisal Document, Implementation Status and Results Report and Working Paper.
IE Citations multiplied by 6
3,691
21,271
0
5,000
10,000
15,000
20,000
25,000
05 06 07 08 09 10 11 12
Tota
l IE
Cit
atio
ns
Fiscal Year
All Documents 2 per. Mov. Avg. (All Documents)
24
1,092
7,684
0
2,000
4,000
6,000
8,000
10,000
05 06 07 08 09 10 11 12
Tota
l IE
Cit
atio
ns
Fiscal Year
Source: WB Archives, author’s calculations Note: The statistics include several types of documents, such as Working Papers and other Research Documents.
Research documents
25
Operational documents
400
2,773
0
500
1,000
1,500
2,000
2,500
3,000
05 06 07 08 09 10 11 12
Tota
l IE
Cit
atio
ns
Fiscal Year
Project Appraisal Documents
25
181
0
20
40
60
80
100
120
140
160
180
200
05 06 07 08 09 10 11 12
Tota
l IE
Cit
atio
ns
Fiscal Year
Country Assistance Strategies
Source: WB Archives, author’s calculations 26
433
3,367
0
500
1,000
1,500
2,000
2,500
3,000
3,500
4,000
05 06 07 08 09 10 11 12
Tota
l IE
Cit
atio
ns
Fiscal Year
Source: WB Archives, author’s calculations Note: The statistics include 3 types of documents: Board Report, Implementation Status and Results Report, and Implementation Completion and Results Report.
Project completion documents
27
Is the cost worth the benefit?
• Zambia best supply chain – IE solution: 83% reduction in stock out
21-25% reduction in malaria mortality
– Pilot: Benefit ($7.4M) / Cost ($3.8M) = 2
– Scaled up: Benefit ($60.7M) / Cost ($12.4M) = 4.9
• Malawi commitment saving account – IE solution: 22% increase in farmer production value
– Pilot: Benefit ($0.7M) / Cost ($0.5M) = 1.4
– If scaled: Benefit ($15.6M) / Cost ($1.7M) = 9.2
29
What does dime do?
Transform development policy
30
1. Run experiments to inform decisions
2. Build agencies’ capacity to do this systematically
3. Draw lessons and share them face-to-face to global audiences
31
dime in action
Inform policy design
Guide mid-course
corrections
Inform adoption and
scale-up
Syst
em
atic
use
of
evid
ence
Train & apply
Learn by doing
Apply knowledge
Capacity
build
ing
IE Product
IE DESIGN
IE IMPLEMENTATION
IE DISSEMINATION
175 IEs in 47 Countries Distribution of dime ’s IEs by GP and CCSA
(number of IEs)
32
32
18
4
2
11
15
14
25
5
21
16
8
4
0 5 10 15 20 25 30 35
Agriculture
Education
Energy & Extractives
Environment & Natural Resources
Finance & Markets
Fragility & Conflict
Governance
Health, Nutition & Population
Social Protection & Labor
Social, Urban, Rural & Resilience
Trade & Competitiveness
Transport & ICT
Water
Policy questions in dime’s portfolio
(% of questions)
33
3%
7%
9%
12%
20%
21%
28%
0% 5% 10% 15% 20% 25% 30%
Top-downaccountability
Demand-sideaccountability
Behavioral biases
Incentives
Constraints
Delivery mechanisms
Packages
• IE has real impacts on developmental effectiveness and project delivery
• High return on investment
• Spillover effects
• i2i opportunity to increase scale and scope
• Open invitation to donors and governments
dime & i2i
34
partners
CRIME LAB
35
The strategic framework for learning and accountability
•Lesson 1: Base development policy decisions on evidence
•Lesson 2: Make learning part of the culture of development co-operation
•Lesson 3: Define a clear role for evaluation
Delivering evaluations effectively
•Lesson 4: Match ambitions with adequate resources
•Lesson 5: Strengthen program design and management systems
•Lesson 6: Ask the right questions and be realistic about expected results
•Lesson 7: Choose the right evaluation tools
•Lesson 8: Work together
•Lesson 9: Help strengthen partner country capacities and use them
Strengthening the culture of learning
•Lesson 10: Act on evaluation findings
•Lesson 11: Communicate evaluation results effectively
•Lesson 12: Evaluate the evaluators
Shared Objectives Evaluating Development Activities, 12 Lessons from the OECD DAC
(OECD 2013)
Human Development
• Education • Health • Social Protection
i2i
pillars and thematic areas
36
Shared Prosperity
• Finance, Private Sector and Jobs
• Agriculture • Infrastructure
Governance
• Public Sector Governance
• Justice • Local Development
Climate Change
• Energy • Environment • Natural Resource
Management • Agriculture • Transport
Fragility & Conflict
• Reintegration • Governance • Gender-based
Violence • Urban crime &
violence
Gender
• Human Capital • Economic
Opportunities • Voice/Agency
5
1 2 3
6 4
Total ODA, Fast Growing Sectors 2006-2012 (USD billion)
37
5.1 5.8
1.9
7.2
5.1
1.6
4.7
9.3 10.3
3.5
16.5
13.9
4.9
11.5
Pop. &Reproductive
Health
Water Supply &Sanitation
Conflict, Peace &Security
Transport &Storage
Energy Banking & FinancialServices
Agriculture,Forestry, Fishing
2006 2012
Source: www.oecd.org/dac/stats
38
Impact Evaluation 4 Peace: Evidence to lower violence and conflict
• In DRC: can short-term employment reduce youth participation in violent and armed activities?
Prioritized themes:
• Urban crime and violence
• Jobs and productive opportunities for youth at risk
• Sexual and gender-based violence
• Public sector /civil service reform
• In Mexico: can behavioral therapy, intensive mentoring and jobs reduce recidivism of youth incarcerated for serious crimes?
• In India: Which of three different strategies can help shift norms to prevent sexual and gender-based violence?
How do we supply reliable, clean energy to support sustainable growth?
Challenge Testing Solutions
Reduce transmission losses
Consume responsibly
Increase reliability
Slow down deforestation
Limit overgrazing and desertification
Mainstream climate smart technologies
Ensure sustainability of infrastructure
Institutions for O&M
Demand-side management
Infrastructure prioritization
Property rights/incentives/
governance
Capacity building and networking
Governance of irrigation
How do we align private, social and environmental objectives to sustainably manage natural resources?
Impact Evaluation in Energy and Environment
Challenge Testing Solutions
ieGovern: Governance & Justice Rigorous evidence on what works in
governance is in short supply… The World Bank has a comparative advantage
for leading IE research in the sector.
79
87
98
103
267
299
380
571
1,551
0 500 1,000 1,500 2,000
Transport & ICT
Governance
Environment & Natural Resources
Water
Urban, Rural, and Social Development
Finance & Markets / Trade & Comp.
Social Protection & Labor
Education
Health, Nutrition, & Population
Number of Impact Evaluations
Glo
bal
Pra
ctic
e
Governance IEs represent less than 3% of globally registered research studies
Source: 3ie Impact Evaluation registry.
Largest Public Sector Modernization lending portfolio among donors
Strong managerial support for IE
2015201 2014: Outreach and thematic expansion 2013: 10 IEs designed and launched in Africa Region
May 2014: 2nd Steering Group meeting
Chairs: Arianna Legovini (DIME) and Bill Dorotinsky
(GGP)
January 2015: Global ieGovern workshop
in Istanbul
Civil Service Reform Justice
Subnational PSM / Service Delivery
New Themes • Citizen
Engagement • Decentralization • Procurement • Public Financial
Management • Tax Admin.
Countries • Colombia • Ethiopia • India • Jamaica • Kazakhstan • Pakistan • Vietnam
- Design interventions to mainstream gender into operations - Run RCTs to measure results - Inform course corrections - Build knowledge across DIME’s knowledge areas.
Gender and Impact Evaluation
Financial Empowerment & Women
Gender Empowerment to Combat Domestic Violence
Addressing Human Capital Gender Gaps through the Lifecycle
Maternal & Child Health Initiative
Malaria Control Booster Project
our gender impact
• build women’s empowerment and agency – women can reduce informational constraints to adoption (Malawi and
Mozambique) – clarifying land rights can be especially productive for women farmers
(Benin) – gender-tailored saving products lift investment constraints (Rwanda)
• address human capital gender gaps through the lifecycle – reproductive health (Nigeria) – vocational training to ease the school-to-work transition (Malawi)
• combat domestic violence – Shifting social norms (India) – Address reproductive coercion through health programs (Mexico)
More cooperation in Operations
In 2007-2010, multi-bilateral ODA jumped from $9 billion to $16.7 billion
Limited Evaluation Resources
• The central evaluation unit of DAC member country has a budget of USD 2.4 million, or 0.16% of that country ODA.
• The evaluation budget of multilateral institutions is USD 10 million, a tiny fraction of lending.
Call for Collaboration
• The 2010 OECD report notes new demands on DAC evaluation units for rigorous impact evaluations, working in a more collaborative way with country partners, and synthesizing findings to report on development results
• i2i offers a framework for collaboration to exploit economies of scale in learning and knowledge sharing
44
Increasing Cooperation in Evaluation: small budgets, large task, economies of scale
Discussion
• In 2013, net official development assistance reached a record of US$138 billion.
• Which development interventions are effective and worth scaling up? Which promising mechanisms are worth testing across different sectors? Impact Evaluations provide guidance.
• The i2i platform can facilitate the rapid expansion of IEs, especially in under-evaluated sectors.
top related