case study 1: what, why and how of impact...
TRANSCRIPT
Case study 1: What, why and how of impact evaluations
Shubhra Mittal
Senior Policy and Training Manager, J-PAL South Asia
Kathmandu, March 28, 2017
I. Introduction to J-PAL
II. Why Evaluate: Case study of pricing of preventive health products
III. Case Study 2: understanding impact of a programme
2
About J-PAL Who we are, what we do
Governments face multiple problems across various sectors… A Typical Social Policy Goal – Improving Learning Outcomes
3
To address poor learning levels, there are many potential solutions:
Free
uniforms
Libraries Cash
grants School
equipment
Information
Campaigns Technology Pedagogy
changes
The fundamental dilemma - How do we know which solution creates the most impact? How do we know which solution is the most cost-effective?
Impact evaluations identify the causal impact of a programme (e.g. a pedagogy
intervention) on an outcome of interest (e.g. learning outcomes) by comparing what happened with the programme with what would have happened without the
programme i.e. a counterfactual.
J-PAL’s mission is to reduce poverty by ensuring that policy is informed by scientific evidence.
Scale-ups Partnerships Dissemination Workshops Courses
Agriculture Crime Education Environment and Energy
Health Finance and Microfinance
Labour Markets
Political Economy and Governance
EVALUATIONS J-PAL affiliates conduct randomised evaluations to test and improve the effectiveness of poverty reduction programmes across sectors.
POLICY J-PAL affiliates and staff disseminate research results and build partnerships with policymakers to ensure policy is driven by evidence and effective programmes are scaled up.
CAPACITY BUILDING J-PAL trains implementers and policymakers on how to become better producers and users of evidence from impact evaluations.
We are a network of 146 affiliated professors from over 49 universities…
6
J-PAL Latin America
and the Caribbean: 86
completed
evaluations; 41
ongoing
J-PAL North America:
136 completed
evaluations; 18
ongoing
J-PAL Africa: 131
completed
evaluations; 102
ongoing
J-PAL Europe:
37 completed
evaluations; 6
ongoing
J-PAL South
Asia: 98
completed
evaluations;
68 ongoing
J-PAL Southeast
Asia: 28
completed
evaluations; 16
ongoing
J-PAL Global:
Headquarters
supporting regional
offices
Nearly 819 ongoing and completed evaluations across 8 sectors in over 78 countries
… across 7 global offices, working with local partners on local issues.
• Over 166 completed and ongoing evaluations across SA; 124 across 13 states of India
• 19 different partnerships with various state
governments
• Partners in India, Bangladesh, Nepal, Pakistan, and Sri Lanka
• 3650+ policymakers & practitioners trained to conduct high quality Monitoring & Evaluation
J-PAL South Asia est. 2007 at the Institute for Financial Management and Research (IFMR), Chennai
7
South Asia senior management team
Iqbal Dhaliwal (ex-IAS) Deputy Director, J-PAL
Scientific Director, SA
Esther Duflo (MIT) Director, J-PAL
Scientific Director, SA
Sanjoy Narayan
Executive Director J-PAL SA
Shobhini Mukerji
Executive Director J-PAL SA
• State partnerships to institutionalise an evidence
based approach with the government
• Supporting the scale-up of programmes that
work
• Collaborating with Govt. departments and
Ministries to rigorously evaluate innovative
solutions
• Sharing evidence on what works (and doesn’t)
• Assisting in M&E capacity building
Fostering Evidence-based Policymaking
8
9
Using evidence to improve
effectiveness of policy
Why evaluate?
Rigorous evaluations have produced important and surprising results
10
• Major programmes not as effective as previously thought
– Fixing supply of health services, inputs to education
• Small interventions proved very cost-effective
– Deworming
• Conventional wisdoms have been undermined
– Incentives for monitoring, community participation
11
Case study1: Choosing between
alternative programme designs Pricing of preventive health products
Pricing of preventive health products The debate in the policy/donor community: free versus subsidised distribution
12
Free distribution?
Cost sharing?
• People who need something are more willing to pay for it
• Paying for goods makes people more likely to use them (sunk cost bias is significant)
• Giving away goods and services for free creates dependency (an entitlement effect)
• Charging fees helps programmes maintain financially sustainability
• Prices prevent access for people who need it the most
• Sunk cost bias may be negligible
• Free samples help people learn about a good’s benefits and they will be willing to pay for them later
• There are positive health externalities that warrant complete subsidisation for maximum public health benefits
Understanding usage patterns for ITNs based on price paid
13
Background
• In Kenya, malaria responsible for one out of every four child deaths
• ITNs shown to reduce child mortality in regions of Africa
• Less than 5 percent of children and pregnant women sleep under an ITN
Programme
• ITN distribution to pregnant women who visited clinics for prenatal care
Cohen and Dupas 2010
Randomised evaluation design: understanding usage patterns for ITNs based on price paid
14
Status quo
Free
Subsidy 97.5%
Subsidy 95.0%
Subsidy 90.0% P
reg
na
nt
wo
me
n v
isitin
g
clin
ic o
ffe
red
ITN
s
Additional discount ($0 to posted price) for women interested to purchase an
ITN
Results: Cost sharing reduced take-up of ITN
16
• Charging 60 cents (10% of actual price) for insecticide treated nets (ITNs) reduced take-up by 60pp relative to free distribution in Kenya (2007)
• No evidence that cost-sharing put ITNs in the hands of women who need it the most
• No evidence that the act of paying for a product makes a recipient more likely to use it (ITNs no more likely to be hanging in the house during a spot check)
Cohen and Dupas 2010
Take-up of preventive products drops as price increases
17
Rigorous evidence has informed pricing of ITNs
J-PAL | EVIDENCE FROM RANDOMIZED EVALUATIONS OF ECONOMIC INTERVENTIONS IN HEALTH 19
20
Case study2: Understanding
programme’s impact Targeting the Ultra-poor
• Ultra-poor women headed households make up 69.4
lakh households in India and have an average monthly income of Rs.1,250
The problem
21
Ultra-poor female headed households
There is no evidence yet that they benefit from traditional credit-based interventions
Defining the Ultra-poor
A potential solution – the ‘Graduation Model’ Carefully sequenced support for the poorest of the poor women in rural communities to graduate out of extreme poverty
22
Programme costs Rs20,000 per beneficiary
Choosing the counterfactual
1. Option 1: Programme participants before the programme
• Why or why not?
2. Option 2: Programme non-participants
• Why or why not?
How do we measure TUP’s impact? Impact = what happened with the programme - what would have happened WITHOUT the programme
23
Designing a randomised evaluation to measure TUP’s impact
24
Status quo (#466)
Offered programme
(#512)
Poorest of the
poorest women
in village
Period Time since
asset
transfer
Baseline Before programme implementation (February 2007- March 2008)
-
Endline
1
Completion of programme (January 2009- November 2009)
1.5 years
Endline
2
One year after programme completion (June 2010-February 2011)
2.5 years
Endline
3
Five years after programme completion (September 2014- March 2015)
7 years
Understanding Intent-to-Treat and Treatment-on-the Treated
26
Programme take-up: 56 percent
1. Intent-to-treat
• Comparing people offered programme with those in the comparison group
• Consumption increases by 15 percent at the end of the programme, and nearly 25 percent five years after the end of the programme
2. Why may we be interested in understanding the ‘treatment-on-the treated’?
Households experience broad and lasting economic impacts
27
• 46% higher consumption than comparison group five years after programme completion
• Consumption pattern changed – spent more on dairy, protein-rich foods and durable goods
• Income increased
• Sources of income diversified
• Food security improved
• Household assets and savings
increased
0
10
20
30
40
50
60
70
80
90
100
At programmecompletion
One year later Seven years later
Comparison group Programme Participants
25%
20%
46%
Replicated in ~20 countries, multisite randomised evaluations in 7 countries
29
India Bangladesh
Pakistan
Ethiopia Ghana
Peru
Honduras
Multisite RCTs funded by the Graduation Program Consortium, i.e., Ford Foundation and Consultative Group to Assist the Poor
Basic programme components
adjusted to fit the individual country
context and implemented by local
organisations
How have government’s responded to this evidence?
Government of Rajasthan
Rajasthan State Livelihoods mission(Rajeevika) is funding the THP programme implementation, by Bandhan Konnagar, in the Manohar Thana block of Rajasthan for 1,000
Odisha
Jharkhand*
Bihar Rajasthan
Government of Jharkhand
Jharkhand Welfare Department, Government of Jharkhand is funding the THP implementation in 2,000 households in two districts (Dumka and Paschim Singhbhum).
Bihar and Odisha, USAID funded DIV funding 4,350 Households in two states, Bihar and Odisha
Madhya Pradesh
West Bengal
Assam
Other states Funded by foundation and donor funding
Concluding thoughts Social programmes + Rigorous evidence = Higher Impact
31
• Understanding the impact caused by the programme
– Are the people better off than they would have been otherwise?
– What are the reasons for success/failure?
– Causal effect can be determined through a rigorous evaluation
• Comparing programmes and choosing the best – What is the most effective way to achieve an outcome?
– Are there common strategies that will succeed across fields?
• Using rigorous evidence of impact of your intended programme to inform decisions
– Expand coverage of programme?
– Withdraw programme?
– More evidence needed?
