The use of impact evaluations in Nepal

Impact evaluations in Nepal

Questions

Ministry of health: Should

the health insurance

program require a co-pay

or not?

Ministry of education:

Does individual teacher

incentive improve test

scores more effectively

than group incentives?

Traffic police: Is a Rs.50

fine for violators of lane-

driving enough to cover

the costs of monitoring? Ministry of health: Does a

public health campaign to

increase awareness

reduce smoking rates

among teenagers?

Impact evaluations in Nepal

Impact evaluations can help inform various aspects of policy

• Evaluation of processes • Are the processes effective, efficient, appropriate, etc.

• Cost-benefit evaluations • Which program produces the most benefits relative to

costs

• Cost-effectiveness evaluations • Which is the best program for per rupee spent?

• Evaluation of impact • What is the (causal) impact of an intervention on an

outcome of interest?

Impact evaluations in the education sector

A look at the education sector

High enrollment in primary However, enrollment rates drop

significantly in secondary schools

What can we do to increase secondary school enrollment?

A. Students drop out to work provide incentives to encourage schooling

B. Parents and students underestimate the value of education implement awareness campaigns

C. Students drop out because they are poorly prepared improve learning in primary grades

Impact evaluations in the education sector

There are an estimated 50,000 children in the labor force

and of these 11,000 are in the carpet industry (ICF, 2012)

Research Question: Can providing incentives for

schooling decrease the incidence of child labor?

Impact evaluations in the education sector

Example 1: Schooling incentives to reduce child labor

Children working in garment factories

(660 children)

Scholarship group: Out-of-pocket education expenditure

paid or reimbursed up to Rs.3,950 per year

Stipend group: scholarship + stipend of Rs.1,000 per month

conditional on 80 percent attendance

Comparison group

Can you guess which

program produced

the greatest impact?

And by how much?

- Edmunds and Shrestha (2015)

Impact evaluations in the education sector

Example 1: Schooling incentives to reduce child labor

Impact evaluations in the education sector

Example 1: Schooling incentives to reduce child labor

• Scholarship treatment increases

attendance in the first few months

of the program but the effect fades

away quickly

• Why?

• Stipend treatment has large effects

on attendance and achievement

• Child participation in carpet

weaving also falls

• The effects dissipate when

schooling is not longer incentivized

Impact evaluations in the education sector

Research Question: Can computer-aided

instruction help improve learning?

Impact evaluations in the education sector

Example 2: Evidence from other countries

Israel 2002

35,000 computers in 905 schools

4th and 8th graders

Computer use

Test scores

India 2007

Two hours per week shared computer

time

4th graders

Math scores Language scores

Romania 2010

Vouchers for low-income families to

purchase computers for

home

English, Math, Romanian

Computer Science

Colombia 2009

Computers for education program

3rd – 9th graders

Test scores

India 2016

Adaptive learning materials

6th to 8th grades

Math and Hindi scores

Impact evaluations in the education sector

Example 2: Evaluation of One-Laptop-Per-Child (OLPC) in Nepal

• The OLPC program – a collaboration between Department of Education and Open Learning Exchange Nepal

• A low cost laptop provided to 2nd, 3rd and 6th grades students from 26 program schools(2,486 students) and 39 control schools (4,279 students) in 6 districts

• Teachers were provided laptops and trained for 10 days on how to teach different subjects using laptops

• Baseline test scores differences – program schools better

No statistically significant effect on mathematics,

some negative effect on English.

Negative effect more pronounced for students in the

bottom tercile of baseline scores

- Sharma (2014)

Impact evaluations in the education sector

Question: Does this mean that laptop programs like

OLPC have no impact on student learning?

Results from Nepal and other countries can help

us design and test a powerful technology-aided

instruction program!

Maternal and child health (MCH) and Early Childhood Development (ECD)

Impact evaluations in the health sector

Massive improvements in maternal and child health between 1996 and 2006

0

5

10

15

20

25

30

35

De

ath

s p

er

10

00

live

bir

ths

Neonatal mortality rate

1996 2006

0

100

200

300

400

500

600

De

ath

s p

er

10

00

00

live

bir

ths

Maternal mortality rate

1996 2006

Substantial decline in neonatal mortality rate Halving of Maternal Mortality Rate

Questions:

(1) What contributed to such improvements?

(2) How do we make further improvements?

First generation intervention

Example 3a: Direct education by health workers on infant care practices after delivery

• Postnatal health care education for mothers on infant care and postnatal family planning practices in Kathmandu district

• One-on-one health education for mothers

• No significant effect on infant feeding, infant care, or immunization.

- Bolam, Manandhar, Shrestha, et al., 2004

A second generation intervention

Example 3b: Participatory intervention with women’s group on birth outcomes

• Clustered randomized controlled trial in Makwanpur district

• A facilitator – literate and locally resident – convened o e wo e ’s group eeti g each month in every ward

• Total of 10 meetings over the year

- Manandhar, Osrin, Shrestha, et al., 2004

Impact evaluations in the health sector

0

10

20

30

40

De

ath

s p

er

10

00

live

bir

ths

Neonatal mortality rate

Treatment Control

0

100

200

300

400

De

ath

s p

er

10

00

00

live

bir

ths

Maternal mortality rate

Treatment Control

• Large positive effects on neonatal mortality rate and maternal mortality rate

• Highly cost effective- $111 per life year saved

Participatory intervention with women’s group on birth outcomes

Hardly surprising! Nepal is one of the exemplars of successful community and local

governance!

- Elinor Ostrom, Nobel Laureate in Economics

Caution before comparing: Different samples and different interventions!

Impact evaluations in ECD

Intervention on parenting practices

Communities

Monthly meetings with info on nutrition and feeding practices

Meetings + cash payment for each pregnant woman and child under

2 years age

Comparison

• Implemented in four districts in

collaborating with Ministry of Federal

Affairs and Local Development

• Randomized control trial with

randomization at the village level –

591 villages in 184 VDCs

• Design of the experiment helps

distinguish between two barriers to

nutrition

• Lack of information

• Financial hardship

Improved maternal knowledge

Improved motor skills in children

- Levere, Bharadwaj, and Acharya, 2004

Impact evaluations in Nepal: Looking Ahead

Impact evaluations in Nepal

Looking ahead

Because of the strong support of the Government of Nepal

• There have been rigorous impact evaluations in Nepal since the 1990s

• There is also tremendous capacity to carry out rigorous impact evaluations • Policy-makers, research institutes, and individual researchers, survey firms etc.

• Donors have committed a lot of resources for impact evaluations in Nepal • Examples: SIEF, World Bank; DFID; 3ie; GDN

Now is the perfect opportunity to leverage the full potential of impact evaluations for efficient policy-making!

Thank you!