Conducting and Using Impact Evaluations: A large-scale Experiment in Ghana

Annie Duflo & Jessica Kiessel March 10, 2015

• The problem and existing lessons • Designing the intervention • Designing the evaluation • Analyzing and understanding results • Conclusion

THE TEACHER COMMUNITY ASSISTANT INITIATIVE

LARGE LEARNING GAPS English Test Scores

24

57

98

11 20 31

0

20

40

60

80

100

P1 P2 P3Grade Level

The context: Heterogeneous levels

HETEROGENEOUS LEVELS

WHAT HAVE WE LEARNED?

Kenya: Adding extra teachers most

effective when class split by ability

India: Pull-out remedial classes for

low-performing students by

community tutors (Balsakhi)

India: Community-based volunteers running remedial classes in

communities (Read India Program)

FOCUSING INSTRUCTION AT THE R IGHT LEVEL IS KEY

• The problem and existing lessons • Designing the intervention • Designing the evaluation • Analyzing and understanding results • Conclusion

THE TEACHER COMMUNITY ASSISTANT INITIATIVE

FROM INDIA TO GHANA

• Key concept

• Context

• Political acceptance

• Sustainability

PROGRAM DESIGN AND SUPPORT

G OVERNMENT’ S QUESTIONS ABOUT MOST EFFECTIVE DESIGN

• The problem and existing lessons • Designing the intervention • Designing the evaluation • Analyzing and understanding results • Conclusion

THE TEACHER COMMUNITY ASSISTANT INITIATIVE

C Assistant-led

remedial classes after

school

D Extra assistant: Randomly split revision group

E Teacher-led

Targeted instruction

Randomly allocate to :

A

Control group

B Assistant-led

remedial classes during

school

500 schools across all 10 regions, 42 Districts

RANDOMIZED EVALUATION DESIGN

RANDOMIZED EVALUATION DESIGN

N ATIONALLY REPRESENTATIVE SAMPLE OF 42 ,000 GRADE 1 -3 PUPILS

Targeted Instruction

Extra materials

Remedial pedagogy & materials

Extra Assistant

Smaller class size

Extra school hours

1. Rem Ed during school hours

X

X

X

X

X

2. Rem Ed after schools

X

X

X

X

X

3. Extra Assistant X

X

X

4.Targeted instruction by teachers

X

X

X

5. Control

INTERVENTION KEY FEATURES

THE FRAMEWORK

2.

Testing Proof of concept

3.

Refining the concept’s

applications across contexts

4.

Testing Scalability

EVALUATION OBJECTIVES

Mobilizing and supporting decision-makers to use evidence

Randomized Controlled Trials

1.

Identifying problems and

potential solutions

• Final outcomes (oral and

written testing tools) • Intermediate variables

(midline observation surveys) • Supporting characteristics

(baseline student, school and community surveys)

13

DATA COLLECTION DATA COLLECTION

• The problem and existing lessons • Designing the intervention • Designing the evaluation • Analyzing and understanding results • Conclusion

THE TEACHER COMMUNITY ASSISTANT INITIATIVE

0.083* 0.075+ 0.047 0.036

0

5

10

15

20

25

30

35

40

45

Control EL2 Treat1 Treat2 Treat3 Treat4

Overall Test Scores (SD)

POSITIVE AND LASTING IMPACTS

0.14** 0.13** 0.11** 0.08*

0

10

20

30

40

50

60

Control Treat1 Treat2 Treat3 Treat4

Overall Test Scores - P3 & P4 (Standard Deviations)

Control Remedial Classes During School

Remedial Classes After

School

Extra Assistants

Teacher-led Targeted

Instruction

POSITIVE AND LASTING IMPACTS

0.18** 0.17* 0.06

0.16**

0

5

10

15

20

25

30

Control EL2 Treat1 Treat2 Treat3 Treat4

Local Lang Reading Skills (P3-4) (Standard Deviations)

Control Remedial Classes During School

Remedial Classes

After School

Extra Assistants

Teacher-led Targeted

Instruction

HIGHEST FOR TARGETED SECTIONS

0%

10%

20%

30%

40%

50%

60%

70%

80%

TCA present Teacherpresent

TCA teachingclass

Teacherteaching class

Attendance and Time on Task: TCAs and Teachers

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

ML 1 ML 2 ML 3 ML 4 ML 5 ML 6

Attendance of TCAs over time

During school

After school

DESPITE IMPLEMENTATION ISSUES

Effects on similar grades after similar amount of time in EL1 and EL2

T1 T2 T3 T4

P2 at EL1 (13 weeks) 0.084 0.091 0.016 0.035

P2 at EL2 (38 weeks) 0.041 0.068 0.053 0.038

IMPLEMENTATION OVER TIME

More than 3

times as effective

No effect

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

111113115117122124131133141143145151161163165172182191193201203

TCA Attendance Teacher Attendance

TCA vs. Teacher Attendance by District

LARGE REGIONAL VARIATIONS LINKED TO IMPLEMENTATION VARIATIONS

All English Loc Lang Math

During school 0.30* 0.26* 0.43* 0.34*

After School 0.21* 0.17* 0.26* 0.17*

All English Loc Lang Math

During school 0.12** 0.13** 0.11 0.1+

After School 0.14** 0.16** 0.14+ 0.1*

• Mostly depends on the school environment

- Likelihood of class taking place?

- Multigrade teaching • After school works better

for girls than boys - From 0.05 to 0.13 SD - Significant for girls in P1-2

Schools with multigrade teaching

Schools with no unused classroom at baseline

DURING OF AFTER SCHOOL?

WHAT’ S NEXT?

Nov 2009

• Shared results

• Funding • Planning • MoU

May 2011- Sept 2013

Evaluation

2014

• Analyze • Share • Next

steps

2014-2015

• Follow up research

• Policy outreach

2005-2011

Results from India and Kenya

2014-2015

Lessons integrated into system