how an impact evaluation will look like - world...

How an impact evaluation will

look like

Impact Evaluation Workshop

Belgrade, Serbia

Robin Audy

June 4, 2015

Key Messages

• Good Impact evaluations require careful planning from the start of a project

The earlier the better

• Collecting data electronically is now accessible to all

It allows new types of indicators

1. The steps of an impact evaluation

1. Identify the problem

3. Design

an

evaluation

4. Do Initial

Survey

5. Randomize

the program

6. Conduct

follow-up

survey

7. Crunch the

numbers

8.

Announce

the result

2. Create a new or

target an ongoing

program

1. The steps of a randomized impact evaluation

4. Do Initial

Survey

5.

Randomize

the program

6. Conduct

follow-up

survey

2. Who do we interview?

Who are the final beneficiaries

Who should be interviewed, and when

How to identify them?

2. Who do we interview?

2. Who do we interview, and when?Imagine a simple conditional cash transfer for the poor

2. Who do we interview, and when?You will target among your population. A census can help determine who is eligible.

2. Who do we interview, and when?From the list of targeted eligible individuals, do your baseline survey

Do the baseline survey before assigning treatment status!

2. Who do we interview, and when?Only then should you assign treatment and control statuses

2. Who do we interview, and when?Perhaps take-up will not be complete among all those in the treatment group.


You could also have individuals from the control who manage to get treatment.


You could also have individuals from the control who manage to get treatment.

At endline, follow all the individuals from the baseline

2. Note about sample representativenessInternal validity vs External validity

2. Baseline surveyAnother type: clustered randomization by village

2. Baseline survey

3. What does a baseline interview look like?

3. What about electronic data collection?Preparation is key!

• Design questions and prepare deployment

• Test, pilot and refine design

• Train and run interviews

4. Resources for electronic data collection

• SurveyCTO www.surveycto.com

• Kobo Collect (www.kobotoolbox.org)

5. Resources for electronic data collection

6. Types of indicators to consider

• Contextual setting variables

• Age, education, poverty, community characteristics

• Intermediate outcomes

• Enrolment in school, frequency of health care visits, sanitation behavior

• Final outcomes

• Measures of learning, health outcomes, measures of deforestation

• Contact information for follow-up

• Map, name, phone number, etc…

6. Types of indicators to consider

24

7. Technology allows for new types of indicators

25


26


8. Randomly assign treatment and comparison


• The lottery assigns who gets what

• Perception of fairness

• Transparency:

-> Big public event with dignitaries and all stakeholders

• Communications plan is important

• Explain equal chances to all


• Imagine we put in place right now a program that tests the attention level of

workshop attendees here today. We administer a test about the knowledge

retained from the presentation.

• For this, we divide all attendees into 3 groups:

• Stretch

• Caffeine

• Control group

• We have ~18 attendees, we want to divide them into 3 equal groups:

Example:

Stretch Caffeine Control Total

6 6 6 18


All 18 participants are assigned to a ticket

A, B or C

It is possible to stratify. For example,

from each row

These tickets can be shown to

stakeholders in advance of

randomization

During the lottery, we randomly assign a

treatment to each letter…

Example:

Person Ticket

12 A

8 A

6 A

1 A

17 A

11 A

13 B

15 B

9 B

3 B

18 B

4 B

7 C

2 C

5 C

16 C

10 C

14 C

8. Randomly assign treatment and comparisonExample:

Treatment Ticket

Stretch ?

Caffeine ?

Control group ?

Person Ticket

12 A

8 A

6 A

1 A

17 A

11 A

13 B

15 B

9 B

3 B

18 B

4 B

7 C

2 C

5 C

16 C

10 C

14 C


• An early childhood development program has 3 components:

• Free access

• Free access + small incentive

• Free access + large incentive

• Control group

• Also, orthogonally Info and No info

• We have 240 participating villages

• We want to divide the villages into 3 equal groups:

Example of a larger program:

Free

access

Free

access +

small

incentive

Free

access +

large

incentive

Control Total

villages

No Info 30 30 30 30 120

Info 30 30 30 30 120


All 240 villages are assigned to a ticket

A, B or C and X or Y

The tickets can be stratified, i.e. same

number of letters by region

These tickets can be shown to

stakeholders in advance of

randomization

In the lottery, we randomly assign a

treatment to each letter

Example:Village Ticket

Bansko AX

Belitza BX

Blagoevgrad CY

Gotse Delchev AX

Garmen CX

Krupnik BY

Mikrevo AY

Petrich BX

Sandanski CY

Satovcha BY

Simitly AY

Yakorouda CX

Ajtos AY

Burgas BY

Karnobat BX

Pomorie AX

Vresovo BY

Troianovo CY

Ahtopol AX

...

8. Randomly assign treatment and comparisonLottery results

8. Verify Baseline balance

Variable NControl

No info

Control

Info

Free

No info

Free

Info

Twenty lev

No info

Twenty lev

Info

Seven lev

No info

Seven lev

Info

Age 57723.90

{0.043}

3.85

[0.557]

3.93

[0.675]

3.80

[0.172]

3.86

[0.607]

3.92

[0.786]

3.89

[0.967]

3.91

[0.860]

Female 57720.460

{0.020}

0.505

[0.131]

0.491

[0.276]

0.479

[0.520]

0.502

[0.184]

0.502

[0.181]

0.454

[0.828]

0.477

[0.535]

Ethnicity: Bulgarian 57720.241

{0.040}

0.220

[0.707]

0.219

[0.709]

0.193

[0.424]

0.154

[0.126]

0.165

[0.181]

0.182

[0.279]

0.202

[0.501]

Ethnicity: Turkish 57720.150

{0.048}

0.131

[0.754]

0.143

[0.908]

0.139

[0.858]

0.227

[0.331]

0.212

[0.408]

0.331**

[0.021]

0.147

[0.959]

Ethnicity: Roma 57720.602

{0.059}

0.629

[0.741]

0.614

[0.882]

0.631

[0.724]

0.614

[0.895]

0.614

[0.881]

0.475

[0.151]

0.611

[0.916]

Ethnicity: Other 57720.001

{0.001}

0.016

[0.206]

0.021

[0.158]

0.015

[0.131]

0.001

[0.979]

0.004

[0.399]

0.008

[0.108]

0.031

[0.206]

Attended KG before 57720.711

{0.043}

0.669

[0.506]

0.666

[0.528]

0.661

[0.439]

0.725

[0.809]

0.706

[0.941]

0.638

[0.256]

0.695

[0.808]

Past 3 days child got read

books5683

0.530

{0.043}

0.439

[0.169]

0.499

[0.653]

0.441

[0.191]

0.451

[0.237]

0.499

[0.631]

0.552

[0.734]

0.482

[0.455]

Past 3 days child got told

story5687

0.723

{0.048}

0.605

[0.104]

0.635

[0.224]

0.685

[0.570]

0.622

[0.120]

0.635

[0.197]

0.738

[0.813]

0.661

[0.347]

Past 3 days child sang 56870.760

{0.049}

0.690

[0.314]

0.710

[0.471]

0.740

[0.753]

0.732

[0.664]

0.743

[0.783]

0.776

[0.807]

0.749

[0.855]

8. Verify Baseline balance

Variable NControl

No info

Control

Info

Free

No info

Age 57723.90

{0.043}

3.85[0.557]

3.93[0.675]

Female 57720.460{0.020}

0.505[0.131]

0.491[0.276]

Ethnicity: Bulgarian 57720.241{0.040}

0.220[0.707]

0.219[0.709]

Ethnicity: Turkish 57720.150{0.048}

0.131[0.754]

0.143[0.908]

9. Follow-up surveys

• Initiate intervention

• Option: collect additional intermediate outcomes

• Option: collect program implementation information

• Option: collect costing information for CBA / CEA analysis (opportunity costs

need to be collected

• Follow up beneficiary survey: back in the fields!

Key Messages

• Good Impact evaluations require careful planning from the start of a project

The earlier the better

Know who, when and how

Plan a transparent, public lottery (especially for large programs)

• Collecting data electronically is now accessible to all

It allows new types of indicators

Questions?

how an impact evaluation will look like - world...

Documents