1

Different Methods of Impact Evaluation

How to measure impact?

• Assessing causality

Impact of program = outcome 1 – outcome 2In practice, we compare two groups, one of

which benefited from the program, the other one did not 2

Event 1e.g. Education

program/ treatment

Event 2 (Effects)

Outcome 1Causes

Event 2 occurs if and only if Event 1 occurred before

Event 1No Education

Program

Event 2Outcome 2Causes

Constructing the counterfactual

• The counterfactual– what would have happened in the absence of the

program? …– … for the people who benefitted from the program: we

don’t observe it – thus, impact evaluation will have to mimic it

• Counterfactual is constructed by selecting a group not affected by the program – this is the main challenge of impact evaluation

• Methods differ by the way the counterfactual is constructed

3

Non-Experimental Methods

4

Non-Experimental Methods

1. Simple Difference2. Multivariate (Multiple) Regression3. “Difference in difference” (Multiple

Regression with Panel Data)4. Matching5. Randomization/RCT’s (advanced topic

covered separately)

5

Simple difference

• Simple difference is a first measure – why is it not sufficient?

• There may be differences between the two groups (age, location, gender, initial endowment, bargaining power)

• So we may want to control for these differences

6

Multi-variate regression

• Suppose we can observe these differences.– Age composition of the group, initial infrastructure

in the school, level of education, …

• We can include all these variables in a regression: Y = a.T + b.Age + c.Infr + d.Edu +…– A regression provides the linear combination of

observable variables that best “mimics” the outcome

– Each coefficient represents the effect of each variable

– Will give us the effect of the treatment everything else being equal, or more exactly every other observable characteristics being equal

7

Multi-variate regression

• Problems of the regression– You may want to include many many variables,

to control for as many characteristics as possible– Problem of sample size (degrees of freedom)

• More important: do you have measures of everything?– Bargaining power, Pro-activeness, Intrinsic

motivation, hopelessness

• There are unobservables8

Panel data

• Simple difference: before/after– Counterfactual = same group before the program– Can we trust this? Assumption = would have

remained the same– Ex: Police project

• Double difference– Control for the situation before the program– Ex: Group 1 = Treatment Group; Group 2 = Control

Group– 2006: Group 1 : 30 Group 2 : 60

2009: Group 1 : 50 (+66%) Group 2 : 90 (+50%)Effect = +16%

- Assumption: they would have grown at the same pace- Not sure…

9

Matching

• We compare pairs of 2 individuals for which the values taken by ALL variables are the same

10

Matching

• Variation: Propensity Score Matching: all the variables do not need to be exactly the same, but you look for individuals which have the same “profile”

• Problems– This matching method requires a big dataset:

find pairs on a sufficient number of variables– What about unobservables?

11

An example

• Case study: US Congress elections, 2002– 60 000 phone calls to potential voters to

encourage them to vote; 25 000 reached– Outcome: did they actually go vote?– 1st method: compare the 25 000 (reached) vs. the

35 000 (not reached)– 2nd method: introduce co-variates in a regression– 3rd method: introduce baseline data (vote in 1998)– 4th method: do a matching

12

1st comparison: we suspect a selection bias

Reached Not Reached

Difference

Female 56.2% 53.8% 2.4 pp*Newly Regist. 7.3% 9.6% -2.3 pp*From Iowa 54.7% 46.7% 8.0 pp*

Voted in 2000

71.7% 63.3% 8.3 pp*

Voted in 1998

46.6% 37.6% 9.0 pp*

13

Selection Bias: differences in observable / unobservable characteristics → differences in

outcome not due to the treatment

Non-Experimental Methods

14

Method Estimated Impact

1 – Simple Difference 10.8 pp *

2 – Multiple regression 6.1 pp *

3 – Multiple regression with panel data (diff-in-diff)

4.5 pp *

4 – Matching 2.8 pp *

5 – Randomized Experiment 0.4 pp

• For more details, please read Case Study: “Get out the vote? Do phone calls encourage voting” under References

15