AADAPT Workshop Latin AmericaBrasilia, November 16-20, 2009

Non-Experimental Methods

Quasi Experimental Methods I

Florence Kondylis

What we know so far

Aim: We want to isolate the causal effect of our interventions on our outcomes of interest Use rigorous evaluation methods to answer our

operational questions Randomizing the assignment to treatment is

the “gold standard” methodology (simple, precise, cheap)

What if we really, really (really??) cannot use it?!

>> Where it makes sense, resort to non-experimental methods

3

Objective

Find a plausible counterfactual Every non-experimental method is

associated with a set of assumptions The stronger the assumption, the

more doubtful our measure of the causal effect

▪ Question our assumptions▪ Reality check, resort to

common sense!

4

Principal Objective▪ Increase maize production

Intervention▪ Fertilizer Vouchers distribution▪ Non-random assignment

Target group▪ Maize producers, land over 1 Ha &

under 5 Ha Main result indicator

▪ Maize yield

Example: Fertilizer Voucher Program

Before After0

2

4

6

8

10

12

14Control GroupTreatment Group

5

(+) Impact of the program

(+) Impact of external factors

Illustration: Fertilizer Voucher Program (1)

Before After0

2

4

6

8

10

12

14Control GroupTreatment Group

6

(+) BIASED Measure of the program impact

Illustration: Fertilizer Voucher Program (2)

“Before-After” doesn’t deliver results we can believe in!

Before After0

2

4

6

8

10

12

14Control GroupTreatment Group

7

« After » difference btwparticipants andnon-participants

Illustration: Fertilizer Voucher Program (3)

« Before» difference btw participants and nonparticipants

>> What’s the impact of our intervention?

8

Difference-in-Differences Identification Strategy (1)Counterfactual: 2 Options1.Non-participant maize yield after the

intervention, accounting for the “before” difference between participants/nonparticipants (the initial gap between groups)

2.Participant maize yield before the intervention, accounting for the “before/after” difference for nonparticipants (the influence of external factors)

1 and 2 are equivalent

Difference-in-DifferencesIdentification Strategy (2)Underlying assumption:Without the intervention, maize yield for participants and non participants’ would have followed the same trend

>> Graphic intuition coming…

10

Data -- Example 1

Average maize yield

(T / Ha)2007 2008 Difference

(2007-2008)

Participants (P) 1.3 1.9 0.6Non-participants (NP)

0.6 1.4 0.8

Difference (P-NP) 0.7 0.5 -0.2

11

Data -- Example 1

Average maize yield

(T / Ha)2007 2008 Difference

(2007-2008)

Participants (P) 1.3 1.9 0.6Non-participants (NP)

0.6 1.4 0.8

Difference (P-NP) 0.7 0.5 -0.2

12

NP2008-NP2007=0.8

Impact = (P2008-P2007) -(NP2008-NP2007)

= 0.6 – 0.8 = -0.2

2007 200800.20.40.60.8

11.21.41.61.8

2

Participants Non-Participants

P2008-P2007=0.6

13

2007 200800.20.40.60.8

11.21.41.61.8

2

Participants Non-Participants

P-NP2008=0.5

Impact = (P-NP)2008-(P-NP)2007= 0.5 -

0.7 = -0.2

P-NP2007=0.7

Assumption of same trend: Graphic Implication

2007 200800.20.40.60.8

11.21.41.61.8

2

Participants Non-Participants

Impact=-0.2

Summary Negative Impact:

Very counter-intuitive: Increased input use should increase yield once external factors are accounted for!

Assumption of same trend very strong 2 groups were, in 2007, producing at very different

levels➤ Question the underlying assumption of same

trend!➤When possible, test assumption of same trend with

data from previous years

2006 2007 20080

0.5

1

1.5

2

2.5

participantsnon-participants

Questioning the Assumption of same trend: Use pre-pr0gram data

>> Reject counterfactual assumption of same trends !

17

Data – Example 2

Average maize yield

(T / Ha)2007 2008 Difference

(2007-2008)

Participants (P) 1.5 2.1 0.6Non-participants (NP)

0.5 0.7 0.2

Difference (P-NP) 1.0 1.4 0.4

182007 20080

0.5

1

1.5

2

2.5

participantsnon-participants

P08-P07=0.6

NP08-NP07=0.2

Impact = (P2008-P2007) -(NP2008-NP2007)

= 0.6 – 0.2 = + 0.4

Assumption of same trend: Graphic Implication

2007 20080

0.5

1

1.5

2

2.5

participantsnon-participants

Impact = +0.4

Conclusion

Positive Impact: More intuitive

Is the assumption of same trend reasonable?

➤ Still need to question the counterfactual assumption of same trends !➤Use data from previous years

Questioning the Assumption of same trend: Use pre-pr0gram data

>>Seems reasonable to accept counterfactual assumption of same trend ?!

2006 2007 20080

0.5

1

1.5

2

2.5

participantsnon-participants

Caveats (1)

Assuming same trend is often problematic No data to test the assumption Even if trends are similar in the past…

▪ Where they always similar (or are we lucky)?

▪ More importantly, will they always be similar?▪ Example: Other project intervenes in our nonparticipant villages…

Caveats (2) What to do?

>> Be descriptive! Check similarity in observable characteristics

▪ If not similar along observables, chances are trends will differ in unpredictable ways

>> Still, we cannot check what we cannot see… And unobservable characteristics might matter more than observable (ability, motivation, etc)

Matching Method + Difference-in-Differences (1)Match participants with non-participants on the basis of

observable characteristicsCounterfactual: Matched comparison group

Each program participant is paired with one or more similar non-participant(s) based on observable characteristics

>> On average, participants and nonparticipants share the same observable characteristics (by construction)

Estimate the effect of our intervention by using difference-in-differences

24

Matching Method (2)

Underlying counterfactual assumptions

After matching, there are no differences between participants and nonparticipants in terms of unobservable characteristics

AND/OR Unobservable characteristics do not affect

the assignment to the treatment, nor the outcomes of interest

How do we do it?

Design a control group by establishing close matches in terms of observable characteristics Carefully select variables along which to

match participants to their control group So that we only retain

▪ Treatment Group: Participants that could find a match

▪ Control Group: Non-participants similar enough to the participants

>> We trim out a portion of our treatment group!

Implications

In most cases, we cannot match everyone Need to understand who is left out

Example

Score

NonparticipantsParticipants

MatchedIndividuals

Wealth

Portion of treatmentgroup trimmed out

28

Conclusion (1)

Advantage of the matching method Does not require randomization

29

Conclusion (2)

Disadvantages: Underlying counterfactual assumption is

not plausible in all contexts, hard to test▪ Use common sense, be descriptive

Requires very high quality data: ▪ Need to control for all factors that influence

program placement/outcome of choice Requires significantly large sample size to

generate comparison group Cannot always match everyone…

30

Summary

Randomized-Controlled-Trials require minimal assumptions and procure intuitive estimates (sample means!)

Non-experimental methods require assumptions that must be carefully tested

More data-intensiveNot always testable

Get creative: Mix-and-match types of methods!

31

Thank You