randomized evaluations - open computing facilitygarret/decal/random1.pdf ·...

Correlations Econometrics Case Study - Progresa

Randomized EvaluationsIntroduction, Methodology, & Basic Econometrics using Mexico’s

Progresa program as a case study

(with thanks to Clair Null, author of 2008 Notes)

Sept. 15, 2009

Not All Correlations are Causal Relationships

Far from it!There are several reasons why X & Y might be correlated:

causality X → Y

reverse causality Y → X

simultaneity X → Y and Y → X

omitted variables / confounding Z → X and Z → Y

spurious correlation

causality X → Y

Practice Correlations

Think about the following correlations and be skeptical of causality.Which of the other 4 types of correlations would you want to rule outbefore you would be confident that the relationships were causal? In thecase of omitted variables, what are some “Z” factors that you’re worriedabout?

1 reverse causality Y → X2 simultaneity X → Y and Y → X3 omitted variables / confounding Z → X and Z → Y4 spurious correlation

Job applicants with names that are common among AfricanAmericans are less likely to get an interview.In a country with few women leaders, voters have low opinions of awoman’s ability to lead.As the planet heats up, there are fewer and fewer pirates.The chronically ill are usually poor.

Job applicants with names that are common among AfricanAmericans are less likely to get an interview.

For causality, would like to know that if the name was the onlything that changed, they’d still be less likely to get aninterview.Is it possible to design an experiment that would let us observethat?

Actually, yes, and it’s remarkably easy. Two economists(Bertrand & Mullainathan) did it by sending out fake resumes.The results are disturbing - those with white-sounding nameswere 50% more likely to be called for an interview.Even worse, while the likelihood of getting an interview isincreasing in a “white” applicant’s credentials, experience &honors mattered much less for “black” applicants. This sort ofdiscrimination could turn into a vicious cycle.

In a country with few women leaders, voters have lowopinions of a woman’s ability to lead.

For causality, would like to know if an exogenous change in thenumber of women leaders would lead to changes in opinions ofa woman’s ability to lead.Is it possible to design an experiment that would let us observethat?

India’s policy of requiring that some leadership positions be filled bywomen gives us a “natural” experiment (Beaman et al.).The essential aspect of this policy (in terms of determining a causalrelationship between exposure to female politicians and opinions ofthem), is that the local governments didn’t get to choose whetheror not they wanted their position to be reserved for a woman. Thepositions to be reserved were randomly assigned.

Perceptions of women leaders’ effectiveness did improve afterrandomly-assigned exposure.Almost twice as many women won unreserved positions in placeswhere the position had been reserved for a woman in the prior twoelections relative to places where the position had been reservedonly once or never at all.

India’s policy of requiring that some leadership positions be filled bywomen gives us a “natural” experiment (Beaman et al.).The essential aspect of this policy (in terms of determining a causalrelationship between exposure to female politicians and opinions ofthem), is that the local governments didn’t get to choose whetheror not they wanted their position to be reserved for a woman. Thepositions to be reserved were randomly assigned.

Perceptions of women leaders’ effectiveness did improve afterrandomly-assigned exposure.Almost twice as many women won unreserved positions in placeswhere the position had been reserved for a woman in the prior twoelections relative to places where the position had been reservedonly once or never at all.

As the planet heats up, there are fewer and fewer pirates.

The point here is that theory helps. If there’s not somereasonable explanation, it’s not very likely to be causal. (Bewary of unreasonable explanations.)

As the planet heats up, there are fewer and fewer pirates.

The point here is that theory helps. If there’s not somereasonable explanation, it’s not very likely to be causal. (Bewary of unreasonable explanations.)

The chronically ill are usually poor.

Again, to explore causality in either direction, would want tosee how health responds to an exogenous change in wealth (orvice versa).Probably going to need an experiment for this one.

need to make some people rich and see if they get healthierthan otherwise identical peopleor make some chronically ill people healthier and see if theyget richer than otherwise identical chronically ill people

In principle, a natural experiment could work (e.g. lotterywinners? people who live near new medical facilities?), but afield experiment is a sure bet.

The chronically ill are usually poor.

Again, to explore causality in either direction, would want tosee how health responds to an exogenous change in wealth (orvice versa).Probably going to need an experiment for this one.

need to make some people rich and see if they get healthierthan otherwise identical peopleor make some chronically ill people healthier and see if theyget richer than otherwise identical chronically ill people

In principle, a natural experiment could work (e.g. lotterywinners? people who live near new medical facilities?), but afield experiment is a sure bet.

The Real Issue: Finding the Right Counterfactual

To determine the causal effect of X on Y we want to observewhat happens to Y when we change X without changinganything else (i.e. Z).In the resume experiment, this was easy to do, because thepeople didn’t really exist.But what if we need to change X in real people’s lives? (callthese people the treatment group)To make the comparison we need someone else whose X didn’tchange but who was otherwise identical to the treatmentgroup. (call these people the control group)We can’t observe the same person both with and without thechange in X (the treatment)–the counterfactual isunobservable.

Randomization to the Rescue!

With a large enough sample, by randomly assigning people totreatment and control groups we can make the two more orless identical (e.g. their X’s & Z’s should be the same onaverage).IMPORTANT: If people get to choose whether or not theywant to be in the treatment group, then there’s no way tomake sure that the people in the treatment group are identicalto the people in the control group.

Even if they look the same in every other characteristic, thefact that the treatment people chose to be treatment peopleand the control people chose to be control people means thatsomething about them was different (there’s some piece of Zthat we’re not able to measure).

In Math...

8 of Duflo et al, using our beat-to-death textbookthing as the backstory...We want to know Y T

i − Y Ci , this is impossible to observe.

We can however, estimate E [Y Ti − Y C

The simple easy-to-get, not-the-same-thing (why?) number is:D = E [Y T

i |treated − Y Ci |control ]

Y Ti was a school that was observed with textbooks, so

having textbooks and being treated are not identical.Add and subtract E [Y C

i |T ]

E [Y Ti |T ]− E [Y C

i |T ]− E [Y Ci |C ] + E [Y C

i |T ] =E [(Y T

i − Y Ci )|T + E [Y C

i |T ]− E [Y Ci |C ]

In Math...

i |T ]

i |T ]− E [Y Ci |C ] + E [Y C

i |T ] =E [(Y T

i − Y Ci )|T + E [Y C

i |T ]− E [Y Ci |C ]

In Math...

i |T ]

i |T ]− E [Y Ci |C ] + E [Y C

i |T ] =E [(Y T

i − Y Ci )|T + E [Y C

i |T ]− E [Y Ci |C ]

More Math

i |T ]− E [Y Ci |C ] + E [Y C

i |T ] =E [(Y T

i − Y Ci )|T + E [Y C

i |T ]− E [Y Ci |C ]

First term is the treatment effect (i.e., what we want.)Second and third terms are the selection bias. “It captures thedifference in potential untreated outcomes between thetreatment and the comparison schools; treatment schools mayhave had different test scores on average even if they had notbeen treated.”Briefly, randomization solves this.“Since the treatment has been randomly assigned, individualsassigned to the treatment and control groups differ inexpectation only through their exposure to the treatment. Hadneither received the treatment, their outcomes would havebeen in expectation the same. This implies that the selectionbias,E [Y C

i |T ]− E [Y Ci |C ], is equal to zero.”

Add Z of Mystery

For simplicity, suppose we’re interested in the effect of X = 1relative to X = 0The outcome Y is a function of the treatment (X ) and someother characteristic (for simplicity let Z = 1 or 0)We write this relationship in mathematical notation as

Yi = a + bXi + cZi + εi

where the i subscript refers to a specific person and the ε is awhite noise “error / disturbance” term that averages out acrossthe populationWhat that says in words for the Progresa case study is:

school attendance (Y ) depends on whether or not the child gets ascholarship (X ), the child’s ability (Z ), and whether or not the

child woke up on the right side of the bed (ε)

Z of Mystery

To measure the effect of X = 1 relative to X = 0, taking intoaccount Z , we compare the expected values (averages) of Yconditional on the levels of X and Z

E[Y |X , Z ] = E[a + bX + cZ + ε|X , Z ]

= E[a|X , Z ] + E[bX |X , Z ] + E[cZ |X , Z ] + E[ε|X , Z ]

= a + bE[X |X , Z ] + cE[Z |X , Z ] + 0

So, for our 4 possible combinations of X and Z , we have

E[Y |X = 1, Z = 1] = a + b + cE[Y |X = 1, Z = 0] = a + bE[Y |X = 0, Z = 1] = a + cE[Y |X = 0, Z = 0] = a

and as expected, the effect of X holding Z constant is b (theeffect of Z holding X constant is c)

Z of Mystery

To measure the effect of X = 1 relative to X = 0, taking intoaccount Z , we compare the expected values (averages) of Yconditional on the levels of X and Z

E[Y |X , Z ] = E[a + bX + cZ + ε|X , Z ]

= E[a|X , Z ] + E[bX |X , Z ] + E[cZ |X , Z ] + E[ε|X , Z ]

= a + bE[X |X , Z ] + cE[Z |X , Z ] + 0

So, for our 4 possible combinations of X and Z , we have

E[Y |X = 1, Z = 1] = a + b + cE[Y |X = 1, Z = 0] = a + bE[Y |X = 0, Z = 1] = a + cE[Y |X = 0, Z = 0] = a

and as expected, the effect of X holding Z constant is b (theeffect of Z holding X constant is c)

Omitted Variable Bias

What if we hadn’t been able to control for ability in therelationship we were just discussing?In that case, we wouldn’t be able to calculateE[Y |X = 1, Z ]− E[Y |X = 0, Z ]

Instead, all we’d be able to calculate isE[Y |X = 1]− E[Y |X = 0]Writing out the gory details, we’d have

E[Y |X = 1]− =(a + b + cE[Z |X = 1]

E[Y |X = 0](a + 0 + cE[Z |X = 0]

)= b︸︷︷︸

true effect

+ c(E[Z |X = 1]− E[Z |X = 0]

)︸︷︷︸OVB

Omitted Variable Bias

E[Y |X = 1]− = b︸︷︷︸true effect

+ c(E[Z |X = 1]− E[Z |X = 0]

)︸︷︷︸OVB

E[Y |X = 0]

In words, if average ability is different for the kids in thetreatment & control groups, then we won’t be able to separatethe effect of the treatment from the effect that their differentability levels are having on their attendance ratesRandomization assures us that E[Z |X = 1] = E[Z |X = 0] sothat there is no omitted variable bias

OVB is What Ails Us

OVB is the real problem. You can sometimes get cluesabout the sign/magnitude.Given a true model Yi = a + bXi + cZi + εi , if you leave outZ, your estimate of b come out as b + c ∗ cov(X ,Z)

var(X )

In a word, if there exists any variable Z that’s correlated withyour variable of interest X and your outcome variable Y, you’rescrewed.Some think that means the solution to OmittedVariable bias isto not omit anything. While that may help a little, there arealways things to omit, so it’s better if you can find an X that’suncorrelated (ie, random).

Proof of Randomization

Can you prove that you randomized? (Can you prove thatthere exists no Z that’s correlated with Y and your X?)

That’s a universal negative, so no, you can’t prove it.But you can gather a whole bunch of Z’s before the programand show they have the same average across treatment andcontrol groups, and that might assuage some fear.

Proof of Randomization

Can you prove that you randomized? (Can you prove thatthere exists no Z that’s correlated with Y and your X?)

That’s a universal negative, so no, you can’t prove it.But you can gather a whole bunch of Z’s before the programand show they have the same average across treatment andcontrol groups, and that might assuage some fear.

Program Design

The goal is to keep kids in school. How can the governmentdo it?

Economics is all about incentives and constraints...

What sort of incentives could the government provide tohouseholds?What sort of constraints could the government relax forhouseholds?

Program Design

The goal is to keep kids in school. How can the governmentdo it?

Economics is all about incentives and constraints...

What sort of incentives could the government provide tohouseholds?What sort of constraints could the government relax forhouseholds?

Alternatives

Supply approaches

Build schools near where kids live (reduce cost of getting toschool)Increase school resources (teacher salaries, meals, etc.)Might not affect enrollment gap between poor & non-poorstudents

Demand approach: conditional cash transfers (CCT)

Subsidies targeted to poor (“means tested”)Compensate families for opportunity cost of child’s labor

school itself is free

Minimize disincentive to work (conditioned only onpre-program income)

Alternatives

Supply approaches

Build schools near where kids live (reduce cost of getting toschool)Increase school resources (teacher salaries, meals, etc.)Might not affect enrollment gap between poor & non-poorstudents

Demand approach: conditional cash transfers (CCT)

Subsidies targeted to poor (“means tested”)Compensate families for opportunity cost of child’s labor

school itself is free

Minimize disincentive to work (conditioned only onpre-program income)

Program Implementation

Initial census to determine eligibility status

about 2/3 of households qualifiedDo we care at all about the non-eligible households?

Monthly educational grants

children enrolled in grades 3-9conditional on 85% attendance rates (confirmed by teacher)increasing in grades and extra bonus for girls in secondaryschool

70-255 pesosgirl in 9th grade ∼ 44% of day-laborer’s monthly earnings

What is Y ?

School enrollment

inequality

achievement (not mentioned in this paper)

Child labor

Fertility

program initially for 3 years only, but if viewed as anentitlement, bigger effect among eligible mothersfor teenage girls, staying in school could reduce fertility

What is Y ?

School enrollment

inequality

Child labor

Fertility

What is Y ?

School enrollment

inequality

Child labor

Fertility

What is Y ?

School enrollment

inequality

Child labor

Fertility

What is Y ?

School enrollment

inequality

Child labor

Fertility

What is Y ?

School enrollment

inequality

Child labor

Fertility

What Z ’s are observable?

Community level

school quality (proxied with # kids/teacher)distance to secondary schooldistance to urban labor market

farther away → lower opportunity cost of timefarther away → less info about returns to schooling

Household level

parent’s educationPRE-program poverty index (what problem with concurrentincome level?)

Child level

agesexgrades completed

What Z ’s are observable?

Community level

school quality (proxied with # kids/teacher)distance to secondary schooldistance to urban labor market

farther away → lower opportunity cost of timefarther away → less info about returns to schooling

Household level

parent’s educationPRE-program poverty index (what problem with concurrentincome level?)

Child level

agesexgrades completed

What Z ’s are unobservable?

Community level

“cohesiveness”local returns to schooling

Household level

parents’ preferences for schooling

Child level

child’s preferences for schoolingability

What Z ’s are unobservable?

Community level

“cohesiveness”local returns to schooling

Household level

parents’ preferences for schooling

Child level

child’s preferences for schoolingability

Deep Thoughts on Randomization

In general, the finer the level at which you randomize, thebetter (more observations to compare to one another)

Is randomizing at the child level a good idea?

You can’t ignore “political” consequences.The idea is that the researchers want to compare participants -but participants are also likely to compare themselves to oneanother in cases where the treatment is observable.

Randomizing at the household level might solve some of theseproblems associated with child-level randomization, but is itthe best solution?

Ultimately, Progresa randomization was at the village level.Does this solve all the problems we might worry about?

Localities phased-in: treatment got it starting in 1998,controls got it starting in 2000

Alternative Methods of Randomizing

Randomized order of phase-intiming is essential (are you trying to measure long-termeffects?)caution: does control group change behavior in expectation ofeventual treatment?

Lotteries for oversubscribed treatmentscaution: only identifies effect of treatment among thoseeligible for randomization

“Within-group”treat subgroupscaution: is control group contaminated?

Encouragement designtreatment available to everyone, but take-up rate variesrandomly assign some people encouragement to receivetreatmentanalytically difficult (only changes probability of treatment)

Results - Enrollment

Program increased schooling by about 2/3 of a yearIs this a big effect?

Depends on the base that this increase builds upon

compared to average of 6.8, pretty big, actually

Many kids would have gone to school anyway

for those who changed enrollment status in response toprogram, effect is even bigger

Reduced inequality between poor & non-poor kids

So, should we expand Progresa everywhere?

Behrman & Todd estimate that increases in earning powerfrom more education exceed costs of program by 40-110%!More cost effective than building schools, at least in ruralMexico (Parker & Coady)Intergenerational effect on educational attainment alsoimportant

But what about external validity?

So, should we expand Progresa everywhere?

Behrman & Todd estimate that increases in earning powerfrom more education exceed costs of program by 40-110%!More cost effective than building schools, at least in ruralMexico (Parker & Coady)Intergenerational effect on educational attainment alsoimportant

But what about external validity?

Re-cap

Correlation is not causationGoal of impact evaluation is to identify causal effectsof X on YOmitted variable bias is a big problemRandomization is the safest way to avoid OVBProgresa case study

randomized at locality level (phased-in pilot)conditional cash transfers increased school attendanceconcern w/ any randomized evaluation is external validity

Return

randomized evaluations - open computing facilitygarret/decal/random1.pdf ·...

Documents