Econometric Evaluation of Social Programs
Part I: Definitions of Treatment Effects
James J. Heckman and Edward J. Vytlacil
Econ 312, Spring 2019
Heckman and Vytlacil Definitions of Treatment Effects
: More General Criteria
• One might compare outcomes in different sets that are ordered.
• Thus if Y (s, ω) is scalar income and we compare outcomes fors ∈ SA with outcomes for s ′ ∈ SB , where SA ∩ SB = ∅, thenone might compare YsA to YsB , where
sA = argmaxs∈SA(Y (s, ω)) ,
sB = argmaxs∈SB(Y (s, ω)) ,
and we suppress the dependence of sA and sB on ω.
Heckman and Vytlacil Definitions of Treatment Effects
• This compares the best in one choice set with the best in theother.
• Another contrast compares the best choice with the next bestchoice.
• To do so, define s ′ = argmaxs∈S (Y (s, ω)) and SB = S r s ′,and define the treatment effect as Ys′ − YsB .
Heckman and Vytlacil Definitions of Treatment Effects
• Social welfare theory constructs aggregates over Ω ornonempty, nonsingleton subsets of Ω (see Sen, 1999).
• Let sp (ω) denote the s ∈ Sp that ω receives under policy p.
• This is a shorthand notation for the element in Sτ determinedby the map p = (a, τ) assigned to agent ω under policy p.
Heckman and Vytlacil Definitions of Treatment Effects
• A comparison of two policy outcomes sp(ω)ω∈Ω andsp′(ω)ω∈Ω, where p 6= p′ for some ω ∈ Ω, can be
expressed as
RG
(Y (sp (ω) , ω)ω∈Ω
)− RG
(Y (sp′(ω), ω)ω∈Ω
)using the social welfare function defined over outcomesRG
(Y (s, ω) , ωω∈Ω
).
Heckman and Vytlacil Definitions of Treatment Effects
• The cost-benefit comparison of two policies p and p′ is
CBp,p′ =
∫ΩW (Y (sp(ω), ω)) dµ(ω)−
∫ΩW(Y(sp′(ω), ω
))dµ(ω),
where p, p′ are two different policies and p′ may correspond toa benchmark of no policy and µ(ω) is the distribution of ω.
• The Benthamite criterion replaces W (Y (s (ω) , ω)) withR(Y (s(ω), ω)) in the preceding expressions and integratesutilities across agents:
Bp,p′ =
∫ΩR(Y (sp(ω), ω))dµ(ω)−
∫ΩR(Y (sp′(ω), ω))dµ(ω).
Heckman and Vytlacil Definitions of Treatment Effects
: The Evaluation Problem
• Assume a well-defined set of individuals ω ∈ Ω and a universe ofcounterfactuals or hypotheticals for each agent Y (s, ω), s ∈ S.
• Different policies p ∈ P give different incentives by assignmentmechanism a to agents who are allocated to treatment by arule τ ∈ T .
Heckman and Vytlacil Definitions of Treatment Effects
Begin material from previous slide presentations(2).
Heckman and Vytlacil Definitions of Treatment Effects
How To Construct Counterfactuals?
• Central problem in the evaluation literature is the absence ofinformation on outcomes for person ω other than the outcomethat is observed.
• Even a perfectly implemented social experiment does not solvethis problem.
• Randomization with full compliance identifies only onecomponent of Y (s, ω)s∈S for any person.
• In addition, some of the s ∈ S may never be observed.
Heckman and Vytlacil Definitions of Treatment Effects
• For each policy regime, at any point in time we observe personω in some state but not in any of the other states.
• Do not observe Y (s ′, ω) for person ω if we observe Y (s, ω),s 6= s ′.
• Let D (s, ω) = 1 if we observe person ω in state s under policyregime p.
• Observed objective outcome
Y (ω) =∑s∈S
D (s, ω)Y (s, ω) . (1)
Heckman and Vytlacil Definitions of Treatment Effects
• The evaluation problem in this model is that we only observeeach individual in one of S possible states.
• We do not know the outcome of the individual in other statesand hence cannot directly form individual level treatmenteffects.
• The selection problem arises because we only observe certainpersons in any state.
• We observe Y (s, ω) only for persons for whom D (s, ω) = 1.
• In general, the outcomes of persons found in S = s are notrepresentative of what the outcomes of people would be if theywere randomly assigned to s.
Heckman and Vytlacil Definitions of Treatment Effects
• The Roy model (1951): Two possible treatment outcomes(S = 0, 1) and a scalar outcome measure and a particularassignment mechanism D (1, ω) = 1 [Y (1, ω) > Y (0, ω)](reveals R(1, ω)− R(0, ω) ≥ 0).
• The economist’s use of choice data distinguishes theeconometric approach from the statistical approach.
Heckman and Vytlacil Definitions of Treatment Effects
How To Construct Counterfactuals?
• Two main avenues of escape from this problem.
• The first avenue, featured in explicitly formulated econometricmodels and often called “structural econometric analysis ”,derives from the Cowles tradition.
• Models Y (s, ω) explicitly in terms of its determinants asspecified by theory.
• This entails describing the random variables characterizing ωand carefully distinguishing what agents know and what theanalyst knows.
Heckman and Vytlacil Definitions of Treatment Effects
How To Construct Counterfactuals?
• This approach also models D(s, ω) and the dependencebetween Y (s, ω) and D(s, ω) produced from variables commonto Y (s, ω) and D (s, ω).
• Specifies a full model and attempts to addressproblems (P1)–(P3).
Heckman and Vytlacil Definitions of Treatment Effects
How To Construct Counterfactuals?
• A second avenue, pursued in the recent treatment effectliterature, redirects attention away from estimating thedeterminants of Y (s, ω) toward estimating some populationversion of individual “causal effects,” without modeling whatfactors give rise to the outcome or the relationship between theoutcomes and the mechanism selecting outcomes.
• Agent valuations of outcomes are typically ignored.
• The treatment effect literature focuses largely on policyproblem (P-1) for the subset of outcomes that is observed.
• Seeks to answer a narrower problem.
Heckman and Vytlacil Definitions of Treatment Effects
• For program (state, treatment) j compared to program (state,treatment) k ,
ATE(j , k) = E (Y (j , ω)− Y (k , ω)) .
TT(j , k) =E (Y (j , ω)− Y (k , ω) | D(j , ω) = 1) . (2)
• These are the traditional parameters for average returns.
• But for economic analysis, marginal returns are more important.
Heckman and Vytlacil Definitions of Treatment Effects
• The distinction between the marginal and average return is acentral concept in economics.
• The Effect Of Treatment for People at the Margin ofIndifference (EOTM) between j and k , given that these arethe best two choices available is, with respect to personalpreferences, and with respect to choice-specific costs C (j , ω).
Heckman and Vytlacil Definitions of Treatment Effects
EOTMR(j , k) = (3)
E
Y (j , ω)−Y (k, ω)
∣∣∣∣∣∣R (Y (j , ω) ,C (j , ω) , ω) = R (Y (k , ω) ,C (k, ω) , ω) ;R (Y (j , ω) ,C (j , ω) , ω)R (Y (k , ω) ,C (k , ω) , ω)
≥ R (Y (`, ω) ,C (`, ω) , ω)
,
` 6= j , k.
Heckman and Vytlacil Definitions of Treatment Effects
• A generalization of this parameter called the MarginalTreatment Effect, introduced into the evaluation literature byBjorklund and Moffitt (1987).
• Return to people at the margin of choice.
• Will discuss methods for identifying this return.
Heckman and Vytlacil Definitions of Treatment Effects
Policy relevant treatment effect
• Effect on aggregate outcomes of one policy regime p ∈ Pcompared to the effect of another policy regime p′ ∈ P :
PRTE: E (Y (sp(ω), ω)− Y (sp′(ω), ω)),where p, p′ ∈ P .
sp(ω) is treatment allocated under policy p.
• Corresponding to this objective outcome is the subjectivecounterpart:
Subjective PRTE: E (R(sp(ω), ω))− E (R(sp′(ω), ω)),where p, p′ ∈ P .
Heckman and Vytlacil Definitions of Treatment Effects
• Modern political economy seeks to know the proportion ofpeople who benefit from policy regime p compared with p′.Voting Criterion:
Pr (Y (sp(ω), ω) > Y (sp′(ω), ω)) .
• For particular treatments within a policy regime p, it is also ofinterest to determine the proportion who benefit from jcompared to k as
Pr (Y (j , ω) > Y (k , ω)) .
• Option values also interesting: option of having access to aprogram.
• Uncertainty and regret.
Heckman and Vytlacil Definitions of Treatment Effects