studying variation in the effect of program participation · 3. instrumental variables in...
TRANSCRIPT
Studying Variation in the Effect of Program Participation
Stephen W. Raudenbush
Presentation at the
“Workshop on Learning from Variation in Program Effects”
Palo Alto, July 19, 2016
The research reported here was supported by a grant from the W.T. Grant Foundation to the University of Chicago entitled “Building Capacity for Evaluating Group-Level Interventions.” Thanks to Sean Reardon and Takako Nomi for their collaboration on these ideas.
Outline1. Pervasiveness of
• Multi-site trials• Non-compliance
2. Instrumental variables in a single-site study• Under homogeneity of impact• Under heterogeneity of impact• Examples
3. Instrumental variables in multi-site studies: • Method 1 Combine 2 ITT Analyses• Method 2: Two-stage generalized least squares• Method 2= “Between-Site Regression!”
4. Design Considerations
5. Modeling Program Participation and Program Impact on Participants
1. Pervasiveness of Multi-Site Trials
Since 2002, IES has funded 175 group-randomized trials
Vast majority are multi-site trials (Spybrook, 2013)
Other recent examples
* National Head Start Evaluation (US Dept of HHS, 2010)
* Moving to Opportunity (Sonbanmatsu, Kling, Duncan, Brooks Gunn, 2006)
* School-based lottery studies (Abdulkadiroglu, Angrist, Dynarski, Kane, and Pathak, 2009).
* Tennessee STAR (Finn and Achilles, 1990)
* Ending Social Promotion (Jacob and Lefgren, 2009)
* Double-Dose Algebra (Nomi and Allensworth, 2009)
* Welfare to Work (Bloom, Hill, Riccio, 2003)
* Small Schools of Choice (MDRC)
Some Recent MS TrialsStudy Levels Assigned
UnitsSites Fixed or
Random sites
National Head Start Eval.
2 Children 198 Program Sites
Random
Moving to Opportunity
2 Families 5 cities Fixed
BostonCharter School Lotteries
2 Children Lottery pools Random
Tennessee STAR
3 Teachers 79 Schools Random
4 R’s 3 Classrooms 18 Schools Random
Double-Dose Algebra
2 Children 60 Schools Random
2. Estimating the Impact of Program Participation in a One- Site Study
T=Random assignment M=Participation
Y=outcome
Figure 1: Conventional Instrumental Variable Model (Homogeneous Treatment Effects)
0/
:EffectITTTotal
so
Single site, heterogeneous treatment effects
TM
Y
Person-specific Causal Model
)(E
)(E
),()(
CovBEB iii
“No Compliance-Covariance” Assumption is Strong!
0/0),(
),()(
ifCovif
CovBE
Alternative Approachfor binary M
“Local Average Treatment Effect” (LATE)or“Complier Average Treatment Effect”
(Bloom, 1984; Angrist, Imbens, and Rubin, 1996)
Principal StratificationStratum M(1) M(0) Г=M(1)-M(0) Y(M(1))-Y(M(0)) Fraction
of popAverageEffect
Compliers 1 0 1 Y(1)=Y(0) γcompliers δcompliers
Always-takers
1 1 0 Y(1)-Y(1)=0 γ always 0
Never-takers
0 0 0 Y(0)-Y(0)=0 γnever 0
Defiers 0 1 -1 Y(0)-Y(1) 0 0
Complier-average treatment effect(“Local average treatment effect”)
randomizedisTifMonITTTMETME
YonITTTYETYENote
so
BE
complierscompliers
neveralwayscomplierscompliers
"")0|()1|("")0|()1|(
/
*0*0)(
In SumWe can estimate the Population-Average
Effect of Participating if we assume Cov(Г,Δ)=0
We can estimate LATE if we assumePr(Г≥0)=1
The latter is a weaker assumption, but does not eliminate the selection problem!
Multiple SitesHow do we take this to multiple sites to
* Estimate average Impact of Program Participation
* Estimate variation in the Impact of Program Participation
* Two methods using simulated data:“Small Schools of Choice” Design (J=200, 80<n<120)
)430.0,194.1:( 2 valuestrue
Method 1: Combine 2 ITT analyses
Step 1: Estimate the Impact of Treatment Assignment on the Outcome
Results
2)(,)(
.).(.
B
jijjijjjij
BVarBE
eeTTBYY
)75.1,21.(499.*96.1770.
forintervalvalueplausible%95
499.249.0ˆ53.0,770.0ˆ
22
j
B
B
se
Step 2: Estimate the Impact of Treatment Assignment on Program Participation
Results2)(,)(
.).(.
G
jijjijjjij
GVarGE
TTGMM
)86,55(.
forintervalvalueplausible%95
078.0061.0ˆ703.0ˆ
22
j
G
G
Step 3: Combine Results
)248.1,267.(483.*96.1095.1)(%95
483.)703(.)078(.
)078(.*)095.1()249(.ˆ
caseourIn
095.1703./770.ˆ/ˆˆ/
22
222
22
2222
DPV
DGif
caseourIn
D
D
In sumTrue Values
Our estimates
430.0,19.1: 2 valuesTrue
487.0ˆ,10.1ˆ 2
Method 2: Two-Stage Generalized Least Squares: Theoretical Model
2
2
)(,)(
.).(.:2Stage
)(,)(
.).(.:1Stage
D
jijjijjjij
G
jijjijjjij
DVarDE
MMDYY
GVarGE
TTGMM
Method 2 in Practice
2
**
2
**
)(,ˆ)(
).(ˆ.
originalusingvariancesRecompute.3
)(,)(
).ˆ(.:.2
).(ˆ.ˆ:.1
D
ijijjijjjij
D
ijijjijjjij
jijOLSjij
DVarDE
MMDYY
M
DVarDE
MMDYYHLMDo
TTGMMCompute
Method 2=“Between Site Regression!”
Cj
Ejjjjj
jjj
jjj
GDGB
whichfrom
GDG
GDB
ˆ)(ˆˆ.....
)(
0)](ˆ[ jj DGERequires
Results
)863.,562(.)746.,316(.%95
6968.
4856.ˆ
0730.0,095.1ˆ
22
2
2
CI
se
Envisioning Variation:LATE Effect
“Head Start” Design (J=200, 10<n<20)
“Small Schools of Choice” Design (J=200, 80<n<120)
“Welfare to Work” Design (J=60, 200<n<1400)
Program Participation Model (“LATE”)
)430.0,194.1:(
280.0ˆ,007.1ˆ:
487.0ˆ,095.1ˆ0ˆ986.0ˆ:
),(~
.).(.:
)(,)(
.).(:
2
2
2
2
2
2
valuestrue
WtW
SSC
HS
ND
MMDYYModelImpact
GVarGE
vvTTGMMmodelionParticipat
j
jijjijjjij
jijjijjjij
Profile Likelihood for LATE: “HS” Design
0 0.13 0.25 0.38 0.50 0.63 0.76 0.88 1.01-1.00
0
1.00
2.00
3.00
Tau
Bet
a
Profile Likelihood for LATE: “SSOC” Design
0 0.13 0.25 0.38 0.50 0.63 0.76 0.88 1.01
-0.50
1.00
2.50
Tau
Bet
a
Profile Likelihood for LATE: “WtW” Design
0 0.13 0.25 0.38 0.50 0.63 0.76 0.88 1.01-1.00
0
1.00
2.00
3.00
Tau
Bet
a
Posterior intervals for site-specific LATE Effects
Posterior Intervals for LATE: “HS” Design
-0.55
0.27
1.09
1.90
2.72
MH
AT
0 50.50 101.00 151.50 202.00
Posterior Intervals for LATE: “SSC” Design
0 50.50 101.00 151.50 202.00-1.21
-0.13
0.96
2.04
3.12
MH
AT
Posterior Intervals for LATE: “W to W” Design
0 15.50 31.00 46.50 62.00-0.37
0.41
1.18
1.95
2.73
MH
AT
Moving Toward Explanationmodeling participation, modeling impact
Total Impact of Assignment=Impact of Assignment on Participation * Impact of participation on Outcome
Within site:
Which persons are most likely to participate?Which persons are most likely to benefit from participation?
Between Sites:
How do we improve site-average participation rate?How do we enhance average benefit of participating?
Models are needed at both levels because sites vary not only in organizational effectiveness but also in client composition
The picture can't be displayed.
jjj DGB