nature gives us correlations… evaluation research (8521) prof. jesse lecy lecture 0 1
TRANSCRIPT
![Page 1: Nature gives us correlations… Evaluation Research (8521) Prof. Jesse Lecy Lecture 0 1](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e745503460f94b746ee/html5/thumbnails/1.jpg)
1
Nature gives us correlations…
Evaluation Research (8521)Prof. Jesse Lecy
Lecture 0
![Page 2: Nature gives us correlations… Evaluation Research (8521) Prof. Jesse Lecy Lecture 0 1](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e745503460f94b746ee/html5/thumbnails/2.jpg)
2
Policy / ProgramBlack Box
Input Outcome
The Program Evaluation Mindset
(something happens here)
![Page 3: Nature gives us correlations… Evaluation Research (8521) Prof. Jesse Lecy Lecture 0 1](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e745503460f94b746ee/html5/thumbnails/3.jpg)
3
Policy / Program
Input Outcome
The Program Evaluation Mindset
𝑂𝑢𝑡𝑐𝑜𝑚𝑒= 𝑓 (𝑝𝑟𝑜𝑔𝑟𝑎𝑚 , 𝑖𝑛𝑝𝑢𝑡 𝑙𝑒𝑣𝑒𝑙 )
𝑂𝑢𝑡𝑐𝑜𝑚𝑒=𝑏0+𝑏1 ∙ 𝐼𝑛𝑝𝑢𝑡+𝜀
(something happens here)
The slopes tells us how much impact we expect a program to have when we spend one additional unit of input.
The outcome is some function of the program and the amount of inputs into the process. It can sometimes be represented by this simple input-output equation.
![Page 4: Nature gives us correlations… Evaluation Research (8521) Prof. Jesse Lecy Lecture 0 1](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e745503460f94b746ee/html5/thumbnails/4.jpg)
50 55 60 65 70
50
10
01
50
Dosage and Response
Caffeine (mm)
He
art
Ra
te (
pe
r m
in)
𝐻𝑒𝑎𝑟𝑡𝑟𝑎𝑡𝑒=𝑏0+𝑏1∙𝐶𝑎𝑓𝑓𝑒𝑖𝑛𝑒+𝜀
Effect
![Page 5: Nature gives us correlations… Evaluation Research (8521) Prof. Jesse Lecy Lecture 0 1](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e745503460f94b746ee/html5/thumbnails/5.jpg)
5
𝐻𝑒𝑎𝑟𝑡𝑟𝑎𝑡𝑒=𝑏0+𝑏1∙𝐶𝑎𝑓𝑓𝑒𝑖𝑛𝑒+𝜀
Heart Rate
Treatment(Caffeine)
Control(No caffeine)
![Page 6: Nature gives us correlations… Evaluation Research (8521) Prof. Jesse Lecy Lecture 0 1](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e745503460f94b746ee/html5/thumbnails/6.jpg)
50 55 60 65 70
50
10
01
50
Dosage and Response
# of Potato Chips
He
art
Ra
te (
pe
r m
in)
![Page 7: Nature gives us correlations… Evaluation Research (8521) Prof. Jesse Lecy Lecture 0 1](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e745503460f94b746ee/html5/thumbnails/7.jpg)
7
http://www.radiolab.org/2010/oct/08/its-alive/ 4:15-
![Page 8: Nature gives us correlations… Evaluation Research (8521) Prof. Jesse Lecy Lecture 0 1](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e745503460f94b746ee/html5/thumbnails/8.jpg)
50 55 60 65 70
50
10
01
50
Dosage and Response
Caffeine (mm)
He
art
Ra
te (
pe
r m
in)
50 55 60 65 70
50
10
01
50
City Density and Productivity
Walking Speed
Nu
mb
er
of P
ate
nts
Pe
r 1
00
00
Re
sid
en
ts
How do we know when the interpretation is causal?
Effect?Effect ?
![Page 9: Nature gives us correlations… Evaluation Research (8521) Prof. Jesse Lecy Lecture 0 1](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e745503460f94b746ee/html5/thumbnails/9.jpg)
9
NATURE GIVES US CORRELATIONS
![Page 10: Nature gives us correlations… Evaluation Research (8521) Prof. Jesse Lecy Lecture 0 1](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e745503460f94b746ee/html5/thumbnails/10.jpg)
0 50 100 150 200
010
030
0
x
y
0 50 100 150 200
010
030
0
z
y
0 50 100 150 200
050
100
200
x
z
x y
z
Example #1
![Page 11: Nature gives us correlations… Evaluation Research (8521) Prof. Jesse Lecy Lecture 0 1](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e745503460f94b746ee/html5/thumbnails/11.jpg)
x y
z
0 100 200 300 400
010
030
0
x
y
0 50 100 150 200
010
030
0
z
y
0 100 200 300 400
050
100
200
x
z
Example #2
![Page 12: Nature gives us correlations… Evaluation Research (8521) Prof. Jesse Lecy Lecture 0 1](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e745503460f94b746ee/html5/thumbnails/12.jpg)
x y
z
0 100 200 300 400
020
060
0
x
y
0 50 100 150 200
020
060
0
z
y
0 100 200 300 400
050
100
200
x
z
Example #3
![Page 13: Nature gives us correlations… Evaluation Research (8521) Prof. Jesse Lecy Lecture 0 1](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e745503460f94b746ee/html5/thumbnails/13.jpg)
x y
x y
0 20 40 60 80 100
-50
50
15
02
50
x
y
-50 0 50 100 150 200 250
02
06
01
00
y
x
![Page 14: Nature gives us correlations… Evaluation Research (8521) Prof. Jesse Lecy Lecture 0 1](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e745503460f94b746ee/html5/thumbnails/14.jpg)
Examples of Poor Causal Inference
![Page 15: Nature gives us correlations… Evaluation Research (8521) Prof. Jesse Lecy Lecture 0 1](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e745503460f94b746ee/html5/thumbnails/15.jpg)
Examples of Poor Causal Inference1. Ice cream consumption causes polio
2. Investments in public buildings creates economic growth
3. Early retirement and health decline
4. Hormone replacement therapy and heart disease:
In a widely-studied example, numerous epidemiological studies showed that women who were taking combined hormone replacement therapy (HRT) also had a lower-than-average incidence of coronary heart disease (CHD), leading doctors to propose that HRT was protective against CHD. But randomized controlled trials showed that HRT caused a small but statistically significant increase in risk of CHD. Re-analysis of the data from the epidemiological studies showed that women undertaking HRT were more likely to be from higher socio-economic groups (ABC1), with better than average diet and exercise regimes. The use of HRT and decreased incidence of coronary heart disease were coincident effects of a common cause (i.e. the benefits associated with a higher socioeconomic status), rather than cause and effect as had been supposed. (Wikipedia)
![Page 16: Nature gives us correlations… Evaluation Research (8521) Prof. Jesse Lecy Lecture 0 1](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e745503460f94b746ee/html5/thumbnails/16.jpg)
Examples of Complex Causal Inference
![Page 17: Nature gives us correlations… Evaluation Research (8521) Prof. Jesse Lecy Lecture 0 1](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e745503460f94b746ee/html5/thumbnails/17.jpg)
17
MODERN PROGRAM EVAL
![Page 18: Nature gives us correlations… Evaluation Research (8521) Prof. Jesse Lecy Lecture 0 1](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e745503460f94b746ee/html5/thumbnails/18.jpg)
18
To Experiment or Not Experiment
http://www.youtube.com/watch?v=exBEFCiWyW0
![Page 19: Nature gives us correlations… Evaluation Research (8521) Prof. Jesse Lecy Lecture 0 1](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e745503460f94b746ee/html5/thumbnails/19.jpg)
19
CASE STUDY – EDUCATION REFORM
![Page 20: Nature gives us correlations… Evaluation Research (8521) Prof. Jesse Lecy Lecture 0 1](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e745503460f94b746ee/html5/thumbnails/20.jpg)
Classroom Size and Performance
http://www.publicschoolreview.com/articles/19
State Laws Limiting Class Size
Notwithstanding the ongoing debate over the pros and cons of reducing class sizes, a number of states have embraced the policy of class size reduction. States have approached class size reduction in a variety of ways. Some have started with pilot programs rather than state-wide mandates. Some states have specified optimum class sizes while other states have enacted mandatory maximums. Some states have limited class size reduction initiatives to certain grades or certain subjects.Here are three examples of the diversity of state law provisions respecting class size reduction.
California – The state of California became a leader in promoting class size reduction in 1996, when it commenced a large-scale class size reduction program with the goal of reducing class size in all kindergarten through third grade classes from 30 to 20 students or less. The cost of the program was $1 billion annually.
![Page 21: Nature gives us correlations… Evaluation Research (8521) Prof. Jesse Lecy Lecture 0 1](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e745503460f94b746ee/html5/thumbnails/21.jpg)
Classroom Size and Performance
http://www.publicschoolreview.com/articles/19
Florida – Florida residents in 2002 voted to amend the Florida Constitution to set the maximum number of students in a classroom. The maximum number varies according to the grade level. For prekindergarten through third grade, fourth grade through eighth grade, and ninth grade through 12th grade, the constitutional maximums are 18, 22, and 25 students, respectively. Schools that are not already in compliance with the maximum levels are required to make progress in reducing class size so that the maximum is not exceeded by 2010. The Florida legislature enacted corresponding legislation, with additional rules and guidelines for schools to achieve the goals by 2010.
Georgia -- Maximum class sizes depend on the grade level and the class subject. For kindergarten, the maximum class size is 18 or, if there is a full-time paraprofessional in the classroom, 20. Funding is available to reduce kindergarten class sizes to 15 students. For grades one through three, the maximum is 21 students; funding is available to reduce the class size to 17 students. For grades four through eight, 28 is the maximum for English, math, science, and social studies. For fine arts and foreign languages in grades K through eight, however, the maximum is 33 students. Maximums of 32 and 35 students are set for grades nine through 12, depending on the subject matter of the course. Local school boards that do not comply with the requirements are subject to lose funding for the entire class or program that is out of compliance.
![Page 22: Nature gives us correlations… Evaluation Research (8521) Prof. Jesse Lecy Lecture 0 1](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e745503460f94b746ee/html5/thumbnails/22.jpg)
Class Size Case Study - The Theory:
Class Size Test Scores
Class Size Test Scores
SES
?
Scenario 1
Scenario 2
-400 -300 -200 -100 0
01
00
20
03
00
40
0
Class Size
Test
Score
![Page 23: Nature gives us correlations… Evaluation Research (8521) Prof. Jesse Lecy Lecture 0 1](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e745503460f94b746ee/html5/thumbnails/23.jpg)
23
Example: Classroom Size
-400 -300 -200 -100 0
-100
-50
050
Class Size
Tes
t S
core
Res
idua
ls
∆Y
∆X
𝑆𝑙𝑜𝑝𝑒=∆𝑌∆ 𝑋
The regression coefficient represents a slope. In policy we think of the slope as an input-output formula. If I decrease class size (input) standardized tests scores increase (output).
Note changes in slopes and standard errors when you add variables.
![Page 24: Nature gives us correlations… Evaluation Research (8521) Prof. Jesse Lecy Lecture 0 1](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e745503460f94b746ee/html5/thumbnails/24.jpg)
The Naïve Model:
-400 -300 -200 -100 0
01
00
20
03
00
40
0
Class Size
Tes
t S
core
eClassSizebbTestScore 10
![Page 25: Nature gives us correlations… Evaluation Research (8521) Prof. Jesse Lecy Lecture 0 1](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e745503460f94b746ee/html5/thumbnails/25.jpg)
With Teacher Skill as a Control:
-400 -300 -200 -100 0
-10
0-5
00
50
Class Size
Tes
t S
core
Res
idua
ls
eTeachSkillbClassSizebbTestScore 210
![Page 26: Nature gives us correlations… Evaluation Research (8521) Prof. Jesse Lecy Lecture 0 1](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e745503460f94b746ee/html5/thumbnails/26.jpg)
-400 -300 -200 -100 0
-20
0-1
00
01
00
Class Size
Tes
t S
core
Res
idua
ls
Add SES to the Model:
eSESbTeachSkillbClassSizebbTestScore 3210
![Page 27: Nature gives us correlations… Evaluation Research (8521) Prof. Jesse Lecy Lecture 0 1](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e745503460f94b746ee/html5/thumbnails/27.jpg)
27
Example: Classroom Size
Why are slopes and standard errors changingwhen we add “control” variables?
-400 -300 -200 -100 0
-200
-100
010
0
Class Size
Tes
t S
core
Res
idua
ls
![Page 28: Nature gives us correlations… Evaluation Research (8521) Prof. Jesse Lecy Lecture 0 1](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e745503460f94b746ee/html5/thumbnails/28.jpg)
How do we interpret results causally?
Class Size Test Scores
SES
Teacher Skill
X
![Page 29: Nature gives us correlations… Evaluation Research (8521) Prof. Jesse Lecy Lecture 0 1](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e745503460f94b746ee/html5/thumbnails/29.jpg)
29
COURSE OUTLINE
![Page 30: Nature gives us correlations… Evaluation Research (8521) Prof. Jesse Lecy Lecture 0 1](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e745503460f94b746ee/html5/thumbnails/30.jpg)
30
The Origins of Modern Program Evaluation
• The “Great Society” introduced unprecedented levels of spending on social services – marks the dawn of the modern welfare state.
• Econometrics also comes of age, creating tools the provide opportunity for rigorous analysis of social programs.
![Page 31: Nature gives us correlations… Evaluation Research (8521) Prof. Jesse Lecy Lecture 0 1](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e745503460f94b746ee/html5/thumbnails/31.jpg)
31
![Page 32: Nature gives us correlations… Evaluation Research (8521) Prof. Jesse Lecy Lecture 0 1](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e745503460f94b746ee/html5/thumbnails/32.jpg)
32
![Page 33: Nature gives us correlations… Evaluation Research (8521) Prof. Jesse Lecy Lecture 0 1](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e745503460f94b746ee/html5/thumbnails/33.jpg)
33
We need effective programs, not expensive programs
![Page 34: Nature gives us correlations… Evaluation Research (8521) Prof. Jesse Lecy Lecture 0 1](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e745503460f94b746ee/html5/thumbnails/34.jpg)
34
Modern Program Evaluation
Course Objectives:
1. Understanding why regressions are biased
Seven Deadly Sins of Regression:
1. Multicollinearity2. Omitted variable bias3. Measurement error4. Selection / attrition5. Misspecification6. Population heterogeneity7. Simultaneity
![Page 35: Nature gives us correlations… Evaluation Research (8521) Prof. Jesse Lecy Lecture 0 1](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e745503460f94b746ee/html5/thumbnails/35.jpg)
35
Modern Program Evaluation
Course Objectives:
2. Understand tools of program evaluation
– Fixed effect models– Instrumental variables
– Matching– Regression discontinuity– Time series – Survival analysis
![Page 36: Nature gives us correlations… Evaluation Research (8521) Prof. Jesse Lecy Lecture 0 1](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e745503460f94b746ee/html5/thumbnails/36.jpg)
36
Modern Program Evaluation
Course Objectives:
3. How to talk to economists (and other bullies)
![Page 37: Nature gives us correlations… Evaluation Research (8521) Prof. Jesse Lecy Lecture 0 1](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e745503460f94b746ee/html5/thumbnails/37.jpg)
37
Modern Program Evaluation
Course Objectives:
4. Correctly apply and critique evaluation designs
Experiments• Pretest-posttest control group• Posttest only control group
Quasi-Experiments• Pretest-posttest comparison group• Posttest only comparison group• Interrupted time series with comparison group
Reflexive Design• Pretest-posttest design• Simple time series
![Page 38: Nature gives us correlations… Evaluation Research (8521) Prof. Jesse Lecy Lecture 0 1](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e745503460f94b746ee/html5/thumbnails/38.jpg)
38
Course Structure
First half: Understanding bias– No text, course notes online– Weekly homework
Second half: Evaluation design– Text is required– Campbell Scores
policy-research.net/programevaluation
![Page 39: Nature gives us correlations… Evaluation Research (8521) Prof. Jesse Lecy Lecture 0 1](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e745503460f94b746ee/html5/thumbnails/39.jpg)
39
Evaluating Internal Validity: The Campbell Scores
A Competing Hypothesis Framework
Omitted Variable Bias• Selection• Nonrandom Attrition
Trends in the Data• Maturation • Secular Trends
Study Calibration• Testing• Regression to the Mean• Seasonality• Study Time-Frame
Contamination Factors• Intervening Events• Measurement Error
![Page 40: Nature gives us correlations… Evaluation Research (8521) Prof. Jesse Lecy Lecture 0 1](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e745503460f94b746ee/html5/thumbnails/40.jpg)
Homework Policy
40
• Homework problems each week 1st half of semester– Graded pass/fail– Submit via D2L please– Work in groups is strongly encouraged
• Campbell Scores are due each class for the second half of the semester
• Midterm Exam (30%)– Confidence intervals, Standard error of regression, Bias
• Final Exam (20%)– Covers evaluation design and Internal validity
No late homework accepted! 50% of final grade.
See syllabus for policy on turning in by email.
![Page 41: Nature gives us correlations… Evaluation Research (8521) Prof. Jesse Lecy Lecture 0 1](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e745503460f94b746ee/html5/thumbnails/41.jpg)
41
Midterm Spring 2012
Grade
Stu
de
nt
Co
un
t
40 60 80 100 120
01
23
45
1st Qu. 75.5Median 89Mean 83.473rd Qu. 96
Midterm Fall 2011
Grade
Stu
de
nt
Co
un
t
40 60 80 100 120
05
10
15
1st Qu. 82Median 86Mean 87.213rd Qu. 97.5
![Page 42: Nature gives us correlations… Evaluation Research (8521) Prof. Jesse Lecy Lecture 0 1](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e745503460f94b746ee/html5/thumbnails/42.jpg)
42
0-2 hours/wk 3-4 hours/wk 4-8 hours/wk 9-14 hours/wk
15+ hours/wk0.00%5.00%
10.00%15.00%20.00%25.00%30.00%35.00%40.00%45.00%50.00%
Average Time Spent on Class by Students: Spring 2012
Hours Per Week
0-2 3-4 4-8 9-14 15+0.00%5.00%
10.00%15.00%20.00%25.00%30.00%35.00%40.00%45.00%50.00%
Average Time Spent on Class by Students: Fall 2011
Hours Per Week