sensitivity of teacher value-added estimates to student and peer control variables
DESCRIPTION
Sensitivity of Teacher Value-Added Estimates to Student and Peer Control Variables. March 2012 Presentation to the Association of Education Finance and Policy Conference Matt Johnson Stephen Lipscomb Brian Gill. VAMs Used Today Differ in Their Specifications. Research Questions. - PowerPoint PPT PresentationTRANSCRIPT
Sensitivity of Teacher Value-Added Estimates to Student and Peer Control
VariablesMarch 2012
Presentation to the Association of Education Finance and Policy Conference
Matt Johnson Stephen Lipscomb Brian Gill
VAMs Used Today Differ in Their Specifications
2
Value-Added ModelStudent
CharacteristicsClassroom
Characteristics
Multiple Years of
Prior Scores
Colorado Growth Model No No Yes
DC IMPACT Yes No No
Florida Yes Yes Yes
New York City Yes Yes No
SAS EVAAS No No Yes
How sensitive are teacher value-added model (VAM) estimates to changes in the model specification?– Student characteristics– Classroom characteristics– Multiple years of prior scores
How sensitive are estimates to loss of students from sample due to missing prior scores?
Research Questions
3
Teacher value-added estimates are not highly sensitive to inclusion of:– Student characteristics (correlation ≥ 0.990)– Multiple years of prior scores (correlation ≥ 0.987)
Estimates are more sensitive to inclusion of classroom characteristics (correlation = 0.915 to .955)
Estimates are not very sensitive to loss of students with missing prior test scores from sample (correlation = 0.992)– Precision increases when two prior scores are used but
fewer teacher VAM estimates are produced
Preview of Main Results
4
Explore sensitivity to several specifications:– Exclude score from two prior years (Yi,t-2)
– Exclude student characteristics (Xi,t)
– Include class average characteristics
Student data from a medium-sized urban district for 2008–2009 to 2010–2011 school years
All models run using the same set of student observations
Instrument using opposite subject prior score to control for measurement error
Baseline Model
5
Student Level Class Level
Free or Reduced-Price Meals • •
Disability • •
Gifted Program Participation • •
Lagged Rate of Attendance • •
Lagged Fraction of Year Suspended • •
Race/Ethnicity •
Gender •
Age/Behind Grade Level •
Average Prior Achievement in Same Subject
•
Standard Deviation of Lagged Achievement
•
Number of Students in Classroom •
Student and Class Characteristics
6
Correlation of 6th-Grade Teacher Estimates Relative to Baseline VAM Specification
7
Math (n = 87)
Reading (n = 99)
Exclude Student Characteristics 0.990 0.996
Exclude Prior Score from t-2 0.993 0.987
Exclude Student Characteristics and Prior Score from t-2
0.978 0.970
Add class average variables 0.955 0.915
Baseline: Student Characteristics and Prior Scores from t-1 and t-2
Findings are based on VAM estimates from 2008–2009 to 2010–2011 on the same sample of students.
Exclude Student Characteristics
1st (Lowest) 2nd 3rd 4th
5th
(Highest)
Baseline Model
1st (Lowest) 95 5 0 0 0
2nd 5 90 5 0 0
3rd 0 5 75 20 0
4th 0 0 20 70 10
5th (Highest) 0 0 0 10 90
Percentage of 6th-Grade Reading Teachers in Effectiveness Quintiles, by VAM Specification
8
Findings are based on VAM estimates for 99 reading teachers in grade 6 from 2008–2009 to 2010–2011 for a medium-sized, urban district. Correlation with baseline = 0.996.
Baseline + Class Average Characteristics
1st (Lowest) 2nd 3rd 4th
5th
(Highest)
Baseline Model
1st (Lowest) 80 20 0 0 0
2nd 5 65 30 0 0
3rd 15 10 50 15 10
4th 0 5 10 65 20
5th (Highest) 0 0 10 20 70
Percentage of 6th-Grade Reading Teachers in Effectiveness Quintiles, by VAM Specification
9
Findings are based on VAM estimates for 99 reading teachers in grade 6 from 2008–2009 to 2010–2011 for a medium-sized, urban district. Correlation with baseline = 0.915.
Benefits of including two prior years:– More accurate measure of student ability– Increase in precision of estimates
Costs of using two prior years:– Students with missing prior scores dropped– Some teachers dropped from sample
Relative magnitude of costs/benefits?
One or Two Years of Prior Scores?
10
Estimate two VAMs using one year of prior scores– First VAM includes all students– Second VAM restricts sample to students with nonmissing
second prior year of scores
Correlation between teacher estimates: 0.992
Percentage of students dropped: 6.2
Percentage of teachers dropped: 3.9
Net increase in precision from using two prior years– Increase in average standard error of estimates: 2.3%
when students with missing scores are dropped– Decrease in average standard error of estimates: 7.6%
when second year of prior scores added
One or Two Years of Prior Scores?
11
Mathematica® is a registered trademark of Mathematica Policy Research.
Please contact– Matt Johnson
– Stephen Lipscomb• [email protected]
– Brian Gill• [email protected]
For More Information
12