school accountability and the distribution of student achievement randall reback barnard college...
TRANSCRIPT
![Page 1: School Accountability and the Distribution of Student Achievement Randall Reback Barnard College Economics Department and Teachers College, Columbia University](https://reader036.vdocument.in/reader036/viewer/2022082712/56649e985503460f94b9ba45/html5/thumbnails/1.jpg)
School Accountability and the Distribution of Student
Achievement
Randall RebackBarnard College Economics Department
and Teachers College,Columbia University
![Page 2: School Accountability and the Distribution of Student Achievement Randall Reback Barnard College Economics Department and Teachers College, Columbia University](https://reader036.vdocument.in/reader036/viewer/2022082712/56649e985503460f94b9ba45/html5/thumbnails/2.jpg)
No Child Left Behind• States must adopt accountability systems that
assign ratings to schools based on student pass rates on exams in elementary, middle school, and high school grades
• School is not making ‘Adequate Yearly Progress’ if a pass rate is not sufficiently high, where the required pass rate increases each year
• Consequences– Stigma, financial rewards/penalties, loss of
local control, changes in property values– Intra-district public school choice provision– Tutoring for economically disadvantaged
children
![Page 3: School Accountability and the Distribution of Student Achievement Randall Reback Barnard College Economics Department and Teachers College, Columbia University](https://reader036.vdocument.in/reader036/viewer/2022082712/56649e985503460f94b9ba45/html5/thumbnails/3.jpg)
Texas Accountability Program
• Precursor to No Child Left Behind• Assigns schools one of four ratings based on
– Dropout Rates– Attendance Rates– Fraction of students who pass exams (overall
and within subgroups by race and family income)
• Testing Incentives are based on Pass Rates, not value-added measures of student-achievement
![Page 4: School Accountability and the Distribution of Student Achievement Randall Reback Barnard College Economics Department and Teachers College, Columbia University](https://reader036.vdocument.in/reader036/viewer/2022082712/56649e985503460f94b9ba45/html5/thumbnails/4.jpg)
Key Provisions of the Texas Accountability System
Minimum TAAS Pass Rate
Maximum Dropout Rate
Minimum Attendance Rate
E R A E R A E R A 1993 90.0% 65.0% 20.0% 1.0% 3.5% N/A 97.0% 95.0% N/A
1994 90.0% 65.0% 25.0% 1.0% 3.5% N/A 94.0% 94.0% N/A
1995 90.0% 70.0% 25.0% 1.0% 3.5% 6.0% 94.0% 94.0% N/A
1996 90.0% 70.0% 30.0% 1.0% 3.5% 6.0% 94.0% 94.0% N/A
1997 90.0% 75.0% 35.0% 1.0% 3.5% 6.0% 94.0% 94.0% N/A
1998 90.0% 80.0% 40.0% 1.0% 3.5% 6.0% 94.0% 94.0% N/A
1999 90.0% 80.0% 45.0% 1.0% 3.5% 6.0% 94.0% 94.0% N/A
![Page 5: School Accountability and the Distribution of Student Achievement Randall Reback Barnard College Economics Department and Teachers College, Columbia University](https://reader036.vdocument.in/reader036/viewer/2022082712/56649e985503460f94b9ba45/html5/thumbnails/5.jpg)
Previous Research Related to School Accountability/Minimum Proficiency
• Relative performance of students at different points in distribution (Holmes, 2004; Deere & Strayer, 2001)
• Achievement trends– Grissmer & Flanagan (98)- Math NAEP in TX– Hanushek & Raymond- Math NAEP – Carnoy, Loeb, Smith (2002)- TX improvements didn’t correspond
with improved 10-12th grade transitions, SAT participation, SAT performance
• Low-performing versus high-performing schools (Jacob, forthcoming• States with or without HS graduation exams (Jacobsen, 1993)• Gaming
– Exemptions: Figlio & Getzler, Cullen & Reback– School meals: Figlio & Winicki– Disciplinary practices: Figlio
![Page 6: School Accountability and the Distribution of Student Achievement Randall Reback Barnard College Economics Department and Teachers College, Columbia University](https://reader036.vdocument.in/reader036/viewer/2022082712/56649e985503460f94b9ba45/html5/thumbnails/6.jpg)
Theoretical Framework• Subject Specific but not Student Specific Inputs (as)• Not Subject Specific but Student Specific Inputs (bi)• Subject Specific and Student Specific Inputs (cs)• Assume only campus-wide Math (m) and Reading (r) pass
rates count. Call all other subjects (z). Schools want to maximize:
TcbafPTcbafPN
iimimimim
N
iiriririr
~)),,((Prob*
~),,((Prob
11
N
iimirizmr Kccbaaa
1
)(
subject to
![Page 7: School Accountability and the Distribution of Student Achievement Randall Reback Barnard College Economics Department and Teachers College, Columbia University](https://reader036.vdocument.in/reader036/viewer/2022082712/56649e985503460f94b9ba45/html5/thumbnails/7.jpg)
The Data• Texas Assessment of Academic Skills
– Math Tested Grades 3-8 and 10– Writing Tested Grades 4, 8, and 10– Test Documents Submitted for Every Student– Includes Student Descriptors
• Campus Level Data on Attendance/Dropouts• Texas Learning Index
– Measures How Student Performs Compared to Grade Level
– I Do Not Measure Test Score Gains for Observations with Prior Year’s Scores Below 30 or Above 84
![Page 8: School Accountability and the Distribution of Student Achievement Randall Reback Barnard College Economics Department and Teachers College, Columbia University](https://reader036.vdocument.in/reader036/viewer/2022082712/56649e985503460f94b9ba45/html5/thumbnails/8.jpg)
Pass Rate Probabilities Based on Prior Year Test Score Range
Previous Year’s Scoring
Range
Probability of Passing Math based on Previous
Math Score Range
Probability of Passing Reading based on Previous
Reading Score Range 30-44 7.3% 14.9% 45-54 17.7% 26.3% 55-64 39.2% 45.8% 65-74 69.0% 69.1% 75-84 92.3% 88.8%
Passing Score=70
![Page 9: School Accountability and the Distribution of Student Achievement Randall Reback Barnard College Economics Department and Teachers College, Columbia University](https://reader036.vdocument.in/reader036/viewer/2022082712/56649e985503460f94b9ba45/html5/thumbnails/9.jpg)
Estimating the Marginal Benefit to the School from a Moderate Increase in a
Student’s Expected Performance (1) estimate the probability that each student
passes by grouping students based on their performance during other years
(2) use these student-level pass probabilities to compute the probability that the school will obtain each rating
(3) find the marginal effect of a moderate improvement in the expected achievement of a particular student on the probability that the school obtains the various ratings….
![Page 10: School Accountability and the Distribution of Student Achievement Randall Reback Barnard College Economics Department and Teachers College, Columbia University](https://reader036.vdocument.in/reader036/viewer/2022082712/56649e985503460f94b9ba45/html5/thumbnails/10.jpg)
How a ‘moderate improvement’ in a student’s achievement is determined
• hypothetical pass probability by re-estimating the student’s pass probability after dropping the bottom X% of the current year score distribution among students with identical prior year scores
• For example: distribution of this year’s Math scores for students scoring 53 last year in Math
0%: 36 Actual pass probability= .20 20%: 49
40%: 55 60%: 59 Pass probability with X% set at 20%: 80%: 70 .2 / .8=.25 100%: 86
![Page 11: School Accountability and the Distribution of Student Achievement Randall Reback Barnard College Economics Department and Teachers College, Columbia University](https://reader036.vdocument.in/reader036/viewer/2022082712/56649e985503460f94b9ba45/html5/thumbnails/11.jpg)
Dependent Variable
• Year-to-year variation in test scores might be greater at certain points of achievement distribution
• Value-added models examining distributional effects SHOULD NOT simply look at changes in levels or in relative place in distribution
• Instead, use conditional Z-score… Z-score among students with similar prior year scores
• This way, results are compared to typical progress at that place in the test score distribution
![Page 12: School Accountability and the Distribution of Student Achievement Randall Reback Barnard College Economics Department and Teachers College, Columbia University](https://reader036.vdocument.in/reader036/viewer/2022082712/56649e985503460f94b9ba45/html5/thumbnails/12.jpg)
Empirical Model #1: Campus-Year Fixed Effects
Student i in grade g during year y at school s
stjittjti
stistjistisi
tjstgi
YCS
RPEERRx
vY
,,,5,4,
3,1,2,1,,,1,,
,1,,,
log
Si,t includes control variables for student characteristics:• Cubic terms for prior year scores in other subject• Racial dummy variables, Low-income family dummy, and Race-income interactions
![Page 13: School Accountability and the Distribution of Student Achievement Randall Reback Barnard College Economics Department and Teachers College, Columbia University](https://reader036.vdocument.in/reader036/viewer/2022082712/56649e985503460f94b9ba45/html5/thumbnails/13.jpg)
Achievement Gains and Marginal Accountability Incentives within
Schools and within the Same Year MATH READING
Model Also Includes Peer Ability Controls?
(1) (2)
YES NO
(3) (4)
YES NO Point Estimate for Student-level Accountability Incentive Variable, ( in equation 6)
.00090 (.00013)
.00099 (.00013)
-.00007 (.00015)
.00009 (.00015)
Effect sizes: Approximate standard deviation change in statewide distribution associated with a 1 standard deviation increase in relative importance of student’s score within the school
.005 .005 -.0004 .0005
![Page 14: School Accountability and the Distribution of Student Achievement Randall Reback Barnard College Economics Department and Teachers College, Columbia University](https://reader036.vdocument.in/reader036/viewer/2022082712/56649e985503460f94b9ba45/html5/thumbnails/14.jpg)
Model #2: Response to Infra-marginal Incentives (Cross-sectional comparisons)
• Schools might consider the impact of improving the expected performance of 5% of the students
• Define as the marginal change in the
schools’ probability of a higher rating if all students in the ‘group’ are expected to do better
stjisttti
stistjistisi
tjstgi
SYX
RPEERRg
vY
,,,6,54,
3,1,2,1,,,1,,
,1,,,
log
si
tj
g
v
,
,
![Page 15: School Accountability and the Distribution of Student Achievement Randall Reback Barnard College Economics Department and Teachers College, Columbia University](https://reader036.vdocument.in/reader036/viewer/2022082712/56649e985503460f94b9ba45/html5/thumbnails/15.jpg)
Achievement Gains and Infra-marginal Accountability Incentives
MATH READING
Includes Peer Ability Controls?
YES NO
YES NO Point Estimate for Student-level Accountability Incentive Variable, (in equation 7)
.00092
(.00006)
.00119
(.00006)
.00005
(.00008)
.00057
(.00007)
Effect sizes: Approximate standard deviation change in student’s place in the statewide distribution associated with a 1 standard deviation increase in the infra-marginal incentive to help the student’s performance
.0111 .0143 .0007 .0074
![Page 16: School Accountability and the Distribution of Student Achievement Randall Reback Barnard College Economics Department and Teachers College, Columbia University](https://reader036.vdocument.in/reader036/viewer/2022082712/56649e985503460f94b9ba45/html5/thumbnails/16.jpg)
Model #3: Incentive to Improve Performance within a Grade-level
• Schools might use inputs that simultaneously affect multiple students
• Define as change in school’s probability of receiving a higher rating if all students in student i‘s grade at the school improve
.
log
,,,7,
,,1,6,5
4,3,1,2,1,,,1,,
,1,,,
stjisi
tjstistt
tististjistisi
tjstgi
h
vRSY
XRPEERRg
vY
si
tj
h
v
,
,
![Page 17: School Accountability and the Distribution of Student Achievement Randall Reback Barnard College Economics Department and Teachers College, Columbia University](https://reader036.vdocument.in/reader036/viewer/2022082712/56649e985503460f94b9ba45/html5/thumbnails/17.jpg)
Approximate Effect Sizes (SD change in Statewide Achievement Distribution from 1 SD increase in
Accountability Incentive
MATH READING
Student-level Incentive
.009 .010 -.001 .002
Grade-level Incentives Interacted with Prior Year
Achievement Level Lowest 0.082 0.086 0.010 0.018 Very Low 0.074 0.077 0.014 0.023 Low 0.068 0.071 0.014 0.020 Marginal 0.060 0.065 0.013 0.017 Proficient
0.061 0.067 0.004 0.005
Peer ability Controls?
YES NO YES NO
![Page 18: School Accountability and the Distribution of Student Achievement Randall Reback Barnard College Economics Department and Teachers College, Columbia University](https://reader036.vdocument.in/reader036/viewer/2022082712/56649e985503460f94b9ba45/html5/thumbnails/18.jpg)
Effects of Sample Selection Due to
Student Exemptions & Grade Repetition • Exemptions
– negative relationship between student-level accountability incentive and likelihood that student is exempted from accountability pool
– suggests that estimated effect of student-level accountability incentive may understate the true effect
– Also suggests that estimated effect of grade-level incentives overstate the effects for lowest achievers and understate for others
• Grade Retention– small, positive relationship between the student-level
accountability incentives and the probability of grade retention
– effect on main results is unclear, but likely small
![Page 19: School Accountability and the Distribution of Student Achievement Randall Reback Barnard College Economics Department and Teachers College, Columbia University](https://reader036.vdocument.in/reader036/viewer/2022082712/56649e985503460f94b9ba45/html5/thumbnails/19.jpg)
Conclusions
• Schools respond to specific incentives of a rating system
• Appear to respond with broad changes in teaching or resource allocation rather than narrowly-targeted changes
• Current findings may understate distributional effects– High achievers (top 50% Reading, top 33%
math) are not included– May be permanent changes rather than
response to short-run incentives• NCLB-style ratings. Are they good? bad?...
depends on one’s preferences.