special data opportunities in florida
DESCRIPTION
Special Data Opportunities in Florida. David N. Figlio University of Florida and National Bureau of Economic Research. What’s so special about Florida?. Florida has developed a remarkable ability to analyze data across a wide variety of settings - PowerPoint PPT PresentationTRANSCRIPT
Special Data Opportunities in Florida
David N. FiglioUniversity of Florida and
National Bureau of Economic Research
What’s so special about Florida?
• Florida has developed a remarkable ability to analyze data across a wide variety of settings
• Facilitated through legislative authority – but reflective of exceptional nurturing of interagency relationships that facilitate data-sharing
• Easier also to collect large datasets directly from school districts in areas not present in state data
• Due in part to the relatively small number of highly-organized school districts that are acculturated to data-sharing and facilitating policy-motivated research
Two main special data opportunities in Florida
• Florida Education and Training Placement Information Program (FETPIP) and K-20 Education Data Warehouse– Established in 1988 to document compliance with
vocational education performance requirements– Expanded to become a system of collecting and
sharing individual-level data across all education sectors (K-12, vocational education, community colleges, state universities)
– Follows education, employment and earnings outcomes longitudinally for students – integrated into a relational database
Two main special data opportunities in Florida
• Independent extensive surveys of school policies and practices– Collaborative effort between David Figlio (U. Florida),
Dan Goldhaber (U. Washington), Jane Hannaway (Urban Institute) and Cecilia Rouse (Princeton U.)
– Attempted census of all public schools in Florida – First conducted in 1999-2000, then followed up in
2001-02 and 2003-04– Surveys of teachers in 2000-01, 2002-03, and 2004-
05 in 288 representative elementary schools– Better than 70 percent response rates in all waves;
over 80 percent longitudinal response rates
Examples of FETPIP data
• Data from public two- and four-year institutions, as well as in-state private colleges and universities
• In-state, data are collected on courses taken, programs of study, and attainment
• Florida is working with the National Student Clearinghouse to include enrollment and credential records for a large fraction of out-of-state college students
Examples of FETPIP data
• Using social security numbers, FETPIP marches student records with other outcome data for all students exiting Florida public institutions as well as some private exits
• Examples include information on further education, job placement, compensation, military service, incarceration
• Result: a remarkable tool for policy generation, evaluation and research
School Surveys
• All “regular” public schools in 2001-02 and 2003-04;
• 70%+ response rate in each year;
• 2,095 schools responded in 2002; 81% of these responded in 2004.
School policies/practices can be grouped into domains
• Policies to improve low-performing students
• Lengthening instructional time• Reduced class size for subject• Minimum time required for
tested subject instruction• Minimum time required for non-
tested subject instruction• Scheduling systems• Additional school resources
• Policies to improve low-performing teachers
• Teacher resources• Teacher incentives• Teacher autonomy• District control• Principal control• School climate
Feeling the Florida Heat? How Low-Performing Schools Respond to
Voucher and Accountability Pressure
Cecilia Elena RousePrinceton University & NBER
Jane HannawayThe Urban Institute
Dan GoldhaberUniversity of Washington
David N. FiglioUniversity of Florida & NBER
There is little evidence on how schools respond to accountability pressure…
• Improved teacher effectiveness and greater focus on basics;
• Teaching to the test;• Cheating;• Reclassification of students;• “Strategic” suspension of
students.
Our question: Do schools change their policies and practices in
response to school accountability and voucher pressure?
Our approach:
• Study effects of school accountability on student test score performance in Florida with a change in the A+ Plan for Education;
• Analyze effects of accountability on schools using longitudinal data on school policies collected from surveys of school principals in 1999-00, 2001-02, and 2003-04;
• Attempt to determine if the policy changes explain the test score effects.
We find…
• Among elementary schools, student achievement significantly increases among F-graded schools;
• F-graded schools appear to respond with policy changes;
• These policy changes appear to explain non-trivial portions of the student gains, particularly in math.
Since 2002, Grade Points =
percent students meeting levels 3+ in reading, writing, and math
+
percent students making “learning gains” in reading and math
+
the percentage of the bottom 25% that have improved scale points in reading.
Table 1: The Distribution of Elementary School Grades, by Year
School Year
School Grade Summer
1999 Summer
2000 Summer
2001 Summer
2002 Summer
2003 Summer
2004
Elementary Schools
A 119 485 389 623 928 974
B 214 180 324 368 360 333
C 713 614 636 452 299 284
D 448 260 215 124 63 67
F 61 4 0 35 18 9
N 0 0 46 68 2 0
Total 1555 1543 1610 1670 1670 1667
Table 2: Transition Matrix in Predicted Grades Based on 2002 Grade Change, Elementary Schools
(row percentages)
Grade in 2002 based on new (summer 2002) grading system
A B C D F
A 0.89 0.11 0.00 0.00 0.00
B 0.48 0.38 0.13 0.26 0.00
C 0.23 0.25 0.46 0.05 0.28
D 0.02 0.01 0.38 0.44 0.15
Simulated grade in
2002 based on
old (summer
2001) grading system F 0.00 0.00 0.00 0.29 0.71
Appendix Table 1 (part 1): Mean School Characteristics in 2002
2002 2004
Schools in
Frame
Schools in Survey
Schools in
Frame
Schools in Survey
Variable (1) (2) (3) (4) Proportion Classified
Disabled 0.156 0.155 0.156 0.159
Proportion Free or Reduced Lunch Eligible
0.541 0.534 0.539 0.531
Proportion Gifted Students 0.041 0.043 0.041 0.042 Proportion LEP 0.087 0.084 0.087 0.084 Stability Rate 0.931 0.932 0.931 0.933 Number of Students 687.289 685.795 687.339 693.963 Special Education
Expenditure per Pupil 8564.039 8458.324 8554.113 8507.815
Regular Education Expenditure per Pupil
4532.851 4495.650 4524.682 4479.649
At-risk Students Expenditure per Pupil
5624.279 5632.600 5644.243 5483.582
Vocational Education Expenditure per Pupil
127.674 120.055 128.304 120.149
Appendix Table 1 (part 2): Mean School Characteristics in 2004
2002 2004
Schools in
Frame
Schools in Survey
Schools in
Frame
Schools in Survey
Variable (1) (2) (3) (4)
School Grade in 2002 = A 0.355 0.366 0.352 0.358
School Grade in 2002 = B 0.220 0.221 0.220 0.228
School Grade in 2002 = C 0.291 0.286 0.289 0.288
School Grade in 2002 = D 0.072 0.068 0.072 0.066
School Grade in 2002 = F 0.024 0.023 0.024 0.022
Number of Schools 2699 2095 2837 2088
Table 3 (part 1): Regression-Discontinuity Estimates of the Effect of Receiving an F Grade in Summer 2002 on
Student Performance
Standardized test score
Model specification High-stakes
reading High-stakes
math Low-stakes
reading Low-stakes
math All elementary cohorts with lagged test scores
Standardized test performance in 2002-03
0.096 (0.029)
0.140 (0.032)
0.059 (0.021)
0.069 (0.025)
Fifth-grade cohort of 2002-03
Standardized test performance in 2002-03
0.099 (0.030)
0.142 (0.045)
0.069 (0.021)
0.076 (0.028)
Standardized test performance in 2002-03 – controlling instead for third grade test scores
0.098 (0.037)
0.136 (0.048)
0.071 (0.027)
0.075 (0.035)
Table 3 (part 2): Regression-Discontinuity Estimates of the Effect of Receiving an F Grade in Summer 2002 on
Student Performance
Standardized test score
Model specification High-stakes
reading High-stakes
math Low-stakes
reading Low-stakes
math
Fifth-grade cohort of 2002-03
Standardized test performance in 2003-04
0.058 (0.029)
0.104 (0.046)
0.044 (0.022)
0.047 (0.026)
Standardized test performance in 2004-05
0.079 (0.029)
0.096 (0.039)
0.058 (0.024)
0.054 (0.026)
-.5
0.5
200 300 400 500 600points_in_2002
Test scores of fifth graders in 2002-03math: quintic in summer 2002 points
-.4
-.2
0.2
.4
200 300 400 500 600points_in_2002
Test scores of fifth graders in 2002-03reading: quintic in summer 2002 points
Table 4 (part 1): Regression-Discontinuity Estimates of the Effect of Receiving an F Grade in Summer 2002 on
Student Performance
Standardized test score Model specification High-stakes
reading High-stakes
math Low-stakes
reading Low-stakes
math
Performance of 2002-03 fifth-grade cohort in three years following policy change (as compared with the preceding cohort)
Fifth grade (or equivalent) performance
0.140 (0.036)
0.212 (0.047)
0.074 (0.026)
0.122 (0.029)
Sixth grade (or equivalent) performance
0.051 (0.026)
0.119 (0.039)
0.053 (0.022)
0.101 (0.024)
Seventh grade (or equivalent) performance
0.088 (0.028)
0.118 (0.034)
0.081 (0.021)
0.078 (0.024)
Table 4 (part 2): Regression-Discontinuity Estimates of the Effect of Receiving an F Grade in Summer 2002 on
Subsequent Student Performance: Alternative specifications
Standardized test score
Model specification High-stakes
reading High-stakes
math Low-stakes
reading Low-stakes
math
Performance of 2002-03 fifth-grade cohort in third year (seventh grade or equivalent) following policy change (as compared with the preceding cohort): Alternative specifications
Comparing “F” schools to “D” schools in summer 2002 only
0.079 (0.045)
0.150 (0.064)
0.151 (0.037)
0.135 (0.039)
Restricting sample to those predicted to earn “D” grade under old system
0.120 (0.034)
0.139 (0.044)
0.117 (0.035)
0.127 (0.037)
Falsification test: Successive cohorts in fourth grade
-0.022 (0.029)
n/a 0.028
(0.029) 0.018
(0.028)
Table 7 (part 1): Seemingly-Unrelated Regression and OLS Results of the Effect of Receiving an F Grade in
Summer 2002 on School Policy in 2003-04 Seemingly-Unrelated Regression Index F vs. All F vs. D Domain (1) (2) (3) (4) (5) Policies to Improve Low-
Performing Students 0.438
(0.097) 0.354
(0.123) 0.339
(0.143) 0.347
(0.143)
0.207 (0.163)
Lengthening Instructional Time 0.278 (0.095)
0.205 (0.105)
0.283 (0.125)
0.285 (0.125)
0.249
(0.136) Reduced Class Size for Subject 0.559
(0.191) 0.535
(0.215) 0.329
(0.244) 0.335
(0.243)
0.082 (0.253)
Minimum Time Required for Tested Subject Instruction
0.213 (0.154)
0.058 (0.167)
0.027 (0.177)
0.028 (0.177)
0.238
(0.237) Minimum Time Required for Non-
Tested Subject Instruction -0.028
(0.185) -0.287 (0.223)
-0.290 (0.259)
-0.273 (0.260)
-0.217 (0.253)
Scheduling Systems 0.208 (0.120)
0.349 (0.116)
0.349 (0.139)
0.355 (0.139)
0.260
(0.139) School Resources 0.034
(0.240) -0.131 (0.354)
-0.193 (0.512)
-0.182 (0.497)
0.254
(0.374) Specifications also control for: 2002 “Simulated Grade” N Y Y Y Y Lagged dependent variable N Y Y Y Y Other school characteristics N Y Y Y Y 2002 grade points N N Y Y Y
Table 7 (part 2): Seemingly-Unrelated Regression and OLS Results of the Effect of Receiving an F Grade in
Summer 2002 on School Policy in 2003-04 Seemingly-Unrelated Regression Index F vs. All F vs. D Domain (1) (2) (3) (4) (5) Policies to Improve Low-
Performing Teachers 0.216
(0.081) 0.123
(0.086) 0.128
(0.093) 0.133
(0.094)
0.192 (0.151)
Teacher Resources 0.728 (0.391)
0.866 (0.453)
1.042 (0.525)
1.021 (0.530)
0.711
(0.651) Teacher Incentives 0.221
(0.161) 0.210
(0.171) 0.102
(0.203) 0.107
(0.203)
-0.042 (0.196)
Teacher Autonomy -0.060 (0.129)
-0.193 (0.141)
-0.083 (0.158)
-0.090 (0.157)
-0.153
(0.193) District Control -0.167
(0.172) -0.029
(0.179) 0.119
(0.209) 0.127
(0.208)
0.083 (0.197)
Principal Control 0.168 (0.129)
-0.067 (0.140)
-0.267 (0.161)
-0.264 (0.161)
-0.383 (0.200)
School Climate -0.455 (0.125)
-0.028 (0.136)
0.108 (0.157)
0.104 (0.156)
0.073
(0.163) Specifications also control for: 2002 “Simulated Grade” N Y Y Y Y Lagged dependent variable N Y Y Y Y Other school characteristics N Y Y Y Y 2002 grade points N N Y Y Y
Table 8 (part 1): OLS Results of the Impact of Receiving an F Grade in Summer 2002 on School
Selected Individual Policies in 2003-04
F vs. All F vs. D
No Covariates With Covariates Variable (1) (2) (3)
Average 1st and 4th grade class size -2.934 (0.733)
-0.501 (0.912)
-0.488 (0.913)
Require tutoring for low-performing students
0.303 (0.095)
0.204
(0.123)
0.200 (0.123)
Require Saturday classes for low-performing students
0.228 (0.055)
0.111
(0.067)
0.115 (0.067)
Use block scheduling 0.250
(0.097)
0.227 (0.115)
0.227
(0.115)
Use common prep period 0.010
(0.054)
0.110 (0.070)
0.113
(0.070)
Use other scheduling structure 0.067
(0.077)
0.179 (0.105)
0.182
(0.105)
Table 8 (part 2): OLS Results of the Impact of Receiving an F Grade in Summer 2002 on School
Selected Individual Policies in 2003-04
F vs. All F vs. D No Covariates With Covariates Variable (1) (2) (3)
Minutes per week for collaborative planning/class preparation
21.829 (45.396)
117.834 (61.733)
116.716 (61.782)
District control of budget spending -0.181 (0.208)
0.418
(0.267)
0.432 (0.267)
Reduced class size gifted academic performance
-0.245 (0.099)
-0.051 (0.127)
-0.051 (0.127)
Whole school reform model -0.056 (0.085)
-0.025 (0.110)
-0.026 (0.110)
Minimum time spent on science 0.102
(0.097)
-0.094 (0.120)
-0.085 (0.120)
Table 9 (part 1): The Effect of Including School Policy/Practice Variables on Regression-Discontinuity Estimates of the Effect of Receiving an F Grade in Summer 2002 on Subsequent Student
Performance: Fifth-Grade Cohort of 2002-03 Standardized test score
Model specification High-stakes
reading High-stakes
math Low-stakes
reading Low-stakes
math Fifth grade (or equivalent) performance Models excluding school policy/practice variables
0.100
(0.029) 0.135
(0.045) 0.067
(0.021) 0.073
(0.028)
Models including school policy/practice variables
0.090
(0.036) 0.102
(0.056) 0.056
(0.031) 0.018
(0.032)
p-value of joint significance of policy variables requiring resources
0.004 0.117 0.010 0.198
p-value of joint significance of policy variables aimed at greater efficiency
0.001 0.002 0.002 0.000
Sixth grade (or equivalent) performance Models excluding school policy/practice variables
0.059
(0.029) 0.102
(0.045) 0.047
(0.022) 0.048
(0.027) Models including school policy/practice variables
0.048
(0.036) 0.055
(0.060) 0.038
(0.028) 0.022
(0.033)
p-value of joint significance of policy variables requiring resources
0.000 0.001 0.019 0.011
p-value of joint significance of policy variables aimed at greater efficiency
0.004 0.000 0.006 0.029
Table 9 (part 2): The Effect of Including School Policy/Practice Variables on Regression-Discontinuity Estimates of the Effect of Receiving an F Grade in Summer 2002 on Subsequent Student
Performance: Fifth-Grade Cohort of 2002-03
Standardized test score
Model specification High-stakes
reading High-stakes
math Low-stakes
reading Low-stakes
math Seventh grade (or equivalent) performance
Models excluding school policy/practice variables
0.079
(0.030) 0.094
(0.039) 0.059
(0.025) 0.045
(0.027)
Models including school policy/practice variables
0.070
(0.037) 0.068
(0.043) 0.039
(0.027) 0.016
(0.029) p-value of joint significance of policy variables requiring resources 0.018 0.032 0.008 0.076
p-value of joint significance of policy variables aimed at greater efficiency 0.005 0.001 0.047 0.011
In sum…
• We estimate effect sizes in reading test scores among “F-graded” schools between 6-10σ and effect sizes in math between 7-14σ.
• We also find that “F-graded” schools appear to focus on low-performing students, lengthen the amount of time devoted to instruction, adopt different ways of organizing the instructional environment of students and teachers, increase resources available to teachers, and decrease principal control.
• These policies may explain at least 10% of the gains in reading and at least 25% of the gains in math.
Caveats….
• F-graded schools receive additional state assistance (e.g., assessment and course materials, increased professional development for teachers);
• While our results suggest that schools respond to accountability in potentially educationally meaningful ways, we do not observe student performance along all dimensions.