Download - Fryer Apollo Presentation
The Impact of Apollo 20 on Student Achievement:
Evidence from Year One by
Roland G. Fryer, Jr. Robert M. Beren Professor of Economics
Harvard University EdLabs NBER
CONFIDENTIAL
Outline Apollo 20
Road-‐map for the talk: A. The Achievement Gap in America B. The Five Tenets: Mo=va=on and Implementa=on
1. More Time in School 2. Small-‐Group Tutoring and Differen=a=on 3. Human Capital Management 4. Data-‐Driven Instruc=on 5. Culture and High Expecta=on
C. Explana=on of Empirical Methodology D. Year One Results E. Verifica=on Checks F. Context and Wrap-‐up
CONFIDENTIAL 3
Accoun&ng for educa&onal achievement dras&cally reduces racial and socioeconomic inequality across a wide range of important life outcomes.
28.0%
0.6% 0.0%
10.0%
20.0%
30.0%
Not Accounting for Educational Achievement
Accounting for Educational Achievement
The Black-White Wage Gap
182%
38%
0%
100%
200%
300%
Not Accounting for Educational Achievement
Accounting for Educational Achievement
The Black-White Incarceration Rate Gap
190%
90%
0%
100%
200%
300%
Not Accounting for Educational Achievement
Accounting for Educational Achievement
The Black-White Unemployment Gap
0.196
0.059
0
0.1
0.2
0.3
Not Accounting for Educational Achievement
Accounting for Educational Achievement
The Black-White Health Gap (in Standard Deviations)
Overview Educa5onal Achievement Drives Life Outcomes
CONFIDENTIAL
Among ci&es that par&cipate in NAEP, the magnitude of racial differences in educa&onal achievement is startling.
Overview The Achievement Gap
0 10 20 30 40 50 60 70 80
Math
White
Black
Hispanic
0 10 20 30 40 50 60 70 80
Reading
White
Black
Hispanic
CONFIDENTIAL
What’s more, “conven&onal wisdom” solu&ons have failed to solve the problem.
22.3 :1
20.4 :1
18.7 :1
17.9 :1 17.2 :1 17.3 :1
16.0 :1 15.6 :1
14:1
16:1
18:1
20:1
22:1
24:1
1970 1975 1980 1985 1990 1995 2000 2005
Student to Teacher Ra5o
$5,243 $6,049 $6,268
$7,347 $8,790 $8,949
$10,508 $11,438
$0
$5,000
$10,000
1970 1975 1980 1985 1990 1995 2000 2005
Total Expenditure Per Pupil (2008-‐09 $)) $12,116
23.5% 27.5% 49.6% 53.1% 56.8% 61.8%
0%
20%
40%
60%
80%
100%
1961 1971 1981 1991 2001 2006
Percentage of Teachers with a Master's Degree or Higher
76.9% 73.6% 71.4% 72.4% 73.4% 69.3% 69.8% 75.4% 74.7%
0%
25%
50%
75%
100%
1970 1975 1980 1985 1990 1995 2000 2005
HS Graduates as a ra5o of 17 year-‐old popula5on
175
200
225
250
275
300
325
1971 1975 1980 1984 1988 1990 1994 1996 1999 2004 2008
Reading and Math Achievement of 9, 13, and 17 year-‐olds, 1971-‐2008
9 year-‐olds
13 year-‐olds
Overview Conven5onal Wisdom
CONFIDENTIAL
Given the current rate of “No Excuses” charter school growth, charters will take over half a century to educate every child in America and close the achievement gap. That’s too long. We are faced with two op=ons:
1. Create a market in which only gap closing charters can survive (great idea, tough poli=cs).
2. Boil down charter school successes into translatable, scalable prac=ces for public schools.
Scaling Up Charter Success Apollo 20
CONFIDENTIAL
Scaling Up Charter Success Apollo 20
One of the original missions of charter schools was to experiment and incubate best prac=ces that could then be transferred to public schools. Dobbie and Fryer (2011a) examine results from 106 charter schools in New York City in order to iden=fy the prac=ces most correlated with student achievement. They find that accoun=ng for five factors explains roughly 40% of the variance in charter school performance.
CONFIDENTIAL 8
The key strategies of Apollo schools.
Scaling Up Charter Success Apollo 20 Strategies
More Time in School • Extended day, week, and school years are all integral components of successful school
models. In the case of Harlem Children’s Zone’s Promise Academy, students have nearly doubled the amount of time on task compared to students in NYC public schools.
Small Group Tutoring and Differentiation • In top performing schools, classroom instruction is supplemented by individualized tutoring,
both after school and during the regular school day.
Human Capital Management • Successful charters reward teachers for performance and hold them accountable if they are
not adding value.
Data-Driven Instruction • In the top charter schools, students are assessed frequently, and then, in small groups, re-
taught the skills they have not yet mastered.
Culture and Expectations • In successful schools, students buy into the school’s mission and into the importance of
their education in improving their lives.
CONFIDENTIAL
Implementa5on Human Capital
In addi=on to finding nine new principals, teacher turnover spiked to 53% in Apollo schools over the summer of 2010. Value-‐added data shows that teachers who returned as Apollo teachers had a much stronger history of increasing student achievement in every subject, rela=ve to those who lea.
-‐0.7
-‐0.6
-‐0.5
-‐0.4
-‐0.3
-‐0.2
-‐0.1
0
0.1
0.2
Language Math Reading Science Social Studies
Teacher Value Added
Lea
Stayed
0%
10%
20%
30%
40%
50%
60%
2005-‐06 2006-‐07 2007-‐08 2008-‐09 2009-‐10
Teacher Departure Rates
Apollo Schools
Comparison Schools
CONFIDENTIAL
Implementa5on Increased Time in School
The school day was extended in Apollo schools during the 2010-‐11 school year: 7:45am – 4:15pm Monday through Thursday, and 7:45am – 3:15pm on Fridays. This was an average of an hour longer per school day. The school year was extended by five school days. Apollo students reported for school on August 16, 2010, while the rest of the district began on August 23, 2010.
BoXom line: The difference between instruc=onal =me in 2009-‐10 and 2010-‐11 amounts to approximately 30 school days – that’s 6 addi5onal weeks of school for students.
0
200
400
600
800
1000
1200
1400
1600
1800
Pre-‐Interven=on Typical NYC Charter Post-‐Interven=on No Excuses NYC Charter
Instruc5onal Hours per Year
CONFIDENTIAL
Implementa5on High Dosage Differen5a5on: Tutoring and Double-‐Dosing
Tutors were recruited both na=onally and locally during summer 2010. • 1158 total applicants (through August 9, 2010) • 516 applicants par=cipated in screening • 319 offered a posi=on • 257 accepted a posi=on in one of the nine schools • 173 tutors qualified for end-‐of-‐year student performance bonuses, earning an
average of $3,346. • 96 tutors are returning for the 2011-‐12 school year • 1156 6th graders and 1585 9th graders received 2:1 tutoring in the 2010-‐11 school
year. Middle school students received approximately 215 hours of tutoring/double-‐dosing High school students received approximately 189 hours of tutoring/double-‐dosing
CONFIDENTIAL
Implementa5on Data-‐Driven Instruc5on
Students were regularly assessed in order to provide teachers with accurate data on student mastery. In addi=on to the HISD 3-‐week interim assessments, the Apollo schools administered comprehensive benchmark assessments according to the schedule: • Middle school reading – December, February/March • Middle school math – December, February/March • Middle school science – December, March • Middle school social studies – December, March • High school ELA – December, January • High school math – December, March • High school science – December, March • High school social studies – December, March Aaer each assessment, teachers received student-‐level data and used this to have one-‐on-‐one goal-‐sehng conversa=ons with students.
CONFIDENTIAL
Implementa5on Culture and Expecta5ons
There was a consistent emphasis on college matricula=on at the Apollo schools, and students were supported in their efforts to apply to college.
Apollo High Schools College Applica5on and Acceptance Rates
2-‐Year 4-‐Year
Applied Accepted Applied Accepted
Jones HS 95% 95% 60% 44%
Kashmere HS 90% 90% 89% 42%
Lee HS 97% 97% 84% 47%
Sharpstown HS 94% 94% 81% 55%
“It feels like we’re actually going to school now.” – Lee HS student
“We have supports this year that we didn’t have in the past.” – Sharpstown HS student
“School is harder…but it’s good because we’ll be ready.” – Key MS student
CONFIDENTIAL
Implementa5on Culture and Expecta5ons
At the end of the 2009-‐10 school year, TNTP interviewed teachers in schools that would become Apollo schools for 2010-‐11. Teachers who returned as Apollo teachers scored higher on every type of ques=on, including their commitment to the “No Excuses” philosophy and beliefs that all students are capable of high achievement.
0
0.5
1
1.5
2
2.5
3
3.5
No Excuses Alignment with Mission
Student Achievement
Commitment to Students
Student Mo=va=on
Interview Response Scores
Lea
Stayed
CONFIDENTIAL
Given that we don’t have a randomized experiment in year one, we use several empirical methods to assess the impact of Apollo 20.
Research Design Empirical Strategies
• Ordinary Least Squares Regression
• Advantages: Simple and easy to interpret • Disadvantages: Assumes linear func=onal form to allow comparisons between students
of different sta=s=cal profiles, cannot control for unobservable differences between treatment and control students
• Nearest Neighbor Matching
• Using the distance func=on defined above, we locate each students’ four closest sta=s=cal neighbors and compare them to the group average.
• Advantages: No func=onal form assump=on, focuses aoen=on on students from similar backgrounds
• Disadvantages: Cannot control for unobservable differences between treatment and control students.
igisgsi XApolloscore εγβββ ++⋅+⋅+= 210,,
)()(),( jiji XXVXXjid −ʹ′−=
CONFIDENTIAL
Research Design Empirical Strategies
• Difference in Differences
• Advantages: By focusing on year-‐over-‐year changes in scores, we account for baseline varia=on that we do not otherwise observe.
• While these methods have their advantages, all three will be biased if students selec=vely opt in or out of Apollo schools.
sigisgsi XApolloscore ,210,, εγβββ ++⋅+⋅+=Δ
CONFIDENTIAL
Research Design Selec5on Issues
Incoming Sixth Graders
Incoming Ninth Graders
CONFIDENTIAL
• To account for the decline in baseline ability in Apollo schools, we need a way to iden=fy which students would have enrolled in the absence of such significant changes. This leads us to our final empirical strategy, which uses a student’s home enrollment zone to predict which students are likely to aoend Apollo schools.
• Two-‐Stage-‐Least-‐Squares Difference in Differences
• Advantages: Solves selec=on issue. • Disadvantages: Possibly vulnerable to cri=ques based on the exclusion
restric=on.
igiigsi XzonedApollo εγβββ ++⋅+⋅+= 210,,
Research Design Empirical Strategies
CONFIDENTIAL
Results Main Results
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
6 7 & 8 9 10 & 11
TAKS Math Results
OLS
NNM
DID
2SLS
In Math, we see posi=ve and sta=s=cally significant results in both middle and high school. The gains in grades that received high-‐dosage tutoring were drama=c.
CONFIDENTIAL
In Reading/ELA, the results are mixed. While high schools performed extremely well, there is liole evidence of success in middle school – indeed some es=mates are nega=ve.
-‐0.1
-‐0.05
0
0.05
0.1
0.15
0.2
0.25
6 7 & 8 9 10 & 11
TAKS Reading
OLS
NNM
DID
2SLS
Results Main Results
CONFIDENTIAL
Verifica5on Alternate Test
The results presented so far are extremely promising. However, some have argued that the gains on state exams can be driven by test-‐specific preparatory ac=vi=es at the expense of general student learning (Jacob 2005, Heilig and Darling-‐Hammond 2008). To provide some evidence on this maoer, we also looked at Stanford 10 scores. This test is na=onally normed and not =ed to any accountability measure, so the incen=ves to teach to the test or otherwise manipulate results are minimal
CONFIDENTIAL
Stanford 10 tests show a similar paoern to TAKS, sugges=ng that our results are not driven by teaching to the test.
-‐0.05
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
6 7 & 8 9 10 & 11
Stanford Math
OLS
NNM
DID
2SLS
-‐0.2
-‐0.15
-‐0.1
-‐0.05
0
0.05
0.1
0.15
0.2
6 7 & 8 9 10 & 11
Stanford Reading
OLS
NNM
DID
2SLS
Verifica5on Alternate Test
CONFIDENTIAL
A second poten=al threat to our results is aori=on from our sample. Our main results use the sample of students who enroll in an Apollo school or a comparison school at the beginning of the 2010-‐11 school year and for whom we have test score data in the spring of 2011. If Apollo students are more likely to miss the Spring 2011 TAKS examina=on, this could bias our es=mates. Given the major and sudden changes that occurred in Apollo schools, it is plausible that parents might choose to move their children to a private school or a charter school like KIPP or YES. In these instances, we would not observe their test scores. To examine this issue, we es=mate the impact of Apollo on the probability of entering our analysis sample.
Verifica5on AXri5on
CONFIDENTIAL
Verifica5on AXri5on
We es=mate that Apollo students are 0.6% more likely to miss an exam than comparable students in comparison schools, though this es=mate is not sta=s=cally significant. Dropping the top 0.6% of Apollo students from the analysis does not change our findings. For the difference-‐in-‐difference es=mates, we must also be able to match students to their 2009-‐10 score to calculate a trend. We find that Apollo students are roughly 4% more likely to be missing baseline scores. Since we are able to replicate our DID results with other models that do not require baseline scores, this is not problema=c.
Outcome Marginal Effect Outcome Marginal Effect
Missing 2011 Math .006 Missing 2010 Math .040***
(.005) (.017)
Missing 2011 Rdg. .006 Missing 2010 Rdg. .038
(.005) (.017)
Notes: Standard Errors reported in parentheses.
CONFIDENTIAL
Apollo schools were also subjected to unprecedented levels of scru=ny and accountability – the nine principals, in par=cular, earned bonuses for mee=ng performance goals and could be terminated if they fell short. One might worry that such a high-‐stakes environment might induce school officials to cheat when repor=ng scores. Accordingly, we examined student responses on the 2010-‐11 TAKS and ranked every grade in every school according to four measures designed to detect suspicious paoerns in student answers:
1. Unusual blocks of consecu=ve iden=cal answers given by mul=ple test-‐takers 2. Unlikely correla=on in answer responses within grades 3. Unusually high variance in these correla=ons 4. Unlikely combina=ons of correct answers – for instance, gehng easy
ques=ons wrong and hard ques=ons correct
Verifica5on Chea5ng Analysis
CONFIDENTIAL
Verifica5on Chea5ng Analysis
Aaer adding the ranks of all schools and grades on each of these four measures, we find that the average Apollo school is in the 43.9 percen=le in math and 43.1 percen=le in reading.
CONFIDENTIAL
Results Subcomponents
0
0.1
0.2
0.3
0.4
0.5
0.6
6 9
Tutoring Effec5veness
Above-‐Grade Comparison
Reading Comparison
-‐0.1
-‐0.05
0
0.05
0.1
0.15
0.2
0.25
0.3
7 8 10 11
Double-‐Dosing Effec5veness
Math
Reading
Unsurprisingly, our triple-‐difference analysis shows that tutoring was highly effec=ve in increasing achievement. The double dosing results, however, are all sta=s=cally zero except for a large posi=ve effect in eighth grade math.
CONFIDENTIAL
Results Context
The Apollo results are strikingly similar to the effects of aoending the Harlem Children’s Zone Promise Academy Middle School, KIPP Lynn Middle School, and a group of “No Excuses” Boston-‐area charters.
Apollo 20 Harlem
Children’s Zone
KIPP Lynn Boston Charter Schools
Oversubscribed All Urban
6th Grd. Math .484 .249
MS Math .234 .229 .346 .359 .148
MS Reading -‐.014 .047 .120 .198 .098
9th Grd. Math .739
HS Math .368 .265 .122
HS Reading .189 .364 .169
Sources: Dobbie and Fryer (2010), Dobbie and Fryer (2011b), Abdulkadiroglu et al (2011), Angrist et al. (2010)
CONFIDENTIAL
Results Return on Investment
Implemen=ng Apollo cost roughly $2,042 per student. A back-‐of-‐the-‐envelope calcula=on shows that that internal rate of return from this investment is 20.16%. Compared to other educa=onal interven=ons, this return is large.
Ini5a5ve IRR
Apollo 20 20.16 %
“No Excuses” Charter School 18.50 %
Early Childhood Educa=on 7.99 %
Reduced Class Size 6.20 %
Sources: Curto and Fryer (2011), Heckman (2010), Krueger (2003)
CONFIDENTIAL
Lessons Learned
The main takeaway from Apollo Program in 2010-‐11 is that a fully loaded Apollo program in which all five tenets are implemented with fidelity produces large, significant gains in student performance. We must think hard about how to scale-‐up tutoring in a manner that maximized student performance while controlling the costs of the program.
CONFIDENTIAL
References
Abdulkadiroglu, A=la, Joshua Angrist, Susan Dynarski, Thomas J. Kane, and Parag Pathak (2011), “Accountability in Public Schools: Evidence from Boston’s Charters and Pilots”, forthcoming in Quarterly Journal of Economics. Angrist, Joshua D., Susan M. Dynarski, Thomas J. Kane, Parag A. Pathak, and Christopher R. Walters. 2010. "Inputs and Impacts in Charter Schools: KIPP Lynn." American Economic Review Papers and Proceedings, 100(2): 239–43. Dobbie, Will and Fryer, Roland G. (2011a), “Are High Quality Schools Enough to Increase Achievement Among the Poor? Evidence From the Harlem Children’s Zone”, Forthcoming in American Economic Journal: Applied Economics. Dobbie, Will and Fryer, Roland G. (2011a), “Are High Quality Schools Enough to Increase Achievement Among the Poor? Evidence From the Harlem Children’s Zone”, Working Paper No. 15473 (NBER, Cambridge, MA) Dobbie, Will and Fryer, Roland G. (2011b), “School Characteris=cs and Student Achievement: Evidence from NYC Charter Schools”, Unpublished Manuscript (Harvard University). Heckman, James J., Seong Hyeok Moon, Rodrigo Pinto, Peter A. Savelyev, and Adam Yavitz, “The Rate of the Return to the HighScope Perry Preschool Program”, Journal of Public Economics, 94: 114–128, (2010). Heilig, Julian V. and Linda Darling-‐Hammand (2008), “Accountability Texas Style: The Progress and Learning of Urban Minority Students in a High-‐Stakes Tes=ng Context”, Educa&onal Evalua&on and Policy Analysis, 30(2): 75-‐110. Jacob, Brian A. (2005), “Accountability, incen=ves and behavior: the impact of high-‐stakes tes=ng in the Chicago Public Schools”, Journal of Public Economics 89: 761-‐796. Krueger, Alan B. (2003) , “Economic Considera=ons and Class Size,” The Economic Journal, 113, F34—F63.