does school choice work? effects on integration and student achievement

1 PPIC PPIC

Does School Choice Work? Does School Choice Work? Effects on Integration Effects on Integration

and Student Achievementand Student Achievement

Julian R. Betts, Lorien A. Rice, Julian R. Betts, Lorien A. Rice, Andrew C. Zau, Y. Emily Tang, and Andrew C. Zau, Y. Emily Tang, and

Cory R. KoedelCory R. Koedel

2 PPIC PPIC

Overview of Entire ReportOverview of Entire Report

Entire book is downloadable from Entire book is downloadable from www.ppic.org

Today will focus on lottery-based assessment Today will focus on lottery-based assessment of impact of choice on math and reading of impact of choice on math and reading achievementachievement

Will briefly highlight other aspects of report:Will briefly highlight other aspects of report:

• ““Who chooses”Who chooses”

• Effects on integrationEffects on integration

• Non-lottery study of charters and Non-lottery study of charters and achievementachievement

3 PPIC PPIC

Specific to this School Choice Study (I)Specific to this School Choice Study (I)

Many of the previous studies of effect of choice on Many of the previous studies of effect of choice on achievement lack convincing statistical identification achievement lack convincing statistical identification

– Compare switchers to those students left behindCompare switchers to those students left behind

– Compare switchers to local students at current Compare switchers to local students at current schoolschool

This study uses lottery data to study the three main This study uses lottery data to study the three main choice programs. Compares lottery winners to choice programs. Compares lottery winners to loserslosers

– Student fixed effects models for charter schools Student fixed effects models for charter schools only that compare individual student to herself in only that compare individual student to herself in years before switch years before switch

4 PPIC PPIC

Choice Programs in San DiegoChoice Programs in San Diego

Busing availableBusing available

– VEEP (Voluntary Ethnic Enrollment Program)VEEP (Voluntary Ethnic Enrollment Program)

– MagnetsMagnets

• Both are legacies of 30-year-old court Both are legacies of 30-year-old court desegregation orderdesegregation order

– NCLB-ordered busing from “failing schools” NCLB-ordered busing from “failing schools”

• Not studied: minor program, tiny enrollmentsNot studied: minor program, tiny enrollments

No busingNo busing

– Choice (SChoice (State-mandated open enrollment program)tate-mandated open enrollment program)

– Charter Schools (1 charter pays district for busing)Charter Schools (1 charter pays district for busing)

5 PPIC PPIC

Between 2001 and 2003, School Choice Between 2001 and 2003, School Choice Enrollment Share Rose from 25% to 28%Enrollment Share Rose from 25% to 28%

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8

VEEP

Choice (Open Enrollment)

Magnet (Non-resident)

NCLB Choice

Charters

Traditional (non-choice)

2003 2001

6 PPIC PPIC

OutlineOutline

Introduction Introduction

Who Chooses to Leave and Why?Who Chooses to Leave and Why?

The Effect of School Choice on IntegrationThe Effect of School Choice on Integration

The Effect of Choice on AchievementThe Effect of Choice on Achievement

Policy ImplicationsPolicy Implications

7 PPIC PPIC

Who Exercises Choice - Who Exercises Choice - Personal CharacteristicsPersonal Characteristics

Non-whitesNon-whites– Sometimes, but not always, more likely to apply Sometimes, but not always, more likely to apply

BlacksBlacks– Always more likely to applyAlways more likely to apply

• Twice the average probability of applying to VEEP high Twice the average probability of applying to VEEP high schoolsschools

English LearnersEnglish Learners– Generally less likely to applyGenerally less likely to apply

Are students with higher test scores, GPA or higher parental Are students with higher test scores, GPA or higher parental education being skimmed off?education being skimmed off?– Mixed evidence, weak effectsMixed evidence, weak effects

8 PPIC PPIC

Who Exercises Choice – Who Exercises Choice – School CharacteristicsSchool Characteristics

Weak evidence that students base application Weak evidence that students base application decisions on academic characteristics of local and decisions on academic characteristics of local and option schools option schools

Distance is typically a barrierDistance is typically a barrier

– At high school level, distance is strongest At high school level, distance is strongest deterrent in deterrent in ChoiceChoice (open enrollment) program, (open enrollment) program, which provides no busingwhich provides no busing

9 PPIC PPIC

OutlineOutline






10 PPIC PPIC

Integration ResultsIntegration Results

Overall, choice programs…Overall, choice programs…

– Integrate along race/ethnicity and parental education linesIntegrate along race/ethnicity and parental education lines

– Segregate slightly along test score and English Learner Segregate slightly along test score and English Learner (EL) lines(EL) lines

Decreasing Rank of integrating efficacyDecreasing Rank of integrating efficacy

– VEEP, Magnets, VEEP, Magnets, ChoiceChoice (no busing) (no busing)

• ChoiceChoice actually lowers integration along race, parental actually lowers integration along race, parental education, test score and language status lineseducation, test score and language status lines

Limited supply of places sharply reduces actual integrationLimited supply of places sharply reduces actual integration

11 PPIC PPIC

OutlineOutline

IntroductionIntroduction





12 PPIC PPIC

OverviewOverview

Lotteries represent a true quasi-experimental Lotteries represent a true quasi-experimental way of measuring impact of choice on way of measuring impact of choice on choosers because selectivity bias is removedchoosers because selectivity bias is removed

Model reading and math test scores one, two Model reading and math test scores one, two and three years after the admissions lotteries and three years after the admissions lotteries for Fall 2001for Fall 2001

– Two measures of math achievement per Two measures of math achievement per year, and three measures of reading year, and three measures of reading achievement per yearachievement per year

– Three types of school choiceThree types of school choice

13 PPIC PPIC

How Lotteries Work in San DiegoHow Lotteries Work in San Diego

Each application is assigned a random number Each application is assigned a random number

Applications to a given school and grade are Applications to a given school and grade are sorted into priority groups (e.g. top priority goes sorted into priority groups (e.g. top priority goes to siblings), so potentially multiple lotteries per to siblings), so potentially multiple lotteries per school/grade.school/grade.

– Table C.1 lists priority groupsTable C.1 lists priority groups

Central administration admits in descending Central administration admits in descending order of priority. Typically one priority group per order of priority. Typically one priority group per grade undergoes a lotterygrade undergoes a lottery

14 PPIC PPIC

Preliminary Analysis for the Fall 2001 Preliminary Analysis for the Fall 2001 Admissions LotteriesAdmissions Lotteries

First, check which potential lotteries are First, check which potential lotteries are “true” lotteries, in sense that proportion p “true” lotteries, in sense that proportion p winning admission obeys 0 < p < 1winning admission obeys 0 < p < 1

– Generally a greater proportion of lotteries Generally a greater proportion of lotteries are “true” in grades above grade 3are “true” in grades above grade 3

• Elementary VEEP in particular: very Elementary VEEP in particular: very small sample of students in true lotteries small sample of students in true lotteries relative to other casesrelative to other cases

• Table C.2 Table C.2

15 PPIC PPIC

Preliminary Analysis for the Fall 2001 Preliminary Analysis for the Fall 2001 Admissions Lotteries (II)Admissions Lotteries (II)

Second, for students in the subsample of Second, for students in the subsample of lotteries that are “true”, need to check that lotteries that are “true”, need to check that lotteries are “fair”.lotteries are “fair”.

– Should be no statistically significant Should be no statistically significant difference in initial (spring 2001) test difference in initial (spring 2001) test scores between lottery winners and losersscores between lottery winners and losers

– A regression of initial test scores on lottery A regression of initial test scores on lottery dummies and a dummy for “winning” dummies and a dummy for “winning” showed the latter to be insignificantshowed the latter to be insignificant

16 PPIC PPIC

Preliminary Analysis for the Fall 2001 Preliminary Analysis for the Fall 2001 Admissions Lotteries (III)Admissions Lotteries (III)

Additional tests that lotteries are “fair”.Additional tests that lotteries are “fair”.

– Repeated for 178 regression samples and Repeated for 178 regression samples and null is rejected at roughly the size of the null is rejected at roughly the size of the test, as expected (e.g. 0.6% of time is test, as expected (e.g. 0.6% of time is rejection at the 1% level)rejection at the 1% level)

– Table C.3Table C.3

17 PPIC PPIC

Lottery Regression ModelsLottery Regression Models

Will focus on models of the “intent to treat”. Will focus on models of the “intent to treat”.

– Let J denote number of true lotteries. Let J denote number of true lotteries.

– We will model the test score for student i We will model the test score for student i in year t, where t is one of the post-lottery in year t, where t is one of the post-lottery years 2002, 2003 and 2004. This student years 2002, 2003 and 2004. This student applies to lottery j, and in year t attends applies to lottery j, and in year t attends school s, so his score is denoted by Sschool s, so his score is denoted by S ijstijst. .

18 PPIC PPIC

Lottery Regression ModelsLottery Regression Models

Simplest model: Simplest model:

Model includes:Model includes:

– a set of dummy variables αa set of dummy variables αjj for the lotteries for the lotteries

– a dummy variable WINa dummy variable WINijtijt and corresponding and corresponding

coefficient β indicating whether the student i, whose coefficient β indicating whether the student i, whose test score is modeled in year t, won lottery j, and test score is modeled in year t, won lottery j, and

– an error term consisting of an error component for an error term consisting of an error component for school s in year t, ηschool s in year t, ηstst, and a white noise error term ε, and a white noise error term εijst:ijst:

J

jijststijtjijst WINS

1

19 PPIC PPIC

Robustness Check: Estimate 4 Robustness Check: Estimate 4 SpecificationsSpecifications

In addition to the previous model (1), add:In addition to the previous model (1), add:

– (2) Same as (1) but add spring 2001 test (2) Same as (1) but add spring 2001 test scorescore

– (3) Same as (2) but add square of spring (3) Same as (2) but add square of spring 2001 test score2001 test score

– (4) Same as (3) but add personal (4) Same as (3) but add personal characteristicscharacteristics

– (5) Same as (4) but add classroom (5) Same as (4) but add classroom characteristics (teacher qualifications etc.) characteristics (teacher qualifications etc.) to see if we can explain any of the to see if we can explain any of the differences accruing to lottery winnersdifferences accruing to lottery winners

20 PPIC PPIC

Preview of ResultsPreview of Results

Few differences across specificationsFew differences across specifications

– Will focus on model (2) which adds lagged Will focus on model (2) which adds lagged test score to improve precisiontest score to improve precision

– Results for all four specifications Results for all four specifications appear in Appendix Cappear in Appendix C

21 PPIC PPIC

Effects of Winning a VEEP Lottery: CST Reading Effects of Winning a VEEP Lottery: CST Reading AchievementAchievement

VEEP Reading

0

500

1000

1500

2000

2500

3000

3500

01-02

02-03

03-04

01-02

02-03

03-04

01-02

02-03

03-04

01-02

02-03

03-04

Elementary

Sample Size

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

1.2

Middle

Proportion of a

Standard Deviation

High All Grades

22 PPIC PPIC

Effects of Winning a Magnet Lottery: CST Effects of Winning a Magnet Lottery: CST Reading AchievementReading Achievement

Magnet Reading

0

500

1000

1500

2000

2500

3000

3500

01-02

02-03

03-04

01-02

02-03

03-04

01-02

02-03

03-04

01-02

02-03

03-04

SampleSize

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

1.2

Middle

Proportion of a

Standard Deviation

Elementary High All Grades

23 PPIC PPIC

Effects of Winning an Open Enrollment (Choice) Effects of Winning an Open Enrollment (Choice) Lottery: CST Reading AchievementLottery: CST Reading Achievement

Choice Reading

0

500

1000

1500

2000

2500

3000

3500

01-02

02-03

03-04

01-02

02-03

03-04

01-02

02-03

03-04

01-02

02-03

03-04

Elementary

SampleSize

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

1.2

Middle

Proportion of a

Standard Deviation

High All Grades

24 PPIC PPIC

Effects of Winning a VEEP Lottery: CST Math Effects of Winning a VEEP Lottery: CST Math AchievementAchievement

VEEP Math

0

500

1000

1500

2000

2500

3000

3500

01-02

02-03

03-04

01-02

02-03

03-04

01-02

02-03

03-04

01-02

02-03

03-04

Elementary

SampleSize

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

Middle

Proportionof a

Standard Deviation

High All Grades

25 PPIC PPIC

Effects of Winning a Magnet Lottery: CST Math Effects of Winning a Magnet Lottery: CST Math AchievementAchievement

Magnet Math

0

500

1000

1500

2000

2500

3000

3500

01-02

02-03

03-04

01-02

02-03

03-04

01-02

02-03

03-04

01-02

02-03

03-04

Elementary

SampleSize

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

Middle

Proportion of a

Standard Deviation

High All Grades

26 PPIC PPIC

Effects of Winning an Open Enrollment (Choice) Effects of Winning an Open Enrollment (Choice) Lottery: CST Math AchievementLottery: CST Math Achievement

Choice Math

0

500

1000

1500

2000

2500

3000

3500

01-02

02-03

03-04

01-02

02-03

03-04

01-02

02-03

03-04

01-02

02-03

03-04

Elementary

SampleSize

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

Middle

Proportionof a

Standard Deviation

High All Grades

27 PPIC PPIC

Results of Choice on AchievementResults of Choice on Achievement

In general, winning a lottery has no statistically In general, winning a lottery has no statistically significant effect on reading or math achievement significant effect on reading or math achievement one to three years afterone to three years after

Important exceptionsImportant exceptions

– In some cases lottery winners fare slightly worse In some cases lottery winners fare slightly worse in first year, but recover to identical achievementin first year, but recover to identical achievement

– Students winning lotteries for magnet high Students winning lotteries for magnet high schools seem to do better in math 2-3 years laterschools seem to do better in math 2-3 years later

• Effect sizes are meaningful: 0.18 and 0.23 in Effect sizes are meaningful: 0.18 and 0.23 in years 2 and 3years 2 and 3– But only for CST and not CAT/6But only for CST and not CAT/6

28 PPIC PPIC

Charter SchoolsCharter SchoolsEffect on Achievement GainsEffect on Achievement Gains

Relative to regular public schoolsRelative to regular public schools

– Charters do not systematically under- or Charters do not systematically under- or overperformoverperform

A few exceptions. Most interestingly…A few exceptions. Most interestingly…

– Often, elementary startup charters underperform Often, elementary startup charters underperform in math and reading for 1 to 3 years, then in math and reading for 1 to 3 years, then perform at same level perform at same level

29 PPIC PPIC

OutlineOutline






30 PPIC PPIC

Conflicting Achievement ResultsConflicting Achievement Results

Choice affected some achievement results, but the more Choice affected some achievement results, but the more general finding was of no effect even three years after winning general finding was of no effect even three years after winning a lotterya lottery

Yet, school choice programs extremely popular in San Diego Yet, school choice programs extremely popular in San Diego with 28% student share by 2003-2004with 28% student share by 2003-2004

Possible explanations:Possible explanations:

– Parents care about math and reading but lack informationParents care about math and reading but lack information

– Parents consider factors other than math and reading, such Parents consider factors other than math and reading, such as socioeconomic climate, other achievement outcomesas socioeconomic climate, other achievement outcomes

31 PPIC PPIC

Conclusions and National ImplicationsConclusions and National Implications

NCLB mandates that students in “failing” schools NCLB mandates that students in “failing” schools be given option to be bused to other schoolsbe given option to be bused to other schools

Question: Is the best way to help students at Question: Is the best way to help students at “failing” schools to move students from the “failing” schools to move students from the school, or to reform the schools in question?school, or to reform the schools in question?

In San Diego, the NCLB Choice program In San Diego, the NCLB Choice program piggybacked on VEEP and magnetspiggybacked on VEEP and magnets

– Effects likely similar, and smallEffects likely similar, and small

32 PPIC PPIC

Conclusions and National Implications Conclusions and National Implications

Additional aid to students in low-performing schools Additional aid to students in low-performing schools may be a better solutionmay be a better solution

– Betts, Zau and King (PPIC, 2005) report some Betts, Zau and King (PPIC, 2005) report some evidence that “within-school” reforms can boost evidence that “within-school” reforms can boost achievementachievement

Choice may provide better option in certain Choice may provide better option in certain situations situations

But presumption that choice automatically confers But presumption that choice automatically confers large achievement gains is probably incorrectlarge achievement gains is probably incorrect

33 PPIC PPIC

Does School Choice Work? Does School Choice Work? Effects Effects

on Integration on Integration and Student Achievementand Student Achievement

Julian R. Betts, Lorien A. Rice, Julian R. Betts, Lorien A. Rice, Andrew C. Zau, Y. Emily Tang, and Andrew C. Zau, Y. Emily Tang, and

Cory R. KoedelCory R. Koedel

34 PPIC PPIC

Supplementary SlidesSupplementary Slides

35 PPIC PPIC

Who Exercises Choice - Methodology Who Exercises Choice - Methodology

Look at applications to VEEP, magnet and Look at applications to VEEP, magnet and ChoiceChoice enrollment programs for 2001-2002enrollment programs for 2001-2002

– Gives truer picture of demand for school choice Gives truer picture of demand for school choice than actual school switchers, because demand than actual school switchers, because demand exceeds supplyexceeds supply

Statistically model the probability that a student Statistically model the probability that a student applies to VEEP; separate models for magnets and applies to VEEP; separate models for magnets and ChoiceChoice (open enrollment) (open enrollment)

36 PPIC PPIC

Change in the Probability of Application for Change in the Probability of Application for Local School Characteristics (Elementary)Local School Characteristics (Elementary)

Elementary School

-12-10-8-6-4-202468

Perc

en

tag

e C

han

ge i

n S

tud

en

ts A

pp

lyin

g

VEEP

Magnet

Choice

37 PPIC PPIC

Change in the Probability of Application for Change in the Probability of Application for Local School Characteristics (Middle)Local School Characteristics (Middle)

Middle School

-12-10-8-6-4-202468

Perc

en

tag

e C

han

ge i

n S

tud

en

ts A

pp

lyin

g

VEEP

Magnet

Choice

38 PPIC PPIC

Change in the Probability of Application for Change in the Probability of Application for Local School Characteristics (High)Local School Characteristics (High)

High School

-12-10-8-6-4-202468

Perc

en

tag

e C

han

ge i

n S

tud

en

ts A

pp

lyin

g

VEEP

Magnet

Choice

39 PPIC PPIC

Change in the Probability of Application for Change in the Probability of Application for Option School Characteristics (Elementary)Option School Characteristics (Elementary)

Elementary School

-12-10-8-6-4-202468

Perc

en

tag

e C

han

ge i

n S

tud

en

ts A

pp

lyin

g

VEEP

Magnet

Choice

40 PPIC PPIC

Change in the Probability of Application for Change in the Probability of Application for Option School Characteristics (Middle)Option School Characteristics (Middle)

Middle School

-12-10-8-6-4-202468

Perc

en

tag

e C

han

ge i

n S

tud

en

ts A

pp

lyin

g

VEEP

Magnet

Choice

41 PPIC PPIC

Change in the Probability of Application for Change in the Probability of Application for Option School Characteristics (High)Option School Characteristics (High)

High School

-12-10-8-6-4-202468

Perc

en

tag

e C

han

ge i

n S

tud

en

ts A

pp

lyin

g

VEEP

Magnet

Choice

42 PPIC PPIC

Relative to White, Male, non-English Learner, Low Parental EducationRelative to White, Male, non-English Learner, Low Parental Education

Who Exercises Choice – Percent Change in Probability of Applying

High School

-1.3

-0.8

-0.3

0.2

0.7

1.2

1.7

2.2

2.7

3.2

GPA Stanford 9Score

Female Black Asian Hispanic EnglishLearner

HighParental

Education

VEEP

Magnet

Choice

43 PPIC PPIC

Black VEEP Applicants’ Change in % White (School Population)Black VEEP Applicants’ Change in % White (School Population) Potential versus Actual Potential versus Actual

0

5

10

15

20

25

30

35

40

45

50

Applications (potential exposure)

LotteryWinners (potential)

Lottery WinnersWho Switch(actual)

44 PPIC PPIC

Charter School Enrollment Share (%) in San Charter School Enrollment Share (%) in San DiegoDiego

0

1

2

3

4

5

6

7

1996-97 1997-98 1998-99 1999-00 2000-01 2001-02 2002-03 2003-04

startup

conversion

45 PPIC PPIC

Charter SchoolsCharter Schools Background Background

Charters vary widely in academic focus and remediation Charters vary widely in academic focus and remediation programsprograms

Exempt from many district regulations and state lawsExempt from many district regulations and state laws

Other general characteristics Other general characteristics – Students Students

• 83% non-white (non-charters: 73%) 83% non-white (non-charters: 73%) • 66% eligible for free/reduced-price meals66% eligible for free/reduced-price meals

(non-charters: 56%)(non-charters: 56%)– Draw from a wider geographic areaDraw from a wider geographic area– Fewer teachers with long experience and advanced Fewer teachers with long experience and advanced

educationeducation– Often have additional costs such as rental of space, Often have additional costs such as rental of space,

rental of school buses rental of school buses

46 PPIC PPIC

Effects of Attending a Charter on Stanford 9 Effects of Attending a Charter on Stanford 9 ScoresScores

-0.2

-0.15

-0.1

-0.05

0

0.05

0.1

Elementary Middle High

Gradespan

Pro

po

rtio

n o

f a

Sta

nd

ard

Dev

iati

on

Math

Reading

47 PPIC PPIC

Effect of Attending a Startup or Conversion Charter SchoolEffect of Attending a Startup or Conversion Charter School

on Stanford 9 Test Score Gainon Stanford 9 Test Score Gain

-0.25

-0.20

-0.15

-0.10

-0.05

0.00

0.05

0.10

0.15

Startup Conversion Startup Conversion

Elementary Middle

Gradespan, Charter Type

Pro

po

rtio

n o

f a

Sta

nd

ard

Dev

iati

on

Math

Reading

48 PPIC PPIC

Policy ImplicationsPolicy ImplicationsIntegration vs. ResegregationIntegration vs. Resegregation

Overall, school choice improves racial/ethnic and Overall, school choice improves racial/ethnic and socioeconomic integrationsocioeconomic integration

Weak evidence that students with higher test Weak evidence that students with higher test scores or more highly educated parents more scores or more highly educated parents more likely to applylikely to apply

– Effects smallEffects small

– Less clear for GPALess clear for GPA

– Does not support notion that choice programs Does not support notion that choice programs “skim off the cream”“skim off the cream”

49 PPIC PPIC

Qualifications of Integration ResultsQualifications of Integration Results

School choice programs could integrate more if School choice programs could integrate more if supply constraints less severesupply constraints less severe

Notable integrative differences between Notable integrative differences between ChoiceChoice——which reduced integration along various lineswhich reduced integration along various lines——and and VEEP and magnet programs VEEP and magnet programs

– VEEP and magnets featureVEEP and magnets feature

• Publicly provided busingPublicly provided busing

• Regulation through geographic matching or Regulation through geographic matching or preferences preferences

50 PPIC PPIC

Policy QuestionPolicy QuestionDoes Competition Force Schools to Improve?Does Competition Force Schools to Improve?

Although not directly studied, student applications Although not directly studied, student applications respond only weakly to average test scores at respond only weakly to average test scores at schoolsschools

If actual choices do not typically boost If actual choices do not typically boost achievement, how can student departures induce achievement, how can student departures induce schools that lose students to boost achievement?schools that lose students to boost achievement?

51 PPIC PPIC

Other Result Implications Other Result Implications

Limited results (one cohort) prevent effective Limited results (one cohort) prevent effective argument against any of the choice programsargument against any of the choice programs

Relative cost effectiveness implicationsRelative cost effectiveness implications

– ChoiceChoice——no public busing costno public busing cost

– VEEP and magnetVEEP and magnet——public busing cost public busing cost

• Positive results for some magnets an offsetting Positive results for some magnets an offsetting factorfactor

– ChartersCharters——Often appear to receive less funding yet Often appear to receive less funding yet perform about as well as regular public schoolsperform about as well as regular public schools

• Urgent need for studies of charter fundingUrgent need for studies of charter funding

does school choice work? effects on integration and student achievement

Documents