The Road to

Charter School Quality US Department of Education

September 30, 2013

Introduction

2

Acknowledgements

Robertson Foundation

Three Anonymous Funders

State Education Agency Partners

Center for Reinventing Public Education

National Alliance for Public Charter Schools

Peer Reviewers

3

Overview of Presentation

• What do we know today about charter school

quality?

• Do charter schools improve over time?

• Are CMOs a reliable path to quality?

• Is the sector as a whole getting better?

• Do policy factors drive quality?

4

Conclusions

Charter school quality is set early in the operating

life of schools.

CMO quality is a function of flagship quality.

Charter sector is improving – by small amounts

Opportunity to segment market for finer views

5

Charter School

Growth and

Replication

Maturation

Replication

Impact

Big Picture

Quality

Trends

Today’s Story

7

Study Approach

What happens to quality as schools mature?

• Examined first growth period for 912 charter schools

• Divided first year observations into 5 Quintiles of

Quality

• Held quintile boundaries constant and watched

schools grow up

8

1-Year Conditional

Probabilities

9

Age of School

If the school’s starting quintile is:

Q1 Q2 Q3 Q4 Q5

In which quintiles does the school appear the following year?

1-2 3-5 1-2 3-5 1-2 3-5 1-2 3-5 1-2 3-5

1 0.66 0.33 0.41 0.60 0.22 0.78 0.13 0.87 0.08 0.92

2 0.72 0.29 0.46 0.54 0.27 0.74 0.14 0.87 0.05 0.95

3 0.77 0.23 0.50 0.51 0.22 0.79 0.09 0.91 0.05 0.95

4 0.74 0.26 0.59 0.40 0.27 0.73 0.15 0.86 0.04 0.95

5 0.80 0.19 0.51 0.49 0.23 0.77 0.09 0.91 0.06 0.94

No. of Schools 1688

*Results shown are for math.

Early signals of quality, both high and low, are consistent predictors of quality over

time.

2-Year Conditional

Probabilities

10

Age of School

If the school’s starting quintile is:

Q1 Q2 Q3 Q4 Q5

In which quintiles does the school appear the following year?

1-2 3-5 1-2 3-5 1-2 3-5 1-2 3-5 1-2 3-5

1 - 2 0.82 0.19 0.74 0.26 0.20 0.80 0.15 0.84 0.00 1.00

2 - 3 0.85 0.15 0.73 0.28 0.18 0.82 0.09 0.91 0.03 0.97

3 - 4 0.91 0.10 0.65 0.35 0.23 0.76 0.08 0.92 0.02 0.99

4 - 5 0.84 0.15 0.56 0.44 0.19 0.82 0.05 0.96 0.04 0.97

No. of Schools 577

*Results shown are for math.

Quality becomes even more consistent when viewed over a two-year time span.

Where the Action Is

11

Age of School

Q2

1-2 3-5

1 0.41 0.60

2 0.46 0.54

3 0.50 0.51

4 0.59 0.40

5 0.51 0.49

Age of School

Q1

1-2 3-5

1 0.18 0.81

2 0.17 0.84

3 1.00 0.00

4 0.20 0.80

5 0.00 1.00

Quintile 2 Elementary Schools

Two groups show improvement over time:

Quintile 2 schools have a 50-50 chance of improving

Elementary schools show improvement but it takes longer

*Results shown are for math.

Maturation

Replication

Impact Big Picture

Quality

Trends

Today’s Story

12

Who Decides to

Replicate?

13

n=1

n=1

n=5

n=8

n=6

0 1 2 3 4 5

*Results shown are for reading.

Quintile of Flagship School in Year Prior to First Replication

Quality of Replication

14

1

2

3

4

5

Flagship in Year Prior to Replication First Replication School, Age 1

Qu

inti

le

CMO 1.1

CMO 1.1 CMO 2.1 CMO 4.3

CMO 1.1 CMO 4.1 CMO 4.2 CMO 4.3

CMO 2.1

CMO 3.1 CMO 3.2 CMO 3.3

CMO 3.1 CMO 5.1

CMO 3.2 CMO 5.2

CMO 3.3

CMO 4.1 CMO 4.2 CMO 4.3

CMO 5.1 CMO 5.2

*Results shown are for reading.

Replication Success

by Year

15

14 13 12

18

24 15

11

5 6

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

2006 2007 2008

New Schools Significantly Stronger thanExisting Portfolio

No Significant Difference between NewSchools and Existing Portfolio

New School Significantly Weaker thanExisting Portfolio

WYSIWYG

Maturation

Replication

Impact Big Picture

Quality

Trends

Today’s Story

16

Is the Charter

Sector

Improving?

17

2009 Aggregate Results

18

Charter Sector Improvement

Examined the current performance of the 16 states

from 2009 study

Four sources of influence must be considered:

1. Charter schools change performance

2. School closings alter the stock of schools

3. New charter schools come on line

4. Traditional public schools change quality

19

Where Does Change Happen?

20

2009

Charter

Schools

Closed

Still Open

New

2009 TPS

2013

Charter

Schools

2013 TPS

80%

20%

90%

10%

16 States-Results

Overall Charter Impact Reading

21

2013 Charter

Impact Reading

Continuing Schools .01**

New Schools -.01**

-.01**

.01**

-36

-18

0

18

36

-.05

.00

.05

Then Both

Now

Days of

LearningStandard

Deviations

** Significant at p ≤ 0.01

TPS

Growth

2009 2013

16 States-Results

Overall Charter Impact Reading

22

-.01**

.01**

-36

-18

0

18

36

-.05

.00

.05

Then Both

Now

Days of

LearningStandard

Deviations

Reading Charter Reading TPS** Significant at p ≤ 0.01

2009 2013

.01**State

Average

Growth

16 States – Results

Overall Charter Impact Math

23

2013 Charter

Impact Math

Continuing Schools -.01**

New Schools -.03**

-.03**

-.01**

-36

-18

0

18

36

-.05

.00

.05

1 2

Days of

LearningStandard

Deviations

** Significant at p ≤ 0.01

TPS

Growth

2009 2013

16 States – Results

Overall Charter Impact Math

24

-.03**

-36

-18

0

18

36

-.05

.00

.05

Then

Days of

LearningStandard

Deviations

Math Charter Math TPS** Significant at p ≤ 0.01

2009 20132009 20132009 20132009 2013

-.01**-.03**

State

Average

Growth

Student Group

Reading

2009

Reading

2013

Math

2009

Math

2013

Black -7 7 -7 0

Hispanic -14 -7 -14 -7

Poverty 7 14 7 22

English Language Learners 36 43 22 36

Special Education 0 14 7 14

16 States - Results

Subgroup Results

25

Student Group

Reading

2009

Reading

2013

Math

2009

Math

2013

Black -7 7 -7 0

Hispanic -14 -7 -14 -7

Poverty 7 14 7 22

36 43 22 36

Special Education 0 14 7 14

Student Group

Reading

2009

Reading

2013

Math

2009

Math

2013

Black -7 7 -7 0

Hispanic -14 -7 -14 -7

Poverty 7 14 7 22

36 43 22 36

Special Education 0 14 7 14

Student Group

Reading

2009

Reading

2013

Math

2009

Math

2013

Black -7 7 -7 0

Hispanic -14 -7 -14 -7

Poverty 7 14 7 22

36 43 22 36

Special Education 0 14 7 14

Student Group

Reading

2009

Reading

2013

Math

2009

Math

2013

Black -7 7 -7 0

Hispanic -14 -7 -14 -7

Poverty 7 14 7 22

36 43 22 36

Special Education 0 14 7 14

Student Group

Reading

2009

Reading

2013

Math

2009

Math

2013

Black -7 7 -7 0

Hispanic -14 -7 -14 -7

Poverty 7 14 7 22

36 43 22 36

Special Education 0 14 7 14

27-State

Analysis

26

-0.008 -0.008

-0.005-0.003

-0.001

.007* .007*

0.010** 0.010** 0.011**

-14

-7

0

7

14

-0.02

-0.01

0.00

0.01

0.02

5 Growth

Periods

4 Growth

Periods

3 Growth

Periods

2 Growth

Periods

1 Growth

Period

Days of

Learning

Standard

Deviations

Math Read* Significant at p ≤ 0.05

** Significant at p ≤ 0.01

27

27 States

Overall Trendline

28

27 States

Reading State Charter Impacts

Math State Charter Impacts

Subgroup Findings

29

Student Group Reading Math

White Negative Negative

Black Positive Positive

Hispanic Similar Similar

Asian Similar Negative

Students in Poverty Positive Positive

English Language Learners (ELL) Positive Positive

Special Education Similar Positive

How does attendance in

a CMO affiliated school compare

to an independent charter?

Students who attend CMO affiliated charter schools had stronger growth in math but weaker growth in reading than those who attend non-CMO affiliated charter schools. All charter schools do better that TPS in reading but lag in math.

30

** p<0.01, * p<0.05

-0.012**

0.007**

-0.005**

0.005**

-0.050

-0.040

-0.030

-0.020

-0.010

0.000

0.010

0.020

0.030

0.040

0.050

Math Reading

Non-CMO Charters Math

CMO Charters Math

Non-CMO Charters Read

CMO Charters Read

TPS gains

set to zero

How does attendance in

a CMO impact math growth for

students in poverty?

31

Poor black and poor Hispanic students attending CMO affiliated schools have stronger math growth than poor black and Hispanic students attending TPS schools.

The increased growth from attending CMO schools is enough to offset the negative coefficients typically associated with being a Hispanic student.

Because minority status and poverty are highly correlated, we included race by poverty status breakdowns in this analysis.

-0.05**

-0.13**

-0.04**

-0.07**

0.02**

-0.04** 0.00

0.01**

-0.16

-0.14

-0.12

-0.10

-0.08

-0.06

-0.04

-0.02

0.00

0.02

0.04

non-Poverty Poverty non-Poverty Poverty non-Poverty Poverty non-Poverty Poverty

Black TPS Black CMO Hispanic TPS Hispanic CMO

Math Growth by Poverty, Race, and School Type

TPS white,

non-poverty

student gains

set to zero

32

Number of Networks

Percent of Networks

Number of Schools

Percent of Schools

Number of Students

Percent of Students

Math

CMO Stronger vs

TPS 62 37% 499 36% 198,199 34%

CMO Same vs TPS 22 13% 240 17% 78,397 14%

CMO Weaker vs TPS 83 50% 636 46% 298,133 52%

Quality Curve

CMO Math

33

Number of Networks

Percent of Networks

Number of Schools

Percent of Schools

Number of Students

Percent of Students

Reading

CMO Stronger vs

TPS 71 43% 679 49% 303,929 51%

CMO Same vs TPS 35 21% 182 13% 42,900 7%

CMO Weaker vs TPS 61 37% 511 37% 244,725 41%

Quality Curve

CMO Reading

WYSIWYG

Maturation

Replication

Impact Big Picture

Quality

Trends

Today’s Story

34

Associated Factors

Characteristic

Was impact of characteristic

significant in subject?

MATH READING

Network Size No Difference No Difference

Maturity of CMO No Difference No Difference

Distance between Flagship and

Schools

No Difference No Difference

Geographic Concentration No Difference No Difference

Multistate CMO Network Negative Negative

Multistate EMO Network Positive Positive

35

-0.04**

0.21**

-0.01**

0.13**

-0.05

0.00

0.05

0.10

0.15

0.20

0.25

non-CSGF Math CSGF Math non-CSGF Read CSGF Read

CSGF school students had much stronger growth than other CMOs or TPS students.

0.1

.2.3

.4

Ran

ge

of Q

ualit

y

-.2 0 .2 .4Average Growth

Other Insights

36

Range of growth is very stable over the entire spectrum of performance.

Charter School Growth Fund is identifying high performers.

WYSIWYG

Maturation

Replication

Impact Big Picture

Quality

Trends

Today’s Story

37

Impact on New Schools

38

-0.04**

-0.01**

-0.03**

-0.06**

-0.09

-0.08

-0.07

-0.06

-0.05

-0.04

-0.03

-0.02

-0.01

0.00

non-CMO Reading CMO Reading non-EMO Reading EMO Reading

New schools which are part of a CMO network had better reading performance than new non-CMO

charter schools.

TPS gains

set to zero

Growth Over Time

39

0.20

0.16

0.00

0.04

-0.17

-0.11

-0.20

-0.15

-0.10

-0.05

0.00

0.05

0.10

0.15

0.20

0.25

Average Starting Effect Size Average Ending Effect Size

Highest Starting Effect Size

Middle Starting Effect Size

Lowest Starting Effect Size

WYSIWYG

Maturation

Replication

Impact Big Picture

Quality

Trends

Today’s Story

40

Conclusions

• Charter school sector has improved slightly

since 2009.

• Some improvement driven by individual schools

getting better, but larger gains come from

closures.

• Significant benefit for disadvantaged students

• Same variation in quality applies to CMOs

• State policy / Authorizer performance matters!!

41

Questions?

42

Thank You

43