Evaluation of the DC Opportunity Scholarship Program: Final Report

Conducted by Westat, University of Arkansas,

Chesapeake Research AssociatesPresented by

Patrick J. Wolf, Principal InvestigatorIES Research Conference

National Harbor, MarylandJune 30, 2010

Overview of Program Part of 3-sector strategy of urban education reform Student Eligibility

Grade K-12, residents of DC Family income ≤ 185% Federal poverty line Priority to students at schools in need of improvement (SINI)

Scholarship Up to $7,500 to cover tuition, school fees, and transportation Renewable for up to 5 years

Participation (2004-2009) 68 of the 88 general purpose private schools in DC 5,547 eligible applicants; 3,738 scholarship recipients; 2,881

initial scholarship users Closed to new participants in spring of 2009

Overview of Study Design

Lottery-based randomized control trial Sample: 2,308 eligible applicants in 2004 and 2005 Data:

Evaluation-administered SAT-9 Surveys: parents, students, principals

Analysis: impact of offer, use of scholarship, attendance at private school

Reports: 2 describing program applicants and our sample 4 estimating impacts (after 1, 2, 3, and 4+ years)

Scholarship Use Over 4+ Years

Just over one-fourth of students offered a scholarship consistently used it throughout the study

Participation drop-off (net): averaged 22% per year Top reasons for dropping out of OSP

Child got into a charter school (22%)

Lack of space (for students transitioning from K-8) (19%)

Moved out of DC (15%)

Transportation problems (14%)

Students more likely to drop out of OSP: lower initial test scores, older grade levels, male, with special needs, more siblings

Treatment Group Scholarship Use by Final Year

Partially used51%

Consistently used27%

Never used22%

In 2008-09

Not in 2008-09

Key Findings

OSP significantly raised graduation rate

No conclusive evidence OSP affected achievement over longer-run In final year, no statistically significant impacts for

students overall, SINI, lower performing, males Third year overall reading gains didn’t persist Impacts on reading test scores for some subgroups, but

results may be due to chance

Positive impacts on ratings of safety and satisfaction from parents but not students

High School Graduation Rates, Overall Sample & SINI Subgroup, Parent Reports, 2008-09

ITT

IOT

ITT Impacts on Reading Achievement Overall, by Years After Application

-2.41

-0.53

0.89

-0.13

1.03

3.17

4.46*3.90

4.46

6.88

8.03 7.94

-4

-3

-2

-1

0

1

2

3

4

5

6

7

8

9

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5After one year After two years After three years After at least four years

* Statistically significant at the 95 percent confidence interval.

ITT Impacts on Math Achievement Overall, by Years After Application

- 0.26

- 3.07

- 2.42

- 3.01

2.74

0.23

0.81 0.70

5.74

3.53

4.034.41

-4

-3

-2

-1

0

1

2

3

4

5

6

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5After one year After two years After three years After at least four years

Parent and Student Reports of Safety and an Orderly School Climate, 2008-09

**Statistically significant at the 99 percent confidence level.

Parent and Student Reports of School Satisfaction, 2008-09

**Statistically significant at the 99 percent confidence level.

Context DC proficiency and NAEP scores up 20 percent 2005-2009 Only 3 percent of DC schoolchildren participated and little

evidence of competitive response from DC schools Reliable studies indicate graduation yields key benefits:

ETS study – increases earnings by $8,500/year, decreases unemployment by one-third

Census study – all else equal, graduates live 7 years longer than dropouts Belfield and Levin – each graduate reduces the cost of crime by $112,000 Cecelia Rouse – each additional graduate saves the nation $260,000

Study limitations: One study, one city, specific program design Both achievement and attainment analyses in final year based on

subgroups Some key outcomes based on parent and student reports