what was learned from a second year of implementation
DESCRIPTION
What Was Learned from a Second Year of Implementation. William Corrin, Senior Research Associate MDRC. IES Research Conference Washington, DC June 8, 2009. Presentation Overview. Project Background Do teachers need more than a year to master new practices? - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: What Was Learned from a Second Year of Implementation](https://reader035.vdocument.in/reader035/viewer/2022062501/56815d9e550346895dcbc59a/html5/thumbnails/1.jpg)
What Was Learned from aSecond Year of Implementation
IES Research Conference
Washington, DCJune 8, 2009
William Corrin, Senior Research AssociateMDRC
![Page 2: What Was Learned from a Second Year of Implementation](https://reader035.vdocument.in/reader035/viewer/2022062501/56815d9e550346895dcbc59a/html5/thumbnails/2.jpg)
Presentation Overview
Project Background Do teachers need more than a year to master new
practices?
- Year 2 Implementation Findings (and Year 1 Comparison)
- Impact findings
- Exploratory findings
o Two-Year Teachers and Replacement Teachers
o Two-Year Teachers: Year 1 vs. Year 2
Summary
![Page 3: What Was Learned from a Second Year of Implementation](https://reader035.vdocument.in/reader035/viewer/2022062501/56815d9e550346895dcbc59a/html5/thumbnails/3.jpg)
Project Background
![Page 4: What Was Learned from a Second Year of Implementation](https://reader035.vdocument.in/reader035/viewer/2022062501/56815d9e550346895dcbc59a/html5/thumbnails/4.jpg)
The ERO Study
The ERO study is an impact evaluation of two supplemental literacy programs targeted at 9th grade students with limited reading skills.
Key research question:
- What are the impacts of the two interventions, together and separately, on ninth-grade students’ reading achievement and reading behaviors?
![Page 5: What Was Learned from a Second Year of Implementation](https://reader035.vdocument.in/reader035/viewer/2022062501/56815d9e550346895dcbc59a/html5/thumbnails/5.jpg)
The Interventions
Two literacy programs targeted at 9th grade students with limited literacy skills (2-5 years below grade level)- Reading Apprenticeship Academic Literacy (RAAL)
- Xtreme Reading (XR)
The programs are supplemental reading classes- Designed as full-year courses that replace a 9th grade
elective class (rather than a core academic class)
The two programs share common goals and instructional principles
![Page 6: What Was Learned from a Second Year of Implementation](https://reader035.vdocument.in/reader035/viewer/2022062501/56815d9e550346895dcbc59a/html5/thumbnails/6.jpg)
Delivery of the Interventions Each high school implemented one of the two programs
for ninth graders for two school years
- 2005-06 (Cohort 1) and 2006-07 (Cohort 2)
One ERO teacher per high school
- Experienced full-time ELA or social studies teacher was trained to teach the ERO reading course
- Responsible for 4 sections of 10-15 students each
Training and technical assistance were provided to teachers by the developers
- Summer training institutes, school-year off-site booster training sessions and on-site coaching
![Page 7: What Was Learned from a Second Year of Implementation](https://reader035.vdocument.in/reader035/viewer/2022062501/56815d9e550346895dcbc59a/html5/thumbnails/7.jpg)
Implementation Data Collection
To answer questions about how well the programs were put into place
Allows us to answer questions about how implementation compared across program years: Do teachers do better with more experience teaching these programs? That is, is implementation stronger in the second year compared to the first?
Allows us to investigate associations between implementation and impacts
![Page 8: What Was Learned from a Second Year of Implementation](https://reader035.vdocument.in/reader035/viewer/2022062501/56815d9e550346895dcbc59a/html5/thumbnails/8.jpg)
Measuring Implementation Fidelity
Two site visits in Year 2 (one per semester; only one site visit in Year 1)
Implementation fidelity was assessed on two dimensions:- Classroom learning environment
- Comprehension instruction
Classroom observation ratings were obtained for 6 constructs common to both programs, and 7 program-specific constructs- Ratings are based on a 3-point scale
![Page 9: What Was Learned from a Second Year of Implementation](https://reader035.vdocument.in/reader035/viewer/2022062501/56815d9e550346895dcbc59a/html5/thumbnails/9.jpg)
Measuring Implementation Fidelity (cont.)
Composite ratings for each of the two dimensions were calculated, and then used to classify sites as:
- “well aligned” (average rating > 2)
- “moderately aligned” (average rating = 1.5 – 1.9)
- “poorly aligned” (average rating < 1.5)
![Page 10: What Was Learned from a Second Year of Implementation](https://reader035.vdocument.in/reader035/viewer/2022062501/56815d9e550346895dcbc59a/html5/thumbnails/10.jpg)
Implementation Findings
![Page 11: What Was Learned from a Second Year of Implementation](https://reader035.vdocument.in/reader035/viewer/2022062501/56815d9e550346895dcbc59a/html5/thumbnails/11.jpg)
Implementation Findings
ERO teachers in Year 2- Of the 34 ERO teachers in Year 2:
o 25 had taught the entire first year of the study (13 RAAL, 12 XR)
o 2 had taught part of the first year (both XR)
o 7 were new to the programs (4 RAAL, 3 XR)
All Year 2 teachers taught the entire year
![Page 12: What Was Learned from a Second Year of Implementation](https://reader035.vdocument.in/reader035/viewer/2022062501/56815d9e550346895dcbc59a/html5/thumbnails/12.jpg)
Implementation Findings (cont.)
Implementation fidelity in Year 2- In terms of classroom learning environment, 1 school
was rated as poorly aligned with program models (vs. 4 schools in Year 1)
- In terms of reading comprehension instruction, 1 school was rated as poorly aligned with program models (vs. 9 schools in Year 1)
- 23 schools were rated as well aligned on both dimensions (vs. 16 schools in Year 1)
- Similar pattern of implementation findings at RAAL and XR schools
![Page 13: What Was Learned from a Second Year of Implementation](https://reader035.vdocument.in/reader035/viewer/2022062501/56815d9e550346895dcbc59a/html5/thumbnails/13.jpg)
Implementation Findings (cont.)
Implementation fidelity rated higher in Year 2- In terms of classroom learning environment, the
average rating at the spring site visit was 2.5 in Year 2 compared to 2.2 in Year 1
- In terms of reading comprehension instruction, the average rating at the spring site visit was 2.3 in Year 2 compared to 1.9
![Page 14: What Was Learned from a Second Year of Implementation](https://reader035.vdocument.in/reader035/viewer/2022062501/56815d9e550346895dcbc59a/html5/thumbnails/14.jpg)
Learning EnvironmentBy Site Visit
![Page 15: What Was Learned from a Second Year of Implementation](https://reader035.vdocument.in/reader035/viewer/2022062501/56815d9e550346895dcbc59a/html5/thumbnails/15.jpg)
Comprehension Instructionby Site Visit
![Page 16: What Was Learned from a Second Year of Implementation](https://reader035.vdocument.in/reader035/viewer/2022062501/56815d9e550346895dcbc59a/html5/thumbnails/16.jpg)
Impact Findings
![Page 17: What Was Learned from a Second Year of Implementation](https://reader035.vdocument.in/reader035/viewer/2022062501/56815d9e550346895dcbc59a/html5/thumbnails/17.jpg)
Random Assignment and Analysis
Random assignment:- School-Level
o 34 schools randomly assigned within 10 districts to RAAL or Xtreme Reading (17 RAAL sites and 17 Xtreme Reading sites)
- Student-Level o In each high school, eligible students were randomly
assigned to either: Enroll in an ERO class (ERO group) or Take one of the regular elective classes (non-ERO
group)
Analysis: Impacts are estimated by comparing the outcomes of students in the ERO and non-ERO group
![Page 18: What Was Learned from a Second Year of Implementation](https://reader035.vdocument.in/reader035/viewer/2022062501/56815d9e550346895dcbc59a/html5/thumbnails/18.jpg)
Impact Findings: Reading Achievement
Reading achievement was measured using the GRADE, a standardized reading assessment.
Overall positive impact on reading comprehension- 0.09 SD for Cohort 1
- 0.08 SD for Cohort 2
No impact on vocabulary- Same result for both cohorts
![Page 19: What Was Learned from a Second Year of Implementation](https://reader035.vdocument.in/reader035/viewer/2022062501/56815d9e550346895dcbc59a/html5/thumbnails/19.jpg)
Impact Findings: Reading Behaviors
Cohort 1- Estimated impacts on three reading behaviors were
not statistically significant (frequency of school-related reading, freq. of non-school-related reading, use of reading strategies)
Cohort 2- Estimated impact on students’ use of the reading
strategies taught by the programs is 0.09 SD and statistically significant
![Page 20: What Was Learned from a Second Year of Implementation](https://reader035.vdocument.in/reader035/viewer/2022062501/56815d9e550346895dcbc59a/html5/thumbnails/20.jpg)
Exploratory Analyses
![Page 21: What Was Learned from a Second Year of Implementation](https://reader035.vdocument.in/reader035/viewer/2022062501/56815d9e550346895dcbc59a/html5/thumbnails/21.jpg)
Implementation Fidelity by Teacher Experience and Year
Implementation ratings by year and by teacher experience:
Implementation Dimension Year
Schools with Two-Year Teachers (N=25)
Schools with Replacement
Teachers (N=9)
Learning Environment
1 2.4 1.7
2 2.5 2.4
Reading Comprehension
Instruction
1 2.0 1.8
2 2.3 2.3
![Page 22: What Was Learned from a Second Year of Implementation](https://reader035.vdocument.in/reader035/viewer/2022062501/56815d9e550346895dcbc59a/html5/thumbnails/22.jpg)
Implementation Fidelity by Teacher Experience and Year
Overall implementation fidelity:
- Of the 25 sites with two-year teachers, 19 had higher overall implementation ratings in the second year compared to the first year
- Of the 9 sites with replacement teachers, 8 sites had higher overall implementation ratings with the replacement teacher compared to the original teacher
![Page 23: What Was Learned from a Second Year of Implementation](https://reader035.vdocument.in/reader035/viewer/2022062501/56815d9e550346895dcbc59a/html5/thumbnails/23.jpg)
Composite Fidelity Scores by Site Visit
![Page 24: What Was Learned from a Second Year of Implementation](https://reader035.vdocument.in/reader035/viewer/2022062501/56815d9e550346895dcbc59a/html5/thumbnails/24.jpg)
Variation in Impacts
Impacts on Student Outcomes by Teacher Experience Teaching ERO: Year 2
Student Outcome
Schools w/ Two-Year Teachers
(N=25)
Schools w/ Replacement
Teachers (N=9)
Reading Comprehension
0.09 SD*
0.06 SD
Use of Reading Strategies
0.10 SD*
0.07 SD
![Page 25: What Was Learned from a Second Year of Implementation](https://reader035.vdocument.in/reader035/viewer/2022062501/56815d9e550346895dcbc59a/html5/thumbnails/25.jpg)
Variation in Impacts (cont.)
Impacts on Student Outcomes by Implementation Year: Schools w/ Two-Year Teachers (N=25) Student Outcome
Year 1
Year 2
Reading Comprehension
0.11 SD*
0.09 SD*
Use of Reading Strategies (diff. betw. yrs. is stat. sig. )
-0.05 SD
0.10 SD*
![Page 26: What Was Learned from a Second Year of Implementation](https://reader035.vdocument.in/reader035/viewer/2022062501/56815d9e550346895dcbc59a/html5/thumbnails/26.jpg)
Summary Supplemental literacy courses for 9th grade students can have a
positive impact on students’ reading comprehension.
These programs can be implemented with reasonable fidelity in a diverse array of high schools, and implementation fidelity can improve over time.
On average, implementation fidelity was stronger in the second year.
There is suggestive, but not convincing, evidence that teachers gained greater mastery of the programs having a second year to teach it.
However, impacts on students reading comprehension remained the same. (Impacts on students’ use of reading strategies were greater.)
![Page 27: What Was Learned from a Second Year of Implementation](https://reader035.vdocument.in/reader035/viewer/2022062501/56815d9e550346895dcbc59a/html5/thumbnails/27.jpg)
Further questions? William Corrin
- (212) 340-8840 Marie-Andree Somers
- (212) 340-8825 Paul Strasberg
- (202) 219-3400