making a difference update on the national i3 evaluation of the implementation and impact of...
TRANSCRIPT
Making a DifferenceMaking a DifferenceUpdate on the National i3 Evaluation of the
Implementation and Impact of Diplomas Now
2
Diplomas Now i3 Validation Study
Goal: Validate effectiveness of the DN model
Research Questions - Phase I (current):
1. What is the impact of Diplomas Now on students’ outcomes, particularly with regard to attendance, in-school behavior, and course performance?
2. What lessons can be learned about implementation of the model during national expansion?
3
Notable about this Evaluation
It’s rigor:• Random assignment design: uses a lottery-like process to
assign schools to two equivalent groups (“DN schools” and “Non-DN schools”)
• Gold standard for evaluation research: Diplomas Now caused any differences in school and student outcomes
• Fairest way to allocate scarce resources
It’s size: • 11 school districts, 62 secondary schools• 32 DN, 30 non-DN
4
Notable about this Study
It’s generalizability: • The study districts are broadly representative of urban
districts across the country• The schools represent some of the most challenged in the
nation• The students represent those traditionally dealing with the
most educational and life challenges
It’s duration:• US DOE Office of Innovation and Improvement awarded
extension grant for the evaluation: follow 2 student cohorts for 4 years
• Importance of continuity of implementation of the program model
5
Diplomas Now i3 Validation Study
• Second report (May 2015) on the first and second years of implementation: “fidelity” and “service contrast”
• The third report (early 2016) will include the impact of Diplomas Now on students’ ABCs:
– Attendance– Behavior– Course Performance
6
Fidelity of Implementation
• How closely the model as implemented in the Diplomas Now schools matched the original design.
• Fidelity measures created with support from DNIST to identify the components of the model and to set thresholds for successful implementation of each component.
• Data from surveys of Diplomas Now field staff and partner organizations’ data systems.
7
Fidelity of Implementation
• In Year 1 and Year 2, the majority of the model was implemented as designed.– Fidelity measurement based on 111 components clustered in 9
“inputs” and reported on a 0-1 scale– Year 1 = 0.61 and Year 2 = 0.62
• While there is still room for growth, the level of fidelity necessary to achieve impacts on student outcomes is still unknown.
9
Service Contrast
• The extent to which the Diplomas Now model as implemented created differences between practices in Diplomas Now and Non-Diplomas Now schools.
• Created survey items that focused on the kinds of activities implemented in DN schools, but used language that any teacher or administrator could understand.
• Teachers and administrators were surveyed in both DN and Non-DN schools.
10
Service Contrast
• Implementation of Diplomas Now made schools different in organization and staff practices than what would have been expected.
• In Year 2, across 29 measures:– 26 indicated positive change made by Diplomas Now– 16 of these differences were statistically significant
• From Year 1 to Year 2, differences increased, often driven by stability in DN schools and declines in Non-DN schools.
11
Outcomes and Impacts by Implementation Quality (Career Academies example)
85 84
0
10
20
30
40
50
60
70
80
90
100
Program A: Strong
Gra
du
ati
on
Ra
tes
(%
) 71
53
0
10
20
30
40
50
60
70
80
90
100
Program C: Weak
Gra
du
ati
on
Ra
te (
%)72
85
0
10
20
30
40
50
60
70
80
90
100
Program B: Moderate
Gra
du
ati
on
Ra
tes
(%
)
Career Academy Group Control Group
12
Diplomas Now i3 Validation Study
• Research Questions – Phase II (extension):
1. What is the impact of Diplomas Now on four-year student outcomes: the 9th grade transition of middle grades students and the graduation of high school students
2. What additional lessons can be learned about implementation of the model during national expansion, particularly about relationships between implementation and outcomes?
13
For More Information…
• See the second evaluation report:
http://www.mdrc.org/publication/moving-down-track
• See the evaluation project page:
http://www.mdrc.org/project/diplomas-now
Contacts:
William Corrin, MDRC, [email protected]
Susan Sepanik, MDRC, [email protected]