the national i 3 evaluation of diplomas now

THE NATIONAL i3 EVALUATION OF DIPLOMAS NOWDN Summer Institute – July 9, 2013

Topics for this session Refresher on the evaluation

Evaluation team i3 context Study design

Overview of data collection Student and staff surveys 2013-14 Fidelity of Implementation

Wave 1 Schools – Spring 2012

Partner Organizations – DN Study

MDRC 40-yr old nonprofit, nonpartisan, education and

social policy research organization dedicated to learning what works to improve programs and policies that affect low-income individuals and communities

ICF International 40-yr old research and consulting firm that seeks

to provide solutions and services that address challenging policy issues

Goals of the i3 Validation Grant Program

Identify some of the most promising school improvement initiatives

Provide support for them to scale up nationally

Research their effectiveness using the most rigorous methodologies available

Document lessons learned about implementation during the scale-up process

Publicize study results to influence national and state policy

Overview of Diplomas Now Study

Overall goal: Validate effectiveness of the Diplomas Now model

Research Questions: 1. What is the impact of Diplomas Now on

students’ outcomes, particularly with regard to attendance, in-school behavior, and course performance?

2. What lessons can be learned about implementation of the model during national expansion?

Study Design: Random Assignment A random assignment design uses

a lottery to assign participating schools to one of two groups DN schools (implementing the DN

program) Non-DN schools (implementing any

other school reform program)

National sample Currently 62 secondary schools in 11

districts across the country participating in the study

Study will compare student outcomes in the 32 middle and high schools that implement DN to those in the 30 schools that do not

Study will document implementation in the 32 DN schools, and also investigate how it compares to any school improvement efforts in the 30 Non-DN schools.

Data CollectionDN Schools Non-DN Schools*

Student Records Data (will be obtained from district)

Student Records Data (will be obtained from district)

Student, Principal, and Teacher Surveys

Student, Principal, and Teacher Surveys

Case Studies (Interviews, focus groups, observations; only at 8 selected schools) * For their participation, Non-DN

Schools will receive $10,000 compensation each year

Surveying: Spring 2014

Wave and ActivityWave 1 Schools - Principals and teachers complete online survey

- 8th / 11th grade students complete survey

Wave 2 Schools- Principals and teachers complete online survey

- 7th / 10th grade students complete survey

Questions and Answers

DIPLOMAS NOW EVALUATION

Fidelity of Implementation: Spring 2012

DRAFT - NOT FOR DISTRIBUTION 11

12Implementation Fidelity Data Sources

DRAFT - NOT FOR DISTRIBUTION

Fidelity of Implementation Data come from the following sources: Diplomas Now Implementation Support

Team (DNIST) Survey School Transformation Facilitator (STF)

Survey Citi Year Program Manager (CYPM) Survey Communities In Schools (CIS) Site

Coordinator (SC) Survey Communities In Schools (CIS) Site Records

13DN Fidelity of Implementation


Fidelity of implementation is based on the DN Logic Model and measured using the Fidelity of Implementation Matrix.

The matrix is built on 111 separate components, 62 of which were identified as critical to adequate implementation.

These components sort into 9 inputs, 6 of which were identified as critical to adequate implementation.



That is…

Input 1

Component X

Component Y

Component Z



And…

Overall

Fidelity

Input 1

Input 2

Input 3

Input 4

Input 5 Inpu

t 6Input 7

Input 8

Input 9

16 DRAFT - NOT FOR DISTRIBUTION

17

Fidelity Matrix: Inputs


Program Staff Training and Professional Development 18 individual components, 15 of which are

critical Integrated On-Site Support (Critical

Input) 11 individual components, 9 of which are

critical Family and Community Involvement

6 individual components, 1 of which is critical

18

Fidelity Matrix: Inputs (cont.)


Tiered Intervention Model (Critical Input) 3 individual components, 2 of which are critical

Strong Learning Environments (Critical Input) 6 individual components, 4 of which are critical

Professional Development and Peer Coaching (Critical Input) 5 individual components, 2 of which are critical

19

Fidelity Matrix: Inputs (cont.)


Curriculum for College Readiness 24 individual components, 4 of which are

critical Student Supports (Critical Input)

24 individual items, 19 of which are critical Student Case Management (Critical

Input) 14 individual items, 5 of which are critical

20

DN Fidelity of Implementation


Fidelity is measured in two ways, by a categorical rating and a continuous score:1. Implementation Rating (categorical

measure): focused on critical components

2. Implementation Score (continuous measure): inclusive of all components

21Implementation Rating


The rating focuses on “critical” components and “critical” inputs. How well did a site implement aspects of the model hypothesized to be most important to improving student outcomes?

Each input (e.g., program staff professional development) of the DN model was rated as either:1. “Successful” - met implementation thresholds

for all “critical” components2. “Developing” - did not meet threshold for one

or more critical components

22Implementation Rating (cont.)


Individual input ratings served as the basis for the site-level fidelity rating, which has been broken up into four categories: 1. Low: successful on less than 3 critical

inputs2. Moderate: successful on at least 3 critical

inputs3. Solid: successful on at least 5 critical inputs4. High: successful on 8 or more inputs

including 5 critical inputs

23Implementation Score


The score measures implementation of all aspects of the DN model, going beyond just the “critical” aspects of the model. How well did a site implement the model overall?

Each input is scored based on how well every one of its components was implemented.

The average of the 9 input scores provides the site-level implementation score (0-1 scale: the proportion of the entire model implemented by a site)

28

Wave 1 Schools - Year 1 Preliminary Findings


Cohort 1: 12 DN Sites 5 High Schools 7 Middle Schools

Cross-Site Implementation Rating % of DN sites with Solid Implementation Rating: 0% % of DN sites with Moderate Implementation Rating: 42% % of DN sites with Low Implementation Rating: 58%

Cross-Site Implementation Score Overall Implementation Score: 0.59

Highs and Lows: Critical Components by Input

NUMBER OF CRITICAL COMPONENTS

INPUT Total

Met by > 75% of sites

Met by < 50% of sites

DN Staff Trng./PD 15 10 4Integr. On-Site Supp.

9 6 0

Family & Cmty. Involv.

1 0 0

Tiered Intervention

2 2 0

Strong Learning Env.

5 1 0

Highs and Lows: Critical Components by Input (cont.)

NUMBER OF CRITICAL COMPONENTS

INPUT Total

Met by > 75% of sites

Met by < 50% of sites

PD and Peer Coach.

2 0 0

Curric. For Coll. Rdy.

4 1 2

Student Supports

19 11 1

Student Case Mngmt.

5 2 1

31Critical Components Met by < 50% of Sites


Program Staff Training and Professional Development

COMPONENT % of Sites Meeting

CY CORPS MEMBER trained in the use of data to identify interventions 42%

CY CORPS MEMBER received on-going support in the use of data to identify intervention supports 33%

Did DN team and school admin and teachers meet prior to the start of the school year for joint planning sessions 25%

11 module online course, approximately 1.5-2 hours per module 25%

32

Critical Components Met by < 50% of Sites


Curriculum for College ReadinessCOMPONENT % of

Sites Meeting

Student Team Literature (MS only, n =7) 29%

Savvy Readers’ Lab (MS only, n =7) 29%

33

Critical Components Met by < 50% of Sites


Student SupportsCOMPONENT % of

Sites Meeting

% of math classrooms with embedded CY CORPS MEMBERS 42%Student Case ManagementCOMPONENT % of

Sites Meeting

All Case Managed students identified with academic needs are provided with Academic Assistance interventions

25%

34

Cohort 1 - Year 1 Preliminary Findings by H.S.


Count of Critical

Inputs MetCount of All Inputs Met Rating Score

Belaire HS 3 4 Moderate .57

Booker T. Washington HS 3 3 Moderate .50

English HS 1 1 Low .39

Newtown HS 1 2 Low .54

Sheepshead Bay HS 1 3 Low .51

35

Cohort 1 - Year 1 Preliminary Findings by M.S.


Count of Critical

Inputs MetCount of All Inputs Met Rating Score

Broadmoor MS 3 4 Moderate .73

Capitol MS 2 3 Low .63

Clinton MS 2 3 Low .53

Dever-McCormack MS 1 1 Low .46

Drew MS 2 3 Low .73

Miami Edison MS 3 5 Moderate .74

Shaw MS 3 4 Moderate .71

National Evaluation Contacts

MDRC Project DirectorWilliam CorrinDeputy Director, K-12 Education(212) [email protected]

ICF Project ManagerAracelis GraySenior Manager, Health, Education and Social Programs(703) [email protected]

mailto:[email protected]

mailto:[email protected]

the national i 3 evaluation of diplomas now

Documents

schools spring

schools principals

schools principals

participating schools

dn programnondn schools

groupsdn schools

selected schools

secondary schools