Dr. Bonnie J. Faddis & Dr. Margaret BeamRMC Research

Fidelity of Implementation and Program Impact

2

PPS Striving Readers Partners

• Grantee: Portland Public Schools

• Professional Developer: University of Kansas, Center for Research on Learning

• Evaluator: RMC Research

3

Targeted Intervention Overview

• Goals:– To increase reading achievement

– To increase reading motivation

• Intervention features: – Xtreme Reading curriculum

– Small class size (15 students/teacher)

– Ongoing professional development

4

Eligibility for Intervention

• Students enrolled in Grades 7–10

• Students reading 2 or more years below grade level on the Oregon State Assessment Test (OSAT)

• Students without OSAT scores who are reading 2 or more years below grade level on the GRADE test

5

Intervention Characteristics

• 9 schools – 5 middle schools– 4 high schools

• 18 targeted intervention teachers (2 per school)

Year 1

• 9 schools – 1 middle school– 2 K–8 schools– 1 6–9 girls’ school– 1 8th grade academy– 4 high schools

• 18 targeted intervention teachers

Year 2

6

Intervention Characteristics

Year 3

• 10 schools– 2 middle schools– 3 K-8 schools– 1 6-10 girls’ school– 4 high schools

• 14 targeted intervention teachers

7

Intervention Staffing Plan

• 25-30 students per grade level assigned to intervention class at each school

• Intervention class paired with language arts or social studies class in high school

• Intervention class integrated with LA/SS block in middle school

• 2 teachers co-taught intervention class and LA/SS class (Years 1-2)

8

Counterfactual

• Xtreme Reading taught during LA/SS block

• Control group students received LA/SS instruction with higher student-teacher ratios

Middle Schools

• Xtreme Reading was an elective course

• Control group students enrolled in a range of other electives across content areas

High Schools

9

Xtreme Reading Model

Metacognition

LINCS Vocabulary Strategy Learning new ways to remember the meaning of vocabulary

Word Mapping Strategy Learning new ways to remember the meaning of vocabulary

Word Identification Strategy Learning how to pronounce multisyllabic words

Self-Questioning Strategy Learning to ask yourself questions, make predictions, and talk about answers as you read

Visual Imagery Strategy Learning to make pictures in your mind while reading a passage

Paraphrasing Strategy Learning to put main ideas and details in your own words

Inference Strategy Learning to ask and answer thoughtful questions as you read, infer, and predict information

Motivation

Possible Selves Increasing student motivation by thinking about important goals for the future

Class Management

Xpect to ACHIEVE Understanding and following clear guidelines that support a successful learning community

Community Learning Skills

The SCORE Skills Social skills for cooperative groups

Talking Together How to participate respectfully in class discussions

10

8 Stages of Instruction

• Describe rationale and strategy steps• Model strategy through think-aloud• Verbally rehearse strategy steps• Guided reading practice• Paired reading practice with fluency tests• Independent practice w/comprehension check• Apply strategy during oral reading with

teacher• Integrate strategy with text from other classes

11

Implementation Results

• Professional Development

• Classroom Implementation

• Teacher Buy-In

• Factors that Facilitated Implementation

• Barriers to Implementation

12

Professional Development Participation

• Group session attendance > 75%– Year 1: 40% of middle school and 25% of high

school Xtreme teachers

– Year 2: 89% of middle school and 75% of high school Xtreme teachers

– Year 3: 100% of middle school and 71% of high school Xtreme teachers

• Added in Year 2 & 3: – Monthly meetings of Xtreme teachers

13

Classroom Implementation

School

Average Fidelity

Year 1 Year 2 Year 3

Middle Schools

Alpha 92% 89% 79%*

Beta 96% 90% 92%

Gamma 72% 84% 93%

Delta 93% 91% 94%*

Epsilon 94% 88% —

Phi — — 95%

Zeta — — 80%*

High Schools

Kappa 83% 63% 92%

Lambda 84% 58% 82%

Sigma 82% 78% 74%*

Theta 45% —* 80%*

14

Teacher Buy-In

Professional Development &

Support*

Perceived Program

Effectiveness**

Middle High Middle High

Year 1 4.2 3.6 4.1 3.6

Year 2 4.2 3.2 4.0 3.7

Year 3 4.4 4.0 4.6 4.0

* Rating Scale: 1 = Strongly disagree; 5 = Strongly agree** Rating Scale: 1 = Not at all helpful; 5 = Very helpful

15

Factors that Facilitated Implementation

• Xtreme teachers liked the curriculum• Students liked the books• Teachers liked the professional

development• Curriculum pacing schedules improved• 80% of Year 2 teachers had experience• 62% of Year 3 teachers had experience

16

Barriers to Implementation

• Organization of Xtreme materials was confusing

• Teacher skills with low achievers• Changes in middle school configurations

reduced target population

• Small school organization in 2 high schools created scheduling problems

• High school counselors resistant

17

Impact Evaluation

• Questions– Was the Striving Readers Xtreme

intervention effective in improving students’ reading achievement?

– Were the effects of the Striving Readers Xtreme intervention similar for middle schools and high schools?

18

Impact Evaluation

• Random AssignmentEligible students were randomly assigned to treatment or control (stratified by school and grade level)

• Analytic DesignIntent-to-treat statistical model using 2-level HLM (students clustered within schools)

19

Impact Evaluation

• Outcome MeasureSpring GRADE test normal curve equivalent (NCE) scores

• Covariates– Fall GRADE pretest normal curve equivalent

(NCE) scores– Cohort– Ethnicity (Black, Hispanic)– English language proficiency– Grade level

20

Statistical Power

• 2-level HLM cluster randomized trial

• N = 1,273

• Power = 80%

• Alpha = .05

• Minimum detectable effect (MDE) = .27

21

Random Assignment

Striving Readers Enrollment

Met eligibility criteria n = 2,438

Randomized

Treatment Group

Allocated to intervention n = 1,223

Excluded from study n = 296

Lost to follow-up n = 328

Analyzed n = 599

Allocation

Ineligibility

Follow-Up

Control Group

Allocated to intervention n = 1,215

Excluded from study n = 267

Lost to follow-up n = 274

Analyzed n = 674

Analyzed n = 674

Analysis

22

Student Sample

Grade Level N7 3478 3139 34210 271Total 1,273

23

Student Characteristics

CharacteristicTreatment

PercentControl Percent

Gender

Male 53 53

Female 47 47

Ethnicity

White 25 25

American Indian 2 2

Hispanic 30 31

African American 28 29

Asian 15 12

Special Education Services 25 25

ELL Services 27 25

Note. Treatment total N = 599; Control total N = 674.

24

Impact Results

Estimated Impact

Group nImpact

() S.E.Effect Size

p-value

Overall 849 2.58 0.79 .15 .002

Middle School

443 4.84 0.90 .29 .000

High School 406 0.16 1.31 .01 .901

25

Impact Results

• Additional Finding

Significant school level variability in treatment effects in both the overall analysis and in the middle school analysis

26

Effects of Implementation

• Question

To what extent do teacher level variables explain school level variability in treatment effects?

27

Analytic Design

• Statistical model using 2-level HLM (students clustered within schools)

• Teacher level variables aggregated up to school level

• Separate models by implementation year

28

Level 2 Variables

• Percent fidelity of implementation

• Percent of professional development training attended

• Years of teaching experience

29

Results

• Fidelity of implementation significantly contributed to school level variability in treatment effects in Years 1 and 3, but not in Year 2

30

Summary

• Program was more effective in middle than in high schools

• Variability in student outcomes partially explained by fidelity of classroom implementation

• Teaching experience and amount of professional development not related to student outcomes