sra flex literacy research - mcgraw-hill education

ResearchRonald C. Martella, Ph.D.

Nancy E. Marchand-Martella, Ph.D.

confidential and proprietary

confidential and proprietary 2

confidential and proprietary 3

confidential and proprietary 4

confidential and proprietary

Research Base:White Papers

5

confidential and proprietary 6

confidential and proprietary 7

confidential and proprietary 8

confidential and proprietary 9

confidential and proprietary 10

confidential and proprietary 11

confidential and proprietary 12

confidential and proprietary 13

confidential and proprietary

DATA

14

confidential and proprietary

Learner Verification Study

15

• Main purpose to provide feedback using pre-publication version of program.

• Imperfect implementation—full program not completed, starts and stops throughout, changes in program and technology improvements.

• Even with focus of study, positive results were seen across diverse settings—non-special education, special education, after-school.

• Social Validation: Teachers and students liked program and reported effects.

• Statistically significant results seen on TOWRE-2.• Positive results on GRADE; statistically significant effects for Grade 5

Total and Comp; statistically significant effects for Vocab at Washington site.

• Increasing trend on DIBELS.

confidential and proprietary 16

Background • Designed by Department of Academic Research as an

Exploratory study• Incorporates 141 students across grades 3 through 8

attending schools in GA, SC, TN, TX, and WA• All students performing at least two years below grade level

in reading • Demographics

– Higher percentage of males (58%) than females – 65% identified as Caucasian– Over half identified with special needs, most of these SLD– 86% free/reduced lunch

confidential and proprietary 17

Implementation

• **Study utilized pre-published version of materials **

• Staggered implementation– Late September: SC– October: GA, TN, WA– Early November: TX

• Setting– GA and TX in non-special education classrooms– TN and SC in special education classrooms – WA in an ‘after-school’ setting

confidential and proprietary 18

Implementation, cont.• Consultants conducted on-site visits to monitor

implementation throughout the study– Strong implementation for SC, TN, and TX

• All program components; 90 – 120 minutes per day for instruction– Slight deviations observed in GA

• All program components; 90 – 120 minutes per day for instruction – Partial implementation in WA

• Digital and Print; 45 minutes – 1 hour for instruction– Initial difficulties with technology/connectivity in WA which

impacted implementation of the Digital Experience

confidential and proprietary 19

Social Validity Questions of Interest• What are teachers’ and students’ attitudes and

perceptions of the program?– Online survey for teachers, end of study– Student interview, end of the study

• What effect does FLEX Literacy have on students’ perceived reading skill and interest in reading?– Pre- and post-program administration of a 9 item

survey

confidential and proprietary 20

Results

• Teacher Survey

confidential and proprietary 21

Teacher Survey

• All teachers (100%) reported that students were engaged during the Print Experience , Digital Experience, and the Project Experience.

• All teachers (100%) reported that routines were easy to follow, students felt successful, and content standards were addressed in the Print Experience.

• All teachers (100%) indicated that the Teacher’s Guide is well-organized, students felt successful, and content standards are addressed in the Project Experience.

confidential and proprietary 22

Teacher Survey, cont.

• Responses on items addressing the Digital Experience varied across sites and largely reflect issues with technology/connectivity that have since been corrected.

• Despite this, the majority of teachers cited the Digital Experience and the opportunity for students to work at their own pace as particular strengths of SRA FLEX Literacy.

confidential and proprietary 23

• What four words would you use to describe SRA FLEX Literacy?

• Individual• Challenging• High-expectations• Collaborative• Intense• Flexible• Thought provoking• Life time skills• Interesting• Relevant• Deliberate• Engaging• A very different format

confidential and proprietary 24

• 100% of teachers indicated they would recommend SRA FLEX Literacy to a colleague.

• Yes. I already have. Teachers are looking for an individualized program for students struggling in reading and this program (mainly the digital) is equivalent to a student who is sitting one on one with a reading tutor. The teacher knows exactly where the student is and 30 students can work at once on their level, rather than hiring a tutor for 30 minutes of small group once or twice a week. More students benefit from individualized instruction resulting in less being labeled with a disability.

confidential and proprietary 25

Results

• Student Interview

confidential and proprietary 26

How did SRA FLEX Literacy help you become a better reader?

• 87% of students reported that they felt SRA FLEX Literacy helped them become a better reader.

Read/Pronounce Words

Read Aloud Asks Questions Other0

10

20

30

40

50

60

70

80

confidential and proprietary 27

90% of students’ responses were positive, indicating they enjoyed the program or thought it improved their reading skills.

• What would you tell other students about SRA FLEX Literacy?Fun/Cool Read Better Easy/Hard DK

0

10

20

30

40

50

60

confidential and proprietary 28

59%

16%

7%

6%

12%

Digital ExperiencePrint ExperienceProject ExperienceLearningDK

What is the best thing about SRA FLEX Literacy?

confidential and proprietary 29

• The majority of students responded that they enjoyed the Project Experience.– Roughly one-third reported that their favorite part

was “working in teams”– Other responses varied considerably, but included

“researching,” “coloring/creating,” “making PowerPoint slides,” and simply, “all of it.”

confidential and proprietary 30

Results

• Student Survey

confidential and proprietary 31

• Short, nine-item survey administered before and after participation in FLEX

• Designed to capture student perceived reading skill and interest in reading– I am a good reader– I like to read at home

• Response categories ranged from Yes! to Not At All

confidential and proprietary 32

All Students

• After participation in FLEX, more students selected Yes!, or Mostly in responding to the statements:– I am a good reader (15 % increase)– When reading, I can easily read all of the words

(14% increase)– When reading, I know what all of the words mean

(10% increase)

confidential and proprietary 33

By Subgroup

• I am a good reader– Grade 5 (25% increase)

• I can read as well as other students– Grade 5 (8% increase)

• When reading, I know what all of the words mean– Grade 5 (20% increase)

• I would enjoy receiving a book as a present– Grade 8 (25% increase)

• When reading, I can easily read all of the words– Grade 8 (33% increase)

confidential and proprietary 34

Results

• Achievement Measures

confidential and proprietary 35

Questions of Interest, cont.

• Association with basic reading, vocabulary, and comprehension skills– Pretest and Posttest using published, norm-

referenced tests

• Association with oral reading fluency skills– Repeated administrations of the Oral Reading

Fluency (ORF) subtest of the DIBELS

confidential and proprietary 36

Quantitative Measures

• Test of Word Reading Efficiency, Second Edition (TOWRE-2)– 2 subtests: Sight Word Efficiency and Phonemic Decoding

Efficiency • Group Reading Assessment and Diagnostic Evaluation

(GRADE)– Comprehension and Vocabulary Composites along with a

Total Score• Oral Reading Fluency (ORF) subtest from the DIBELS

– One-minute reading assessment of reading fluency (words per minute)

confidential and proprietary 37

TOWRE-2 and GRADE

• Group administered• Contain parallel forms for pretest and posttest• High reliability/validity• Give norms for off-grade level assessment• Provide standard scores/are norm-referenced– Allow for comparison across various tests and groups– Can be manipulated for use in statistical tests– Easily interpretable – Often utilized in the identification of students with

disabilities

confidential and proprietary 38

A word about “statistical significance”

• Indicates whether or not there is a difference between groups simply due to chance

• Is impacted in part by sample size– If treatment effects are small (and present), larger

samples are more powerful in detecting them.– With smaller samples, treatment effects (if

present) must be relatively larger to be detected.

confidential and proprietary 39

…and Effect Size• Describes the magnitude of the difference• Represents the average difference between groups,

expressed in standard deviation units – For example, an effect size of .30 means “three-tenths of a

standard deviation”• Effect size is descriptive for that particular group• Cohen’s d

– ‘Small’: .20– ‘Medium’: .50– ‘Large’: .80 and greater– For educational research, .25 is considered minimal level

confidential and proprietary 40

Learner Verification Study (continued)

• Effect sizes are measures of magnitude—i.e., how large are the effects.• Effect sizes were positive. Note that .25 effect size is the minimal level for

educational significance (Adams & Engelmann, 1996); effect sizes over 1.00 are relatively rare in education.

Some example results:Grade 5

Effect size = .42 on GRADE Comp Effect size = .38 on GRADE Total

Grade 8Effect size = 1.49 on TOWRE-2

TennesseeEffect size = 1.37 on TOWRE-2

WashingtonEffect size = .73 on TOWRE-2 Effect size = .52 on GRADE Vocab

confidential and proprietary 41

All Students• Statistical significance

achieved for TOWRE-2, effect size is small.

• Non-statistically significant increases for the GRADE.

• For all measures, pretest and posttest scores fall within the Low Average range.

confidential and proprietary 42

Student Performance by Grade Level

• Grades 3, 4, 6, and 7– Increases noted

• Grade 5– Statistically significant increases for GRADE

Comprehension (d = .42, small) and GRADE Total (d = .38, small effect).

• Grade 8– Statistically significant difference for TOWRE-2 (d =

1.49, large effect)

confidential and proprietary 43

Grade 5• Statistical significance

achieved for GRADE Total and GRADE Comp; effect size is small.

• Non-statistically significant increases for the TOWRE-2 and GRADE Vocab.

• The average comprehension score increased from the Low range to the Low Average range.

confidential and proprietary 44

Grade 8• Statistical significance

achieved for TOWRE-2,; effect size is large.

• Stable performance for the GRADE.

• The average TOWRE-2 score increased from the Low Average range to the Average range.

confidential and proprietary 45

Student Performance By Site

• Increases shown in Georgia, South Carolina, and Texas

• Tennessee– Statistically significant increases for the TOWRE-2 (d = 1.37,

large effect)

• Washington– Statistically significant increases for the TOWRE-2 (d = .73,

medium effect) and GRADE Vocabulary (d = .52, medium effect).

confidential and proprietary 46

Tennessee• Statistical significance

achieved for TOWRE-2; effect size is large.

• Non-statistically significant increases for the GRADE.

• The average TOWRE-2 score increased from the Low Average range to the Average range.

confidential and proprietary 47

Washington• Statistical significance

achieved for TOWRE-2; effect size is medium.

• Statistical significance achieved for GRADE Vocabulary; effect size is medium.

confidential and proprietary 48

SLD Category• Statistical significance

achieved for the TOWRE-2, effect size is large.

• Non-statistically significant increases are noted on the GRADE Total and GRADE Comprehension.

• Stable performance on GRADE Vocabulary.

• The average TOWRE-2 score increased from the Low range to the Low Average range.

confidential and proprietary 49

Results

• DIBELS• Oral Reading Fluency

confidential and proprietary 50

Oral Reading Fluency

• General upward trend across grade levels• The average gain in words per minute by

grade level ranged from a 4% to a 59% increase

• Strongest results for grade 5

confidential and proprietary 51

GRADE 5ORF

• At baseline, students read an average of 63 words per minute.

• Students read an average of 100 words per minute on the Spring benchmark.

• Students gained, on average, 37 words per minute. This reflects a 59% increase over baseline levels.

confidential and proprietary 52

Results

• LEXILE Measures

confidential and proprietary 53

confidential and proprietary 54

confidential and proprietary 55

Because the total number of lessons students receive may have an effect on their scores, the correlation between Lexile® growth and percentage of lessons completed for the program was calculated.

The correlation between Lexile® growth and percentage of yearly lessons completed for the program was statistically significant, r = .559, n = 69, p < .001 (two-tailed). This correlation indicates there was a statistically significant, positive relationship between Lexile® growth and the percentage of yearly lessons completed.

confidential and proprietary 56

Summary

• The majority of students and teachers reported that they enjoyed using SRA FLEX Literacy, would recommend it to others, and felt successful while using it.

• There were some positive shifts in perceived reading skill and interest in reading, for several groups.

• Pre-post program differences on the TOWRE-2 were determined significant for several groups.

• Oral Reading Fluency scores show an upward trend, over time.

confidential and proprietary

Washington School District Study

57

• Large study in Eastern Washington School District.• 55%-75% free and reduced price lunch.• Primary purpose to determine student academic gains.• Full program implemented for entire year.• Up to 150 students across Grades 6, 7, and 8.• Intervention students.• Multiple measures including Washington State MSP (Measurements of

Student Progress) test results, AIMSWeb, TOWRE-2, GRADE, in-program assessments.

• Possible math test results on MSP.• Compare with students who are not in intervention program but with similar

demographics.