Improving Comprehension Online Project, 2005-08

Designing and testing a universally designed strategic digital literacy environment for diverse learners

Bridget Dalton, Vanderbilt University

Patrick Proctor, Boston College

IES Research Conference

Washington, DC ~ June 11, 2008

A goal 2 development award

to CAST, Inc.

Research team• Bridget Dalton (Co-PI), Vanderbilt

University, and Elaine Mo, Kristin Robinson, Ge Vue, Mary O’Malley, & Boris Goldowski, CAST, Inc.

• Patrick Proctor (Co-PI), Yi-Chien Li, & Kevin O’Connor, Boston College

• Catherine Snow (Co-PI), Paola Uccelli, Sabina Neugebauer, Lorena Landeo Schenone, Harvard Graduate School of Education

• School partners: 3 semi-urban and 1 urban school in northeastern Massachusetts

Project goal

• To develop and test a universally designed

(Rose & Meyer, 2002) strategic digital reading approach (Dalton & Proctor, 2007) to improving reading achievement of 5th grade students, including bilingual students and struggling readers

Multiple perspectives required

Vocabulary

Digital literacy

environmentsUniversal

Design for Learning

New literacies

Bilingualism2LA

Reading comprehension

icon

Universal design for learning (Rose & Meyer, 2002)

Design for the broadest range of learners from the beginning; avoid retrofitting

Provide multiple means of • Representation• Expression• Engagement

Assumptions

• Shift to universal design perspective – Attention to diversity and individual difference

benefits individual and society

• New literacies, while more complex, are more flexible and inclusive– potential to level the playing field for those

who have not fared well with print literacy

Rand Reading Study Group’s (2002) reading comprehension heuristic

text reader

activity

Sociocultural context

comprehension

text reader

activity

Sociocultural context

comprehension

Strategic Digital Reading (Dalton & Proctor, 2007)

textreader

activity

Sociocultural context

comprehension

Comprehension in a new literacies landscape: Strategic digital reading

text

reader

activity

Sociocultural context

comprehension

How does ICON support diverse learners in relation to…

• Representation?

• Expression?

• Engagement?

• What is unique for ELLs?

• What features/supports are essential for some; good for many/all?

Iterative design, formative feedback and testing

Y1. Develop

Vocabulary

Y2. Compare Vocabulary, Strategies & Combo

Y3. Compare Combo

Vs. Control

Y1. Develop

Vocabulary

Y2. Compare Vocabulary, Strategies & Combo Versions

Y3. Compare Combo

Vs. Control

ICON optimal prototype (Yr. 3)

Embedded Strategies

Spanish language support

Coaches

Level 1 coaches provide text-specific models and think alouds. As skill increases, students select strategies and coaches provide generic think-alouds.

Vocabulary: Connect It!

Vocabulary: Language Alert

More than 60% of the power words are Spanish English cognates

Vocabulary: Web It!

Vocabulary: Caption It!

All 3 years/studies: Feasibility, appeal & usability

• Teachers and students view ICON as a helpful reading tool, easy to use, & engaging

• Technical support required; bandwidth issue• Variation in teacher enactment of ICON suggests need for

additional study• English proficiency levels influence ways in which

students use ICON and extent to which additional support is needed– Peer collaboration one means of support

• Increased sensitivity to learner (needs, use of system, performance) is likely to benefit all

Y1 Study of Semantic Depth• 35 students, 24 bilinguals (Spanish and other low-

incidence languages), 11 English monolinguals

• Oral language skills (WJ picture vocab+listening comprehension)

• Reading skills (WJ passage comp + MCAS ELA score)

• Average semantic depth score for 8 target words (Anxiously, Bitter, Dense, Grasp, Ignore, Menacing, Powerless, Relieved)

Yr. 1 study of semantic depth(Proctor, Uccelli, Dalton, & Snow, in press)

Effective teaching and learning activities targeted for further analysis: Caption-It

Semantic depth was a significant predictor of reading performance, mediated by interaction

with English oral language proficiency

Figure 1. Effect of semantic depth predicting reading latent variable as a function of oral language proficiency

-1.5

-1

-0.5

0

0.5

1

1.5

2

2.5

0 1 2 3 4 5

Semantic Depth Score

Re

ad

ing

La

ten

t S

co

re

10% Oral Lg

50% Oral Lg

90% Oral Lg

*No effect for language status (bilingual vs. monolingual)

What did we learn from Y1 vocabulary study?

Caption It: Encouraging but preliminary – As both an activity and an assessment, it appeared to reduce the monolingual/bilingual gap in students’ performance, though oral language was heavily implicated

Semantic Depth: Promising but far from final Semantic depth showed a positive association with reading

comprehension, beyond the contribution of decoding and oral language skills

Semantic depth seems to play a more prominent role as oral language skills improve

Semantic depth refers to a cluster of skills: associations among these and with other dimensions of vocabulary depth need to be explored.

Year 2 study: Strategies vs. vocabulary vs. combo

• Very hard to find research that compares effects of vocabulary versus comprehension instruction

– Likely because the two are so strongly intertwined

• Given that vocabulary is a primary focus of instruction

for ELL students, we found this question intriguing, and asked 2 basic questions:

1. Does assignment to condition (Vocabulary-only, Strategy-only, Combo) affect students’ performance on standardized and researcher-developed measures of vocabulary and comprehension?

2. Do the effects vary by language status (monolingual, Spanish-English bilingual, other bilingual)?

Our hypotheses

For standardized and researcher-developed

vocabulary:

Combo > Vocabulary > Strategy

For standardized and researcher-developed

comprehension:

Combo > Strategy > Vocabulary

Y2 study of vocab vs strategies: What matters and for whom?

(Dalton, Proctor, Uccelli, Mo, & Snow, in preparation)

• 106 students, 21 Spanish-English bilinguals, 17 other-English bilinguals, 68 monolinguals in 6 classrooms, 3 districts

• Random assignment to condition (vocabulary, strategy, combination)

• 14-week intervention• Pre-post standardized testing,

embedded vocabulary and comprehension quizzes

Y2 effect size overview

What did we learn from Y2 study of vocabulary vs. reading

strategies vs. combined?

• Overall, hypotheses held, and combination version showed strongest results across standardized and researcher measures

• Did the small sample size mask learner by treatment interactions?

• Theoretically, an interaction between student characteristics (reader type and/or language status) would make sense

– Sample size may be too small– Student controls access to support and may not be

making good decisions about when and how to use support

– Thresholds of language proficiency

Year 3 study: Combo vs controlHaving established general effectiveness, time to move toa comparison between treatment and control using optimalversion of ICON

• Quasi-experimental study

• 12 classrooms, classrooms assigned randomly to treatment or control condition, n = 227 (108 control, 119 intervention; 10.5% other bilinguals, 48.5% Spanish-English bilinguals, 41% English monolinguals)

• For intervention group, 2 x 50 minutes per week, for 16 weeks

• For control group, across the three districts, standard literacy curriculum included reading strategies focus, but limited vocabulary instruction

• Initial training of teachers and students by research team, gradual release of ICON prototype teaching responsibility

• Analyses conducted at student level, randomization at teacher level

Y3 measures

• Gates-MacGinitie reading vocabulary and comprehension subtests pre- and post-intervention

• Aprenda reading vocabulary - pre• Researcher-designed breadth of vocabulary

(targeted words) post-intervention– 20-item multiple choice assessment

• Reseacher-designed depth of vocabulary, post-intervention– 5-item definition, drawing, & captioning assessment

Y3 results

General results:• No effect of condition on standardized

measures; significant voc and comp gain for both groups

• Strong effect of condition on researcher developed measures

Condition Depth/BreadthStandard

Vocab and comp

Condition effects on ICON vocabulary breadth

0

10

20

30

40

50

60

70

80

Mono BilSp BilOth

Control

Experimental

Significant effect of condition on ICON voc. Breadth F(1,205) = 56.62, p < .001

Significant difference between Spanish bilinguals and English monolinguals (t = 5.1, p < .001)

Strong readers significantly outperform average (t = 5.1, p < .001) and struggling (t = 12.6, p < .001)

No interactions by language or reader status and condition

Condition effects on ICON vocabulary depth

Exp. significantly outperform Control on vocabulary depth

F(1,224) = 101.4, p < .001

English monolinguals significantly outperform Spanish bilinguals (t = 5.3, p < .001) and non-Spanish bilinguals (t = 2.2, p < .05)

Strong readers significantly outperform average (t = 5.0, p < .001) and struggling (t = 9.4, p < .001) readers

No interactions by language status, BUT average-reader X condition interaction (p = .048)

Spanish-English bilinguals and ICON depth of vocabulary

0

1

2

3

4

5

6

7

8

400 500 600 700

Aprenda Vocabulary Score

Su

m E

ng

lish

Vo

ca

bu

lary

De

pth

S

co

re control vocdepth

intervention vocdepth

For intervention Spanish-English bilinguals, Spanish Vocabulary scores explained English Vocabulary Depth performance, after controlling for condition and prior English proficiency.

Conclusions and next steps

• Continue to analyze Year 3 data– Worklog responses, multimodal retellings, student

feature use, teacher use of feedback support.

• For whom does this intervention work best? – Goal 3: effects for Spanish-English bilinguals and

struggling readers are intriguing– Goal 2: Work for transfer. Design for increased

sensitivity to learner characteristics, especially language proficiency.

• Distal effects on standardized measures