A beginning understanding of the interplay between offline and online reading comprehension ability when adolescents read on the Internet

Julie Coiro, University of Rhode Island jcoiro@snet.net

International Reading Association Research Conference (May 3, 2008)

Where are we headed?

Introduction to the current situationTheoretical Frameworks Summarizing the increasing complexities that appear to characterize online reading comprehensionQuantitative measures, procedures, and findingsQualitative measures, procedures, and findings What do these findings mean for literacy theory, research, and classroom practice? Time for discussion (15 minutes)

The current situation

We do little to help struggling adolescent readers and their teachers (Biancarosa & Snow, 2004; Partnership for Reading, 2003; RAND Reading Study Group [RRSG], 2002).

As a result, there is a large and increasing gap in adolescents’ reading achievement (NCES, 2001; 2002; 2003).

At the same time, Internet technologies have increasingly become a part of our daily lives (e.g., Rainee & Hilton, 2005;

Friedman, 2005), introducing additional challenges to reading (Coiro, 2003; Coiro & Dobler, 2007; NICHD, 2000; RRSG, 2002).

The current situation

Unfortunately, we know little about…which skills/strategies/practices contribute the most to successful online reading comprehension; which students demonstrate these skills and which do not;how to characterize, measure, teach, and chart progress of these skills/strategies/practices in the classroom.

Unbelievably, we are doing little to learn more….No states include the reading comprehension demands of the Internet in their standards or in state reading assessments (Leu, Ataya, & Coiro, 2002).

What’s the big deal?

Researchers urged to move beyond group comparison studies to focus more closely on the attributes of skilled and less-skilled online readers (Abrami & Bernard, 2006; Dillon & Greene, 2003).

It is crucial that we learn more about the nature of online reading comprehension or risk even larger gaps in reading achievement and increasing numbers of readers who struggle in the networked information contexts that will dominate their future (e.g., ETS, 2003; IRA, 2001; Leu, 2007).

Purpose of the study

This sequential mixed-methods study sought to:

Investigate the extent to which new online reading proficiencies may be required to comprehend information on the Internet;

Explore the nature of online reading comprehension ability among three adolescent readers who read online at different levels of proficiency.

Theoretical Frameworks

Reading comprehension is an active, constructive process of making meaning (Pearson et al., 1992; Pressley et al., 1989; RRSG, 2002).

New literacies perspective (Coiro et al., in press; Leu, Kinzer, Coiro, and Cammack, 2004)

Adapting diachronic perspectives of literacy practices (Gutierrez & Stone, 2000)

Developmental theory of reading (Alexander, 1997; 2003; 2005)

Developmental Perspectives

Alexander (2006) argued, “profiles of successful and struggling readers are reflective of developmental forces” (p. 413).

Stages of acclimation, competency, and proficiency may characterize readers over a lifetimeMight these levels be useful for distinguishing strategy use among lower, average, and higher performing online readers over the course of a reading task as they read for information on the Internet?

Increasing Complexities of Text Structure

Impacts Reading Comprehension

Offline Information TextComprehensionHypertext Comprehension

Internet Text Comprehension (Online Reading Comprehension)

Offline Narrative Text Comprehension

Offline Text Structures And Reading Comprehension

A range of active, strategic processes are required to comprehend narrative text

• (Duke & Pearson, 2002; Paris et al., 1991; Pressley & Afflerbach, 1995)

Information texts pose additional challenges with complex concepts, specialized vocabulary, and unfamiliar text structures

• (Guthrie & Mosenthal, 1987; Kintsch, 1990; Meyer, Brant, & Bluth, 1980)

Reader characteristics play a central role in reading comprehension (RRSG, 2002; Guthrie; Alexander).

Key Reader Characteristics

Reading comprehension ability

Skilled readers apply a range of cognitive strategies such as determining key ideas, critically analyzing information, making inferences, etc. (Pearson, Roehler, Dole, & Duffy, 1992; RRSG, 2002).

Strategy use depends on the readers’ purposes and the types of texts they read (Duke & Pearson, 2002; Pressley, 2000)

Key Reader Characteristics

Prior knowledge

• The knowledge a reader brings to any text or learning situation (e.g., Anderson & Pearson, 1984)

• Skilled readers draw on general, topic-, and text-specific knowledge to make predictions, reason strategically, and remember main ideas (e.g., Alexander, Kulikowich, & Schulze, 1991; Alexander, Shallert, & Hare, 1991).

Key Reader Characteristics

Dispositions

• Habits of mind; tendencies to approach and/or respond to situations in certain ways (see Katz,

1988; Carr & Claxton, 2002)

• Distinct aspects of reading dispositions including beliefs, goals, values, purposes, and needs can influence comprehension

– (e.g., Baker & Wigfield, 1999; Guthrie & Wigfield, 1997; Horner & Shewry, 2002; RRSG, 2002)

Added Complexities of Informational Hypertext Structures

Hypertexts (closed systems) present multiple, non-linear pathways (Landow, 1994) Hypertexts extend the range of possible interconnections and pathways between texts (often, some paths are better than others) (Bolter, 1991; Burbules & Callister, 2000; Reinking, 1997) Content hidden beneath multiple layers and connected by links with fewer surrounding context cues (requiring high levels of inferencing) (Foltz, 1996; Spyrakis, 2000)

Author’s intertextual connections may be different than the reader’s connections, creating more complexity (Caney, 1999)

Reader Characteristics and Hypertext Comprehension

Hypertexts…“require skills and abilities beyond those required for comprehension of conventional, linear print” (RRSG, 2002)

Prior knowledge influences navigational patterns and comprehension recall

• e.g., Lawless & Kulikowich (1998); Lawless, Mills, & Brown, (2003); Yang (1997)

Affective dispositions differ between more and less successful hypertext readers

• e.g., Balcytiene (1999); Yang (1997)

Added Complexities of Internet (Online) Text Structures

Online texts are not bound within a closed system with only one organizational structure (Lawless & Schrader, 2007)

Online texts change daily in structure, form, and content (Zakon, 2005) Online texts often contain hidden social, economic, and political agendas not typically found in closed hypertext learning systems (Cope & Kalantzis, 2000; Fabos, 2008) Online texts introduce infinite intertextual connections and intercultural negotiations (Snyder &

Bulfin, 2008; Warshauer & Ware, 2008)

Online Text Comprehension

Hill & Hannafin (1997) found metacognitive strategies, prior knowledge, and perceived self-efficacy influenced how adults learned from Internet text

Studies suggest many adolescents are ill-equipped to deal with new online reading comprehension demands:

Querying search engines • e.g., Eagleton & Guinee (2002); Sutherland-Smith (2002)

Understanding search results • e.g., Coiro & Dobler (2004); Henry (2006); Leu et al. (2004)

Critically evaluating online information • e.g., Burbules & Callister (2000); Fabos (2004)

Online Text Comprehension

Three studies have directly explored Internet use as a complex, multifaceted process of reading comprehension

Coiro & Dobler (2007) found skilled readers employed both similar and more complex applications of (1) prior knowledge; (2) inferential reasoning strategies; and (3) self-regulated reading processes.

Coiro & Dobler (2004) found traditionally skilled readers with Internet reading experience were aware of and demonstrated strategic reading processes to a higher degree than their less-skilled peers.

Internet Text Comprehension

The New Literacies Research Team (2005) used this knowledge to design strategy interventions with seventh grade students and measure growth in online reading and science learning.

Found higher achievement levels in online reading comprehension and conceptual science knowledgeDeveloped a preliminary, psychometrically sound measure of online reading comprehension (ORCA-Blog)Performance in online reading skills (ORCA-Blog) did not correlate with traditional reading skills (DRP), suggesting that the two instruments measured complementary, but orthogonal skill sets.

Initial Evidence of something “new”

(r=0.19, n = 89, N.S.)

Leu, D. Castek, J., Hartman, D., Coiro, J., Henry, L., Kulikowich, J., Lyver, S. (2005).

Offline Reading = CT StateReading Test

Online ReadingComprehension= ORCA Blog

Internet Text Comprehension

Some research suggests the skills sets are similar, but more complex (e.g., Coiro & Dobler, in press).

Other research suggests the skill sets are complementary but orthogonal (e.g., NLRT, 2005). The current study builds on these findings to investigate the extent to which new (and old) comprehension proficiencies may be required on the Internet.

Key Research Questions

RQ1: In a regression analysis, does performance on one measure of online reading comprehension ability significantly predict performance on a second, parallel measure of online reading comprehension ability over and above (a) offline reading comprehension ability and (b) prior knowledge?

RQ2: What specific patterns of skill and strategy use appear to distinguish three students of varying levels of online reading comprehension ability as they engaged in a series of online information requests about science content?

Quantitative Sample

510 7th graders from a convenience sample sorted by strata into two groups (economically advantaged and economically challenged)

60 students randomly selected from each strata

Final sample included 118 ethnically, economically, and academically diverse seventh graders

Diverse Quantitative Sample

Quantitative Measures of Reader Characteristics

Offline reading comprehension ability

Prior knowledge

Online reading comprehension ability

Measuring offline reading comprehension ability

Connecticut Mastery Test (CMT)Standardized reading scores (r = .85)50 percent: • Forming an initial understanding• Developing an interpretation• Demonstrating a critical stance

50 percent: • The Degrees of Reading Power (DRP)

Measuring offline reading comprehension ability

Measuring Prior Knowledge

What do you know about…the lungsthe breathing processoxygencarbon monoxide poisoninganimationreliable information

Topic-Specific

Task-Specific

Conceptual measure with six items scored with a three point rubric (r = .849)

Range 3.5 - 34 Mean = 14.93 (SD = 7.12)

Measuring Online Reading Comprehension Ability

Online Reading Comprehension Assessment (ORCA-Scenarios I and II)

Three integrated tasks in a “Treasure Hunt” that asked students to locate, evaluate, synthesize, and communicate information using the Internet

Twenty open-ended items; each scored using a 4-point rubric (0-3 points) for a maximum score of 60 points

ORCA Scenario I (r = .918) and Scenario II (r = .909)

ORCA-Scenario 1

ORCA-Scenario 2

Quantitative Procedures

118 students completed measure of prior knowledge and ORCA-Scenario I (recorded with Camtasia)

Standardized reading scores were collected from each district16 weeks later, completed same measure of prior knowledge and ORCA-Scenario II (recorded with Camtasia)

This 2nd PK measure was used in regression model

Findings: Predicting Online Reading Comprehension

R2

Offline Reading Comprehension

Additional R2

Prior Knowledge

Additional R2

Online Reading Comprehension

Total R2

.351*

35.1%

.074 .154* .579*

N = 118

Offline Reading Comprehension = CT State Reading Test

Online Reading Comprehension = ORCA Scenario I and II (in Quia interface)

Effect Size .33 - large effect Cohen (1988)

Regression Analysis

7.4% 15.4% 57.9%

Something different, new, unique?

Something predicts online reading comprehension over and above offline reading comprehension & prior knowledgeMultiple possibilities (combinations) of offline and online reading ability Alternative explanation: Process-based strategy use vs. product-based skills (but it’s still reading comprehension)

Tested for Interaction Effects

There was a significant negative interaction effect between prior knowledge and online reading comprehension (ORCA-Scenario I), t(105) = -2.28, p < .05.

When entered into the whole regression model as a fourth predictor variable, this interaction effect explained a small but additional 1.9 percent of variance in the model, which was significant FD(1,104) = 4.782, p < .05

Findings: Predicting Online Reading Comprehension

Findings - Main Effects of Prior Knowledge

Among readers with…

Effect of prior knowledge

Result

HIGH online reading comprehension

PK had no significant effect on ORCA2 performance

t(106)=.079

p > .05

AVERAGE online reading comprehension

PK had no significant effect on ORCA2 performance

t(106)=1.965

p > .05

LOW online reading comprehension

PK had a positive & significant effect on ORCA2 performance

t(106)=2.898

p < .01**

A changing role for prior knowledge?

PK had a positive and significant effect only among students with low online reading ability; for those with average or high online reading ability, PK had no significant effect on online reading performance

This suggests…higher levels of online reading ability may help compensate for lower levels of topic-specific prior knowledge when adolescents complete online reading-for-information tasks.

Part II

So what more can we learn from a richer qualitative look at the patterns of strategy use among three diverse online readers?

How might these patterns begin to help us better understand the “stories behind the numbers”?

Qualitative Sample

Three focal students selected based on a combination of their online and offline reading performance (after Time 1)

A higher performing online reader and higher performing offline reader [Nicole]A lower performing online reader and lower performing offline reader [Christian]A higher performing online reader but lower performing offline reader [Mikaela]

Qualitative Procedures

Directly after ORCA 2, conducted retrospective think-aloud interviews with three focal students

Internet use at home & school

Their personal response to the task

General reading habits offline and online

Elaborate on strategy use while video recording played back

Semi-structured follow-up interviews with each student’s reading/language arts teacher and computer teacher

Teacher’s perceptions and curriculum components

Qualitative Data Analysis

Iterative stages of a diachronic, developmental, contrastive, case study design

Case study of three different readers

Diachronic analysis to identify phases of online reading behaviors along a chronological timeline

Contrastive case analysis of strategy use

Pinpoint developmental aspects of online reading comprehension along a continuum within six phases of online reading

Six phases of online reading (Reading for Information in ORCA-Scenarios I and II)

Phase 1. Understanding the task directionsPhase 2. Selecting and revising search termsPhase 3. Reading to evaluate the relevancy of

hyperlinks in search engines and websitesPhase 4. Reading to critically evaluate accuracy,

reliability, and commercial bias within and across websites

Phase 5. Synthesizing information across three websites to make a best choice

Phase 6. Communicating answers to online information requests

ACTION THINK-ALOUDEMERGINGPATTERNS

Analyzing patterns within subphases of online reading

Qualitative Analysis: Developmental Trends???

Lower performing online reader

Average performing online reader

Higher performing online reader

Task directions

Search terms

Evaluate relevancy

Evaluate accuracy, reliability and stance

Synthesize

Communicate

?12 3

Finding #1 Qualitative Finding #1

A developmental progression of reading comprehension skills and strategies distinguished the

three readers’ performance within each phase

Lower-performing online reader

Average-performing online reader

Higher-performing online reader

He was not sure what to pay attention to - he was confused so he just kept reading. He acknowledged this difficulty in the interview, but had a limited repertoire of strategies to address his confusion about where to start.

She explained she was reading it slowly to make sure it made sense. She was reading all the questions and writing them down to help remember. She was aware of her difficulty, and used a compensatory strategy (write down clues).

She was questioning the task and monitoring her understanding of the topic and the vocabulary. She mentioned the 2nd task was more challenging and wasn’t quite sure which detail to focus on first.

Phase 1: Understanding the task directions [monitoring understanding]

Lower Average Higher

Locating .com strategy Whole phrase Uses keywords

Evaluating Reliability

Struggled to locate “About Author”

page

Judged reliability based on length of

coverageExamined author’s level

of integrity/expertise

EvaluatingCommercial Bias

Ran out of time and did not have opportunity to demonstrate strategy use

Detected related advertising but over-generalized caution

(believed all information was fake)

Detected related advertising and

expressed balanced caution (may be

exxagerated/slanted)

Communicating Copied the long address by hand

Copy/pasted address with mouse shortcut

At each phase, developmental differences appeared to characterize three students with

different overall levels of online reading proficiency

Go to COIRO IRA2008 Part 3

Understanding Task Directions(Developmental Differences)

High (50 seconds) “I was reading the clues and I was like, Oh, United Kingdom! I like the United Kingdom! But then I got to the asthma part and I was like, “what does asthmas have to do with carbon monoxide poisoning, and then I saw it. And then the animation part - I love having animation!”

Avg. (3 min. 30 sec) “I read the clues and I read the questions, and then up here [pointing to the link to the MSN homepage], it goes to the link. I read the question and I wrote down (on the paper next to her) what the questions are, so I don’t have to click back and forth later on to see if I got the right thing.”

Low (3 min 26 sec) “Well, my plan is to like, to check it out, what to do, which one to look at, and where to look at it…I was thinking that I wouldn’t be able to find it - that it was going to be hard”.

LOCATING [Nicole: Task 2] Higher Performing Online Reader

0:00-0:20

1:11. The address is http://www.njpies.org/co

0:20-0:30

0:30-0:47

LOCATING [Christian: Task 2] Lower Performing Online Reader

0:00-4:154:15-5:30

6:15-8:20

10:00-10:45 “In questions 1-4, I just can’t find it.”

MOST RELIABLE: [Nicole - higher performing] This site is most reliable because it gives a lot of information. It is

also done by a college institute.

MOST RELIABLE: [Mikaela - average performing] This site is most reliable because the site is all about the facts on carbon monoxide poisoning and because you learn more about carbon monoxide poisoning.

MOST RELIABLE: [Christian - lower performing][ran out of time looking for author’s name] I was all shocked ‘ cuz I was like kinda lost, cause it’s kinda hard

Critically Evaluating Reliability(Developmental Differences)

LEAST RELIABLE: [Nicole - higher performing] This site looks like they’re trying to sell me stuff - I didn’t think it was the least reliable, but I didn’t think it was the most reliable - it was in the middle. Cause if they try to sell you something, that doesn’t always mean it’s not reliable. It means that it gives you a slight thought that it might not be reliable, but it might just be trying to help you, like picture carbon monoxide detectors now and think the life lost may be yours next time.

LEAST RELIABLE: [Mikaela - average performing] This site is the least reliable because 1) the site has the least facts and 2) because the site is the least accurate for people to learn anything because there’s only 2 1/2 paragraphs.

LEAST RELIABLE: [Christian - lower performing] This site said “For Immediate Release” - I think they just want to put that out really quick - it means they really don’t want to help, they just put information out fast.

Critically Evaluating Reliability (Developmental Differences)

Qualitative Finding #2 Developmental differences appeared to be affected by five key dimensions of offline and online

reading comprehension ability including: (1) fluency; (2) self-regulated reading; (3) inferential reasoning; (4) critical reasoning; and (5) metacognitive knowledge about what, how, and when to employ particular online reading processes

Remember: Strategic knowledge = declarative, procedural, and conditional See Figure 1 in

Handout for Details

Offline PK Online Total R2

35.1% 7.4% 15.4% 57.9%

The Interplay Between Offline and Online Reading Ability

Offline PK Online Total R2

35.1% 7.4% 15.4% 57.9%

Primarily offline

Primarily online

Primarily offline with new online procedures and contexts

Combine offline and new online

Primarily offline with new online procedures and contexts

Primarily offline with new online procedures and contexts

1. Understanding task directions

2. Selecting & revising search terms

3. Evaluating relevancy of hyperlinks

6. Communicating

4. Evaluating accuracy, reliability and commercial bias

5. Synthesizing

35.1% 15.4%plus

Online reading comprehension might be conceived as overlapping and highly integrated dimensions of strategic offline and online reading processes rather than a linear sequence of isolated reading skills.

Thus, online reading comprehension instruction should weave offline and online reading strategy practice within authentic tasks that integrate online locating, evaluating, synthesis, and communication.

Qualitative Finding #3

Implications for Literacy Theory

Preliminary evidence of the psychological reality of new literacies required to comprehend information on the Internet (Leu et al, 2004).

Further informs long-debated theoretical questions about reading comprehension by providing preliminary evidence that online reading comprehension may best be characterized as both:

a highly integrated set of strategic reading processes (see Thorndike, 1974) that can be organized into sets of different sub-skills (see Davis, 1972) related to six phases of online reading

Implications for Literacy Research

Findings contribute to emerging work that seeks to identify the range of skills and strategies that effectively characterize online reading comprehension (TICA Project, 2005-2007).

Findings help us begin to more precisely understand the possible similarities and complexities / differences between offline and online reading comprehension processes revealed in previous work (e.g., Coiro & Dobler, 2007; Leu et al, 2005).

Findings extend work identifying qualitative shifts in strategic offline processing that distinguish novices from experts (e.g., Alexander et al.) to offer preliminary patterns of qualitative shifts in online comprehension strategy use that appeared to distinguish higher-performing online readers from lower-performing online readers.

Implications for Classroom Practice

Inform diagnosis and remediation within each phase of online reading comprehension ability

We can no longer assume that one reader will do equally well or be equally challenged across all dimensions of an online reading task (or that how you read offline predicts entirely how you read online)

Preliminary set of criteria for determining individual strengths and weaknesses can inform realistic next steps for remediation within and across the phases of online reading comprehension

Implications for Classroom Practice

Inform instruction of online reading comprehension

Design leveled (Mosenthal, 1996) and explicit instruction in:

• Locating & communicating skills to enable access to online texts

• Critical evaluation skills for all levels of readers to facilitate deeper questioning of texts and sources

• Synthesis strategies that consider the overlaps between evaluation, synthesis, and communication skills

We need to be cautious in our interpretations…

Tasks did not represent the full extent of online literacy experiences or new literacy components

Focused on externally assigned questions as opposed to self-selected topics of inquiry

Narrow view of evaluation, synthesis, and communicationScoring system reflected the task demands

Can’t really generalize from three case studies that in some ways were more similar than different (but findings can inform a replication study with larger N)

Significance of the Study

Having the skills to comprehend information on the Internet will play a central role in academic success in an information age.

A better understanding of the nature of the new skills & strategies of online reading comprehension can:

Inform an emerging theory of new literaciesProvide rich insights into online reading profilesInform the development of diagnostic instruments and better instructional strategies, particularly for those students who struggle the most with online reading

Pursuing the ideas that emerged may prompt new focus for addressing several of the literacy challenges we face today.

Thank you. Questions? Concerns? Ideas?

Julie CoiroUniversity of Rhode Island

http://www.newliteracies.uconn.edu/coirodissertation/