selection of irrelevant navigation paths during digital ...€¦ · ladislao salmerón eri lectura,...

Selection of irrelevant navigation paths during digital reading:

Underlying cognitive processes

Ladislao Salmerón

ERI Lectura, University of Valencia, Spain

ProcessDataConf2019

Overview

• Navigation paths

– Indicators

– Interpretation

– Context dependencies

• Study

• Conclusions

ProcessDataConf2019

Navigation paths

ProcessDataConf2019

· Major difference between print and digital reading is the presence of hyperlinks that create non-sequential page structures (OECD, 2011, 2015)

· Strong association between quality of students’ navigation and digital reading …

… at country (OECD, 2015) and at individual (e.g.

Naumann & Salmerón, 2016) level. … even after accounting for performance in print reading.

In digital reading, process data is already (partially) accounted in performance scores.

Navigation paths

ProcessDataConf2019

(OECD, 2015)

Navigation paths

ProcessDataConf2019

· “Savvy digital readers are expected to navigate across pages that contain information relevant for the question to be answered, or the task to be solved”

· Process indicators of navigation quality (Naumann, 2015)

1. Number of relevant pages visited2. Total number of pages visited3. Derived indicators (% relevant pages)

1

2 3

4

A page is Relevant if informationfor the response must be:-located in p. 4-related p. 1 to p.4 -related 1-2-3-4

Navigation paths

ProcessDataConf2019

A page is irrelevant if itdoesn’t containinformation useful to answer the question, or isnot in the path guiding to a relevant one

Navigation paths

ProcessDataConf2019

· Cognitive processes underlying navigation:

a) To relevant pages > inferential process

· Print reading comprehension (Hahnel, Goldhammer, Kröhne,

& Naumann, 2017, 2018; Naumann, Richter, Christmann, & Groeben, 2008; Naumann & Salmerón, 2016; OECD, 2011, 2015; Salmerón & García, 2011)

· Progression across high-school years (Keil & Kominsky,

2013; Salmerón, García & Vidal-Abarca, 2018)

Navigation paths

ProcessDataConf2019

· Cognitive processes underlying navigation:

b) To irrelevant pages:· Generally interpreted as ‘Lostness’ (Shapiro &

Niederhauser, 2004)

· Its isolated effect has been less studied in the past

Navigation paths

ProcessDataConf2019

· Scenarios in the Digital reading assessment in PISA

include highly differentiated sections.

Navigation paths

ProcessDataConf2019

· Can visits to irrelevant pages also represent students’

attempts to capture a complete view of the theme?

- Self-regulated vs. cue dependent navigation

(Balcytiene, 1999)

· This aspect should be more obvious in digital texts that

are highly topically interconnected, such as Wikipedia

texts, than in scenarios with highly differentiated sections,

such as web portals.

Goal

ProcessDataConf2019

· Test if visits to irrelevant pages differ across digital

contexts (less / more interconnected):

-Relationship with print reading skills

-Relationship with digital reading scores

-Potential mediation between the former

Study

ProcessDataConf2019

Participants555 students (12-16 years old) · 5 schools (2 rural, 3 urban)· 40 intact classes

Exclusion criteria: learning disability, curricular adaptation

In class participation. Two sessions of 50’

Grade 7th 1548th 1419th 14310th 137

ProcessDataConf2019

• Students’ print comprehension skills (reading literacy) were assessed using a standardized paper and pencil test (Llorens et al., 2011).

• Texts: – a textbook excerpt about global warming, – an article from a popular science magazine about the language

of the bees, – a pamphlet about the postural ergonomics of work chairs, – two letters to the editor about the pros and cons of nuclear

energy,– a newspaper diagram about traffic accidents.

• Each unit included 3-5 questions covering the different aspects suggested by the PISA framework

• Cronbach’s alpha = .80.

Method

Method

ProcessDataConf2019

· Two web reading scenarios were designed following thePISA (OECD, 2009) scheme.· Participants were given one question at a time, and weretold to use the information from the web pages provided to answer the question.· Questions were constructed following three aspects(OECD, 2009): access and retrieve, integrate and interprete, and reflect and evaluate.· Heterogeneous relationship between question and relevant hyperlinks: word match, paraphrases, irrelevant word matches (Cerdán, Gilabert, & Vidal-Abarca, 2011)

Method

ProcessDataConf2019

· Web portal scenario:-Portal dedicated to young people-Divided into five categories (sports, courses, technology, nature and health), with three subcategories each. -12 questions required that participants clicked on at least two (maximum seven) links to access relevant information to answer the question, and involved both textual and graphic information, such as tables or maps.

Method

#19. You would like to learn how to sing, so you decide to enroll in a course offered by the Youth Portal. Search theaddress for the center. a) St. Del Padre Vitoriab) St. Mayor 25c) St. De Donoso Cortésd) St. Goya

ProcessDataConf2019

Example question from Web portal scenario

ProcessDataConf2019

Method

ProcessDataConf2019

· Wikipedia scenario:-Main document on the topic of the French Revolution -Included 10 subsections and 24 embedded hyperlinks to additional Wikipedia pages. - 6 questions required that participants clicked on at least one (maximum two) embedded hyperlink(s) to access relevant information to answer the question.

Method

#13. Why the policy of the Antient Regime was against the new social ideas from the French Revolution? Because…

a) … contrary to the Ancient Regime, the new society valued only daily lifeproblems.b) …social ideas from the Ancient Regime were based on the power of thearmy while new ideas were based on reason.c)… in the new regime the philosophers governed society and in the oldregime monarchs and the Church ruled.d) … the new social ideas were based on reason, and the old in divine law.

ProcessDataConf2019

Example question from Wikipedia scenario

ProcessDataConf2019

Method

• Log-files indicators:

– Number of unique pages visited

– Relevant and irrelevant to the task

– By scenario (Wikipedia and Web portal)

ProcessDataConf2019

Results

ProcessDataConf2019

Min Max Mean SD Asymmetry

Skewness

Unique Relevantvisits Wikipedia

0 1.67 .37 .33 .41 -.74

Unique Irrelevantvisits Wikipedia

0 .33 .06 .10 1.28 .72

Unique Relevantvisits Web Portal

0 3.33 2.0 .74 -1.17 .63

Unique Irrelevantvisits Web Portal

0 .33 .10 .40 .39 -.59

Descriptive scores for navigation indicators, averaged by question

ProcessDataConf2019

Print readingcomprehension

Relevant visits

Digital readingcomprehension

Irrelevant visits

Results

r = .45*

r = -.15*

r = .53*

r = -.13*

Web portal scenario:Pearson correlations

Results

Web portal scenario

Visits Visits

Pri

nt

Rea

din

g co

mp

reh

ensi

on

Dig

ital

Rea

din

g co

mp

reh

ensi

on

Unique relevant visitsUnique irrelevant visits

ProcessDataConf2019

ProcessDataConf2019

Print readingcomprehension

Relevant visits

Digital readingcomprehension

Irrelevant visits

Results

r = .40*

r = .15*

r = .61*

r = .31*

Wikipedia scenario:Pearson correlations

Results

Wikipedia scenarioUnique relevant visitsUnique irrelevant visits

Visits Visits

Pri

nt

Rea

din

g co

mp

reh

ensi

on

Dig

ital

Rea

din

g co

mp

reh

ensi

on

ProcessDataConf2019

ProcessDataConf2019

Print readingcomprehension

Relevant visits

Digital readingcomprehension

Irrelevant visits

Results

.45*

-.15*

.42*

-.17*

Web portal scenario:Mediation analysis (Bootstraping, Preacher & Hayes (2008))

c’= .34*

Indirect effects:-relevant visits: .18 [.14, .24]-irrelevant visits: .03 [.01, .05]

R2adj = .43*

ProcessDataConf2019

Print readingcomprehension

Relevant visits

Digital readingcomprehension

Irrelevant visits

Results

.40*

.15*

.56*

-.02, n.s.

Wikipedia scenario:Mediation analysis (Bootstraping, Preacher & Hayes (2008))

c’= .22*

Indirect effects:-relevant visits: .22 [.17, .27]-irrelevant visits: -.01 [-.02, .01]

R2adj = .44*

Conclusions

• Irrelevant visits in the Web portal scenario, with highly differenciated topic structure:

– Negative correlations with print readingcomprehension and digital reading performance

– Indirect effect of print reading comprehension to digital reading performance.

ProcessDataConf2019

Conclusions

• Irrelevant visits in the Wikipedia scenario, with highly interconnected topic structure:

– Positive correlations with print readingcomprehension and digital reading performance

– It adds no variance to digital readingperformance.

ProcessDataConf2019

Conclusions

• Underlying cognitive processes?

– In loosely interconnected texts: Distraction?(must be independent of inferential processes, accounted for by relevant visits effects).

– In highly interconnected texts: Byproduct of inferential processes (exploration, curiosity, Balcytiene, 1999)

• Need to complement log-files withqualitative data (verbal protocols, interviews)

ProcessDataConf2019

Conclusions

• Practical implications:

– Consider ‘irrelevant visits’ with caution when building indices of ‘navigation quality’.

– Navigation indices may be dependent on the interconnectedness of the scenarios

– ‘Unique relevant visits’ alone are a strong predictor of performance across scenarios

ProcessDataConf2019

Thanks for your attention

www.uv.es/lectura

www.uv.es/lasalgon

Twitter: LSalmeron_UV

ProcessDataConf2019

Unique relevant visitsUnique irrelevant visits

Materials

• Q: at your school garden your class is cultivating some plants on danger. Do you think you could cultivate a ‘Clover water’?

Spanish plants on danger

One of these [extingueshedplants] is the ‘Clover water’. Scientists have found spores of it, which allows its use in the lab and its reintroduction in nature.

Materials

• Q: The Youth Portal just posted a report on the extinction danger of mammals. What is the extinction level in Canada?

Animals in danger of extinction across the planet

Extinction risk variesaccording to the continent