alternative measures of knowledge structure: a s measures of text structure and of reading...

1

Alternative measures of knowledge structure: as measures of text structure

and of reading comprehension

May 14, 2012BSI

Nijmegen, Nederland

Roy [email protected]

Clariana, R.B. (2010). Multi-decision approaches for eliciting knowledge structure. In D. Ifenthaler, P. Pirnay-Dummer, & N.M. Seel (Eds.), Computer-Based Diagnostics and Systematic Analysis of Knowledge (Chapter 4, pp. 41-59). New York, NY: Springer. link

mailto:[email protected]

http://www.personal.psu.edu/rbc4/Multidecision_chapter.pdf

2

Overview

• Introduction• I am an instructional designer and a connectionist,

so my language may be a little different, also slow me down if my accent is difficult

• My intent today is to describe my research on several approaches for measuring Knowledge Structure (KS) and along the way, describe tools, and maybe show extra ways of thinking about text, knowledge, comprehension, and learning

3

KS: Encompassing theoretical positions

• Cognitive structures (de Jong & Ferguson-Hessler, 1986; Fenker, 1975; Korz & Schulz, 2010; Naveh-Benjamin, McKeachie, Lin, & Tucker, 1986; Shavelson, 1972)

• Conceptual networks (Goldsmith et al., 1991)• Conceptual representations (Geeslin & Shavelson, 1975; Novick &

Hmelo, 1994); (McKeithen, Reitman, Rueter, & Hirtle, 1981)• Conceptual structures (Geeslin & Shavelson, 1975; Novick &

Hmelo, 1994) • Knowledge organization and knowledge structures (McKeithen et

al., 1981)• Semantic structures (Gentner, 1983; Riddoch & Humphreys, 1999).

4

KS: Encompassing theoretical positions

• Spatial knowledge (de Jong & Ferguson-Hessler, 1996; Dunbar & Joffe, 1997; Jee, Gentner, Forbus, Sageman, & Uttal, 2009; Korz & Schulz, 2010; Schuldes, Boland, Roth, Strube, Krömker, & Frank, 2011)

• Categorical knowledge (Candidi, Vicario, Abreu, & Aglioti, 2010; Matsuka, Yamauchi, Hanson, & Hanson, 2005; Stone & Valentine, 2007; Wang, Rong, & Yu, 2008)

• Conceptual knowledge (de Jong & Ferguson-Hessler, 1996; Edwards, 1993; Gallese & Lakoff, 2005; Hallett, Nunes, & Bryant, 2010; Rittle-Johnson & Star, 2009)

5

KS: My sandbox modelOur symbolic connectionist view:• Knowledge structure (or structural knowledge) refers to how

information elements are organized, in people and in artifacts

• A departure from most theories, we propose that knowledge structure is pre-propositional, but that KS is the precursor of meaningful expression and the underpinning of thought

• Said differently, knowledge structure is the mental lexicon that consists of weighted associations (that can be represented as vectors) between knowledge elements

6

KS is worth measuring• Measures of content knowledge structure have been

empirically and theoretically related to memory, classroom learning, insight, category judgment, rhyme, novice-to-expert transition (Nash, Bravaco, & Simonson, 2006) and reading comprehension (Britton & Gulgoz, 1991; Guthrie, Wigfield, Barbosa, Perencevich, Taboada, Davis, Scafiddi, & Tonks, 2004; Ozgungor & Guthrie, 2004), and

• And findings for combining individual knowledge structures to form group mental models (Cureeu, P.L., Schalk, R., & Schruijer, S., 2010; DeChurch & Mesmer-Magnus, 2010; Johnson & O’Connor, 2008; Mohammed, Ferzandi, & Hamilton, 2010; Pirnay-Dummer, Ifenthaler, & Spector, 2010).

7

Applied to reading comprehension, KS as a measure of the situation model

Ferstl & Kintsch (1999)• Textbase (the text’s semantic content and

structure, van Dijk & Kintsch, 1983)• Situation model (the integration of the

‘episodic’ text memory with prior domain knowledge, van Dijk & Kintsch, 1983); also called mental model of the text, the text model, the discourse model

Ferstl, E.C., & Kintsch, W. (1999). Learning from text: structural knowledge assessment in the study of discourse comprehension. In van Oostendorp and Goldman (eds.), The construction of mental representations during reading. Mahwah, NJ: Lawrence Earlbaum.

8

Visually


needs

concerns

feelings

empowerment

relationship

motivation

focus

productivity

pay

plan

contingency

classical

efficiency humanistic

measure

leadership

managementsuccess

individual

company

TQM

quality

customers

goal

work

environment

employee

service

needs

concerns

feelings

empowerment

relationship

motivation

focus

productivity

pay

plan

contingency

classical

efficiency

humanistic

leadership

management

success

individual

company

TQM

quality

customers

goal

work

environment

employee

Text base

Updated situation model

(post list recall)

needs

concerns

feelings

empowerment

relationship

motivation

individual

productivity pay

plan

contingency

classical

efficiency

humanistic

measure

leadership

managementsuccess

focus

company

TQM

quality customers

goal

work

situation

employee

Situation model(pre list recall)

A KS measure of the situation model

• Ferstl & Kintsch (1999) used pre-and-post-reading list-cued partially-free recall to elicit KS of the birthday story (which obtains asymmetric matrices)

• Participants – 42 undergraduate students (CU Boulder)• Pre-reading cued-association KS task: Students were presented

by computer a 60 word list of birthday-related terms to view one at a time (randomized), and then were given the list on paper with 3 blanks beside each list term and were asked to write in the 3 terms from the list that come to mind

• Reading: Students then read the 600-word long birthday story• Post-reading cued-association KS task: i.e., same as pre-task, fill

in the list

9Clariana, R.B., & Koul, R. (2008). The effects of learner prior knowledge when creating concept maps from a text passage. International Journal of Instructional Media, 35 (2), 229-236.

10

Results

• Established that the KS cued association paradigm was appropriate for assessing background knowledge and text memory

• This KS approach facilitated interpretation, depicting how the text ‘added to’ the post reading situation model (see their figure 10.4, p.260); provided a different or other way to think about reading comprehension (p.268)

• Test-retest reliability may be a problem for this KS approach


Another KS measure of the text base (or situation model?)

• Clariana & Koul (2008), we asked students to draw concept maps (KS) of a text

• Participants – 16 graduate students in a science instructional methods course (Penn State GV)

• First, students discussed concept maps in class• Then working in dyads (8 pairs), students were given a 255

word passage on the heart and circulatory system and were asked to create a concept map of it

• KS data sources – 8 dyad concept maps of the text– 1 expert concept map of the text– A Pathfinder network (PFNet) map of the text automatically formed

by ALA-Reader software

11Clariana, R.B., & Koul, R. (2008). The effects of learner prior knowledge when creating concept maps from a text passage. International Journal of Instructional Media, 35 (2), 229-236.

12

Data

• 26 terms identified across all of the maps and text• (Text concept map), dyads’ concept map link lines

entered into a 26 x 26 half matrix• Matrix analyzed using Pathfinder Knot

lungs oxygenated deoxygenated

pulmonary artery

pulmonary vein

left atrium right ventricle

moves through

to the passes into

to the Link Array

a b c d e f ga left atrium - b lungs 0 - c oxygenate 0 1 - d pulmonary artery 0 1 0 - e pulmonary vein 1 1 0 0 - f deoxgenate 0 1 0 0 0 - g right ventricle 0 0 0 1 0 0 -

(n2-n)/2 pair-wise comparisonsClariana, R.B., & Koul, R. (2008). The effects of learner prior knowledge when creating concept maps from a text passage. International Journal of Instructional Media, 35 (2), 229-236.

13

Data as percent overlap

• Percent overlap was calculated as links in common divided by the average total links

2 544

% overlap = 4 / ((6+8)/2) % overlap = 4 / 7% overlap = 57%

e.g., Dyad PFNete.g., Expert PFNet

Clariana, R.B., & Koul, R. (2008). The effects of learner prior knowledge when creating concept maps from a text passage. International Journal of Instructional Media, 35 (2), 229-236.

14

Data as percent overlapTable 1. The average percent of agreement for each pair of concept map networks (the number of network propositions are shown in parentheses). Non-science majors Science major D1 D4 D5 D7 D8 D2* D3* D6* Text Dyad 1 (18) -- Dyad 4 (3) 0% -- Dyad 5 (6) 0% 22% -- Dyad 7 (13) 7% 38% 11% -- Dyad 8 (9) 0% 0% 0% 0% -- Dyad 2* (22) 10% 0% 36% 11% 0% -- Dyad 3* (11) 7% 18% 24% 8% 0% 61% -- Dyad 6* (12) 7% 0% 22% 8% 0% 65% 87% -- Text (28) 13% 13% 6% 24% 5% 52% 46% 55% -- Expert map (16) 12% 0% 9% 14% 8% 58% 59% 64% 71%

* dyads with a science major

In the epigraph to Educational Psychology: A Cognitive View, Ausubel (1968) says, “The most important single factor influencing learning is what the learner already knows.”

An aspect of measurement reliability and validity

low

good

ALA-Reader


15

The strong influence ofprior domain knowledge

Figure 3. The relationship between the number of propositions in the dyad concept maps and the average percent agreement with the 255-word text passage (* shows dyads with a science major).

0%

10%

20%

30%

40%

50%

60%

70%

80%

0 5 10 15 20 25

Perc

ent A

gree

men

t with

the

255-

wor

d te

xt

Number of Concept Map Propositions

D1

D2*D6*

D3*

D5 D8

D4

D7

Expert map


Only those with prior domain knowledge could adequately ‘capture’ the text

16

ALA-Reader papersALA-Reader converts text KS

Clariana, R.B., Wallace, P.E., & Godshalk, V.M. (2009). Deriving and measuring group knowledge structure from essays: The effects of anaphoric reference. Educational Technology Research and Development, 57, 725-737.

Clariana, R.B., & Wallace, P. E. (2007). A computer-based approach for deriving and measuring individual and team knowledge structure from essay questions. Journal of Educational Computing Research, 37 (3), 209-225.

Koul, R., Clariana, R.B., & Salehi, R. (2005). Comparing several human and computer-based methods for scoring concept maps and essays. Journal of Educational Computing Research, 32 (3), 261-273.

Clariana, R.B. (2010). Deriving group knowledge structure from semantic maps and from essays. In D. Ifenthaler, P. Pirnay-Dummer, & N.M. Seel (Eds.), Computer-Based Diagnostics and Systematic Analysis of Knowledge (Chapter 7, pp. 117-130). New York, NY: Springer.

Also see HIMAT/DEEP software and Hamlet software

17

KS for influencing learning

• e.g., Trumpower et al. (2010) used knowledge structure of computer programming represented as network graphs to pinpoint knowledge gaps

• KS elicited as pair-wise comparisons and data-reduced to networks using Pathfinder KNOT

• Learners’ networks then compared to an expert referent network

Trumpower, D.L., Sharara, H., & Goldsmith, T.E. (2010). Specificity of Structural Assessment of Knowledge. Journal of Technology, Learning, and Assessment, 8(5). Retrieved from http://www.jtla.org.

18

KS for influencing learning

• The problems were intended to be complex enough so that the solution depended on integration of several interrelated concepts (relational)

• The presence of subsets of links in participants’ PFnets differentially predicted performance on two types of problems, thereby providing evidence of the specificity of knowledge structure

Trumpower, D.L., Sharara, H., & Goldsmith, T.E. (2010). Specificity of Structural Assessment of Knowledge. Journal of Technology, Learning, and Assessment, 8(5). Retrieved from http://www.jtla.org.

19

Protein structure as an analogy of knowledge structure in reading comprehension

Christian Anfinsen received the Nobel Prize in Chemistry in 1972: • Linear sequence of amino acids

enzyme structure enzyme functionIs like:• Linear sequence of words in a text

knowledge structure retrieval function

20

AA Linear sequence enzyme structure function

APRKFFVGGNWKMNGKRKSLGELIHTLDGAKLSADTEVVCGAPSIYLDFARQKLDAKIGVAAQNCYKVPKGAFTGEISPAMIKDIGAAWVILGHSERRHVFGESDELIGQKVAHALAEGLGVIACIGEKLDEREAGITEKVVFQETKAIADNVKDWSKVVLAYEPVWAIGTGKTATPQQAQEVHEKLRGWLKTHVSDAVAVQSRIIYGGSVTGGNCKELASQHDVDGFLVGGASLKPEFVDIINAKH

Triose Phosphate Isomerase: http://www.cs.wustl.edu/~taoju/research/shapematch-final.pdf

http://www.cs.wustl.edu/~taoju/research/shapematch-final.pdf



21

Read linear sequence of words in text

Hyona, J., & Lorch, R.F. (2004). Effects of topic headings on text processing: evidence from adult readers’ eye fixation patterns. Learning and Instruction, 14, 131–152.

Figure 1, p.136

22

Knowledge structure

Imminent extinction pandas the climate

today

exclusively

in the wildlive

Imminent extinction

pandas the climate

Retrieval functionA B (propositional knowledge):Where do pandas live? In the wild

A B,C,D (relational knowledge):What do we know about pandas today? Pandas are heading towards extinction in the wild due to climate change

Retrieval structure

linear

23

Read KS Retrieval function

Relational

Retrieval structure Retrieval functionA B (propositional knowledge):Where do pandas live? In the wild

A B,C,D (relational knowledge):What do we know about pandas today? Pandas are heading towards extinction in the wild due to climate change

24

Summary of the introduction• KS cuts across theories, we support connectionist views• KS is worth measuring, it correlates with many kinds of

performance• KS can be measured in different ways• KS has been used to visually represent the reading

comprehension situation model• KS has been used to visually represent the text structure• Specific KS structure leads to specific cognitive

performance• Enzyme Analogy: linear chain structure function

Measuring knowledge structureMy foundation and trajectory for measuring KS:• Vygotsky (in Luria, 1979); Miller (1969) card-sorting

approaches • Deese’s (1965) ideas on the structure of association in

language and thought • Kintsch and Landauer’s ideas on representing text

structure, and latent semantic analysis• Recent neural network representations (e.g., Elman,

1995)

Jonassen, Beissner, and Yacci (1993) 25

written text

similarityratings

freerecallconcept maps

Clariana & Koul, 2008

Ferstl & Kintsch, 1999

Trumpower, Sharara, & Goldsmith, 2010

Dave Jonassen’s summary of KS measures…

Knowledgerepresentation

Knowledgecomparison

Knowledgeelicitation

Jonassen, Beissner, & Yacci (1993), page 2226

Elicit responses represent responses compare response

Dave Jonassen’s summary …

graphbuilding

similarityratings

semanticproximity

wordassociations

cardsort

orderedrecall

freerecall

additivetrees

hierarchicalclustering

orderedtrees minimum

spanningtrees

linkweighted

Pathfindernets

NetworksDimensional

principalcomponents

MDS – multidimensional scaling

clusteranalysis

expert/novice

qualitativegraph

comparisons

quantitativegraph

comparisons

relatednesscoefficients

scalingsolutions

C of PFNets

Trees


Knowledgecomparison



To show different KRlet’s do an example …

concept mapswritten text

28

Knowledge Representation (KR)• Multidimensional scaling (MDS) - Family of distance and

scalar-product (factor) models. Re-scales a set of dis/similarity data into distances and produces the low-dimensional configuration that generated them

(e.g., see: http://www.tonycoxon.com/EssexSummerSchool/MDS-whynot.pdf)

• Pathfinder Knowledge Network Organizing Tool (KNOT) algorithms take estimates of the proximities between pairs of items as input and define a network representation of the items. The network (a PFNET) consists of the items as nodes and a set of links (which may be either directed or undirected for symmetrical or non-symmetrical proximity estimates) connecting pairs of the nodes.

(See: http://interlinkinc.net/KNOT.html)

http://www.tonycoxon.com/EssexSummerSchool/MDS-whynot.pdf

http://www.tonycoxon.com/EssexSummerSchool/MDS-whynot.pdf

http://interlinkinc.net/KNOT.html



Pathfinder Network (PFNet) analysis

• Pathfinder seeks the least weighted path to connect all terms, shoots for n-1 links if possible

• Pathfinder is a mathematical approach for representing and comparing networks, see: http://interlinkinc.net/index.html

• Pathfinder data reduction is based on the least weighted path between nodes (terms), so for example, Deese’s 171 data points become 18 data points. Only the salient or important data is retained.

• Pathfinder PFNet uses, for example:– Library reference analysis– Use google to search to see many more examples of how

Pathfinder can be used

29Note that Ferstl & Kintsch (1999) used Pathfinder

http://interlinkinc.net/index.html

Deese (1965), free recall data (p.56)

30

moth

insect

wing

bird

fly yellow

flower

bug

cocoon

color

blue

bees

summer

sunshine

garden

sky

nature

spring

butterfly

moth 100 12 12 12 11 1 0 4 11 0 0 2 2 5 1 1 1 1 15insect 12 100 9 9 17 1 1 33 10 1 1 3 0 0 0 0 1 0 12wing 12 9 100 44 19 0 0 3 2 0 0 10 0 0 0 0 3 0 13bird 12 9 44 100 21 1 0 3 2 1 1 10 0 1 0 1 5 0 12fly 11 17 19 21 100 1 1 8 6 1 2 6 0 3 0 2 4 0 11yellow 1 1 0 1 1 100 7 0 0 17 23 2 2 7 5 2 4 3 5flower 0 1 0 0 1 7 100 2 0 3 7 2 1 6 18 2 6 2 6bug 4 33 3 3 8 0 2 100 7 0 0 5 0 0 0 0 2 0 4cocoon 11 10 2 2 6 0 0 7 100 0 0 4 1 1 1 0 2 0 22color 0 1 0 1 1 17 3 0 0 100 32 0 0 2 0 8 0 0 0blue 0 1 0 1 2 23 7 0 0 32 100 1 2 4 4 46 3 2 2bees 2 3 10 10 6 2 2 5 4 0 1 100 1 2 3 0 4 2 7summer 2 0 0 0 0 2 1 0 1 0 2 1 100 5 2 0 1 10 0sunshine 5 0 0 1 3 7 6 0 1 2 4 2 5 100 2 3 2 15 4garden 1 0 0 0 0 5 18 0 1 0 4 3 2 2 100 0 4 4 2sky 1 0 0 1 2 2 2 0 0 8 46 0 0 3 0 100 0 1 0nature 1 1 3 5 4 4 6 2 2 0 3 4 1 2 4 0 100 2 3spring 1 0 0 0 0 3 2 0 0 0 2 2 10 15 4 1 2 100 2butterfly 15 12 13 12 11 5 6 4 22 0 2 7 0 4 2 0 3 2 100

Deese, J. (1965). The structure of associations in language and thought. Baltimore, MD: John Hopkins Press, page 56

Full array (n * n): 19 x 19 = 361Half array ((n2 – n)/2): ((19 x 19) –19 )/2 = 171

100 participants are shown a list of related words, one at a time, and asked to free recall a related term

Deese (1965), free recall data (p.56)

mot

h

inse

ct

win

g

bird

fly yello

w

flow

er

bug

coco

on

colo

r

blue

bees

sum

mer

suns

hine

gard

en

sky

natu

re

sprin

g

butte

rfly

moth 100 12 12 12 11 1 0 4 11 0 0 2 2 5 1 1 1 1 15insect 12 100 9 9 17 1 1 33 10 1 1 3 0 0 0 0 1 0 12wing 12 9 100 44 19 0 0 3 2 0 0 10 0 0 0 0 3 0 13bird 12 9 44 100 21 1 0 3 2 1 1 10 0 1 0 1 5 0 12fly 11 17 19 21 100 1 1 8 6 1 2 6 0 3 0 2 4 0 11yellow 1 1 0 1 1 100 7 0 0 17 23 2 2 7 5 2 4 3 5flower 0 1 0 0 1 7 100 2 0 3 7 2 1 6 18 2 6 2 6bug 4 33 3 3 8 0 2 100 7 0 0 5 0 0 0 0 2 0 4cocoon 11 10 2 2 6 0 0 7 100 0 0 4 1 1 1 0 2 0 22color 0 1 0 1 1 17 3 0 0 100 32 0 0 2 0 8 0 0 0blue 0 1 0 1 2 23 7 0 0 32 100 1 2 4 4 46 3 2 2bees 2 3 10 10 6 2 2 5 4 0 1 100 1 2 3 0 4 2 7summer 2 0 0 0 0 2 1 0 1 0 2 1 100 5 2 0 1 10 0sunshine 5 0 0 1 3 7 6 0 1 2 4 2 5 100 2 3 2 15 4garden 1 0 0 0 0 5 18 0 1 0 4 3 2 2 100 0 4 4 2sky 1 0 0 1 2 2 2 0 0 8 46 0 0 3 0 100 0 1 0nature 1 1 3 5 4 4 6 2 2 0 3 4 1 2 4 0 100 2 3spring 1 0 0 0 0 3 2 0 0 0 2 2 10 15 4 1 2 100 2butterfly 15 12 13 12 11 5 6 4 22 0 2 7 0 4 2 0 3 2 100

Deese, J. (1965). The structure of associations in language and thought. Baltimore, MD: John Hopkins Press, page 56

Full array (n * n): 19 x 19 = 361Half array ((n2 – n)/2): ((19 x 19) –19 )/2 = 171

31

Using MDS in SPSS

• Start SPSS and open this Deese data file• Under Analyze, select Scale, then select

Multidimensional Scaling (ALSCAL)…1. Move Variable from left to right2. Create distances from data3. Model4. Options How to - next page

32

Select all of these

33

MDS of the Deese data

34

-2 -1 0 1

Dimension 1

-1,5

-1,0

-0,5

0,0

0,5

1,0

1,5

Dim

ensi

on 2

moth

insect

wingbirdflyyellow

flower

bug

cocoon

colorblue

bees

summer

sunshinegarden

sky

naturespring

butterfl

Euclidean distance model

Derived Stimulus Configuration

Both are “correct solutions”.WARNING!!

The Hague

Amsterdam

Utrecht

Eindhoven

Nijmegen

Side issue, the MDS obtains alternate visual representations (e.g., enantiomorphism)

Like geographic data, for example, MDS may be oriented in different ways

(describe Ellen Taricani’s 2002 dissertation, handing out teacher maps post-reading is a bad idea)35

The Hague

Amsterdam

Utrecht

Eindhoven

Nijmegen

How good is the MDS representation for displaying the relationship raw data?

• Many dimensions (in this case 19) reduced to 2 dimensions

• Check the “stress” value to estimate how strained the results are

-2 -1 0 1

Dimension 1

-1,5

-1,0

-0,5

0,0

0,5

1,0

1,5

Dim

ensi

on 2

moth

insect

wingbirdflyyellow

flower

bug

cocoon

colorblue

bees

summer

sunshinegarden

sky

naturespring

butterfl



MDS is an algorithmic, power, approach rather than based on a distribution model, so no assumptions about data structure are required…

36

PFNet of Deese data

37

summer

springsunshine

yellowcolor

blue

sky

flower

garden

nature

butterfly

cocoon moth

wing

beesbird

fly

bug

insect

MDS and PFNet of the exact same data from Deese

summer

springsunshine

yellowcolor

blue

sky

flower

garden

nature

butterfly

cocoon moth

wing

beesbird

fly

bug

insect

summer

springsunshine

yellowcolor

blue

sky

flower

garden

nature

butterfly

cocoon moth

wing

beesbird

fly

bug

insect

-2 -1 0 1

Dimension 1

-1,5

-1,0

-0,5

0,0

0,5

1,0

1,5

Dim

ensi

on 2

moth

insect

wingbirdflyyellow

flower

bug

cocoon

colorblue

bees

summer

sunshinegarden

sky

naturespring

butterfl



Pathfinder KNOT PFNet(i.e., local structure, verbatim,

proposition specific)

SPSS MDS(i.e., global structure,relational, fuzzy, gist) 38

MDS and PFNet of the exact same data from Deese

Pathfinder KNOT PFNet(i.e., local structure, verbatim,

proposition specific)

SPSS MDS(i.e., global structure,relational, fuzzy, gist) 39

summer

springsunshine

yellowcolor

blue

sky

flower

garden

nature

butterfly

cocoon moth

wing

beesbird

fly

bug

insect

summer

springsunshine

yellowcolor

blue

sky

flower

garden

nature

butterfly

cocoon moth

wing

beesbird

fly

bug

insect

-2 -1 0 1

Dimension 1

-1,5

-1,0

-0,5

0,0

0,5

1,0

1,5

Dim

ensi

on 2

moth

insect

wingbirdflyyellow

flower

bug

cocoon

colorblue

bees

summer

sunshinegarden

sky

naturespring

butterfl



Blue lines reproduce the PFNet links

MDS and PFNet data reduction• MDS uses all of the raw data to reduce the dimensions in the

representation; if the stress is not too large, global clustering is likely to be good but local clustering less so, and the MDS distances between terms within a tight cluster of terms are more likely to misrepresent the relatedness raw data.

• Pathfinder uses only the strongest relationship data (typically 80% of the raw data is discarded). Pathfinder analysis provides “a fuller representation of the salient semantic structures than minimal spanning trees, but also a more accurate representation of local structures than multidimensional scaling techniques.” (Chen, 1999, p. 408)

40

Dave Jonassen’s summary …

graphbuilding

similarityratings

semanticproximity

wordassociations

cardsort

orderedrecall

freerecall

additivetrees

hierarchicalclustering

orderedtrees minimum

spanningtrees

linkweighted

Pathfindernets

NetworksDimensional

principalcomponents

MDS – multidimensional scaling

clusteranalysis

expert/novice

qualitativegraph

comparisons

quantitativegraph

comparisons

relatednesscoefficients

scalingsolutions

C of PFNets

Trees


Knowledgecomparison



concept mapswritten text

Sabine Klois used …distance

data

Poindexter and Clariana• Participants – undergraduate students in an intro

Educational Psychology course (Penn State Erie)• Setup – complete a demographic survey and how to

make a concept map lesson• Text based lesson interventions – instructional text on

the “human heart” with either proposition specific or relational lesson approach

• KS measured as ‘distances’ between terms in a concept map (a form of card sorting) and also concept map link data, but analyzed with Pathfinder KNOT

Poindexter, M. T., & Clariana, R. B. (2006). The influence of relational and proposition-specific processing on structural knowledge and traditional learning outcomes. International Journal of Instructional Media, 33 (2), 177-184.

42

Treatments• Relational condition, participants were required to

“unscramble” sentences (following Einstein, McDaniel, Bowers, & Stevens, 1984) in one paragraph in each of the five sections or about 20% of the total text content

• Proposition-specific condition (following Hamilton, 1985), participants answered three or four adjunct constructed response questions (taken nearly verbatim from the text) provided at the end of each of the five sections, for a total of 17 questions covering about 20% of the total text content (no feedback was provided).

43Poindexter, M. T., & Clariana, R. B. (2006). The influence of relational and proposition-specific processing on structural knowledge and traditional learning outcomes. International Journal of Instructional Media, 33 (2), 177-184.

DK and KS Posttests• DK - Declarative Knowledge (Dwyer, 1976)

– Identification drawing test (20)– Terminology multiple-choice items (20), declarative

knowledge, e.g., the lesson text states A B, the posttest asks A ?(B, x, y, z) (explicitly stated)

– Comprehension multiple-choice items (20), inference required, e.g., given A B and B C in the lesson text, posttest asks A ?(C, x, y, z) (implicit, not stated)

• KS - Knowledge structure– Concept map link-based common scores– Concept map distance-based common scores


45

Note that declarative knowledge multiple-choice posttest items are sensitive to the linear order of the

lesson text

If the lesson text is A B, paraphrasing the stem (A’) and/or transposing stem and response (B A) to create posttest questions influences performance.

When MC posttest is:• Identical to lesson (A B): 77%• Transposed from lesson (B A): 71%• Paraphrased from lesson (A’ B): 69%• Both T & P from lesson (B A’): 67%

posttest

Bormuth, J. R., Manning, J., Carr, J., & Pearson, D. (1970). Children’s comprehension of between and within sentence syntactic structure. Journal of Educational Psychology, 61, 349–357.

Clariana, R.B. & Koul, R. (2006). The effects of different forms of feedback on fuzzy and verbatim memory of science principles. British Journal of Educational Psychology, 76 (2), 259-270.

Recording link and distance data in a concept map

46


pulmonary artery

pulmonary vein


Link Array


Distance Array


moves through

to the passes into

to the

Student’s concept map

(n2-n)/2 pair-wise comparisons

Distance raw data reduction by Pathfinder KNOT

47

Pathfinder Network

a b c d e f ga left atrium - b lungs 0 - c oxygenate 0 1 - d pulmonary artery 0 1 0 - e pulmonary vein

1

1 0 0 - f deoxgenate 0 1 0 0 0 - g right ventricle

0

0 0 1 0 0 -

Distance Array



pulmonary artery

pulmonary vein


Pathfinder network(based on distances)

(21 distance data points reduced to 6 link data points)

Example of link and distance PFNets for the same concept map

48


pulmonary artery

pulmonary vein


Pathfinder network(from distance data)


pulmonary artery

pulmonary vein


moves through

to the passes into

to the

Student’s concept map(i.e., link data)

Means and sd

49

Treatments Posttests ID TERM COMP Map-prop Map-assoc control 15.1 12.3 7.3 14.1 9.0

(4.4) (4.6) (5.4) (4.6) (3.6)

proposition- 16.3 14.6 13.8 16.5 11.5 specific

(5.6) (5.7) (3.7) (8.3) (3.4)

relational 17.0 12.7 12.4 13.9 10.7 (2.6) (3.5) (3.0) (9.4) (4.6)

Map-link Map-dist

Poindexter, M. T., & Clariana, R. B. (2006). The influence of relational and proposition-specific processing on structural knowledge and traditional learning outcomes. International Journal of Instructional Media, 33 (2), 177-184.

Analysis• MANOVA (relational, proposition-specific, and control)

and five dependent variables including ID, TERM, COMP, Map-prop, and Map-assoc.

• COMP was significance, F = 5.25, MSe = 17.836, p = 0.015, none of the other dependent variables were significance.

• Follow-up Scheffé tests revealed that the proposition-specific group’s COMP mean was significantly greater than the control group’s COMP mean (see previous Table).


Correlations

ID TERM COMP Prop ID -- TERM 0.71 -- COMP 0.50 0.74 -- Map-prop 0.56 0.77 0.53 -- Map-assoc 0.45 0.69 0.71 0.73 All sig. at p<.05

Compare to Taricani & Clariana

next

Map-link

Map-linkMap-distance


(drawing)(MC)(MC)

VerbatimA B

InferenceA C

Compare the correlation results to a related follow-up investigation

Taricani, E. M. & Clariana, R. B. (2006). A technique for automatically scoring open-ended concept maps. Educational Technology Research and Development, 53 (4), 61-78.

Taricani & Clariana (2006) TermCompLink data 0.78 0.54Distance data 0.48 0.61

52

Poindexter & Clariana (2006) TermCompLink data 0.77 0.53Distance data 0.69 0.71

Clariana and Marker (2007)• Participants – 68 graduate students in INSYS intro

ISD course• Computer-based lesson – text, graphics, and

questions on instructional design, either asked to generate headings for each section or not

• Seven sections referred to as A through G, each cover a component of the Dick and Carey model

• KS as a sorting task and a new listwise task

Clariana, R.B., & Marker, A. (2007). Generating topic headings during reading of screen-based text facilitates learning of structural knowledge and impairs learning of lower-level knowledge. Journal of Educational Computing Research, 37 (2), 173-191. link

53

http://www.personal.psu.edu/rbc4/JECR_37_2.pdf

Posttests• Declarative Knowledge – 30-item constructed

response terminology test, 15 items from lesson sections B, D, and F (called “used”) and 15 items from A, C, E, and G (called “not used”)

• Knowledge structure – Posttest focuses on 15 terms used in sections B, D, and F– Listwise rating task agreement scores (compared to

linear and cluster referent)– Sorting task agreement scores (compared to linear

and cluster referent)

List and sorting used by Sabine Klois, note: sorting not the same as card sortingClariana, R.B., & Marker, A. (2007). Generating topic headings during reading of screen-based text facilitates learning of structural knowledge and impairs learning of lower-level knowledge. Journal of Educational Computing Research, 37 (2), 173-191. link

54


Listwise rating task …(available at: www.personal.psu.edu/rbc4)


55

http://www.personal.psu.edu/rbc4


Sorting task …

Drag related terms closer together and unrelated terms further apart.When done, click CONTINUE

CONTINUE

Goal analysis

Verbal informationConcept

Intellectual skill

Psychomotor skill

Target populationLearner analysis

Entry behaviors

Performance context

Transfer

Preinstructional activities

Delivery system

Job aid

Instructional strategy

Feedback


56


An example student PFNet

PerformanceContext

D4

TargetPopulation

D1

LearnerAnalysis

D2.

TransferD5

Deliverysystem F2

Job aidF3

FeedbackF5

Pre-instructionalactivities F1

InstructionalStrategy F4

EntryBehaviors

D3

GoalAnalysis

B1

VerbalInformation

B2

ConceptB3

IntellectualSkill B4

PsychomotorSkill B5

Show how to count linear and nonlinear here …


57


58

Means and standard deviations



59

Analysis• The cued recall and sorting task posttest data were analyzed by

a 2 (Treatment: Headings vs. No Headings) × 2 (Posttest: cued recall and sorting task) mixed ANOVA. The first is a between-subjects factor and the second is the within subjects factor.

• The Treatment main effect was not significant, F(1, 61) = 0.220, MSE = 0.045, p = .94. The Posttest repeated measure was significant, F(1, 61) = 18.874, MSE = 0.022, p < .001, showing that the mean cued recall test score (M = 0.59) was greater than the mean sorting task score (M = 0.47). Finally, the anticipated disordinal interaction of Treatment and Posttest factors was significant, F(1, 61) = 5.119, MSE = 0.022, p = .027



60

Generate headings when reading: better ‘structure’ but worse ‘recall’


Declarative knowledge

Knowledge structure (KS)


61 of 54

Comparison of listwise and sorting KS

2.5

2.7

2.9

3.1

3.3

3.5

3.7

3.9

4.1

Linear Non-linear

no HeadHeadings

Linear Non-linear

Sorting task(more relational)

i.e., A1 A3 or A4 or A5

Listwise task(more linear)i.e., A1 A2



62

Correlations of interestTable 1. The No Headers and Headers treatment group correlations (from Clariana & Marker, 2007).

A B C D E No Header Treatment Group (N = 32)

A. CR Posttest (15 max.) 1 B. Sorting task (linear) 0.24 1 C. Sorting task (non linear) -0.02 -0.37 * 1 D. Listwise task (linear) 0.62 ** 0.30 -0.21 1 E. Listwise task (non linear) 0.08 0.04 0.20 0.00 1

Header Treatment Group (N = 31) A. CR Posttest (15 max.) 1 B. Sorting task (linear) 0.22 1 C. Sorting task (non linear) 0.49 ** 0.09 1 D. Listwise task (linear) 0.44 * 0.36 * 0.39 * 1 E. Listwise task (non linear) 0.37 * 0.30 0.30 0.04 1 p<.05; ** p<.01



63

Brain scans – in proficient readers, text with no headings requires right hemisphere activity to achieve coherence (more

work), some students will not be able to form coherence

http://brain.oxfordjournals.org/cgi/reprint/122/7/1317

headings no headings

RH LH RH LH

“Consistent with previous studies…the right middle temporal regions may be especially important for integrative processes needed to achieve global coherence during discourse processing.” (p.1317 St. George, Kutas, Martinez, & Sereno, 1999)

http://brain.oxfordjournals.org/cgi/reprint/122/7/1317

64

Review - Generate headings when reading: better ‘structure’ but worse ‘recall’


Declarative knowledge

Knowledge structure (KS)


65

Comments• The better structured knowledge of the Headings group (i.e., more

like the author’s text schema) should allow the learners to more flexibly use that knowledge (Jonassen & Wang, 1993) which should influence the reader’s ability to form inferences and comprehend the lesson text, but this apparently comes at the expense of text details.

• These results are consistent with and help explain previous investigations that have reported that learners who generate headings score lower than no-headings control groups on lower-order outcomes but score higher on inference and comprehension tests (Dee-Lucas & DiVesta, 1980; Jonassen et al., 1985; Wittrock & Kelly, 1984). (These papers are listed on the next screen)

66

Generative learning (relational lesson tasks) DK KS reversal reference list

• Dee-Lucas, D. & DiVesta, F. F. (1980). Learner-generated organizational aids: Effects on learning from text. Journal of Educational Psychology, 72(3), 304-311.

• Jonassen, D. H., Hartley, J., & Trueman, M. (1985, April). The effects of learner generated versus experimenter-provided headings on immediate and delayed recall and comprehension. Chicago: American Educational Research Association (ERIC ED 254 567).

• Wittrock, M. C., & Kelly, R. (1984). Teaching reading comprehension to adults in basic skills courses. Final Report, Project No. MDA 903-82-C-0169). University of California, Los Angeles.

67

MDS explanation:Read with terms A ® I

words a b c d e f g h Ia 1 0 0 0 0 0 0 0 0b 1 1 0 0 0 0 0 0 0c 0 1 1 0 0 0 0 0 0d 0 0 1 1 0 0 0 0 0e 0 0 0 1 1 0 0 0 0f 0 0 0 0 1 1 0 0 0

g 0 0 0 0 0 1 1 0 0h 0 0 0 0 0 0 1 1 0I 0 0 0 0 0 0 0 1 1

Link Array(no color)

I

H

G

F

D

C

B

A

E

MDS

Connectivity Matrix (Kintsch, 1998)

68

Same reading with terms A ® I, but with section headings

words a b c d e f g h Ia 1 0 0 0 0 0 0 0 0b 1 1 0 0 0 0 0 0 0c 0 1 1 0 0 0 0 0 0d 0 0 1 1 0 0 0 0 0e 0 0 0 1 1 0 0 0 0f 0 0 0 0 1 1 0 0 0

g 0 0 0 0 0 1 1 0 0h 0 0 0 0 0 0 1 1 0I 0 0 0 0 0 0 0 1 1

blue 1 1 1 0 0 0 0 0 0red 0 0 0 1 1 1 0 0 0

green 0 0 0 0 0 0 1 1 1

IH

G

FE D

CBA

blue red green

blue

red

green

Link Array(with headings) MDS

Headings (i.e., color names)

69

MDS of connectivity matrices

?…. Context (like topic headings) may alter memory structure in a regular way, and we can think about it visually.

IH

G

FE D

CBA

blue

red

green

I

H

G

F

D

C

B

A

E

No color names MDS Color names MDS

tighterclusters

70

Explanation using Lawrence Frase’s matrix multiplication to explain inference

Frase, L.T. (1969). Structural analysis of the knowledge that results from thinking about text. Journal of Educational Psychology, 60 (6, monograph, part 2), 1-16.

Read A B and B C, model of the effects of context (as headings) on verbatim and inference activation

(also notice B-A, C-A, and B-C activations)

A B C cont

ext

row -> sends control panelA 0.9 0.3 0 0.3 column -> receives context 1 name = context tryB 0 0.9 0.3 0.3 reading A->B prop association strength = 0.3 0.3C 0 0 0.9 0.3 context association strength = 0.3 0.4

context 0.3 0.3 0.3 1 term A association strength = 0.9 0.9term B association strength = 0.9 0.9term C association strength = 0.9 0.9

mmultno context no context A B C

A 0.8 0.5 0.1 A 1 0.7 0.1 outputB 0 0.8 0.5 B 0 1 0.7 (no context) - (context)C 0 0 0.8 C 0 0 1 verbatim A->B = -0.033 - means context better

inference A->C = -0.089 - means context betterw context A B C w context A B C

A 0.9 0.6 0.2 0.7 A 1 0.7 0.2B 0.1 0.9 0.6 0.7 B 0.1 1 0.7C 0.1 0.1 0.9 0.6 C 0.1 0.1 1

context 0.6 0.7 0.7 1.3

71 of 54

Clariana and Prestera (2009)• Background color as a weak context variable• Participants – 80 graduate students in INSYS intro

instructional design course• Computer-based lesson – text, graphics, and

questions with feedback on ISD, presented in 5 sections, each section covered a component of the Dick and Carey model (items with feedback should present STRONG AB effects)

• Intervention – lesson presented either with or without a color band on the left margin (this use of color should have WEAK relational effects)

Clariana, R.B., & Prestera, G.E. (2009). The effects of lesson screen background color on declarative and structural knowledge. Journal of Educational Computing Research, 40 (3), 281 -293. link

http://www.personal.psu.edu/rbc4/JECR_color.pdf

72

Example lesson screen


ColororNo color


73 of 54

Posttests• Declarative Knowledge vocabulary posttest – 18

constructed response items (fill in the blank) and 18 multiple choice items terminology test (strong AB)

• Knowledge structure posttest – sort the 36 vocabulary terms (same sorting task as Clariana & Marker (2006) above)

• Results: The anticipated disordinal interaction of Subtest and Lesson Color was significant, F(1, 71) = 5.008, MSe = 0.618, p = .028, with lesson color enhancing structural knowledge scores and inhibiting declarative knowledge scores.



74

Lesson and posttest means



75

Another disordinal interaction of declarative and structural knowledge



76

Section summary

• Different measurement approaches are better for prompting memory for linear or cluster KS

• Linear lesson tasks establish linear KS and relational (generative) lesson tasks establish relational KS

• Models can account for verbatim and inference outcomes

• Next section - Alternative measures of KS

77

For KS, more terms may be better

Pred

ictiv

e V

alid

ity

0.00

0.10

0.20

0.300.40

0.50

0.60

0.70

0 5 10 15 20 25 30

Number of terms

• Goldsmith et al. (1991) the relationship between the number of terms included in Pathfinder network analysis (elicited as pair-wise) and the predictive ability of the resulting PFNets to predict end-of-course grades.

• But only if these are really IMPORTANT terms (Clariana & Taricani, 2010)

Goldsmith, T.E., Johnson, P.J., & Acton, W.H. (1991). Assessing structural knowledge. Journal of Educational Psychology, 83 (1), 88-96. Clariana, R.B., & Taricani, E. M. (2010). The consequences of increasing the number of sterms used to score open-ended concept maps. International Journal of Instructional Media, 37 (2), 163-173. link

http://www.personal.psu.edu/rbc4/IJIM_37_2.pdf

Raw data reductionby Pathfinder KNOT

78

0 5 10 15 20 25 30 350

50

100

150

200

250

300

350

400

450

500

Terms = 10Raw data = 45

PFNet = 9PFNet as % of raw data = 20%





Number of terms (n)

Raw

dat

a (h

alf a

rray

, (n2 -n

)/2

)

Methods that elicit pairwise association fatigue with more then 20 to 30 terms)

KNOT tries to form a path with n-1 links

79

KS measurement• More terms are better but the problem with eliciting KS using

pairwise comparisons (more than 20!)• So, we need a valid and efficient measure of KS … recall from

above that:• Recall that Ferstl & Kintsch (1999) used a more efficient cued-

recall list approach (3 recalls for each term)• Clariana & Marker (2007) added a ‘listwise’ approach, with one

recognition retrieval for each term and a ‘sorting’ approach (dragging all terms around on the screen at the same time)

• Do ‘listwise’ and ‘sorting’ results compare with the more traditional and accepted ‘pairwise’ approach? If yes, then these two can handle large lists of terms.

80 of 54

Clariana and Wallace (2009)• Compared pairwise, listwise, and sorting• Participants – 84 undergraduate students in

business• All students completed 3 computer-delivered KS

measures – listwise, pairwise, and sorting (randomized) using the 15 major concepts of the course

• Students grouped for analysis into high and low groups based on a media split of their end-of-course multiple-choice exam

Clariana, R.B., & Wallace, P. E. (2009). A comparison of pairwise, listwise, and clustering approaches for eliciting structural knowledge in information systems courses. International Journal of Instructional Media, 36, 139–143.

81

The three approaches

computer literacyinternet

networks

applications

WWW

communications

ergonomics

input output

system unit

CPUoperating system

privacyethics

Drag related terms closer together and unrelated terms farther apart.When done. click CONTINUE.

Continue

pairwise

listwise

sorting


82

Sorting and listing are faster than pairwise

• Time to complete the tasks:– pair-wise approach, X = 447.4 s (sd = 140.6)– list-wise approach, X = 193.3 s (sd = 79.6)– Sorting approach, X = 115.5 s (sd = 62.7)

• Concurrent / convergent validity: Do the 3 elicitation tasks obtain similar raw data and PFNet data?


83

Individuals’ raw data arrayswere not similar (correlations)

Therefore, the 3 approaches do not elicit the same raw data associations, individuals’ raw data seems to be idiosyncratic or flaky or noisy; however the group average raw data are much more alike (averaging within a group ‘smooths out’ idiosyncrasy)

Table 3. Relatedness correlations of individual and group average raw proximity data.

Group Individuals Group Average P x L P x S L x S P x L P x S L x S Low (n = 41) 0.31 -0.21 -0.30 0.68 -0.63 -0.79 (.09) (.15) (.14) na na na High (n = 43) 0.31 -0.25 -0.29 0.68 -0.67 -0.78 (.16) (.19) (.13) na na na

P – pair-wise, L – list-wise, S – sorting


84

% overlap based on ‘group average’ PFNet common scores (intersection)


PL PH LL LH SL SH lin nonlinPairwise low (PL) --

Pairwise high (PH) 64% --

Listwise low (LL) 79% 57% --

Listwise high (LH) 71% 79% 79% --

Sort low (SL) 57% 43% 71% 64% --

Sort high (SH) 43% 43% 57% 57% 64% --

linear (lin) 50% 29% 36% 36% 29% 29% --

non linear (nonlin) 9% 9% 9% 9% 9% 9% 0% --

pairwise listwise sorting referent

85

Sabine’s experts

Pairwise Listwise Sorting

% overlap Expert_A

Expert_B

Expert_C

Expert_D

Expert_ave

Expert_A

Expert_B

Expert_C

Expert_D

Expert_ave

Expert_A

Expert_B

Expert_C

Expert_D

Expert_ave

Expert_A one one oneExpert_B 0.46 one 0.60 one 0.43 oneExpert_C 0.27 0.44 one 0.67 0.73 one 0.29 0.43 oneExpert_D 0.43 0.57 0.42 one 0.53 0.53 0.53 one 0.43 0.21 0.36 oneExpert_ave 0.71 0.56 0.52 0.54 one 0.79 0.79 0.79 0.58 one 0.43 0.57 0.64 0.36 one

each avg. = 0.47 0.51 0.41 0.49 0.58 0.65 0.66 0.68 0.54 0.74 0.39 0.41 0.43 0.34 0.50All avg. = 0.49 0.65 0.41

86

Next directions for KS research?

• Continue to find valid and efficient KS approaches

• And close with a few provocative comments …

87

1st year undergraduate textbook in ISTan obvious ‘collage’

Web reading F-pattern?

Heatmaps from user eyetracking studies of three websites. The areas where users looked the most are colored red; the yellow areas indicate fewer views, followed by the least-viewed blue areas. Gray areas didn't attract any fixations.

http://www.useit.com/alertbox/reading_pattern.html88

http://www.useit.com/alertbox/reading_pattern.html

http://www.useit.com/alertbox/reading_pattern.html

Gaze plot of the 4 main classes of web search reading behaviorsearch-dominant navigation-dominant

tool-dominant successful

http://www.useit.com/alertbox/fancy-formatting.html89

http://www.useit.com/alertbox/fancy-formatting.html

http://www.useit.com/alertbox/fancy-formatting.html

Sources of eye-tracking

• http://www.miratech.com/blog/eye-tracking-lecture-web.html

• http://www.youtube.com/watch?v=X60VPJDLAeM&feature=player_embedded

90

http://www.youtube.com/watch?v=X60VPJDLAeM&feature=player_embedded







91

Altered reading due to web experience?

• If students are not reading linearly, or are using (or not using) headings and other text signals (color, underline, highlights) differently, then the KS will be different

• Specific KS can accomplish specific kinds of mental ‘work’ and other KS other work (the protein analogy)

• So determining how today’s students read hypertext and web materials, and whether this transfer back to paper-based text is an important question

• KS is one tool that can complement existing measures and help explain this

92

Term activation across sentences

terms0.92 knight0.92 rode0.92 forest0.92 country0.92 dragon0.92 princess0.92 kidnap0.92 free (freed)0.92 marry (married)0.92 hurried0.92 fought0.92 death (killed)0.92 armor0.92 thankful

1

4

7

10

13

0

1

2

3

4

5

6

7

Axis

Title

1

2

3

4

5

6

7

8

9

10

11

12

13

alternative measures of knowledge structure: a s measures of text structure and of reading...

Documents

conceptual knowledge

knowledge structure

structural knowledge

knowledge elements5ks

knowledge organization

eliciting knowledge

knowledge structures

categorical knowledge