On using eyetracking data to evaluate theories of on-line sentence processing:

The case of reduced relative clausesCharles Clifton, Jr.grateful acknowledgments to

Mohamed Mohamed

Matt Traxler

Rihana Williams

Keith Rayner

Robin Morris

Sungryong Koh

Lyn Frazier

Eye movements and sentence comprehension

• Existing measures not a transparent window into cognitive processes– Lexical processing: some good ideas about

mapping– Comprehension: another story– Consider one case of garden-path sentences…

Garden Paths in Reduced Relative Clause Sentences

(the curse of Tom Bever)

• The horse raced past the barn fell.

• But sometimes reduced relative clauses are easy to comprehend.

McKoon & Ratcliff, 2003

• The window broken by John couldn’t be repaired

• The window examined by John needed repairing

• Both cases: (easy) reduced relative construction denotes an entity that participates in an externally caused event

Trueswell, Tanenhaus, & Garnsey, 1994

• The defendant examined by the lawyer proved to be unreliable– Produces clear reading disruption in disambiguating

region compared to full relative clause (The defendant who was…)

• The evidence examined by the lawyer proved to be unreliable– Initial NP inanimate, and a good theme but a poor agent– Claimed to eliminate difficulty of reduced relative clause

• TTG 1994 actually a followup to Ferreira & Clifton, 1986

Animacy

Animate Inanimate

Fir

st P

ass

Tim

e, M

s/ch

ar

0

20

30

40

50

Ferreira & Clifton 1987, Disambiguating Region

ReducedRel

FullRel

Animate: The defendant examined by the lawyer…

Inanimate: The evidence examined by the lawyer…

Criticism: Trueswell, Tanenhaus, & Garnsey, 1994

• Some of the Ferreira & Clifton inanimate sentences weren’t all that implausible as main clause sentences– The trash smelled…. (pretty bad)– The car towed…(the trailer)

• TTG, better materials, first noun normed for plausibility as agent and as patient of first verb

Subject Noun

Animate Inanimate

Fir

st P

ass

Tim

e, m

s

0

400

500

600

Trueswell, Tanenhaus, & GarnseyDisambiguating Region

RedRel

FullRel

Why the differences?

• Maybe TTG were right – the F&C materials were bad

• But maybe something else: parafoveal preview of “by the…”– Evidence from Burgess, Spivey, McRae, etc, in

self-paced reading– Possible display limitations in F&C (42

character display, line break typically before PP would have prevented effective preview)

Clifton, Traxler, Mohamed, Williams, Morris, Rayner, in press

• Redo Ferreira & Clifton using Trueswell et al. materials (thanks to John et al.)

• Add boundary change manipulation to permit or deny parafoveal preview of “by the noun”

• Perform additional analyses

Clifton et al. details

• Used materials from Trueswell, Tanenhaus, & Garnsey

• 8 conditions– Animate (“difficult”) vs inanimate (“easy”) initial noun

– Reduced vs full (control) relative clause

– Parafoveal preview vs. no preview of “by the” (reader saw random 6-letter string instead of “by the”)

• Measured eye movements while reading for comprehension (24 Ss)

Fir

st P

ass

Tim

e, m

s

0

400

500

600

700

800Disambiguating Region, FP

Animate Inanimate Animate Inanimate Preview No Preview

RedRel

FullRel

Fir

st P

ass

Tim

e, m

s

0

400

500

600

700

800Disambiguating Region, FP

Animate Inanimate Animate Inanimate Preview No Preview

RedRel

FullRel

Overall: 66 ms ambiguity effect for animate nouns, 34 ms ambiguity effect for inanimate nouns (sig. smaller, but > 0)

Reg

ress

ion

Pat

h D

ura

tion

, ms

0

400

500

600

700

800Disambiguating Region, RPD

Animate Inanimate Animate Inanimate Preview No Preview

RedRel

FullRel

Overall: 68 ms ambiguity effect for animate nouns, 83 ms ambiguity effect for inanimate nouns (n.s. different)

Interim Conclusion and Question

• Looking only at first pass times gave an incomplete picture of sentence comprehension– RPD measure indicated garden-pathing even

with inanimate initial noun– Trueswell et al. experiment not an adequate

basis to reject modular serial parsing model• But do the increased RPDs reflect the usual

behavior of the eyes?

Pro

por

tion

Fir

st P

ass

Reg

ress

ion

s O

ut

0.0

5.0

10.0

15.0

20.0

25.0Disambiguating Region, Reg Out

Animate Inanimate Animate Inanimate Preview No Preview

RedRel

FullRel

More regressions for No Preview than Preview, and for Animate than Inanimate, but apparent effect of Reduction and interactions not significant.

Pro

por

tion

Fir

st P

ass

Reg

ress

ion

s O

ut

0.0

5.0

10.0

15.0

20.0

25.0Disambiguating Region, Reg Out

Animate Inanimate Animate Inanimate Preview No Preview

RedRel

FullRel

Reg

ress

ion

Pat

h D

ura

tion

, ms.

, Ou

tsid

e R

egio

n

0

20

40

60

80

100

120Disambiguating Region, RPD Outside

Animate Inanimate Animate Inanimate Preview No Preview

RedRel

FullRel

Sec

ond

Pas

s T

ime,

ms.

, fro

m L

eft

0

20

40

60

80

100

120Disambiguating Region, Second Pass from Left

Animate Inanimate Animate Inanimate Preview No Preview

RedRel

FullRel A similar pattern for: regression path duration (outside the region) second pass on target (coming from left)

Combined data: First pass regressions

• Combine data with parallel experiment conducted at U South Carolina– Always preview of “by the” phrase– Vary high vs low span readers

• Total n = 68, more stable data

Condition

Anim InanPro

por

tion

Fir

st P

ass

Reg

ress

ion

s

0.0

2.0

4.0

6.0

8.0

10.0

12.0

14.0Combined Data, Regressions Out

RedRel

FullRel

Significant effects of animacy, ambiguity, no interaction

Summary

• Eye movement measures show comparable ambiguity effects for sentences with animate and with inanimate subjects– It is premature to conclude that inanimacy

blocks garden-pathing– Other data are needed to evaluate models that

give logical priority to syntactic structure

A problem for understanding eye movements

• Regression path durations are inflated by processing difficulty– This inflation can only come from trials where

there were regressions

• But regressions were very infrequent, < 10% of the trials

A similar effect in other experiments

• Reduced relative clauses with various classes of verbs (manner of motion, theta-grid changing, unaccusatives)

• No garden-pathing in first pass times, big garden-pathing in RPD, but < 30% regressions

Conclusions

• Sentence processing difficulty is only variably mirrored in the ET record– Doesn’t always show up in FPT

• One clear response to difficulty: a regression– But only happens 5-30% of time

• A common response to difficulty: fuggettabottit– Just get the words, `mam

I’d like to be shown wrong, but…

• ET response to comprehension difficulty is variable: linger, regress, nothing

• Effects seen in means may actually occur on only a minority of trials

• Eye movements may be controlled by word recognition, and only exceptionally affected by higher-level factors

Put on those rose-colored glasses

• Maybe all these problems will disappear if people read texts they really care about, not just single sentences

• But I wouldn’t bet on it

Stevenson & Merlo, 1997

• The troops marched across the fields all day resented the general.– Manner of movement verbs hard in reduced

relatives; transitive use of these verbs requires an operation of “syntactic causativisation”

• But…The witch melted in the Wizard of Oz was played by a famous actress.– Seems easy. Suggestion: Unaccusative verbs

OK in reduced relatives

Eyetracking Experiment

• Sentences with Manner of Movement (MOM) verbs

• Sentences with verbs whose thematic grid changes (agent-theme to goal-theme) with passivization

• Sentences with unaccusative verbs (theme in both active and passive subject position)

MOM (manner of movement) sentences

• The horse raced along the beach frightened the onlookers

• The craft sailed to the island served as a makeshift ferry

• Controls– The horse that was raced to to the beach frightened the

onlookers. (full relative clause)– The horse raced along the beach and frightened the

onlookers. (main clause)– Disambiguating region indicated in boldface

Thematic Grid Change Sentences

• The secretary brought the forms filled them out.

• The client asked the question received no thanks.– Main clause structure: Agent – verb – theme– Relative clause structure: Goal – verb – theme– Control: The client asked the question but

received no thanks

Unaccusative Verbs

• The leaves burned today smoked a lot.

• The potatoes baked in the oven smelled terrific.– Main clause structure: theme – verb– Relative clause structure: theme – verb– Control: The potatoes baked in the oven and

smelled terrific.

Predictions

• Manner of movement sentences: Hard, on everybody’s story

• Thematic grid change sentences: Comparably hard, if thematic grid, not causal structure, the root

• Unaccusative sentences: Relatively easy, if thematic grid change not syntactic structure change the underlying problem

First Pass Times

0

400

500

600

Fir

st P

ass

Tim

e, D

isam

big

Reg

,ms

MOM Thematic Unaccusative Change

Legend

RedRelFullRelMainCl

Regression Path Durations

Verb Type

MOM ThemeChg Unaccus

Reg

ress

ion

Pat

h D

urat

ion,

ms

0

400

600

800

1000

1200

1400

Disambiguating Region

Legend

RedRelFullRelMainCl

Comprehension Accuracy

• Questions were asked following 2/3 of the items (half of these were relevant to the RC)– MOM questions: The boy marched to school looked unhappy.

• How did the boy get to school?Someone marched him there/He marched there on his own

– Theta-grid changing questions: The client asked the question received no thanks.

• Who asked the question?The client / someone

– Unaccusative questions: The potatoes baked in the oven smelled terrific

• What was happening with the potatoes?Someone had baked them in the oven / They were sitting in the oven,

baking.

Verb Type

MOM ThemeChg

Pro

port

ion

Cor

rect

0.0

0.2

0.4

0.6

0.8

1.0Question-answering Accuracy

RedRel

FullRel

MainCl

Note: accuracy for Unaccusative verb sentences is not shown. The questions were bad – only 67% correct for main clause items.

Conclusions

• Intuitions of sentence processing difficulty not transparently mirrored in the ET record

• Typical response to difficulty: a regression– May be specific to this experiment; lots of hard,

tricky RC sentences– But even here, only about 1/3 of the trials

• Another response to difficulty: fughetabottit– Just get the words, `mam

For the future: I’d like to be shown wrong, but…

• ET response to comprehension difficulty seems to be variable: linger, regress, nothing

• Eye movements may generally be controlled by word recognition, and only exceptionally affected by higher-level factors

• Effects seen in means may actually occur on only a minority of trials

• Maybe all these problems will disappear if people read texts they really care about, not just single sentences

• But I wouldn’t bet on it

Reg

ress

ion

Pat

h D

ura

tion

, ms.

, Ou

tsid

e R

egio

n

0

20

40

60

80

100

120Disambiguating Region, RPD Outside

Animate Inanimate Animate Inanimate Preview No Preview

RedRel

FullRel

Seco

nd P

ass

Tim

e, m

s., f

rom

Lef

t

0

20

40

60

80

100

120Disambiguating Region, Second Pass from Left

Animate Inanimate Animate Inanimate Preview No Preview

RedRel

FullRel

MacDonald, Pearlmutter, Seidenberg, 1994

• In all cases, the examples cited here were not the only reduced relatives in these articles.

• The pattern of results obtained with global reading times is reflected by differences located on the critical disambiguating phrase.

Ferreira & Clifton 1986 – An early attempt to find easy RRCs

• Ambiguous animate: The defendant examined by the lawyer was unreliable

• Ambiguous inanimate: The evidence examined by the lawyer was unreliable

• Controls: The defendant/evidence who/that was examined by the lawyer was unreliable.– Disambiguating region indicated in boldface