Anticipatory eye-movements in a visual world: Effects of

Context

Heather Ferguson

Tony Sanford & Christoph Scheepers

GLASGOWLANGUAGE

PROCESSING

Counterfactuals

Counterfactual thinking:- Cases of possibly valid reasoning from premises that

are false in actuality (Fauconnier & Turner, 2003)

“If I had woken up earlier this morning then I would have eaten breakfast”

Counterfactuals in cognitive science

Fauconnier (1994; 2002)- Forced incompatibility between two mental

spaces: reality & hypotheticalLeslie (1987)

- Counterfactual thinking is a special case of theory of mind

Kahneman & Miller (1986)- Modifying past experiences manipulates

expected norms

Anomaly detection in reading

Semantic anomalies & eye-movements (Ni et al, 1998; Braze et al, 2002):– generate longer reading times prior to a gradual

increase in regressive eye-movements

Semantic anomalies & ERP (Kutas & Hillyard, 1980):– N400 effect to pragmatic violations

Discourse-dependent context anomalies & ERP (Van Berkum, Hagoort & Brown, 1999):– elicit a large N400 effect

Counterfactuals: Experimental items

[1] RW- inconsistent If cats are hungry they usually pester their owners until they get fed.Families could feed their cat a bowl of carrots and it would gobble it down happily.

[2] RW- consistentIf cats are hungry they usually pester their owners until they get fed.Families could feed their cat a bowl of fish and it would gobble it down happily.

[3] CW- inconsistentIf cats were vegetarians they would be cheaper for owners to look after.Families could feed their cat a bowl of fish and it would gobble it down happily.

[4] CW- consistentIf cats were vegetarians they would be cheaper for owners to look after.Families could feed their cat a bowl of carrots and it would gobble it down happily.

Eye-movements: Summary

RW violations can be ‘neutralised’ within an appropriate pre-specified CW context

RW-congruent items can lead to the experience of an anomaly following an inconsistent CW

context - prior context is rapidly utilised

Disruption to early processing for RW violations regardless of prior context

RW inconsistencies are detected faster than CW inconsistencies– effects for a RW inconsistency are detected in the first-pass RT

Eye-movements: Summary

First-pass reading times- critial region

220

230

240

250

260

270

280

290

300

310

CW RW

Firs

t-pa

ss r

eadi

ng ti

me

(mse

c)

Inconsistent

Consistent

ERPs: Summary

‘world’

inconsistencies have been detected

- BUT, gives limited information on interference from

RW knowledge

N400 effect

Strong frontal theta following ‘world’ inconsistencies

Temporal gamma for RW violations

Time Frequency: Summary19-Jan-2007xlim=[-0.148 1]ylim=[3 40]zlim=[0 2]

16-Jan-2007 xlim=[-0.148 1]ylim=[3 40] zlim=[0 2]

10-Jan-2007 xlim=[-0.148 1]ylim=[3 40] zlim=[0 2]

10-Jan-2007 xlim=[-0.148 1]ylim=[3 40] zlim=[0 2]

RW-consistent RW-inconsistent

CW-consistent CW-inconsistent

Counterfactual reasoning could account, at least in part, for theory of mind understanding

Significant correlations have been found between false belief and counterfactual reasoning tasks

BUT, ability in counterfactual reasoning is a necessary but not sufficient component of successful performance in the false belief tasks (Peterson & Bowler, 2000)

Counterfactuals & Theory of Mind

The ability to understand and predict other peoples’ behaviour by attributing independent mental states to them

Neurology of Theory of mind Limitations to methods of research to date

– See Saxe (2006), ‘Why and how to study Theory of Mind with fMRI’

– No on-line studies of ToM processing

Theory of Mind

Visual-world research:– Eye-movements around a visual scene are closely

time-locked with related auditory input

Visual World Paradigm

- Altmann & Kamide (1999):

“The boy will move the cake.”

“The boy will eat the cake.”

Visual World Paradigm

Participant looked faster to the cake in the “eat” condition than in the “move” condition.

Participants anticipate forthcoming linguistic reference to objects in the scene - i.e. they looked at the cake before the could hear it

Altmann & Kamide concluded that visual attention is drawn to critical objects in the scene before they are actually mentioned in the auditory input

Can people use their knowledge of the wider discourse to over-ride real-world knowledge to predict specific upcoming words as the current sentence unfolds?

24 experimental items 28 native English speakers (16 female) Eyelink II head-mounted eye-tracker Within-subjects 2 (CW/ RW) x 2 (consistent/

inconsistent) design

Visual World Paradigm

Counterfactuals: Experimental items

[1] RW- inconsistent If cats are hungry they usually pester their owners until they get fed.Families could feed their cat a bowl of carrots and it would gobble it down happily.

[2] RW- consistentIf cats are hungry they usually pester their owners until they get fed.Families could feed their cat a bowl of fish and it would gobble it down happily.

[3] CW- inconsistentIf cats were vegetarians they would be cheaper for owners to look after.Families could feed their cat a bowl of fish and it would gobble it down happily.

[4] CW- consistentIf cats were vegetarians they would be cheaper for owners to look after.Families could feed their cat a bowl of carrots and it would gobble it down happily.

Counterfactuals: Example

Sentence 1

+ RW context

CW context

Sentence 2

Counterfactuals: Example

RW- anomalous

RW- congruent

Temporal location of fixations was recorded at 20ms intervals for each condition

k-Means cluster analysis:

- groups similar eye-movement patterns in the region preceding and immediately following the verbal onset of the target word for statistical analysis

Methods of analysis

Counterfactuals: Results

-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

-1500 -1300 -1100 -900 -700 -500 -300 -100 100 300 500 700 900 1100 1300 1500 1700 1900

Time from Target Noun Onset

Lo

g (

P(R

W)/

P(C

W))

RW-ConsistentRW-InconsistentCW-ConsistentCW-Inconsistent

CW Referent

RW Referent

Clu

ster

1 O

nset

Clu

ster

2 O

nset

Clu

ster

3 O

nset

Clu

ster

4 O

nset

Clu

ster

5 O

nset

Clu

ster

6 O

nset

Clu

ster

7 O

nset

Clu

ster

8 O

nset

Clu

ster

9 O

nset

Clu

ster

10

Ons

et

(-200ms) – 400ms: Main effect of context460ms – 2000ms: Main effect of consistencyNo (context x consistency) interaction at any point

Participants anticipated towards a contextually relevant referent from 200ms prior to critical word onset

Shortly after the onset of the target word, however, the relevant ‘appropriate’ referent was visually favoured

The results provide the first evidence that real-world biases in the visual world paradigm can be ‘neutralised’ by a counterfactual context

No effects of gender were detected

Counterfactuals: Summary

Can people use their knowledge of the wider discourse to over-ride real-world knowledge to predict specific upcoming words as the current sentence unfolds?

24 experimental items 28 native English speakers (16 female) Eyelink II head-mounted eye-tracker Within-subjects 2 (Reality/ Belief) x 2 (inconsistent/

consistent) design

Visual World Paradigm

Janet unpacked the belongings and put the postcard in the cupboard.

[1] Reality-inconsistentLater, Janet saw Barry move the postcard from the cupboard to the drawer. Later, Janet wanted to see the postcard so she looked in the cupboard and smiled.

[2] Reality-consistentLater, Janet saw Barry move the postcard from the cupboard to the drawer. Later, Janet wanted to see the postcard so she looked in the drawer and smiled.

[3] Belief-inconsistent While Janet was busy, Barry moved the postcard from the cupboard to the drawer. Later, Janet wanted to see the postcard so she looked in the drawer and smiled.

[3] Belief-consistent While Janet was busy, Barry moved the postcard from the cupboard to the drawer. Later, Janet wanted to see the postcard so she looked in the cupboard and smiled.

Beliefs of others: Experimental items

Beliefs of others: Example

Sentence 1

Beliefs of others: Example

Sentence 2

Reality context

Belief context

Beliefs of others: Example

Sentence 3

RW- anomalous

RW- congruent

Beliefs of others: Results

(-560ms) – 500ms: Main effect of context860ms – 1500ms: Main effect of context280ms – 1500ms: Main effect of consistencyNo (context x consistency) interaction at any point

-2.5

-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

2.5

-1500 -1300 -1100 -900 -700 -500 -300 -100 100 300 500 700 900 1100 1300 1500

Time from Target Noun Onset

Lo

g (

P(R

eali

ty)/

P(B

elie

f))

Reality-ConsistentReality-InconsistentBelief-ConsistentBelief-Inconsistent

Clu

ster

1 O

nset

Clu

ster

2 O

nset

Clu

ster

3 O

nset

Clu

ster

4 O

nset

Clu

ster

5 O

nset

Clu

ster

6 O

nset

Clu

ster

7 O

nset

Clu

ster

8 O

nset

Clu

ster

9 O

nset

Clu

ster

10

Ons

et

Clu

ster

11

Ons

et

Clu

ster

12

Ons

et

Reality Referent

Belief Referent

Gender effect (-1500ms) – (-560ms): (gender x context) interaction

Beliefs of others: Results

-2.5

-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

2.5

-1500 -1300 -1100 -900 -700 -500 -300 -100 100 300 500 700 900 1100 1300 1500

Time from Target Noun Onset

Lo

g (

P(R

eali

ty)/

P(B

elie

f))

Reality-ConsistentReality-InconsistentBelief-ConsistentBelief-Inconsistent

Clu

ster

1 O

nset

Clu

ster

2 O

nset

Clu

ster

3 O

nset

Clu

ster

4 O

nset

Clu

ster

5 O

nset

Clu

ster

6 O

nset

Clu

ster

7 O

nset

Clu

ster

8 O

nset

Clu

ster

9 O

nset

Clu

ster

10

Ons

et

Clu

ster

11

Ons

et

Clu

ster

12

Ons

et

Reality Referent

Belief Referent

Beliefs of others: GenderFemales ToM

-2.5

-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

2.5

-1500 -1300 -1100 -900 -700 -500 -300 -100 100 300 500 700 900 1100 1300 1500

Reality-InconsistentReality-ConsistentBelief-InconsistentBelief-Consistent

Reality Referent

Belief Referent

Males ToM

-2.5

-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

2.5

-1500 -1300 -1100 -900 -700 -500 -300 -100 100 300 500 700 900 1100 1300 1500

Reality-InconsistentReality-ConsistentBelief-InconsistentBelief-Consistent

Reality Referent

Belief Referent

As a group, participants anticipated towards a contextually relevant referent from 560ms prior to critical word onset

Shortly after the onset of the target word, the relevant ‘appropriate’ referent was visually favoured

HOWEVER, this prediction of others’ behaviour seems to have been led by female participants

Here, females show better performance at predicting others’ actions based on their intentions, beliefs and desires than males

Beliefs of others: Summary

First on-line investigation into the comprehension of the beliefs of others

Also shows evidence to support a gender bias in cognition (see Baron-Cohen, 2002)- Females are superior than males on tests of social judgement and measures of empathy and cooperation (Baron-Cohen, 1996; Charman, Ruffman & Clements, 2002)

- Males excel in practical skills, e.g. mathematical reasoning, mental rotation, spatial tasks and target-directed motor skills

Beliefs of others: Summary

EM, ERP and time frequency data support a two-stage mental model theory of discourse comprehension

In a visual world, prior context is integrated even faster and participants expect context-relevant continuations

– Real-world biases in the visual world paradigm can be ‘neutralised’ by a prior context

– Although this does not mean that the RW violation has been missed all together

Results support a theory of a gender bias in cognition and suggest an advantage for females on a theory of mind task

This significant gender bias was not present when predicting events in a counterfactual world, suggesting different processes underlying the two tasks

Conclusions

Thank you!