Mirjana Sekicki

Christine Ankener

Maria Staudte

+ the gaze cue

Costs and gains of using

visual context for

referent prediction

Costs and gains of using visual context for referent prediction

Shannon (1948)

• Comprehension difficulty is relative to the predictability of a word in a given context.

• linguistic expressions that are highly predictable, in a given context, convey less info than those which are surprising.

2

Costs and gains of using visual context for referent prediction

Shannon (1948)

• Comprehension difficulty is relative to the predictability of a word in a given context.

• linguistic expressions that are highly predictable, in a given context, convey less info than those which are surprising.

linguistic

script knowledge

visual

3

Costs and gains of using visual context for referent prediction

We were curious …

• cognitive load at the different stages of prediction

1. prediction making

2. prediction (dis)confirming

we were curious…

The woman irons the t-shirt.

• Quantification of the role of visual context in prediction making

4

Costs and gains of using visual context for referent prediction

• We conducted eye-tracking experiments in the visual world and measured:

• Cognitive Load

• prediction forming

• prediction (dis)confirming

• Anticipatory eye-movements

5

Costs and gains of using visual context for referent prediction

The Index of Cognitive Activity

• Wavelet analysis on the pupil dilation record removes all large oscillations and retains only the small and rapid dilations linked to cognitive load (Marshall, 2000)

• Sensitive to Cognitive load induced by language processing (Demberg & Sayeed, 2016)

6

So, we conducted:

1. Ling Study

2. Vis Study

7

Table 1: Sample items and corresponding offline results.

Item noun condition plausibility cloze probability%M (SD) M (SD)

(1) The woman irons soon the a) t-shirt plausible 1.12 (0.68) 13.67 (18.06)b) sock possible 2.76 (2.17) 0.16 (00.54)c) armchair anomalous 6.35 (1.43) <.01

(2) The woman describes soon the a) t-shirt possible 1.65 (1.50) NAb) sock possible 1.90 (1.80) NAc) armchair possible 1.89 (1.78) NA

sure sensitivity. Besides traditional methods, such as readingand reaction time measurements, we also applied a relativelynovel measure: the Index of Cognitive Activity (ICA). ICAis able to capture task related pupil jitter, linked to cognitiveload and predictability (Demberg & Sayeed, 2016). One ofour aims was to examine if ICA can be used to assess dif-ferences in cognitive load induced by plausibility and pre-dictability variation. The advantage of ICA lies in the possi-bility to have the listener view things other than the word sheis currently processing. This would enable the exploration ofvisual context effects on cognitive load and predictability.

Experimental MaterialsWe ran 2 offline and 4 online studies, using different mea-sures on a set of 36 stimuli items. The aim was to empiri-cally establish a solid indication of cognitive load as a func-tion of plausibility and (in part) predictability. The influenceof cloze-probability on cognitive load was reduced by usinglow constraining contexts.

Experimental items All items were German independentmain clauses with uniform syntactic structure (NP-V-ADV-NP). The first NP (sentence subject) was always neutral toavoid an impact on plausibility. The verbs which introducedthe strength of constraint on the subsequent noun were re-strictive (1) or non-restrictive (2). The two different verb cat-egories create constraint and no-constraint contexts to elicitstronger (1) or weaker (2) predictions. The adverb (gleich)following the verb, serves as spill-over region in order to giveparticipants more time to form predictions about the upcom-ing noun. Example: Die Frau bugelt / beschreibt gleich dasT�Shirt / die Socke / den Sessel (see Table 1 for details).

Both of our verb categories were then paired with threedifferent nouns (sentence object) of approximately same fre-quency. Noun frequencies were extracted from word listsDeReWo2 of the German research corpus (DeReKo) and heldconstant for nouns within one item in order to exclude animpact of frequency. Plausibility and predictability were as-sessed by two offline studies described below.

Filler items Fillers were plausible sentences of German,similar to condition (1a). Since a high amount of semanti-cally anomalous sentences could artificially increase the tol-

2(DeReKo corpus size > 28 billion words. Source: Korpusbasierte WortgrundformenlisteDeReWo,2012, v-ww-bll-320000g-2012-12-31-1.0, http://www.ids-mannheim.de/derewo)

erance for implausible sentences, no anomalous fillers wereused. Fillers differed, however, from experimental items inlength and syntactic structure to introduce syntactic variationfor the comprehender. Half of the fillers were followed byyes/no comprehension questions.Analysis The statistical analyses of the data collected fromall experiments were conducted using R statistical program-ming environment (R Core Team, 2013) and the lme4 pack-age (Bates, Maechler, Bolker, & Walker, 2015). Statisticalsignificance was assessed by stepwise reduction of the maxi-mal model’s fixed effects structure.

The Verb and Object fixed factors used for the analysesof all the experiments were contrast coded, thus creating thefollowing factors: O1 which denotes the difference betweenobjects (a) vs. (b); O2 which denotes the difference betweenobjects (b) vs. (c); and their interaction with the Verb, namely,VxO1 for the interaction between the Verb and O1, and VxO2for the interaction between the Verb and O2.

Offline Experiments

First, a plausibility rating determined how plausible on aver-age a noun is in its sentence context. Second, a classical sen-tence completion task for cloze probability norming assessedto what extend the verb constraint would also increase pre-dictability (as in cloze probability) of plausible vs. possiblenouns. Both studies were set up using a web form. Answerswere submitted anonymously via the form or using pen andpaper. We controlled for the unique participation of partici-pants.Plausibility ratingMethod 14 independent German native speakers with ageranging from 18 to 58 years participated in the study volun-tarily without reimbursement. One list contained all our stim-uli sentences in randomized order and without fillers, so thateach participant scored every item once. 216 sentences wereused for our 36 items in 6 conditions (see Table 1). In thetask description, participants were asked to rate the sentencesaccording to their plausibility on a 7�point Likert scale. Thescale ranged from very plausible (1) to not plausible / diffi-cult to imagine (7).Results We substantially changed 3 items which elicitedscores that noticeably deviated from the average rating pat-

pretests

8

So, we conducted:

1. Ling Study

2. Vis Study

9

Results

Results

●

●

●

●

●

●

12

13

14

15

16

17

18

19

20

21

22

Subject Verb ObjectThe woman irons/describes soon the t−shirt / the sock.

Mea

n IC

A −

Valu

es Verbs:●● Audio: iron

Audio: describe

Visual: iron

Visual: describe

●

●

●

●

●

●

12

13

14

15

16

17

18

19

20

21

22

Subject Verb ObjectThe woman irons/describes soon the t−shirt / the sock.

Mea

n IC

A −

Valu

es Verbs:●● Audio: iron

Audio: describe

Visual: iron

Visual: describe

LING

LING

VIS

VIS

ICA

Ling vs. Vis

11

Results

●

●

●●

●

●

16

17

18

19

20

21

22

Subject Verb ObjectThe woman irons/describes soon the t−shirt / the sock.

Mea

n IC

A va

lues Conditions

●● iron t−shirt

iron sock

describe t−shirt

describe sock

●

●

●●

●

●

16

17

18

19

20

21

22

Subject Verb ObjectThe woman irons/describes soon the t−shirt / the sock.

Mea

n IC

A va

lues Conditions

●● iron t−shirt

iron sock

describe t−shirt

describe sock

ICA

Visual Study

12

Results

●

●

●●

●

●

16

17

18

19

20

21

22

Subject Verb ObjectThe woman irons/describes soon the t−shirt / the sock.

Mea

n IC

A va

lues Conditions

●● iron t−shirt

iron sock

describe t−shirt

describe sock

●

●

●●

●

●

16

17

18

19

20

21

22

Subject Verb ObjectThe woman irons/describes soon the t−shirt / the sock.

Mea

n IC

A va

lues Conditions

●● iron t−shirt

iron sock

describe t−shirt

describe sock

ICA

Visual Study

13

Results

●

●

●●

●

●

16

17

18

19

20

21

22

Subject Verb ObjectThe woman irons/describes soon the t−shirt / the sock.

Mea

n IC

A va

lues Conditions

●● iron t−shirt

iron sock

describe t−shirt

describe sock

●

●

●●

●

●

16

17

18

19

20

21

22

Subject Verb ObjectThe woman irons/describes soon the t−shirt / the sock.

Mea

n IC

A va

lues Conditions

●● iron t−shirt

iron sock

describe t−shirt

describe sock

ICA

Visual Study

14

Results

●

●

●●

●

●

16

17

18

19

20

21

22

Subject Verb ObjectThe woman irons/describes soon the t−shirt / the sock.

Mea

n IC

A va

lues Conditions

●● iron t−shirt

iron sock

describe t−shirt

describe sock

●

●

●●

●

●

16

17

18

19

20

21

22

Subject Verb ObjectThe woman irons/describes soon the t−shirt / the sock.

Mea

n IC

A va

lues Conditions

●● iron t−shirt

iron sock

describe t−shirt

describe sock

ICA

Visual Study

15

Results: Vis Study

describe sock describe t−shirt iron sock iron t−shirt

0.00

0.25

0.50

0.75

1.00

100011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

0010

0011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

0010

0011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

0010

0011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

00

Trial Time (ms)

Prop

ortio

n O

f Fix

atio

ns

Objects:t−shirt

sock

distractor1

distractor2

describe sock describe t−shirt iron sock iron t−shirt

0.00

0.25

0.50

0.75

1.00

100011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

0010

0011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

0010

0011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

0010

0011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

00

Trial Time (ms)

Prop

ortio

n O

f Fix

atio

ns Objects:t−shirt

sock

distractor1

distractor2

face

Referent Gaze Condition

describe sock describe t-shirt iron sock iron t-shirt

Proportion of fixations

16

We saw …

• linguistic + visual context -> concrete predictions

• Verb acts as a pointer to categories of objects

• Situated communication: Gaze cue an inseparable part of the visual context

• Speakers look at entities shortly before mentioning them. (Griffin & Bock, 2000; Meyer et al., 1998)

• Listeners rapidly inspect objects as they are mentioned. (Tanenhaus et al., 1995)

So, we started wondering about … gaze

17

The gaze cue

• Is gaze cue part of the context for the spoken referent?

• If so, how does it influence the surprisal on the referent?

• Is there surprisal on the gaze cue?

18

So, we conducted…

3. Gaze Study

“The woman irons”

“soon the t-shirt.”

no gaze

referent gaze

Gaze conditions:

19

Example trial

Results

●

●

●

●

●

●

●

●

16

17

18

19

20

21

22

23

Subject Verb Adverb ObjectThe woman irons/describes soon the t−shirt / the sock.

Mea

n IC

A va

lues

Referent gaze.Conditions:●● iron t−shirt

iron sock

describe t−shirt

describe sock

●

●

●

●

●

●

●

●

16

17

18

19

20

21

22

23

Subject Verb Adverb ObjectThe woman irons/describes soon the t−shirt / the sock.

Mea

n IC

A va

lues

Referent gaze.Conditions:●● iron t−shirt

iron sock

describe t−shirt

describe sock

●●

●

●

●

●

●

●

16

17

18

19

20

21

22

23

Subject Verb Adverb ObjectThe woman irons/describes soon the t−shirt / the sock.

Mea

n IC

A va

lues

No gaze.Conditions:●● iron t−shirt

iron sock

describe t−shirt

describe sock

●●

●

●

●

●

●

●

16

17

18

19

20

21

22

23

Subject Verb Adverb ObjectThe woman irons/describes soon the t−shirt / the sock.

Mea

n IC

A va

lues

No gaze.Conditions:●● iron t−shirt

iron sock

describe t−shirt

describe sock

ICA

●

●

●●

●

●

16

17

18

19

20

21

22

Subject Verb ObjectThe woman irons/describes soon the t−shirt / the sock.

Mea

n IC

A va

lues Conditions

●● iron t−shirt

iron sock

describe t−shirt

describe sock

●

●

●●

●

●

16

17

18

19

20

21

22

Subject Verb ObjectThe woman irons/describes soon the t−shirt / the sock.

Mea

n IC

A va

lues Conditions

●● iron t−shirt

iron sock

describe t−shirt

describe sock

26

describe sock describe t−shirt iron sock iron t−shirt

0.00

0.25

0.50

0.75

1.00

100011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

0010

0011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

0010

0011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

0010

0011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

00

Trial Time (ms)

Prop

ortio

n O

f Fix

atio

ns Objects:t−shirt

sock

distractor1

distractor2

face

No Gaze Condition

describe sock describe t−shirt iron sock iron t−shirt

0.00

0.25

0.50

0.75

1.00

100011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

0010

0011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

0010

0011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

0010

0011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

00

Trial Time (ms)

Prop

ortio

n O

f Fix

atio

ns Objects:t−shirt

sock

distractor1

distractor2

face

Referent Gaze Condition

describe sock describe t−shirt iron sock iron t−shirt

0.00

0.25

0.50

0.75

1.00

100011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

0010

0011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

0010

0011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

0010

0011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

00

Trial Time (ms)

Prop

ortio

n O

f Fix

atio

ns Objects:t−shirt

sock

distractor1

distractor2

face

Referent Gaze Conditiondescribe sock describe t-shirt describe t-shirtdescribe sock

Proportion of fixations

describe sock describe t−shirt iron sock iron t−shirt

0.00

0.25

0.50

0.75

1.00

100011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

0010

0011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

0010

0011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

0010

0011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

00

Trial Time (ms)

Prop

ortio

n O

f Fix

atio

ns

Objects:t−shirt

sock

distractor1

distractor2

describe sock describe t−shirt iron sock iron t−shirt

0.00

0.25

0.50

0.75

1.00

100011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

0010

0011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

0010

0011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

0010

0011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

00

Trial Time (ms)

Prop

ortio

n O

f Fix

atio

nsObjects:

t−shirt

sock

distractor1

distractor2

27

describe sock describe t−shirt iron sock iron t−shirt

0.00

0.25

0.50

0.75

1.00

100011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

0010

0011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

0010

0011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

0010

0011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

00

Trial Time (ms)

Prop

ortio

n O

f Fix

atio

ns Objects:t−shirt

sock

distractor1

distractor2

face

No Gaze Condition describe sock describe t−shirt iron sock iron t−shirt

0.00

0.25

0.50

0.75

1.00

100011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

0010

0011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

0010

0011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

0010

0011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

00

Trial Time (ms)

Prop

ortio

n O

f Fix

atio

ns Objects:t−shirt

sock

distractor1

distractor2

face

Referent Gaze Condition

describe sock describe t−shirt iron sock iron t−shirt

0.00

0.25

0.50

0.75

1.00

100011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

0010

0011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

0010

0011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

0010

0011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

00

Trial Time (ms)

Prop

ortio

n O

f Fix

atio

ns Objects:t−shirt

sock

distractor1

distractor2

face

Referent Gaze Conditiondescribe sock describe t−shirt iron sock iron t−shirt

0.00

0.25

0.50

0.75

1.00

100011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

0010

0011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

0010

0011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

0010

0011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

00

Trial Time (ms)

Prop

ortio

n O

f Fix

atio

ns Objects:t−shirt

sock

distractor1

distractor2

face

Referent Gaze Condition

describe sock describe t−shirt iron sock iron t−shirt

0.00

0.25

0.50

0.75

1.00

100011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

0010

0011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

0010

0011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

0010

0011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

00

Trial Time (ms)

Prop

ortio

n O

f Fix

atio

ns Objects:t−shirt

sock

distractor1

distractor2

face

Referent Gaze Condition

iron sock iron t-shirt iron t-shirtiron sock

Proportion of fixations

describe sock describe t−shirt iron sock iron t−shirt

0.00

0.25

0.50

0.75

1.00

100011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

0010

0011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

0010

0011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

0010

0011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

00

Trial Time (ms)

Prop

ortio

n O

f Fix

atio

ns

Objects:t−shirt

sock

distractor1

distractor2

describe sock describe t−shirt iron sock iron t−shirt

0.00

0.25

0.50

0.75

1.00

100011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

0010

0011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

0010

0011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

0010

0011

0012

0013

0014

0015

0016

0017

0018

0019

0020

0021

0022

0023

0024

0025

0026

0027

0028

0029

0030

0031

0032

0033

0034

0035

0036

0037

0038

0039

0040

0041

0042

0043

0044

0045

0046

0047

0048

0049

0050

00

Trial Time (ms)

Prop

ortio

n O

f Fix

atio

nsObjects:

t−shirt

sock

distractor1

distractor2

28

• Visual context concrete predictions

• Surprisal effects @ prediction (dis)confirming

• No cognitive load differences @ prediction forming:

• no active predictions?

• load too low?

• No cognitive load differences @ the gaze cue:

• what if gaze becomes really surprising (incongruent)?

Conclusions

29