alignment of attention in mediated communicationihkoesli/pdf/colour-11-vab2007... · 4. does...

1

Visu

elle

Auf

mer

ksam

keit

und

Blic

kbew

egun

gen

AG Neuroinformatik

INTRODUCTION

METHOD

RESULTS

CONCLUSIONS

Alignment of Attention

in Mediated Communication

SFB 673

Visu

elle

Auf

mer

ksam

keit

und

Blic

kbew

egun

gen

AG Neuroinformatik

INTRODUCTIONINTRODUCTION

METHOD

RESULTS

CONCLUSIONS

What is mediated communication?

Does a suitable research paradigm existfor investigating mediated communication?

2

Visu

elle

Auf

mer

ksam

keit

und

Blic

kbew

egun

gen

AG NeuroinformatikMotivation


METHOD

RESULTS

CONCLUSIONS

Customer support

Source: Demag & STMZ

Visu

elle

Auf

mer

ksam

keit

und

Blic

kbew

egun

gen

AG NeuroinformatikChallenge


METHOD

RESULTS

CONCLUSIONS

Mismatch detection in mediated communication

3

Visu

elle

Auf

mer

ksam

keit

und

Blic

kbew

egun

gen

AG NeuroinformatikParadigm

Visual search


METHOD

RESULTS

CONCLUSIONS

Comparative visual search • Tasks

- Visual scanning

• Goal

- Mismatch identification

• Strategy / Procedure

- Analysis of individual objects or object features- Memorisation- Shift of attention- Comparison and validation

(e.g. Pomplun et al., 2006)

Visu

elle

Auf

mer

ksam

keit

und

Blic

kbew

egun

gen

AG NeuroinformatikParadigm

Visual search


METHOD

RESULTS

CONCLUSIONS

• Tasks

- Visual scanning

- Verbal description

• Goal

- Mismatch identification

Comparative visual search

- Verbal guidance

- Collaborative effort

• Strategy / Procedure

- Analysis of individual objects or object features- Description- Identification: Matching of description and stimulus- Comparison and validation

+ +communication

“… in the top left corner …”

requires alignment

dialogue structure

4

Visu

elle

Auf

mer

ksam

keit

und

Blic

kbew

egun

gen

AG Neuroinformatik

1. Can we identify typical search patterns? Do these depend on the mismatch dimension?

2. (How) do partners align their individual search patterns? Do consistent search patterns develop?

3. Do preferable, efficient search patterns exist?

4. Does non-verbal communication affect the search, e.g. gaze contact or gestures?

5. Can we compensate for unavailable communication channels?

Research Questions


METHOD

RESULTS

CONCLUSIONS

Visu

elle

Auf

mer

ksam

keit

und

Blic

kbew

egun

gen

AG Neuroinformatik

Participants• 48 native German speakers

• 50% male-male, 50% female-female pairs

Stimuli• 16 search image pairs

• search images with four objects at quadrant locations

• exactly one mismatch between image pairs

INTRODUCTION

METHODMETHOD

RESULTS

CONCLUSIONS

Experiment Details

Subject A Subject B

5

Visu

elle

Auf

mer

ksam

keit

und

Blic

kbew

egun

gen

AG Neuroinformatik

INTRODUCTION

METHODMETHOD

RESULTS

CONCLUSIONS

Experiment Details

Procedure

➵

t

Visu

elle

Auf

mer

ksam

keit

und

Blic

kbew

egun

gen

AG Neuroinformatik

INTRODUCTION

METHODMETHOD

RESULTS

CONCLUSIONS

Experiment Details

Setting• Subjects’ face-to-face situation OR screen

between subjects

Apparatus• Two Interactive Minds / LC Technologies

EyeGaze eye-tracking systems

• Remote binocular tracking, 120 Hz (interlaced)

• TCP/IP link for experiment (display) synchronisation

6

Visu

elle

Auf

mer

ksam

keit

und

Blic

kbew

egun

gen

AG Neuroinformatik

INTRODUCTION

METHODMETHOD

RESULTS

CONCLUSIONS

Experiment Details

SettingVi

suel

le A

ufm

erks

amke

it un

d Bl

ickb

eweg

unge

n

AG Neuroinformatik

Independent variables• Mismatch dimension

INTRODUCTION

METHODMETHOD

RESULTS

CONCLUSIONS

Experiment Details

• Group

- Face-to-face

- Behind screen

Dependent variables• Response correctness & time

• Typical EM parameters, e.g. number of fixations, gaze duration & saccade length: distribution of attention

Scan path analysis

- Colour

- Typicality

- Completeness

- Orientation

7

Visu

elle

Auf

mer

ksam

keit

und

Blic

kbew

egun

gen

AG NeuroinformatikSearch Patterns

INTRODUCTION

METHOD

RESULTSRESULTS

CONCLUSIONS

Sample gaze trajectories

View subject A View subject B

Face-to-face

Visu

elle

Auf

mer

ksam

keit

und

Blic

kbew

egun

gen


INTRODUCTION

METHOD

RESULTSRESULTS

CONCLUSIONS


View subject A View subject B

Face-to-face

8

Visu

elle

Auf

mer

ksam

keit

und

Blic

kbew

egun

gen


INTRODUCTION

METHOD

RESULTSRESULTS

CONCLUSIONS


View subject B View subject A

Behind screen

Visu

elle

Auf

mer

ksam

keit

und

Blic

kbew

egun

gen


INTRODUCTION

METHOD

RESULTSRESULTS

CONCLUSIONS

Distribution of attention

Areas of interest and transitions

9

Visu

elle

Auf

mer

ksam

keit

und

Blic

kbew

egun

gen


INTRODUCTION

METHOD

RESULTSRESULTS

CONCLUSIONS


I

IIIIV

II


Visu

elle

Auf

mer

ksam

keit

und

Blic

kbew

egun

gen


INTRODUCTION

METHOD

RESULTSRESULTS

CONCLUSIONS


I

IIIIV

III / II

II / I

III / II II / IIIIV / I I / IV

IV / III

III / IV

IV / III / III

III / III / IV


10

Visu

elle

Auf

mer

ksam

keit

und

Blic

kbew

egun

gen


INTRODUCTION

METHOD

RESULTSRESULTS

CONCLUSIONS



I

IIIIV

II I

IIIIV

II I

IIIIV

II I

IIIIV

II

I

IIIIV

II I

IIIIV

II I

IIIIV

II I

IIIIV

II

I

IIIIV

II I

IIIIV

II I

IIIIV

II I

IIIIV

II

I

IIIIV

II I

IIIIV

II I

IIIIV

II I

IIIIV

II0.15

0.15

0.13

0.12

0.03

0.08

0.040.090.06 0.04

0.02 0.05

0.15

0.12

0.03

0.08

0.040.090.06 0.04

0.02 0.05

0.15

0.15

0.03

0.08

0.040.090.06 0.04

0.02 0.05

0.15

0.12

0.03

0.08

0.040.090.06 0.04

0.02 0.05

0.13 0.130.13

0.13 0.12 0.13

0.15

0.15

0.13

0.12

0.03

0.08

0.040.090.06 0.04

0.02 0.05

0.15

0.12

0.03

0.08

0.040.090.06 0.04

0.02 0.05

0.15

0.15

0.03

0.08

0.040.090.06 0.04

0.02 0.05

0.15

0.12

0.03

0.08

0.040.090.06 0.04

0.02 0.05

0.13 0.130.13

0.13 0.12 0.13

0.15

0.15

0.13

0.12

0.03

0.08

0.040.090.06 0.04

0.02 0.05

0.15

0.12

0.03

0.08

0.040.090.06 0.04

0.02 0.05

0.15

0.15

0.03

0.08

0.040.090.06 0.04

0.02 0.05

0.15

0.12

0.03

0.08

0.040.090.06 0.04

0.02 0.05

0.13 0.130.13

0.13 0.12 0.13

0.15

0.15

0.13

0.12

0.03

0.08

0.040.090.06 0.04

0.02 0.05

0.15

0.12

0.03

0.08

0.040.090.06 0.04

0.02 0.05

0.15

0.15

0.03

0.08

0.040.090.06 0.04

0.02 0.05

0.15

0.12

0.03

0.08

0.040.090.06 0.04

0.02 0.05

0.13 0.130.13

0.13 0.12 0.13

Visu

elle

Auf

mer

ksam

keit

und

Blic

kbew

egun

gen


INTRODUCTION

METHOD

RESULTSRESULTS

CONCLUSIONS


I

IIIIV

II0.16

0.05

0.05 0.140.12 0.08

0.08

0.15

0.030.02

0.040.08

Areas of interest and transitions: typical frequencies

11

Visu

elle

Auf

mer

ksam

keit

und

Blic

kbew

egun

gen


INTRODUCTION

METHOD

RESULTSRESULTS

CONCLUSIONS


I

IIIIV

II0.16

0.05

0.05 0.140.12 0.08

0.08

0.15

0.030.02

0.040.08

Areas of interest and transitions: the search “loop”

Visu

elle

Auf

mer

ksam

keit

und

Blic

kbew

egun

gen

AG Neuroinformatik

T(23) = 48.66; p = 0.01INTRODUCTION

METHOD

RESULTSRESULTS

CONCLUSIONS

0

0,5

1

1,5

2

1 2 3 subseq.

search "loop"

rel.

freq

uenc

y


Search Patterns

Pattern I

Fixations: Within-quadrant vs. between-quadrants F(3; 69) = 2139.90; p < 0.001

Pattern II

0

0,5

1

1,5

2

1 2 3 subseq.

search "loop"

rel.

freq

uenc

y

(76% of all trials)

(24% of all trials)

F(3; 69) = 90.87; p = 0.03

12

Visu

elle

Auf

mer

ksam

keit

und

Blic

kbew

egun

gen

AG Neuroinformatik

INTRODUCTION

METHOD

RESULTSRESULTS

CONCLUSIONS

0

0,5

1

1,5

2

1 2 3 subseq.

search "loop"

rel.

freq

uenc

y


Search Patterns

Pattern I

Saccades: Within-quadrant vs. between-quadrants

0

100

200

300

400

500

600

1 2 3 subseq.

search "loop"

sacc

ade

leng

th (p

xls)

F(3; 69) = 1987.2; p < 0.001

with

in q

uadr

ant

betw

een

quad

rant

s F(3; 69) = 6.92; p = 0.09F(3; 69) = 113.61; p = 0.02

Visu

elle

Auf

mer

ksam

keit

und

Blic

kbew

egun

gen

AG Neuroinformatik

T(23) = 18.57; p = 0.03

T(23) = 46.06; p = 0.01

INTRODUCTION

METHOD

RESULTSRESULTS

CONCLUSIONS

0

0,5

1

1,5

2

1 2 3 subseq.

search "loop"

rel.

freq

uenc

y


Search Patterns

Pattern I vs. II

Saccades: Within-quadrant vs. between-quadrants

0

100

200

300

400

500

600

1 2 3 subseq.

search "loop"

sacc

ade

leng

th (p

xls)

F(3; 69) = 1987.2; p < 0.001F(3; 69) = 200.71; p = 0.02

F(3; 69) = 8.11; p = 0.09F(3; 69) = 24.88; p = 0.07

13

Visu

elle

Auf

mer

ksam

keit

und

Blic

kbew

egun

gen

AG Neuroinformatik

INTRODUCTION

METHOD

RESULTSRESULTS

CONCLUSIONS

0

50

100

150

200

colour orientation typicality completeness

mismatch dimension

dete

ctio

n tim

e (s

)

F(3; 33) = 14.92; p < 0.001

Detection time

Group & Mismatch Dimension Effects

0

50

100

150

200


mismatch dimension

dete

ctio

n tim

e (s

)

F(3; 33) = 4.33; p = 0.033

Face-to-face

Behind screen

Visu

elle

Auf

mer

ksam

keit

und

Blic

kbew

egun

gen

AG Neuroinformatik

INTRODUCTION

METHOD

RESULTSRESULTS

CONCLUSIONS

Gaze duration

Face-to-face

Behind screen


0

50

100

150

200


mismatch dimension

gaze

dur

atio

n (s

)

F(3; 33) = 14.92; p < 0.001

0

50

100

150

200


mismatch dimension

gaze

dur

atio

n (s

)

F(3; 33) = 4.33; p = 0.033

14

Visu

elle

Auf

mer

ksam

keit

und

Blic

kbew

egun

gen

AG Neuroinformatik

INTRODUCTION

METHOD

RESULTSRESULTS

CONCLUSIONS

Gaze duration

Face-to-face

Behind screen


0

50

100

150

200


mismatch dimension

gaze

dur

atio

n (s

)

F(3; 33) = 14.92; p < 0.001

0

50

100

150

200


mismatch dimension

gaze

dur

atio

n (s

)

F(3; 33) = 4.33; p = 0.033

gaze contact

“gaze contact”

Visu

elle

Auf

mer

ksam

keit

und

Blic

kbew

egun

gen

AG Neuroinformatik

INTRODUCTION

METHOD

RESULTSRESULTS

CONCLUSIONS

0

100

200

300

400

500


mismatch dimension

num

ber o

f fix

atio

ns

Number of fixations

0

100

200

300

400

500


mismatch dimension

num

ber o

f fix

atio

ns

Face-to-face

Behind screen

F(3; 33) = 30.52; p < 0.001

F(3; 33) = 10.88; p = 0.02


15

Visu

elle

Auf

mer

ksam

keit

und

Blic

kbew

egun

gen

AG Neuroinformatik

INTRODUCTION

METHOD

RESULTSRESULTS

CONCLUSIONS

0

20

40

60

80

100


mismatch dimension

sacc

ade

leng

th (p

xls)

Saccade length


0

20

40

60

80

100


mismatch dimension

sacc

ade

leng

th (p

xls)

Face-to-face

Behind screen

F(3; 33) = 27.33; p = 0.005

F(3; 33) = 35.38; p = 0.003

Visu

elle

Auf

mer

ksam

keit

und

Blic

kbew

egun

gen

AG Neuroinformatik

INTRODUCTION

METHOD

RESULTSRESULTS

CONCLUSIONS

0

50

100

150

I II

search pattern

dete

ctio

n tim

e (s

)

T(23) = 8.92; p < 0.001

Detection time & search patterns

Search Efficiency

0

50

100

150


mismatch dimension

dete

ctio

n tim

e (s

)

Pattern I vs. II

16

Visu

elle

Auf

mer

ksam

keit

und

Blic

kbew

egun

gen

AG NeuroinformatikSummary & Additional Findings

INTRODUCTION

METHOD

RESULTSRESULTS

CONCLUSIONS

• Individual differences in search patterns exist between subject pairs:

I. Partners initially scan all objects briefly and, if mismatch was not found, subsequently discuss objects in detail serially. This is generally quicker than

II. detailed serial object processing without initial “overview” scan.

• Search patterns are consistent only within trials. Partners do not usually stick to a single search pattern during the experiment.

• Colour differences are most easily detected, finding object typicality mismatches takes longest.

• Mismatch dimension from previous trial are often checked first (“recency effect”), leading to singular, unexpectedly short RTs for “difficult” mismatch dimension.

• Search patterns are guided by verbal object descriptions. Partners rarely discuss search strategies beforehand, turn-taking and verbal guidance are automised.

• Searches are rarely dominated by a single interlocutor, only if one partner is very inactive. This “monologue” pattern takes longer to accomplish mismatch detection.

• Approx. 6% of all fixations land on partner, most frequently so when partners search reassurance for verbal utterances or when verbal references are unclear.

• No significant effect on search times was found between face-to-face and behind-screen groups. Partners hardly gesture, even in face-to-face situations.

Visu

elle

Auf

mer

ksam

keit

und

Blic

kbew

egun

gen

AG Neuroinformatik

• A multi-pass search strategy with rather than without an initial coarse overview scan generally detects mismatches in collaborative visual search quicker. The rapid initial scene overview helps to detect obvious differences and should thus not be omitted.

• A colour-coding mechanism even for other object dimensions such as orientation, typicality or completeness may increase mismatch detection speeds.

• In repeated search tasks, visual search may benefit from recency effects, in particular in case of “hard-to-spot” differences.

• A natural communication pattern between partners should not be replaced by a “check-list” search initiated/dominated by one partner – unless required due to partner inactivity.

• The role of gaze contact remains unclear. Although partners more frequently look at each other in “critical” situations, this does not seem to directly affect search performance – but could be a “personal comfort” factor.

• The lack of (attending to) gestures comes as a surprise, however, might be due to experimental conditions/setting. The face-to-face scenario should be improved for future studies and ensure less interrupted views for partners.

INTRODUCTION

METHOD

RESULTS

CONCLUSIONSCONCLUSIONS

Conclusions

alignment of attention in mediated communicationihkoesli/pdf/colour-11-vab2007... · 4. does...

Documents