Blink

Acknowledgements

References

Co-registration of eye movements and EEG to study semantic congruency during scene perception

Have you seen this object?

YES NO

X 10

X 10

time

+

< 5 s gaze contingent

4 s

1 s

X 5

Object• congruent• incongruent• new

CONCLUSIONS

Protocol

Filtering: 0.1 – 100Hz + notch 60 Hz

* *

Occipital

Region Of Interest (ROI): congruent, incongruent or control objectEye-movements Electroencephalography (EEG)→ Incongruent objects fixated more often and longer than congruent objects [Henderson, Weeks and Hollingworth 1999] [Underwood and Foulsham 2006]

[Underwood, Humphreys and Cross 2007] [Võ and Henderson 2009]

→ Incongruent objects elicited a stronger centro-frontal N300–N400→ Early congruency effect, dominant on the left hemisphere electrodes[Mudrik, Lamy & Deouell 2010][Ganis & Kutas 2003] [Demiral, Malcolm & Henderson 2010]

[Mudrik, Lamy & Deouell 2010]

→ Incongruent objects were more fixated and for a longer duration than congruent objects. Replication of eye movement results BUT differences also present between congruent and control objects.

→ The P1 EFRP and a late EFRP emerging around 260 ms after the fixation onset were modulated by the different objects. No difference between congruent and incongruent objects BUT difference between congruent/incongruent and control objects.

The top-down encoding of the scene was built during the first eye fixations; a mismatch between the attribute of the objects and the features of the scene affected the scene exploration. These results

suggest that top-down information influences early object processing during natural viewing.

• Demiral, Ş. B., Malcolm, G. L., & Henderson, J. M. (2012). ERP correlates of spatially incongruent object identification during scene viewing: Contextual expectancy versus simultaneous processing. Neuropsychologia, 50(7), 1271-1285.• Devillez, H., Guyader, N., & Guérin-Dugué, A. (2015). An eye fixation–related potentials analysis of the P300 potential for fixations onto a target object when exploring natural scenes. Journal of Vision, 15(13), 20-20.• Ganis, G., & Kutas, M. (2003). An electrophysiological study of scene effects on object identification. Cognitive Brain Research, 16(2), 123-144. • Henderson, J. M., Weeks Jr, P. A., & Hollingworth, A. (1999). The effects of semantic consistency on eye movements during complex scene viewing. Journal of experimental psychology: Human perception and performance, 25(1), 210• Kaunitz, L. N., Kamienkowski, J. E., Varatharajah, A., Sigman, M., Quiroga, R. Q., & Ison, M. J. (2014). Looking for a face in the crowd: Fixation-related potentials in an eye-movement visual search task. NeuroImage, 89, 297-305..• Mudrik, L., Lamy, D., & Deouell, L. Y. (2010). ERP evidence for context congruity effects during simultaneous object–scene processing. Neuropsychologia, 48(2), 507-517.• Underwood, G., & Foulsham, T. (2006). Visual saliency and semantic incongruency influence eye movements when inspecting pictures. The Quarterly journal of experimental psychology, 59(11), 1931-1949.• Underwood, G., Humphreys, L., & Cross, E. (2007). Congruency, saliency and gist in the inspection of objects in natural scenes. Eye movements: A window on mind and brain, 564-579.• Võ, M. L. H., & Henderson, J. M. (2009). Does gravity matter? Effects of semantic and syntactic inconsistencies on the allocation of attention during scene perception. Journal of Vision, 9(3), 24.

This research was supported by Grants N00014-14-1-0670 and N00014-16-1-2128 from the Office of Naval Research (ONR).Thanks to Chris Bird, Krystin Corby, and Shaina Martis for research support.

Reference: averageStatistics: cluster-based permutation test

Empirical distribution of fixation duration/saccade amplitude

matched to the global distribution by removing fixations/saccades

[Devillez, Guyader and Guérin-Dugué 2015]

PRE-PROCESSINGArtifacts rejection: ICA Eye-movement artifacts limitation

Vertical component

Horizontal component

Saccade

METHODS

OBJECTIVES

→ Investigated the effect of semantic congruency on scene processing using eye-fixation related potentials (EFRPs)→ Simultaneously registered EEG and eye-tracking (ET) signals of participants exploring natural scenes during 4 sec in preparation for a recognition memory test

→Direct access to the timing of the processes involved during fixations→Direct access to the scene exploration across the scanpath

Before

After

Stimuli

timefixation n-1fixation n fixation n+1

Overlap of potentials elicited by consecutive fixations

EFRP according to current fixation duration

Eye movement parameters have been shown to influence EFRPs[Kaunitz et al., 2014]

*

Object processing is affected by the gist of the scene within which it is embedded and the prior knowledge about the objects.

*

Hélène Devillez, Randall C. O’Reilly, Tim CurranDepartment of Psychology and Neuroscience, University of Colorado Boulder

Next stepUse pictures of staged scenes

rather than digitally manipulated pictures.

Occipital cluster

54 → 85 ms

Participants: 30 participants, 10 F – 20 M, Age: 19.48 ± 1.94Recordings: Eyelink 1000 & EGI 128 electrodes

Control objectsOne of the three most salient objects (based on experimental saliency

maps)

Control I from incongruent scenes Control II from incongruent scenes

Fixation durations/saccade amplitudes gathered to form

one global distribution

Object fixated Rank of the FOI

EYE MOVEMENT RESULTS

BEHAVIORAL RESULTS

ANALYSIS

Reaction timeAccuracy

Count

*

*

Total fixation

Duration

*

*

*

*

*

*

First-pass gaze

Fixations of Interest (FOI)

Fixation Of Interest First fixation in the

ROI

First-pass gazeAll fixations between

first entry and first exit in the ROI

Total fixation All fixations within the ROI from scene

onset to scene offset

Frontal

EEG RESULTSEFRP on the FOI

Occipital cluster

267 → 341 ms

Frontal cluster

261 → 310 ms

* p < .05

* p < .05

p < .05

X 50

CongruentIncongruent

Scenes

Objects

[Võ and Henderson 2009]

Correcttrials