The role of auditory-visual integration in object recognition

Clara Suied1, Nicolas Bonneel2 and Isabelle Viaud-Delmon1

1CNRS – UPMC UMR 7593 Hôpital de la Salpêtrière, Paris, France2REVES / Inria Sophia-Antipolis, France

Acoustics’08

Research supported by the EU IST FP6 Open FET project CROSSMOD

Recognition of natural object

• Recognizing a natural object involves pooling information from various sensory modalities

• And to ignore information from competing objects

How do these multisensory information interact to form a unique object concept?

Object recognition

• To direct action to objects, spatial information needs to be encoded and this might interact with object perception– For recognition tasks with the spatial dimension not relevant to the task, conflicting results

(Gondan et al., 2005; Teder-Salerjarvi et al., 2005)

• Realistic object are of interest in the study of multisensory integration, since a given object can be identified through any of several single modalities– Little behavioural studies with realistic objects (e.g. Molholm et al., 2004 for an ERP study; Laurienti et al.,

2004 for linguistic-type stimuli)

•Identification task: go/no-go–When the target (telephone) is either heard or seen, press the button as fast as possible

–Withold response when distractor (frog) is presented alone

Main experiment: Object Recognition

Go conditions

A+40

RING RING

A+0V+

•Unimodal

RING RING

A+0V+ A+40V+

•Bimodal semantically congruent

CROAK CROAK RING RING

A+0V- A-0V+ A-40V+A+40V-

•Bimodal semantically incongruent

No-Go conditions

A-40

CROAK CROAK

A-0V-

•Unimodal

A-0V- A-40V-

•Bimodal semantically congruent

CROAK CROAK

Experimental questions

• Spatial alignment necessary for fast object recognition?

• Larger auditory-visual integration for realistic objects?

• Effect of distractors (semantic congruence) on performance?

Results

Bimodal Visual target Auditory target

Experimental questions

• Spatial alignment necessary for fast object recognition?

• Larger auditory-visual integration for realistic objects?

• Effect of distractors (semantic congruence) on performance?

Spatial alignment

Bimodal Visual target Auditory target

2 (spatial alignment) x 4 (conditions) repeated-measures ANOVA

•Main effect of the spatial alignment (F1,19=17.68; p<0.0005)

•Main effect of the condition (F3,57=65.36; ε= 0.8; p<0.0001)

•But NO INTERACTION

the spatial effect is a Stimulus-Response Compatibility (Simon and Craft, 1970; Simon et al., 1981; Lu and Proctor, 1995)

Spatial alignment

Spatial alignment does not facilitate object recognition

Experimental questions

• Spatial alignment necessary for fast object recognition?

• Larger auditory-visual integration for realistic objects?

• Effect of distractors (semantic congruence) on performance?

Auditory-visual integration

p < 0.0001

Auditory-visual integration

AV integration and not statistical facilitation

Race Model (Miller, 1982)

• Computation of the effect size of the AV integration observed in the A+0V+ condition

(Cohen’s d; Cohen, 1988)

• Comparison with the size of AV integration previously observed in the literature

Size of the AV integration

2)]()),(min([

),min(

0

0

vAAV

VAAV

RTRTRT

RTRTRTd

Large AV integration

Experimental questions

• Spatial alignment necessary for fast object recognition?

• Larger auditory-visual integration for realistic objects?

• Effect of distractors (semantic congruence) on performance?

p < 0.005

Role of a distractor on object recognition

Auditory distractor Visual distractor

• When the distractor is visual– No performance cost when processing an auditory target

• When the distractor is auditory– There is a performance cost when processing a visual target

It seems impossible to ignore an auditory distractor

Role of a distractor on object recognition

Conclusion

• Large bimodal integration effect– Size of the visual object, realism, 3D and large display, immersive

• No effect of spatial alignment on object recognition– Spatial alignment important for saccade generation or signal detection(Stein and Meredith, 1993; Hughes et al., 1994; Frens et al., 1995; Harrington and Peck, 1998)

– Object recognition is a function where spatial alignment is not essential

It could reflect the fact that this function probably involves brain regions containing neurons with broad spatial receptive fields

• A possible asymmetry in the attentional filtering of irrelevant auditory and visual information– Similar asymmetry for cueing effect in detection tasks (Schmitt et al., 2000)

– Alerting role of the auditory system?

AV integration and not statistical facilitation

Race Model (Miller, 1982)