multisensory interplay

Multisensory Interplay

Jon Driver and Toemme Noesselt

Sensory Research

Sensory Research

Behavioral Consequences of Multimodality

• Joint estimates of single propertyo Spatial Ventriloquismo Auditory Drivingo McGurk Effect

• Can modalities affect each other without creating a single unified percept?o Touch at a location can help

perception of coloro Sound-induced Illusory Flasho Orientation discrimination

improves with multiple beeps

Convergence Zones

• Superior Colliculuso  inputs from somatosensory, auditory, and visual areaso  Super or Sub-additive responses found for combination

Additivity in SC

• most likely with weak unisensory inputso Ceiling effect?o Neural limitations?

• Late onset in development

• Depends on multisensory cortex

Convergence Zones

Testing for Convergence

• Anatomical studies:o  Direct connections between different sensory areas

• Single-cell studies:o Response to stimulation from different modalities

• Neuroimaging:o Large-scale responses based on BOLD signal

Influences on 'Sensory-Specific' Areas

• Growing body of research shows that sensory-specific areas might be an artifact of the studies done with them

• Examine studies that use:o fMRIo EEGo Invasive recording in animal models

fMRI

• Caveat:o fMRI has been shown to respond to attention and

imagery

• For example, speech may be imagined when viewing lip movements 

fMRI analysis

• Inspired by Stein (SC), some look for sub-, super-additive responses

• Maybe linearity is normal, though, so some use max or mean criteria

• Difficult because of spatial resolution

Convergence in V1?

Amedi, Jacobsen, Hendler, Malach, and Zohary, 2002

ERP results example

 • Tactile stimulation

• Visual cue

• ERP extracted

ERP studies

• visual N1 enhanced when tactile stimulation occurred at same location as a visual event

• visual P1 modified by task-irrelevant sound

• P1 modified by attend-visual relative to attend-tactile conditions

ERP studies

• ERPs show early multi-sensory effects (~30 ms)

• Poor localization

• Potential methodological confounds

Invasive Studies• Current-source densities

(CSD) reflect local PSPs

• Region of auditory association cortex

• Location and timing of stimulation consistent with auditory feed-forward, visual feed-back 

Invasive Studies

• Posture may affect responses to auditory signals in A1

• Tactile stimuli modulate initial response to auditory signals in A1

Multisensory Interplay

• Examples of converging zones of multi-sensory input

• Examples of interplay: one modality affects another

• What frameworks does this evidence suggest?

Possible Frameworks

• A) All Multisensory

• B) Bimodal Brain Areas

• C) Critical Feedback Circuitry

All Multisensory

• Unlikely to be completely undifferentiated

• Even primary sensory areas responsive to multiple modalities

All Multisensory

• Thalamus might be source of multisensory interplay

• Tactile stimulation can affect first neural response in A1, hypothesized from thalamus

• Found in gerbils, hard to study in humans

All Multisensory

• Direct coritco-cortical influences

• Anatomical evidence: single synapse from AC-VC and AC-SSC, AC-OC

• However, not as many as to conventional Multi-sensory areas

• Role still unclear

All Multisensory

• Still overwhelmingly "sensory-specific"

Bimodal Brain Areas

• Less extreme version of account A

• Similar to current account, with more multi-sensory regions

• Parallel multi- and single-modality processing could explain early EEG modulation

Bimodal Brain Areas

• Different areas in auditory areas may be connected to distinct visual areas

• Bimodal interplay would be affected by transduction time, explaining BOLD response time differences 

Feedback Circuitry

• Effects in primary areas might be feedback from convergence zones

• Evidence from effective connectivity in fMRI, tactile stimulation increases visual response

• Evidence from EEG source-localization: STS - VC

Feedback Circuitry

• Evidence from invasive recordings: late A1 stimulation from vision (speculation?)

• Feedback can be tested directly, but very little has been done

Remarks

• Perhaps the rival frameworks are all valid for certain situations

• Perhaps primary cortex responses in the blind and deaf can help tease out what the multisensory roles are

• New techniques will allow testing causal interplay