1

Current Topics in PerceptionPSYD51

Multisensory IntegrationSession 1

January 6, 2009

2

Current Topics in PerceptionPSYD51

Blurb: multisensory integrationRecent research topicMultimodal information in the worldMaking sense out of it requires multisensory integration

3

Current Topics in PerceptionPSYD51

Multisensory Integration (PSYD51), LEC 01 Place: HW 309 Time: Tuesdays, 1–3pm Intranet

The Course:

Dr. Matthias Niemeier Office: S-572 Phone: 416-287-7466email: niemeier@utsc.utoronto.caOffice hours: Tuesdays, 12–1pm, or per appointment

The Instructor:

4

Current Topics in PerceptionPSYD51

45% Thought papers (9 x 5%)20% Presentation15% In-class participation20% Research proposal

Grading

PrerequisitesPSYB51 – Perception and CognitionPSYC08 – Advanced Data Analysis in Psychology

No exams, active participation

ReadingsNo textbookResearch papers (see reading list)

5

Grading System PSYD51Paper Presentations

20% of the total mark1 paper from reading listFirst come first serve – please email mePdf’s on the intranetFree presentation (notes)Three parts:

a) Brief intro (~5 min)b) Summary of methods, results & conclusions. Figures on

the intranet (~25 min)c) Group discussion led by presenter (~15 min)

6

Grading System PSYD51 Thought Papers

45% of the total mark (9 papers x 5%)Read both research papers of the week, thought paper on one“How to write a thought paper” => intranetScoring systemDue for the respective session of the seminar

1-2 pages, double-spaced2 sentences summary maximumYour own thoughtsIntro (1 paragraph, “Tell them what you are going to tell them”) –body (2-4 paragraphs, “Tell them”) – conclusion (1 paragraph, “Tell them what you told them”)

7

Grading System PSYD51 In-class Participation

15% of the total mark – so please be there and be prepared to say something in class!!Help the presenterDo you agree with the authors? Are you impressed? Something that you didn’t understand? Can you help clarifying things? Is there a reason to believe that the paper is invalid? Can you play devil’s advocate? Suggestions for future research?

8

Grading System PSYD51 Research Proposal

20% of the total mark “How to write a research proposal” => intranetScoring systemDue for the last day of the seminar8 pages, double-spacedTitle page – Abstract – Introduction – Methods –Predicted Results – References

9

Comparing Senses

aroma (flavour)dissolved chemicals

odour (scent)airborne

chemicals

deformationphysical objects

soundpressure

wave

lightelectromagnetic

wave

physical quality

of stimuli

tonguenoseskineareyeorgantastesmelltouchhearingvisionsense

Other channels of sensory input? Temperature, pain? Balance? other senses? Perhaps a better physical classification: electromagnetic, mechanical, chemical stimuli

10

Combining and integrating sensory information

Combining vs. integrating

Example: knock on woodSensory combination: interactions between non-redundant sensory signalsSensory integration: interaction between redundant ones

V: vision, N: neck muscle info, A: audition, P: proprioception from arm, L:location signal

11

Combining and integrating sensory information

Why combining/integrating sensory sources?Combined alarm systems (jungle, darkness, braking)Object identification: information from different modalities support each otherVisual-auditory localisation (twilight, peripheral visual field, occlusion, cluttered scenes)Reliable info for control of locomotion(e.g. visual-proprioceptive cross-adaptation: Pelah and Barlow 1996)Coherence of the world: object unity, perceiving one’s own presence in space (e.g. cue conflict > motion sickness)

Large interest from engineering: sensory fusion in robots, VR, video gamesHow do nervous systems combine sensory information ?

12

Multisensory neuronesSingle cell recordings in cat superior colliculusInput from various cortical regions (visual, auditory, somatosensory, motor...)

Alignment of receptive fields >> integrated multisensory mapsSpatial coincidence detectors

Multisensory receptive fields

13

Multisensory areasin the cortex

How/where are the different senses integrated?fMRI, a tool to study some cases of multisensory integration: most prominent, e.g., integration of visual & auditory info (Calvert et al. 1998)

red: viewing mouth movements without sound blue: listening to speech yellow: activated by both signals

Regions for crossmodal identification

Interaction of light and sound

An experiment

14

to demonstrate visuo-auditory integration: ‘streaming and bouncing’ (Sekuler et al. 1997)

Two objects moving straight across each other are perceived as moving on intersecting linear trajectories

A sound at the intersection point changes perception dramatically: the objects appear to collide and bounce

15

Reading emotions

Observers asked to rate the sadness of a series of faces with different expressions between happy and sad

Sadness ratings follow a psychometric function: from low to high for happy to sad facesCombining the face display with a sad or happy voice shifts the psychometric curve to higher or lower sadness ratings

16

Preliminary Schedulefor the Course

6 Jan Introduction13 Jan Neural mechanisms of multisensory integration20 Jan Neural mechanisms of multisensory integration II:

Supramodal areas (?)27 Jan Multisensory object perception3 Feb Optimal multisensory integration10 Feb Optimal sensorimotor combination24 Feb Vision and sound in space3 Mar Vision and touch in space10 Mar Body schema17 Mar Learning & plasticity24 Mar Neuropsychological deficits31 Mar Synaesthesia

17

Outlook on the Course Topics

Jan 13: Neural mechanisms of multisensory integrationLateral connections: Anatomy: cortical auditory input into V1 (Falchier et al. 2002).fMRI: Polymodal motion processing in posterior parietal and premotor cortex (Bremmer et al. 2001).

Jan 20: Crossmodal cueing of attentionSpence & Driver (1997): 1-way cross-modal dependence in covert orienting; audition influences visionWard et al. (2000): auditory cues fail to orient visual attention

18

Outlook on the Course Topics

Jan 27: Multisensory object perceptionViewpoint dependence in visual and haptic object recognition (Newell et al. 2001).Visuo-haptic object-related activation in the ventral visual pathway (Amedi et al. 2001).

19

Outlook on the Course Topics

Feb 3: Optimal multisensory integrationIntroduction to MLEHumans integrate visual and haptic information in a statistically optimal fashion (Ernst et al. 2002).

20

Outlook on the Course Topics

Optimal transsaccadic integration (Niemeier et al. 2003).Statistical decision theory and rapid, goal-directed movements (Trommershauseret al. 2003).

Feb 10: Optimal sensorimotor combination

21

Outlook on the Course Topics

Optimal ventriloquist effect (Alais et al. 2004).http://www.youtube.com/watch?v=hsfnP9DK-ZgRecalibration of audiovisual time perception (Fujisaki et al. 2004).

Feb 24: Vision & sound in space

Coordinate transformations for representing visual and tactile locations in parietal cortex (Avillac et al. 2005).Influences of gaze direction on crossmodal effects (Macaluso et al. 2002).

Mar 3: Vision & touch in space

22

Outlook on the Course Topics

Mar 17: Learning & plasticityExperience for the development of multisensory integration (Wallace et al. 2004).

Plasticity in sound localization induced by compressed spatial vision (Zwiers et al. 2003).

Out of body experience & rubber gloves (Pavani et al. 2000).fMRI and my hand (Ehrsson et al. 2004).

Mar 10: Body schema

23

Outlook on the Course Topics

Mar 24: Neuropsychological deficitsSplit-brain: Remapping of space (Spence et al. 2001).Impaired crossmodal interactions in a patient with Balintsyndrome (Valenza et al. 2004).

Mar 31: SynaesthesiaPriming and colour-number synesthesia (Mattingley et al. 2001).Neurocognitive mechanisms of synesthesia (Hubbard et al. 2004).

Doorbell ringing Dog barking