Visual Adaptation And Spatial Visual Adaptation And Spatial Auditory ProcessingAuditory Processing

Peter Lokša, Norbert Kopčo

Collaborators:Jenni Groh, Barb Shinn-Cunningham, I-Fan Lin

2

Introduction

Visual stimuli can affect the perception of sound location

e.g. the Ventriloquism Effect

Way to go

Red Sox! Way to go

Red Sox!

But does effect persist?But does effect persist?- barn owls: prism adaptation (Knudsen et al.)- monkeys: “ventriloquism aftereffect” (Woods and

Recanzone, Curr. Biol. 2004)

e.g. the Ventriloquism Afterffect

3

What is reference frame

Eye-centered?

Head (ear) -centered?

Oculomotor?

?

?

What is the reference frame?

4

Methods

Basic idea:

1. Pre-adaptation baseline: Measure auditory saccade accuracy

2. Adaptation phase: Present combined visual-auditory stimuli, with visual location shifted

3. Compare auditory saccade accuracy pre- and post-adaptation

5

Method: center

Audiovisual display Expected behavior

Stimulus Location (°)

Mag

nitu

de (

°)

Fix head to face 0°Induce shift: - in only one region of space- from a single fixation point

Test to see if shift generalizes to the same sub-region in:- head-centered space- eye-centered space

Experiment divided into 1-hour blocks:(12 for humans, 16 for monkeys)

Within a block, 3 types ofrandomly interleaved trials:- Training AV stim: 50%- Test A-only stimuli: 50% (25% from trained, 25% shifted FP)

FPLEDs

Speakers

6

Follow-up

Eye-centered?

Head (ear) -centered?

Oculomotor?

?

?

Are results independent of where ventriloquism is introduced?- Spatial auditory resolution is lower in the periphery- Maier et al. 2009 – binaural cue adaptation only in center

7

Method: periphery

Expected behavior

Stimulus Location (°)

Mag

nitu

de (

°)

Fix head to face 0°Induce shift: - in only one region of space- from a single fixation point

Test to see if shift generalizes to the same sub-region in:- head-centered space- eye-centered space

Experiment divided into 1-hour blocks:(12 for humans, 16 for monkeys)

Within a block, 3 types ofrandomly interleaved trials:- Training AV stim: 50%- Test A-only stimuli: 50% (25% from trained, 25% shifted FP)

Audiovisual display

FPLEDs

Speakers

8

Results: AV-shifted data

-30 -15 0 15 30

0

1

2

Afte

reffe

ct M

ag

nitu

de

[°]

HumansCenral Adaptation

A)

-30 -15 0 15 30

Peripheral Adaptation

B)

-24 -12 0 12 24

0

1

2 TrainingRegion

TrainingFixation

Non-Tr.Fixation

MonkeysCentral Adaptaion

C)

-24 -12 0 12 24

Biasfrom

Training FPNon-Tr. FPDifference

MonkeyF W Avg.

Peripheral Adaptation

D)

-30 -15 0 15 30

-1

0

1

2

sep6 20

Diff

. in

Ma

gn

itud

es

[°]

E)

-30 -15 0 15 30

sep6 20

F)

-24 -12 0 12 24

-1

0

1

2

Actual Target Location [°]

G)

-24 -12 0 12 24

H) Bias if Coord. Frame Eye-centered Head-centered

sep11111100

Results: AV-aligned Baseline

-30 -15 0 15 30-2

-1

0

1

2

3

4

5E) Exp 1: Central

Actual Target Azimuth (°)

Diff

eren

ce B

etw

een

Res

ps f

rom

TrF

P (

+8°

) an

d N

onT

rFP

(-8

°)T

rFP

Res

ps.

are

shift

ed (

re.

Non

TrF

P R

esps

.)T

o th

e Le

ft<

-- -

>>

To

The

Rig

ht

-30 -15 0 15 30-2

-1

0

1

2

3

4

5F) Exp2: Peripheral

Actual Target Azimuth (°)

Diff

eren

ce B

etw

een

Res

ps f

rom

TrF

P (

+8°

) an

d N

onT

rFP

(-8

°)T

rFP

Res

ps.

are

shift

ed (

re.

Non

TrF

P R

esps

.)T

o th

e Le

ft<

-- -

>>

To

The

Rig

ht-24-18-12 -6 0 6 12 18 24

-9

-6

-3

0

3

6

9

C) Exp. 1: Central

-24-18-12 -6 0 6 12 18 24

-9

-6

-3

0

3

6

9

D) Exp. 2: Peripheral

Wysk NTrFP

Wysk TrFPFelix NTrFP

Felix TrFP

-24 -18 -12 -6 0 6 12 18 24-2

-1

0

1

2

3

4

5G) Exp 1: Central

-24 -18 -12 -6 0 6 12 18 24-2

-1

0

1

2

3

4

5H) Exp 2: Peri

-30-22.5-15-7.5 0 7.5 1522.530

-6

-3

0

3

6

A) Exp 1: Central Adaptation

Bia

s in

re

spo

nse

s re

. ac

tual

ta

rge

t lo

cati

on

(°)

Lef

twar

d s

hif

t <

-- -

-> R

ighw

ard

shi

ft

Actual Target Location (°)-30-22.5-15-7.5 0 7.5 1522.530

-6

-3

0

3

6

B) Exp 2: Peri Adaptation

Bia

s in

re

spo

nse

s re

. ac

tual

ta

rge

t lo

cati

on

(°)

Lef

twar

d s

hif

t <

-- -

-> R

ighw

ard

shi

ft

Actual Target Location (°)

Responses from Non-Training FP (-8°)Responses from Training FP (+8°)

Humnas:Monkeys:

10

Summary

The main results are consistent across species and regions:

- it’s possible to locally introduce ventriloquism aftereffect

- the reference frame is a mixture between head- and eye-centered

But

Visual stimuli influence saccade responses to auditory stimuli even when the AV-stimuli are spatially aligned:

- Expanding the auditory spatial map outside the trained area

- In an head-centered way when AV training is in center

- In a way that is modulated by eye-centered representation when AV training is in periphery

Collaborators

Jenni Groh, Duke University

I-Fan Lin, NTT Tokio

Barb Shinn-Cunningham, BU