1Percep

tion&

Atten

tion

Percep

tioniseffo

rtlessbutits

underly

ingmech

anism

sare

incred

ibly

sophisticated

.

•Biologyofthevisu

alsystem

•Rep

resentatio

nsin

prim

aryvisu

alcortex

andHeb

bian

learning

•Object

recognitio

n

•Atten

tion:Interactio

nsbetw

eensystem

sinvolved

inobject

recognitio

nan

dsp

atialprocessin

g

2Percep

tion&

Atten

tion

Somemotiv

atingquestio

ns:

1.Whydoes

prim

aryvisu

alcortex

encodeorien

tedbars

oflig

ht?

2.Whyisvisu

alsystem

split

into

what/

where

path

way

s?

3.Whydoes

parietal

dam

agecau

seatten

tionproblem

s(neg

lect)?

4.How

dowereco

gnize

objects

(across

locatio

ns,sizes,ro

tations

with

wild

lydifferen

tretin

alim

ages)?

3Overv

iewoftheVisu

alSystem

Hierarch

iesofsp

ecializedvisu

alpath

way

s,startingin

retina,to

LGN

(thalam

us),to

V1&

up:

opticchiasm

temporal

temporal

nasal

LGN

V1

V2,V

4...

V2,V

4...

right fieldleft field

4TwoStream

s:Ven

tral“w

hat”

vs.Dorsal

“where”

V1

V2V

3V

4T

EO

TF

TE

PO

V3A

MT

FS

T

MS

T

VIP

PGP

G

TE

V1

p

mm

d

5TheRetin

a

Retin

aisnotapassiv

e“cam

era”

Key

prin

ciple:

contrast

enhan

cemen

tthat

emphasizes

chan

ges

over

space

&tim

e.

−+−

−−

−+

+

+

+

a) On−

centerb) O

ff−center

+

−−

+−

+

− +

−

+

retinal

outputgan

glio

ncells

6LGN

oftheThalam

us

A“relay

station”,b

utso

much

more.

•Organ

izesdifferen

ttypes

ofinform

ationinto

differen

tlay

ers.

•Perfo

rmsdynam

icprocessin

g:mag

nocellu

larmotio

n

processin

gcells,atten

tional

processin

g.

•On-an

doff-cen

terinform

ationfro

mretin

aispreserv

edin

LGN

7Prim

aryVisu

alCortex

(V1):

EdgeDetecto

rs

V1combines

LGN

(thalam

us)

inputsinto

orien

teded

gedetecto

rs:

−+−

−−

−+

+

+

+

a) On−

centerb) O

ff−center

+

−−

+−

+

− +

−

+

−−

−−

+−−

−−

+−−

−−

+−−

−−

+

+−

•Edges

differ

inorien

tation,size

(spatial

frequen

cy),an

d

positio

n.

•Forcoheren

tvisio

n,need

todetect

vary

ingdeg

reesofall

these.

8Prim

aryVisu

alCortex

(V1):

EdgeDetecto

rs

V1combines

LGN

(thalam

us)

minputsinto

orien

teded

ge

detecto

rs :

−+−

−−

−+

+

+

+

a) On−

centerb) O

ff−center

+

−−

+−

+

− +

−

+

−−

−−

+−−

−−

+−−

−−

+−−

−−

+

+−

V1

edgedetector

•Edges

differ

inorien

tation,size

(spatial

frequen

cy),an

dpositio

n.

•Forcoheren

tvisio

n,need

todetect

vary

ingdeg

reesofall

these.

9Prim

aryVisu

alCortex

(V1):

Topograp

hy

LR

LR

4 2−3 blobs

orientations

occularity

hypercolumns

Pinwheel

Hyperco

lumn:Fullset

ofcodingforeach

positio

nPinwheel

can

arisefro

mlearn

ingan

dlateral

connectiv

ity:nothard

-wired

!

10Rero

utin

gofVisu

alInfo

toAudito

ryCortex

•Sharm

a,Angelu

cci&

Sur(2000),N

ature

Rero

uted

fibers

from

Retin

a→

audito

rythalam

us(M

GN)→

A1

•Ifvisu

alproperties

arelearn

ed,th

eysh

ould

dev

elopin

A1.

11Rero

utin

gofVisu

alOrien

tationModules

inA1

Ba-d

:Orien

tationmap

s,dark

-highact

forgiven

orien

tation

(botto

m

right).

C:co

mposite

map

oforien

tationpreferen

ces

D:red

dots=pinwheel

centers

12Visu

alBeh

aviorAfter

Rero

utin

gRightVisu

alField

vonMelch

ner,

Pallas

&Sur(2000)

13Visu

alAcu

ityAfter

Rero

utin

g

→Solearn

ingis

powerfu

l,butso

isev

olutio

n!

14A

Questio

n

What

mak

esvisu

alcortex

visu

alcortex

?Whydoes

itrep

resent

orien

tedbars

oflig

ht?

15Prim

aryVisu

alRep

resentatio

ns

Key

idea:

Orien

teded

gedetecto

rscan

dev

elopfro

mHeb

bian

correlatio

nal

learningbased

onnatu

ralvisu

alscen

es.

16TheModel:

Sim

ulatin

goneHyperco

lumn

•Natu

ralvisu

alscen

esare

prep

rocessed

bypassin

gthem

(separately

)throughlay

ersofon-cen

teran

doff-cen

terinputs

•Hidden

layer:

edgedetecto

rsseen

inlay

ers2/

3ofV1;

Lay

er4(in

put)just

represen

tsunorien

tedon/offinputs

likeLGN

(butcan

bemodulated

by

attentio

n)

17TheModel:

Sim

ulatin

goneHyperco

lumn

•Heb

bian

learningonly

•KWTA

inhib

competitio

nforsp

ecialization(see

Ch4)

18[v1rf.p

roj]

19TheRecep

tiveField

s0

12

34

56

78

910

1112

13

Red

=on-cen

ter>

off-cen

ter,Blue=off-cen

ter>

on-cen

ter

20

2122

23Somedifferen

ces,butpinwheels

stillem

erge

24Percep

tionan

dAtten

tion

1.Whydoes

prim

aryvisu

alcortex

encodeorien

tedbars

oflig

ht?

Correlatio

nal

learningbased

onnatu

ralvisu

alscen

es.

Refl

ectsreliab

lepresen

ceofed

ges

innatu

ralim

ages,w

hich

vary

in

size,positio

n,orien

tationan

dpolarity.

→model

showshow

docu

men

tedV1properties

canresu

ltfro

m

interactio

nsbetw

eenlearn

ing,arch

itecture

(connectiv

ity),an

d

structu

reofen

viro

nmen

t.

25Percep

tionan

dAtten

tion

1.Whydoes

prim

aryvisu

alcortex

encodeorien

tedbars

oflig

ht?

Correlatio

nal

learningbased

onnatu

ralvisu

alscen

es.

2.How

dowereco

gnize

objects

(across

locatio

ns,sizes,ro

tations

with

wild

lydifferen

tretin

alim

ages)?

3.Whyisvisu

alsystem

split

into

what/

where

path

way

s?

4.Whydoes

parietal

dam

agecau

seatten

tionproblem

s(neg

lect)?

26TheObject

Reco

gnitio

nProblem

Problem

:Reco

gnize

object

regard

lessof:locatio

n,size,ro

tation.

Sam

e

Diff

Thisishard

becau

sedifferen

tpattern

sin

samelocatio

ncan

overlap

alot,w

hile

thesam

epattern

sin

differen

tlocatio

ns/

sizes/rotatio

ns

cannotoverlap

atall!

2728 G

radual

Invarian

ceTran

sform

ations(Fukush

ima,’80)

Increasin

grecep

tivefield

sizeen

ables:

Conjunctio

noffeatu

res(to

form

more

complex

objects);

and

Collap

singover

locatio

ninform

ation(“sp

atialinvarian

ce”)

29 Grad

ual

Invarian

ceTran

sform

ations(Fukush

ima,’80)

ifdid

spatial

invarian

cein

onefell

swoop:bindingproblem

-can

’ttell

Tfro

mL

30 Grad

ual

Invarian

ceTran

sform

ations(Fukush

ima,’80)

Goal:

Units

atthetopofthehierarch

ysh

ould

represen

tcomplex

object

features

inalocatio

nan

dsize

invarian

tfash

ion

(alsowan

tben

efits

oftop-downam

plifi

cation,pattern

completio

n,d

istributed

repsetc)

31TheModel

LGN

_On

LGN

_Off

V1 V2

V4/IT

Output

V1=orien

tedlin

e(ed

ge)

detecto

rs,hard

-coded

V2units

encodeconjunctio

nsofV1ed

ges

across

asu

bset

ofsp

ace

Each

V4unitpay

satten

tionto

allofV2

32TheObjects

01

23

4

56

78

9

1011

1213

14

1516

1718

19

Each

object

ispresen

tedat

multip

lelocatio

ns,sizes

Netw

ork’sjobisto

activate

theap

propriate

Outputunit(0-19)

for

eachobject,reg

ardless

oflocatio

nan

dsize

33[objrec.p

roj]

34

3536

3738

3940

4142

4344

Gen

eralization

•Can

thenetw

ork

gen

eralizeto

unseen

view

sofstu

died

objects?

•In

other

words:Does

trainingthenet

toreco

gnize

aset

of

objects

inasize/

locatio

ninvarian

tfash

ionhelp

itreco

gnize

new

objects

inasize/

locatio

ninvarian

tfash

ion?

•Proced

ure:

–Tak

eanet

trained

on18

objects

–Train

with

2new

objects

inonly

somelocatio

ns/

sizes

–Test

thenet

with

nonstu

died

“view

s”(sizes/

locatio

ns)

of

new

objects

4546

Gen

eralization

•Can

thenetw

ork

gen

eralizeto

unseen

view

sofstu

died

objects?

yes

•Approx.75%

correct

onnovel

view

sfollo

wingtrain

ingon10%

ofpossib

lesizes/

locatio

nsExplan

ation:Distrib

uted

represen

tationsan

dHeb

blearn

ing!

•V4rep

resents

objectfeatures

inalocatio

n/size

invarian

tway

•Each

object

activates

adistrib

uted

pattern

ofthese

invarian

t

feature

detecto

rs

4748

4950

5152

5354

01

23

4

56

78

9

1011

1213

14

1516

1718

19

Yeah

,butthese

objects

arereg

ularly

shap

ed,straig

htlin

es...

what

aboutreal

objects?

55

56“E

mer”

therobotreco

gnizin

gobjects..

Video

Dem

o

57

O’Reilly,

CogSci2009

58why?

bidirconssu

pportattracto

rsacro

ssmultip

lelev

elsofnet

toam

plify

consisten

t

info

59A

Challen

ge

60State

oftheArt

61Still

missin

g...

Motio

n

•Neu

ronsin

areaMTvery

sensitiv

eto

motio

n

•Lotsofwork

onhow

downstream

areasinteg

ratemotio

nsig

nals

across

timeto

detect

coheren

ce(e.g

.Shad

len,

New

some,etc)

•Thomas

Serre

has

shownthat

motio

nsig

nals

very

reliable

for

discrim

inatin

gbetw

eenparticu

laractio

ns(eg

throwinga

baseb

all)

•Should

beab

leto

solveproblem

via

bidirectio

nal

influen

ceof

motio

ninteg

rationsig

nals,o

bject

recognitio

n,an

dsp

atialatten

tion(next)....

62Percep

tionan

dAtten

tion

1.Whydoes

prim

aryvisu

alcortex

encodeorien

tedbars

oflig

ht?

Correlatio

nal

learningbased

onnatu

ralvisu

alscen

es.

2.How

dowereco

gnize

objects

(across

locatio

ns,sizes,ro

tations

with

wild

lydifferen

tretin

alim

ages)?

Tran

sform

ations:

increasin

gly

complex

featural

encodings,in

creasinglev

elsof

spatial

invarian

ce;Distrib

uted

represen

tations.

3.Whyisvisu

alsystem

split

into

what/

where

path

way

s?

4.Whydoes

parietal

dam

agecau

seatten

tionproblem

s(neg

lect)?

63Spatial

Atten

tion:Unilateral

Neg

lect

Patien

tcopyinga

scene

Self

portrait,

copying,

linebisectio

ntask

s:In

allcases,p

atientswith

parietal/

temporal

lesionsseem

toforget

about1/

2ofsp

ace!butthey

stillsee

it!

6465

6667

6869

7071

7273

74Effects

ofParietal

Lesio

nsonPosn

erTask

40−

60−

80−

100−

120−01

2

Intact

Lesioned

Neutral

Valid

Invalid

•Patien

tsperfo

rmnorm

allyin

the“n

eutral”

(nocu

e)conditio

n,

regard

lessofwhere

thetarg

etispresen

ted

•Patien

tsben

efitjustas

much

ascontro

lsfro

mvalid

cues

•Patien

tsare

hurtmore

than

contro

lsbyinvalid

cues

75Possib

leModels

+

Alert

Interrupt

Localize

Disengage

Move

Engage

Inhibit

Object

V1

(features x location)

Spatial

Atten

tionem

erges

from

bidirectio

nal

constrain

tsatisfactio

n&

inhibito

rycompetitio

n.

76Sim

ple

Model

Input

V1

Spat1

Spat2

Obj1

Obj2

targ

cue

Output

Object 1 (C

ue)

Object 2 (T

arget)

77[attn

simple.p

roj]

78Posn

erTask

Data

Valid

Invalid

Diff

AdultNorm

al350

39040

Elderly

Norm

al540

60060

Patien

ts640

760120

Elderly

norm

alized(*.65)

350390

40Patien

tsnorm

alized(*.55)

350418

68

79Posn

erTask

Sim

s

•Themodel

explain

sthebasic

findingthat

valid

cues

speed

target

processin

g,while

invalid

cues

hurt

•Also

explain

sfindingthat

patien

tswith

small

unilateral

parietal

lesionsben

efitnorm

allyfro

mvalid

cues

inipsilateral

field

butare

disp

roportio

nately

hurtbyinvalid

cues.

•Noneed

toposit

“disen

gag

e”module!

•Also

explain

sfindingofneglect

ofcontralateral

visu

alfield

afterlarg

e,unilateral

parietal

lesionswhen

somestim

ulusis

presen

tin

ipsilateral

field

(“extin

ction”)

80More

Posn

erLesio

nFun

•Retu

rningto

patien

twith

leftparietal

lesion...

•What

hap

pen

sifcu

esare

presen

tedin

contralateral(affected

)

hem

ifield

?(“R

everse

Posn

er”)

81More

Posn

erLesio

nFun

Retu

rningto

patien

twith

leftparietal

lesion...

•What

hap

pen

sifcu

esare

presen

tedin

contralateral(affected

)hem

ifield

?

Pred

ictions:

•Smaller

ben

efitforvalid

cues

•Patien

tssh

ould

behurtless

than

contro

lsbyinvalid

cues.

82[attn

simple.p

roj]

83Inhibitio

nofRetu

rn

•Typically,targ

etdetectio

nisfaster

ontrials

with

valid

vs

invalid

cues

•H

owever

,ifthecu

eispresen

tedforalonger

time(eg

.500

ms),

perfo

rman

ceisfaster

on

invalidvsvalid

trials

•Can

explain

interm

sofaccom

modation

(neu

ralfatig

ue)

84[attn

simple.p

roj]

85Sim

ple

model:

toosim

ple?

•Has

uniqueone-to

-onemap

pingsbetw

eenlow-lev

elvisu

alfeatu

resan

dobject

represen

tations(notrealistic)

•Does

notad

dress

issueofsp

atialatten

tionwhen

tryingto

perceiv

emultip

leobjects

simultan

eously

•“C

omplex

”model

combines

more

realisticmodel

ofobject

recognitio

n(startin

gfro

mLGN)with

simple

attentio

nmodel

→Can

use

spatial

attentio

nto

restrictobject

processin

gpath

way

tooneobject

atatim

e,enab

lingitto

sequen

tiallyprocess

multip

leobjects.

•Lesio

nsofen

tiresp

atialpath

way

cause

simultan

agnosia:

inab

ilityto

concu

rrently

recognize

twoobjects

86Complex

Model

LGN

_On

LGN

_Off

V1

V2

V4

/IT Output

Spat1

Spat2

Target

Spat1

has

recurren

tprojnsto

encourag

efocu

sononereg

ionofsp

ace

Butonly

mech

anism

forsw

itchingisacco

mmodatio

n...

87Percep

tionan

dAtten

tion

1.Whydoes

prim

aryvisu

alcortex

encodeorien

tedbars

oflig

ht?

Correlatio

nal

learningbased

onnatu

ralvisu

alscen

es.

2.How

dowereco

gnize

objects

(across

locatio

ns,sizes,ro

tations

with

wild

lydifferen

tretin

alim

ages)?

Tran

sform

ations:

increasin

gly

complex

featural

encodings,in

creasinglev

elsof

spatial

invarian

ce;Distrib

uted

represen

tations.

3.Whyisvisu

alsystem

split

into

what/

where

path

way

s?

Tran

sform

ations:em

phasizin

gan

dcollap

singacro

ssdifferen

ttypes

ofrelev

antdistin

ctions

4.Whydoes

parietal

dam

agecau

seatten

tionproblem

s(neg

lect)?

Atten

tionas

anem

ergen

tproperty

ofcompetitio

n

88Gen

eralIssu

esin

Atten

tion

Atten

tion:

•Prio

ritizesprocessin

g.

•Coordinates

processin

gacro

ssdifferen

tareas.

•Solves

bindingproblem

svia

coordinatio

n.

Butatten

tionsh

ould

bemuch

more

flexible

than

justsp

atialbias!

Later:

how

toincorporate

goals,rein

forcem

entprobab

ility,into

attentio

nal

allocatio

n