35 visual stability during saccadic eye movements...nant stimuli (varying in color but not...

Y

35 VisualStabilityDuringSaccadic

EyeMovements

concettamorroneanddavidburr

morroneandburr:visualstabilityduringsaccadiceyemovements 511

abstract We frequently reposition our gaze by making rapidballistic eye movements called saccades to position the fovea onobjectsofinterest.Whilethestrategyishighlyefficientforthevisualsystem,allowingittoanalyzethewholevisualfieldwiththehighresolution of the fovea, it poses several problems for perception.Saccadescauserapid,large-fieldmotionontheretina,potentiallyconfusablewithlarge-fieldmotionintheexternalworld.Theyalsochangetherelationshipbetweenexternalspaceandretinaposition,confounding informationabout visualdirection.Muchefforthasbeenmadeinrecentyearstoattempttounderstandtheeffectsofsaccadesonvisualfunction.Electrophysiological,imaging,andpsy-chophysical evidence suggests that saccades trigger two distinctneural processes: a suppression of visual sensitivity, specific tomotionanalysis,probablymediatedbythemagnocellularpathway,and a gross perceptual distortion of visual space just before therepositioning of gaze. While our knowledge of how the visualsystem copes with the potentially damaging effects of continualsaccadiceyemovementshasincreasedconsiderablyoverthepastfewdecades,manyinterestingavenuesofresearchremainopen.

Visionisalwaysclearandstable,despitecontinualsaccadiceyemovements,thatis,ballisticmovementsoftheeyesthatrepositionourgaze two to three timesa second.Saccadesmaybemadedeliberately,butnormallytheyareautomaticandpassunnoticed.Anobserveratasportingevent,someoneconversingwithacompanion,orapersonreadingabookusually makes many saccades without knowing that theyhaveoccurred.Notonlydoes theactualmovementof theeyesescapenotice;sotoodoesthemotionofimagesastheysweepacrosstheretinaandthefactthatgazeitselfhasbeenrepositioned. The world seems to stay put. Comparableimage motion that is produced externally, rather than bymovementsoftheobserver’sowneyes,hasanalarmingeffecton the observer’s sense of stability. The problem of visualstability is an old one that has fascinated many scientists,includingDescartes,vonHelmholtz,Mach,andSherrington,and indeed goes back at least to the 11th-century Persianscholar Alhazen: “For if the eye moves in front of visible

objectswhiletheyarebeingcontemplated,theformofeveryoneof theobjects facingtheeye...willmoveontheeyesasthelattermoves.Butsighthasbecomeaccustomedtothemotionoftheobjects’formsonitssurfacewhentheobjectsare stationary,and thereforedoesnot judge theobjects tobe inmotion” (Alhazen,1083).Butonlyrecentlyhave thetoolsbecomeavailabletomonitoreyemovementsaccuratelyandtomeasuretheireffectsqualitatively.

Theproblemofvisualstabilitycanbebroadlydividedintothreeseparateissues:Whywedonotperceivethemotionoftheretinalimageproducedastheeyesweepsoverthevisualfield?Howdowecopedynamically“on-line”withthecon-tinual changes in the retinal image produced by eachsaccade?How(andwhere)doweconstructastablespatiotopicrepresentation of the world centred in real-world externalcoordinates from the successive “snapshots” of each fixa-tion? Although the problem of visual stability is far fromsolved,tantalizingprogresshasbeenmadeoverthelastfewyears,someofwhichwillbehighlightedinthischapter.

Saccadic suppression

Partofthegeneralproblemofvisualstabilityiswhythefastmotion of the retinal image generated the movement oftheeyescompletelyescapesnotice.Comparablewide-fieldmotiongeneratedexternallyishighlyvisibleandsomewhatdisturbing(Burr,Holt,Johnstone,&Ross,1982).Ithaslongbeen suspected that vision is somehow suppressed duringsaccades (Holt, 1903), but the nature of the suppressionhasremainedelusive.Nowitisclearthatthesuppressionisneither a “central anaesthesia” of the visual system (Holt,1903), nor a “gray-out of the world” due to fast motion(Campbell&Wurtz,1978;Dodge,1900;Woodworth,1906),asthismotionisactuallyvisible—extremelysoatlowspatialfrequencies (Burr & Ross, 1982). What happens is thatsomestimuliareactivelysuppressedbysaccadeswhileothersarenot.Stimulioflowspatialfrequenciesareverydifficulttodetect ifflashed justprior toa saccade,while stimuliofhigh spatial frequencies remainequallyvisible (Burretal.,1982;Volkmann,Riggs,White,&Moore,1978).Equilumi-nant stimuli (varying in color but not luminance) are notsuppressedduringsaccadesandcanevenbeenhanced(Burr,

concettamorrone FacoltàdiPsicologia,UniversitàVita-Salute“SanRaffaele,”Milan,Italy.davidburr DipartimentodiPsicologia,UniversitàDegliStudidiFirenze,Florence,Italy,DepartmentofPsychology,UniversityofWesternAustralia,Perth,Australia.

Gazzaniga_35_Ch35.indd 511 3/5/2009 9:00:27 PM

Y

512 sensationandperception

Morrone, & Ross, 1994), implying that the parvocellularpathway,essentialforchromaticdiscrimination,isleftunim-paired, while the magnocellular pathway is specificallysuppressed.

Saccadicsuppressionfollowsaspecificandverytighttimecourse,illustratedinfigure35.1A(replottedfromDiamond,Ross,&Morrone,2000).Sensitivity for seeing low-spatial-frequency, luminance-modulated stimuli declines 25msbeforesaccadiconset,reachingaminimumattheonsetofthesaccade,thenrapidlyrecoveringtonormallevels50msafterward.Doesthesuppressionresult fromacentralnon-visual “corollary discharge” signal (discussed in the nextsection),orcoulditresultsimplyfromthevisual“masking”effects?Thiswouldseemunlikely,asgreatcarewastakentoensureauniformsurround.However,thequestionisimpor-tant,sotobecertainthatthesaccadeitselfwasessentialforthe suppression, we simulated saccadic eye movements byviewing the stimulus setup throughamirror thatcouldberotated at saccadic speeds. When the background wasuniform, with minimal visual referents, the simulated sac-cadeshadlittleornoeffectonsensitivity (opensymbolsoffigure35.1A).

Butthatisnottosaythatmaskingdoesnotoccurundermorenaturalconditions.Whentheteststimulusisembed-dedwithinatexturedscreen,simulatedsaccadesdodecreasecontrastsensitivity(figure35.1B).Indeed,themaximumsup-pression is nearly as great as that caused by real saccadesandlastsmuchlonger.Thissuggeststhatafterthesaccade,

-300 -200 -100 0 100 200 300

1

10

30

-300 -200 -100 0 100 200 300

1

10

30

-300 -200 -100 0 100 200 3000.1

1

10

Con

tras

t sen

sitiv

ity

B

Nor

mal

ised

Firi

ng

Time (ms)

A

C

Figure35.1 Theeffectofsaccadesonhumancontrastsensitivityand firing rate in monkey MT. (A) Solid squares show contrastsensitivityfordiscriminating(intwo-alternativeforcedchoice)thebrightnessofabrief,low-frequency,luminance-modulatedgratingpatchasa functionof time relative to saccadiconset.Theback-ground was of mean luminance, with very few visual referentspresent.Sensitivityisseverelyreduced(bymorethanalog-unit)atsaccadiconset.Theopencirclesshowmeasurementsmadeiniden-ticalconditions,butinsteadofmakingasaccade,amirrormovedatthesamespeedandamplitudeasthesaccade;thishadverylittleeffectonsensitivity.(B )Asforpanel.Aexceptthatthebackgroundwasahigh-contrastrandomcheckpattern.Withastructuredback-ground,thesimulatedsaccadedidreducevisibility,presumablybymasking,withtheeffectlastinglongerthanitdidforarealsaccade.The gray-shaded area indicates the region where sensitivity wasgreater during the saccade than in fixation. (C ) Firing rate of atypicalMTneuroninanawakemonkeyinresponsetostimulationtoabriefstimulus,asafunctionoftimerelativetosaccadiconset.ThepatternoftheresponseissimilartothepsychophysicalresultsofpanelA;thetimingdoesnotmatchexactly,butthisisonlyonecell,notanaverage,anddoesnottakeneurallatenciesintoaccount.Theenhancementafterthesaccademayallowforthemorerapidrecoveryfrommaskingduringrealratherthansimulatedsaccades(the difference between the solid and open symbols in panel B).(ReproducedwithpermissionfromDiamondetal.,2000;Ibbotsonetal.,2008.)

sensitivity is greater than that expected with comparablemotion without the saccade, possibly implying a post-saccadicfacilitation(consistentwiththephysiology).

Thatrealsaccadescauseadifferentpatternofresultsfromsimulatedsaccadesshowsthatsuppressionresultsatleastinpart from an active, extraretinal signals. Interestingly, theamountofsuppressionvarieswithage,beingmuchstrongerin adolescent children than in adults (Bruno, Brambati,Perani, & Morrone, 2006), even though in adolescence,motionperceptionandmaskingarelargelyadultlike(Maurer,Lewis, & Mondloch, 2005; Parrish, Giaschi, Boden, &Dougherty,2005).Thisindicatesthatthemechanismsthatmediatesuppressionarestilldevelopingatthisage.Becausethe saccadic motor system is also not completely matureduringadolescence(Fischer,Biscaldi,&Gezeck,1997),thisisfurtherevidencethattheextraretinalsignalthatisrespon-sibleformediatingthesaccadicsuppressionmaybe linkedtothemotorsystem.

Psychophysicalstudiesindicatethatsaccadicsuppressionoccurs early in the visual system (Burr et al., 1994), at orbefore the site of contrast masking and before low-level


Y


motionprocessing(Burr,Morgan,&Morrone,1999).Thilo,Santoro,Walsh,andBlakemore(2003)addressedthisques-tion more directly with a clever electrophysiological tech-nique.ReplicatinganoldstudybyRiggs,Merton,&Morton(1974), they showed that visual phosphenes producedby electrical stimulation of the eye are suppressed duringsaccades. But phosphenes of cortical origin—V1 or V2—generatedwiththetechniqueoftranscranialmagneticstim-ulation were not suppressed. This strongly suggests thatsaccadicsuppressionoccursearly,beforethesiteofgenera-tion of cortical phosphenes, probably within the lateralgeniculate nucleus (LGN) or perhaps within V1 itself. Arecent functional resonance magnetic imaging (fMRI)(Sylvester, Haynes, & Rees, 2005) study that measuredblood oxygen level–dependent (BOLD) activity of LGNwhilesubjectsmadesaccadesoverafieldofconstantillumi-nation (toavoid thegenerationof spurious retinalmotion)showedaclearsuppressioninbothLGNandV1,reinforc-ingearly suggestionof saccadic suppression in thedark inV1 (Bodis-Wollner,Bucher,&Seelos,1999).There isalsofMRI evidence for postthalamic modulation by saccades.BOLDactivitytoluminancestimuliisrelativelysuppressedcompared with that of chromatic stimuli during saccades,but theattenuationvariesbetweenareas (Kleiser,Seitz,&Krekelberg,2004),beingstronginMT—asexpected—butalsostronginV4,acorticalareathatreceivesmoreparvo-cellularthanmagnocellularinput.Thereisalsoevidenceofsuppression in higher neural levels, in areas that are nor-mallyassociatedwithattention(Bristow,Haynes,Sylvester,Frith,&Rees,2005,Kleiseretal.,2004).Thisisinteresting,asitcouldbethesuppressionofthehigh-order“attention-related”areasthatpreventsthesenseofmotionenteringintoawareness,causingstartle.

The electrophysiology of saccadic suppression is morecomplex.Electrophysiologicalstudiesshowthatthemajorityofcells inV1respondvigorouslytothemovementcreatedbysaccades;however,somecellsdonotrespondtosaccade-createdmotionbutonlytorealmotionintheexternalworld.Thesecellsaretheminority,about10%inV1,15%inV2,and 40% in V3A (Galletti & Fattori, 2003; Wurtz, 2008).Recently,Reppas,Usrey,andReid(2002)haveshownthatvoluntarysaccadesinduceprofoundchangesintheresponseofLGNcells,particularlymagno-cells.Activityisdepressedaround the time of the saccade, and there is also a largerand long-lasting enhancement after the saccade. There isalsoclearevidenceforastrongsuppressioninthecolliculusandpulvinarthatcouldbeimportantforthesuppressionoffastmotion(Wurtz,2008).

Perhapsthedatathatcanbemostreadilycomparedwiththe psychophysical sensitivities are those of Ibbotson,Crowder,Cloherty,Price, andMustari (2008),whomeas-ured responsivenessofMT/MSTcells toabrief stimulus,likethatusedinthepsychophysicsexperiments.Anexample

cell is replotted in figure 35.1C. This cell showed a verystrongandrobustsuppressionbeforethestartofthesaccade,followedbyaclearenhancementlastingsome200msaftertheterminationofthesaccade.Whileitisdifficulttomakea quantitative comparison between psychophysical thresh-oldmeasurements (figure35.1A)and thefiringrateofonerepresentativeMTcell (figure35.1C ), it is interesting thatmodulation of MT/MST response follows a time coursesimilartothatofsensitivityforabrieflow-spatialfrequencystimulus,presumablydetectedbymagnocellular/MT-MSTpathways. The very strong postsaccadic enhancement ofMTcellscouldexplaintherelativelyhighersensitivityafterreal saccades compared with after simulated saccades (thedifferencebetweenopenandsolidsymbolsoffigure35.1B).Another interestingresultreportedforMTneurons is thatin addition to being suppressed, many neurons seem toreversetheirpreferreddirectionselectivity(Thiele,Henning,Kubischik,&Hoffmann,2002).Thisoddbehaviorcouldbeimportant in “canceling” motion information, helping tokeeptheworldstill.

Toconclude,itisnotsurprisingthatsaccadicsuppressionshouldoccuratdifferentlevels.Manybasicsensoryphenom-ena, suchas gain control, occurnot at a single sitebut atvirtuallyeverypossiblelocation:photoreceptors,retinalgan-glioncells,LGNcells,andcortex(Shapley&Enroth-Cugell,1984). Indeed, the parallels between saccadic suppressionand contrast gain control are strong, suggesting that theymightsharesimilarmechanisms.Duringsaccades,thetem-poral impulseresponse to luminance,butnot toequilumi-nant, stimuli becomes faster and more transient (Burr &Morrone,1996).LGN(Reppasetal.,2002)andMT/MST(Ibbotsonetal.,2008)cellsshowasimilarresponsepattern,with faster and more transient impulse response functionsduringsaccades.Theseresultssuggestthatsaccadicsuppres-sionmightactbyattenuatingthecontrastgainoftheneu-ronal response, causinga faster impulse response (Shapley&Victor,1981).Changingcontrastgainmakesneuronslessresponsive to low-contrast stimuli,decreasingtheeffective-ness of the spurious noise caused by the saccade, hencefacilitating their recovery to normal sensitivity. That sac-cadicsuppressionoperatesviagaincontrolmechanismsthatareinconsistentwiththeselectivesuppressionofthemagno-cellularpathway,asMcellshavemuchstrongergaincontrolmechanismsthanPcells(Sclar,Maunsell,&Lennie,1990).Thiswouldcertainlybeanelegantandeconomicalsolutiontotheproblemofsaccadicsuppression,takingadvantageofmechanisms that are already in place for other functions.Theideathatgaincontrolexplainsboththesuppressionandrapidrecoveryduringsaccadeshasbeenimplementedinamodelthatsimulatesquantitativelythetimecourseofcon-trastsensitivityinnormalandsimulatedsaccade(Diamondetal.,2000).Inthisview,saccadicsuppressionsubservestwoimportant roles: the suppression of image motion, which

1


Y


would otherwise be disturbing, and the rapid return tonormalsensitivityafterthesaccade.

Dynamic updating of internal spatial maps

Besides the (relatively) simple problem of suppressing themotioncausedbythefast-movingimageontheretina,thebrain must also take into account the saccadic movementwhen determining the instantaneous position of objects inspace.LikeAlhazen,vonHelmholtz(1866)recognizedthat“theeffortofwillinvolvedintryingtoaltertheadjustmentof the eyes” could be used to help stabilize perception.Modelsbasedonsimilarideasofcompensationofeyemove-ments were proposed in the 1950s by Sperry (1950) withthe concept of corollary discharge and by Von Holst andMittelstädt (1954)ofwith theconceptefferencecopy:Theeffort of will of making the eye movements (corollary dis-charge)issubtractedfromtheretinalsignaltocanceltheeyemovement and stabilize perception. Now we know thatretinal motion signal cannot be easily compensated, giventhe sophisticated analysis performed by motion detectors.However, there isevidence for theexistenceofacorollarydischarge signal that must be instrumental in maintainingvisualstability.

Considerablepsychophysicalevidenceexists foracorol-lary discharge in humans, going back to the 1960s, whenLeonardMatinandothersreportedlargetransientchangesinspatial localizationat the timeof saccades.Whenaskedtoreportthepositionofatargetthatwasflashedduringasaccade, subjectsmislocalized it,primarily in thedirectionofthesaccade(Honda,1989;Mateeff,1978;Matin&Pearce,1965).Thelocalizationerroristypicallyontheorderofhalfthesaccadicsize.Later,MateeffandHondameasuredthetime course and showed that the error starts about 50msbeforethesaccadiconsetandcontinueswellafterfixationisregained.Theerrorbeforethesaccadiconsethasbeentakenasanindicationoftheexistenceofaslowandsluggishcorol-lary discharge signal that compensates partly for the eyemovement; the internal representationof theposition andtheactualpositionof thegazedonotmatchanderrors inthelocalizationofabrieftargetaregenerated.

Wehaveexamined saccadicmislocalization inphotopicconditions using equiluminant stimuli (that remain visibleduring saccades). This approach revealed a bizarre result:Atthetimeofsaccades,visualspaceisnotsomuchshiftedinthedirectionofthesaccadebutcompressedtowardthesac-cadictarget(Morrone,Ross,&Burr,1997;Ross,Morrone,&Burr,1997)(seefigure35.2A).Objectsthatareflashedatsaccadiconsettoarangeofpositions,fromclosetofixationtopositionswellbeyondthesaccadictarget,areallperceivedatornearthesaccadictarget.Theeffectisprimarilyparalleltothesaccadedirection(Rossetal.,1997),althoughasmallcompression is also observed in the orthogonal direction

-40

-20

0

20

40

-200 -100 0 100 200

-200 -100 0 100 200

1

2

3

4

Target

Fix

App

aren

t pos

ition

(de

gs)

A

Num

ber

of b

ars

Time (ms)

B

Figure 35.2 Effect of saccades on apparent bar position andnumber.(A)Perceivedpositionofnarrowgreenbars,brieflyflashedon a red background at various times relative to the onset of asaccadefrom−10to+10degrees.Thephysicalpositionofthebars(shownbythedashedlines)couldbe−20degrees(forthetrianglesymbols), 0 degrees (square symbols) and 20 degrees (roundsymbols).Theeffectofthesaccades(maximalatsaccadiconset)istoshifttheapparentpositionofthebartowardsthesaccadictarget,wheretheeyesland.Forstimuliat0or−20degreestheshiftisinthedirectionofthesaccade,butforstimuliat+20degreestheshiftisintheotherdirection.Inallcases,theshiftistowardsthesaccadiclandingpoint.(B )Reportednumberofbarsseen,asafunctionofpresentationtime(relativetosaccadiconset).Avariablenumberofbars (0, 1, 2, 3 or 4) were presented simultaneously in positionsstraddling 10 degrees either side of the saccadic target site. Theresults reported here are for trials in which four bars were pre-sented;butwhenpresentednearsaccadiconset,thefourcollapseontoeachother, soonlyonewas seen.Theotherbarswerenotsuppressed,becauseonebarwasalwaysreportedasone,andzerobarswere reportedaszero (no falsepositives). (ReproducedwithpermissionfromRossetal.,1997).

(Kaiser&Lappe,2004).Theseresultsareintriguingbecausetheyindicatethattheprocessdescribedmathematicallybyasimpletranslationoftheinternalcoordinatesystemisnotplausible;perhaps thesystemcannotperformthetransfor-mationofspacewithoutadditionalperceptualcosts.

Figure35.2Bshowsthatsaccadiccompressionissostrongthatfourbarsspreadover20degreesareperceivedasbeingfusedintoasinglebar.Discriminationofshape(Matsumiya


Y


&Uchikawa,2001)or colours (Lappe,Kuhlmann,Oerke,&Kaiser,2006)ofthebarsisstillpossible,butcountingthemand perceiving them in separate positions are not. Some-timestheshapeororientationoftheflashedobjectcanalsochange,appearingsmallerandmorevertical(forhorizontalsaccades),althoughtheseeffectshavebeenhardertoquan-tify.Thefactthatthefeatureitselfisnotlostorcompressedsuggests that themislocalizationoccursatarelativelyhighlevelofanalysisafterfeatureextraction.

It has been suggested that saccadic compression occursonlywhenvisualreferencesarepresentandisabsentinthedark (Lappe,Awater,&Krekelberg,2000).However, sub-sequentstudies(Awater&Lappe,2006)haveshownthatthisis not necessarily true. In the dark or in a transient darkcondition achieved by a brief blackout on saccadic onset,compression does occur but can be obscured because inthese conditions, there is also amislocalizationof the sac-cadictarget(Morrone,Ma-Wyatt,&Ross,2005).Whenthisistakenintoaccount,compressionoccursinbothlightanddark, with and without visual references. Several studieshaveshownthatvisualreferencesperse (suchasscatteredpointsonthemonitor)donotaffectcompression.However,presenting the same brief stimulus twice perisaccadically,eventodifferentretinallocations,greatlyreducesmislocali-zation(Cai,Pouget,Schlag-Rey,&Schlag,1997;Morroneet al., 1997; Pola, 2007; Sogo & Osaka, 2002), suggestingthat the visual system has a mechanism for maintainingpositionalconstancyofobjectsacrosssaccades.

A related phenomenon is that if the saccadic target isdisplacedafterthesaccadehasbeeninitiated,thedisplace-ment(ofupto30%saccadesize) isnotnoted(Bridgeman,Hendry,&Stark,1975).However,ifthereisabriefgapinthereappearanceofthetargetinthedisplacedposition,thedisplacement is immediately apparent (Deubel, Schneider,&Bridgeman,1996).Thisobservation led to the idea thatthesystemassumesobjectstabilityintheabsenceofcontraryinformation,probablybycomparingpresaccadicandpost-saccadic positions with some form of short-term memorybuffer.Theseresultssuggestthatthevisualsystemdoestakeadvantageofstaticvisualreferencestohelpmaintainstabil-ityacrosssaccades,butthedetailsofhowtheseareselectedarestoredinsomeformofmemorybufferoflimitedcapacityhaveyettobedetermined.

Ithas recentlybeenargued that the insensitivity to sac-cadic target displacement (Bridgeman et al., 1975) can beexplainedbyoptimalsensorimotorintegrationbetweentheretinalsignalandthatextraretinalcorollarydischargesignals(Niemeier,Crawford,&Tweed,2003).At the timeofsac-cades,spatialinformationabouteyeposition,whichisneces-sary to localize objects in external space, is unreliable.Thereforespatialinformationduringthisperiodisgivenlessweightthanisinformationbeforeandafterthesaccade.Thetransientdistortionsofthekindshowninfigure35.2Amay

alsobe consistentwith statisticallyoptimal,or “Bayesian,”integrationof information.A recent studyhas shownhowthis couldbe the case, by studyingaudiovisual integrationduringsaccades.Auditorystimuliareusuallyfarmorediffi-culttolocalizeinspacethanarevisualstimuli.Whenvisionand sound are in conflict, vision dominates (the “ventrilo-quisteffect”)aspredictedbyoptimalintegration.However,when visual stimuli are artificially degraded by blurring,auditioncandominate(Alais&Burr,2004),againconsistentwithoptimalintegration.Becausesaccadeshavelittleeffectonauditoryspaceperception (Harris&Lieberman,1996),they are a useful tool to study saccadic mislocalization.Indeed, audiovisual stimuli (bars and beeps presentedtogetherinthesamespatialposition)aremislocalizedmuchlessthanarevisualstimulithatarepresentedalone,suggest-ingthatvisualinformationisgivenalowweightduringsac-cades,andthiscanleadtomislocalizationoftransientstimuli(Binda,Bruno,Burr,&Morrone,2007).Notonlydoestheidea explain qualitatively the mislocalization, it explainsquantitatively the mislocalization of bimodal audiovisualstimulioverthewholetimecourserelativetosaccadeonset(figure35.3).

Bindaandcolleagues(2007)goontodevelopaBayesianmodel of saccadic mislocalization, simply assuming, likeNiemeierandcolleagues(2003),anincreaseinnoisinessoftheeyepositionsignalatthetimeofsaccades.Atthisstage,themodelaccountsonlyfortheshiftinthedirectionofthesaccade, not the accompanying compression. This wouldrequireafurtherassumption,suchasa“prior”or“defaultrule,” forobjects tobeseenat the fovea.While thisseemsreasonableandhasbeensuggestedinothercontexts(Deubel,Schneider,&Bridgeman,2002;MacKay,1973),itremainsspeculationatthisstage.

So the functional role of spatial compression remainsunclear.However,itisinterestingthatsaccadiccompressionispositivelycorrelatedwithpeaksaccadicvelocity:Individu-alswithhighsaccadicvelocityshowlargecompression,whilesubjectswith slow saccadicvelocity showmainlya shift inthe saccadic direction (but the effect is not related to thespuriousvisualmotion).Thissuggestsastronglinkbetweenperception at the time of saccades and the motor system,probably mediated by the corollary discharge signal. It isalso interesting tonote that the temporaldynamicsof sac-cadic mislocalization are very similar to those of saccadicsuppression (compare figures 35.1 and 35.2), indicating acommon mechanism, probably the corollary dischargesignal. It would be interesting to test whether there is acorrelationbetweensaccadicvelocityandthemagnitudeofsuppression.

Saccadescausedramaticperceptuallocalizationillusions,butwhensubjectsarerequiredtoindicatetheirresponsebyamotoraction—secondarysaccadesorblindhammering—their responses are near-veridical (Hallett & Lightstone,


Y


1976a, 1976b; Hansen & Skavenski, 1977, 1985). Otherstudies (e.g., Bridgeman, Lewis, Heit, & Nagle, 1979) alsoreported that subjects can point accurately to targets thatwere displaced perisaccadically, even though the subjectdidnotperceivethechangeintargetposition.However,afewexperimentshavefailedtoreplicatetheoriginaldissocia-tionbetweenmotor accuracy andperceptual errorduringsaccades, reporting localization errors for both tasks(Bockisch & Miller, 1999; Dassonville, Schlag, & Schlag-Rey, 1992, 1995; Honda, 1991; Miller, 1996; Schlag &Schlag-Rey, 1995). Recently, Burr, Morrone, and Ross(2001)andMorroneetal.(2005a)reportedacleardissocia-tionbetweenverbalreportsandblindpointingforsaccadiccompression. The plot of figure 35.4 shows that brieflyflashed stimuli were perceived clearly in false positions,causing the characteristic compression (solid symbols); butwhenaskedtopointblindlyat thestimuli,with thescreentemporallyobscuredbyliquidcrystalshutter,observersdidsoveridically(opensymbols).

Interestingly, analogous effects have been reported inaudition.Althoughsaccadiceyemovementsdonotaffectthelocalizationoftones,saccadicheadmovementsdo(Leung,Alais&Carlile,2008).Soundsarecompressed toward theendpointof thehead turn.However, if subjectsareask to

pointtotheapparentsoundsource(byheadturn),thecom-pressiondisappears,asitdoesforvision(Burretal.,2001).

However,forvisualjudgments,introducingclearlyvisiblepostsaccadic references under normal lighting conditionscausesbothverbalreportandpointingtoshowcompression.Thissuggeststhatvisionhasaccesstotwomaps,onesubjecttodistortionand theothernot.Themotormapshowsnocompressionexceptwhenvisual referencesremain inviewfor a substantial time after saccade, indicating that thesemaps are updated postsaccadically, while for perceptualjudgments,theupdatingoccursbeforeandduringtheactualsaccade. Both maps contribute to determining the weightgiventoeachmap.Perhapsthepopulardistinctionbetweenconsciousperceptionandaction(Goodale&Milner,1992;Trevarthen,1968)isatbestanoversimplification.

Butwhereinthebraindothesemapsreside?Isthereanyevidencethatadynamicallyupdatedspatiotopicmapactu-allyexists?Electrophysiologicalstudieshavereportedseveraltransientperisaccadicphenomenon.Inthelateralintrapari-etalcortex(LIP),receptivefieldschangepositionalselectivity(Duhamel,Colby,&Goldberg,1992)justbeforeamonkeymakesasaccadiceyemovement,anticipatingthechangeingaze.Thisisillustratedinfigure35.5A,showingtheresponseofanLIPcelltostimuliflashedtothereceptivefieldposition

0

10

0

10

-100 -50 0 50 -100 -50 0 50-100 -50 0 50

0

-10

0

-10

Pre

cisi

onth

resh

old

(deg

) D F

Time (ms)

EP

erce

ived

posi

tion

(deg

) A C

Bimodal Audition VisionB

Figure 35.3 Illustration of how saccadic mislocalization canresultfromoptimal“Bayesian”fusion.Inatwo-alternativeforcedchoice, subjectswereasked to reportwhetheraperisaccadic testbar that was displayed midway between fixation and saccadictarget seemed to be located to the right or left of a presaccadicprobebar. (For fulldetails, seeBindaetal.,2007.)Psychometricfunctionswerefittedtothesedatatogiveanestimateofperceivedposition and also of precision of localization. The upper curvesshowhowperceivedpositionvariedwithtime(relativetosaccadiconset).Visualstimulipresentedontheirown(A)showedthechar-acteristicmislocalization,likethatoffigure35.1A.Auditorystimuliwere not at all affected by the saccade (C ). However, when the

sound was played contemporaneously with the bar display, themislocalization of the bar was reduced (B). The lower curvesshow the localization thresholds. Again, sound was unaffectedby saccades, but the precision of visual localization was reduceddrastically near saccadic onset. During the bimodal audiovisualpresentation,precisionimprovedtotheextentofbeingbetterthaneither the visual or auditory unimodal localization precision.Indeed,thisperformance,bothforperceivedpositionandforpreci-sionthresholds,wasveryclosetotheBayesianprediction,indicatedbythethickgrayline.Thedottedhorizontallinesindicateperform-ance during fixation. (Reproduced with permission from Bindaetal.,2007.)


Y


andwhatwillbecome thereceptivefieldafter the saccadehas been made (“future receptive field”). Note that theresponse in thecurrentreceptivefieldstarts toreduceandthatinthefuturereceptivefieldstartstoincrease,longbeforetheeyehasactuallymovedtorepositiontheretinalimage.Thisistermedpredictive remapping.

This phenomenon occurs not only in LIP, but also inmany other visual areas, including the superior colliculus(Walker, Fitzgibbon, & Goldberg, 1995) and area V3(Nakamura&Colby,2002),withareaV4showingasome-whatdifferentbehavior(Toliasetal.,2001).Ithasevenbeensuggestedthat10%ofneuronsinprimaryvisualcortex(V1)show dynamic updating of receptive fields (Nakamura &Colby, 2002). The origin of the phenomenon has beenstudied in the frontal eye field (FEF), and firm evidencedemonstrates that it is mediated by a corollary dischargesignal,probablyoriginatinginthecolliculusandmediodor-salthalamus(Sommer&Wurtz,2002,2006).Deactivationofthenucleusabolishesthepredictiveupdatingoftherecep-tive field. The corollary discharge signal arrives nearly100msbeforethattheupdatingstartsintheFEF,indicatingthecomplexityofthereorganization.

Despite these recent efforts, there are several aspects ofthe remapping phenomenon that remain unclear. Forexample,betweenthetimethattheneuronstartstorespondtostimuliintheupdatedpositionandwhenitregainspost-saccadicaly retinotopic specificity, are receptive fieldsanchoredinatransientlycraniotopicmap?Dothereceptivefields undergo changes in size during the remapping? Arethe neurons that are susceptible to remapping randomly

intermingledwiththosethatdonotremap,oristheresomespecificorganization?

Clever psychophysical studies have also demonstratedremapping in humans (Burr & Morrone, 2005; Melcher,2005,2007),bystudyingthespatialselectivityofvisualafter-effects. Most aftereffects are spatially selective. But is theselectivity in retinotopic or spatiotopic coordinates? Byimposingasaccadebetweentheadaptorandthetest,Melcherwasabletoshowthattheselectivitywasbothretinotopicandspatiotopic.Thedegreetowhichadaptationwasspatiotopicvariedwiththecomplexityoftheaftereffect.Simpleadapta-tion aftereffects, like contrast (thought to be mediated byprimaryvisualcortex)wereprimarilyretinotopic,whilemorecomplex aftereffects (such as faces) were primarily spati-otopic; aftereffects of intermediate complexity, like the tiltaftereffect,werebothretinotopicandspatiotopic.

Adaptationtechniques (Melcher,2007)canalsobeusedtorevealthedynamicsoftheupdating,bybrieflypresentingthe test justprior toa saccade (figure35.5B).Longbeforethe saccade,adaptation ismaximalwhen testandadaptorarepresentedtothesameposition,atfixation,withverylittleadaptationatthepositionofsaccadictarget.However,whenthetestispresentedperisaccadicallybutbeforetheeyeshavemoved,themaximumadaptationoccursfortestsnearsac-cadictarget,thepositionthatwillcorrespondtotheadaptedretinaaftertheeyeshavemoved.ThesimilarityofthetimecoursesoftheadaptationandtheresponseoftheLIPneuronstrongly imply that Melcher’s experiment reveals the psy-chophysicalcounterpartof the“predictiveremapping.”Atpresent,itisstilluncertainexactlyhowthistransientupdat-

Figure 35.4 No spatial compression for motor responses. (A)Subjects viewed a cathode-ray tube monitor through a liquidcrystalshutter.Oncommand,theymadea15-degreesaccadefrom−7.5to+7.5degrees(dashedlinesinpanelB),andabarwasbrieflydisplayedjustpriortosaccadiconset.Shortlyafterthesaccadewascompleted,theshutterclosed,andsubjectsrespondedbyjabbingatthetouchscreenwithabriskballisticmovement,thehandbeing

hiddenfromview. (B)Theopensquaresshowtheresults for thejabbingresponseforstimulipresentedjustpriortosaccadiconset(−30<t<0ms).Theresponsesarenearveridical.Thesolidcirclesshow results for verbal reports, under identical conditions. Asshowninfigure35.2, there isaverystrongcompression,withallstimuli within 10 degrees of the saccadic target seen at saccadictarget.(ReproducedwithpermissionfromBurretal.,2001.)


Y


Thedynamicsoftheremappingreceptivefieldisalsoverysimilar to thatofperisaccadicmislocalization (figure35.2),suggestingthatacommonmechanismcouldbedrivingallthesephenomena.However, thereare severalproblems inrelatingthetwosetsofdataquantitatively.Withinaframe-work of labeled-line theory, a neuron that is placed in aspecific anatomical position in a cortical map will, whenstimulated,signalthepresenceofastimulusatthatposition.However,ifitrespondspresaccadicallytoastimulusfallingin the future receptive field (one displaced in the directionof the saccade), it should still signal this stimulus locationas being in the normal location, that is shifted in thedirectionoppositetothesaccade:buttheresults(figures35.2and35.3)showthattheprimaryresultistheperceptionofashift inthesamedirectionofthesaccade.Therearetwopossibleschemastoresolvethisapparentcontradiction.Thefirst is toconsider that theremappedactivityof the futurereceptivefieldistheneuronalresponseofthecorollarydis-charge signal,mapped in retinal coordinates (Bindaet al.,2007). This activity is present only if a visual stimulus ispresent; it is active only when important informationneeds to be updated, reducing also the complexity of thephenomenon.Within this framework, theaddition (fusion)of thisactivitywith theretinotopicactivityofvisualcortexcouldgeneratetheshiftofapparentpositionsintheappro-priatedirection, aswehave recentlydemonstrated for theaudiovisual targets (figure35.3).Theotherpossibility is toconsiderthattheremappedneuronalactivityisnotreferredto the exact time of the stimulus presentation but is readafter that the saccade is complete, in a form of postdiction(Eagleman&Sejnowski,2000).Thiswouldalsoimplythatalsoperceptualtimeshouldbealteredbysaccades,asindeeditis(seebelow).

At the time of the saccade, the timing of the neuronalresponsechangesdramatically.InallcellsofareasV3AandFEFthatremapduringsaccades, theirremappedresponseisfasterthanthatduringfixation(Nakamura&Colby,2002).Similarly, the latenciesofneurons inareasMTandMSTare shorter in response to real saccade than to simulatedsaccades (Price, Ibbotson, Ono, & Mustari, 2005). Theseeffectshavepsychophysical implications:Saccades cause acompression, and even an inversion, of perceived time(Morrone,Ross,&Burr,2005).Whenaskedtocomparetheperceiveddurationofatemporalintervalpresentedaroundthetimeofasaccadewithonepresented2safterward,sub-jectsjudgeditmuchshorter,abouthalftheduration(figure35.6).Againthetimecourseofthisdistortionisquitetightand, after taking into account the duration of the stimuli,similartothatofthespatialcompression.

Preliminary data (Binda, Burr, & Morrone, 2008) alsoindicatethattheperceivedtimeatsaccadiconset,measuredusing an auditory tone, is delayed of about 100ms, whileabout50msbeforesaccade,thelatencyisreducedbyabout

Figure 35.5 Predictive remapping in an LIP cell and humanobservers. (A)Theresponseofa“remapping”cellofareaLIPofthemacaquearound the timeof the saccade tobrief stimulidis-playedinthe“current” (presaccadic)receptivefield (opencircles)andtostimuliflashedinwhatwillbecomeitsreceptivefieldafterthesaccadeismade.Theresponsetostimuliinthecurrentrecep-tivefieldbeginstodecreasebeforetheeyesactuallymove.Aroundthe same time, the response in the “future” position begins toincrease,longbeforetheeyeshaveactuallydisplacedthereceptivefield. (B) An experiment showing analogous behavior in humanpsychophysics.Subjectsadaptedtoatiltedgrating,thenmeasuredtheaftereffecttoagratingpresentedinthesame(retinal)position(“current,”opencircles)ortothepositionthatwillcorrespondtotheretinalpositionoftheadaptorafterasaccadehasbeenmade(“future,”solidsquares).Longbeforethesaccade,thereisnoadap-tationinthefuturefield,andthereisfulladaptationinthecurrentfield (normalized to unity). Like the cell firing rate, adaptationeffectsinthecurrentfieldbegintoreduce,andthoseinthefuturefieldbegintoincrease,beforetheeyeshaveactuallymoved.Wellafterthesaccadeisterminated,theeffectsdonotdropcompletelytozero,becausethispositioncorrespondstothespatiotopicposi-tionof theadaptor,andorientationadaptationhasa spatiotopiccomponent (Melcher, 2005). (Reproduced with permission fromKusunoki&Goldberg,2003,Melcher,2007.)

-400 -200 0 200 400 600

-400 -200 0 200 400 600

0.0

0.2

0.4

0.6

0.0

0.5

1.0

future current

Nor

mal

ized

Act

ivit A

Afte

r-ef

fect

mag

nitu

re

Time (ms)

B

ingofreceptivefieldsleadstovisualstability,butitisclearlyimportant to test the future activity before that the directinputwillexcitetheneuronafterthesaccade.Itcouldbridgetheperceptionbetweenthetwofixations,butcanthisphe-nomenonexplainperisaccadicmislocalizations?


Y


20ms,consistentwith the inversionof timedataandwiththefactthatduringtheremappingneuronallatencybecameshorterofabout40ms,explainingtheinversion.Inaddition,theyalsoindicatethatstimulithatarepresentedatsaccadiconsetare coincidentwith stimulipresented soonafter sac-cades, facilitatingtheinterpretationoftheirpositioninthepostsaccadiccoordinatesystem.

Space and time are generally studied separately andthoughtofasseparateandindependentdimensions.However,aswehaveobserved,both spaceand timeundergo severetransientdistortionsatthetimeofsaccades,asobjectsbecomecompressed toward the saccadic target (Ross et al., 1997),and perceived temporal durations are severely shrunk(Morroneetal.,2005b).Aswasdiscussedabove,therelation-shipbetweenperceptualshiftsandreceptivefieldupdatingisfarfromclear,andcompressionoftimeandofspaceisevenmoredifficulttounderstand.Nevertheless,wecanadvanceafewfirmpropertiesthatmighthelptoexplaincompression.As the transient changesboth in spaceand in time followverysimilardynamics,theymightwellbemanifestationsofacommonneuralcause,adistortioninthespace-timemetric(Morrone,Ross,&Burr,2005).Compressionofrelativedis-tancesinspaceandtimeareconsistentwithareductionofspatialand temporal sampling.This isalsooneof the fewconcepts that would explain the perisaccadic increase insensitivity for size (Santoro, Burr, & Morrone, 2002) andduration(Morrone,Ross,&Burr,2005)judgments.

If together with the undersampling, the receptive fieldbecomestransientlyorientedinspace-timesuchthatstimulipresentedbeforethesaccadeandnearthefixationareinte-gratedwithstimulipresentedlaterforpositionfarfromfixa-

tion, we could provide a description of the origin of allperisaccadicphenomena.Therotationinspaceandtimeofthe neuronal selectivity is a concept that has strong andimportant analogies to the physical rotation on space andtimethatoccursofmotionatrelativisticspeeds,discussedindetail elsewhere (Morrone, Ross, & Burr, 2008). Unfortu-nately,thedynamicsofthechangesinreceptivefieldsduringsaccades are not yet well enough described to pursue thisideamuchfurtheratpresent.

Transsaccadic integration and craniotopic maps

Ournormalexperiencecomesfromtheinformationderivedfromonefixationbeingtransferredtothenext,evenwhenaparticularobjectorpartofthescenebecomeshidden.Theo-riesabouttranssaccadicintegrationhaveaboundedoverthepastdecades.Earlyideas(e.g.,Jonides,Irwin,&Yantis,1982)assumeda“transsaccadicmemorybuffer”thataccumulatedhigh-precisioninformationfromeachsaccadetoconstructadetailedrepresentationoftheworld(likepinningstampstoadonkey).These ideas felloutof favor, largelybecauseoftheimplicitimplicationthatthevisualsystemmustconstructsome form of stable Cartesian theater to be viewed bya homunculus. More recent theories have swung to theoppositeextreme,assumingthatperceptualstabilitydepends,paradoxically,onthe lackof internalrepresentationof theworld(O’Regan&Noe,2001).Observersarelargelyinsensi-tivetotranssaccadicchangesinthevisualscene,questioninghow much detailed visual information can be gleaned bymakinganeyemovementondemand;manyhaveassumedthatnovisualmemoryisnecessaryatall(Findlay&Gilchrist,

Figure 35.6 Time is also compressed during saccades. (A) Thesubjectwasasked tocompare thedurationof the intervalof twotestflashes(separatedby100ms)withapostsaccadicprobeofvari-ableduration thatappeared2s later. (B )Theapparentdurationwasthencalculatedfrompsychometricfunctions.Aroundthetime

of saccadic onset, apparent duration was about half the physicalduration.Thedashedlineshowsthedurationmatchduringfixa-tion. (Reproducedwithpermission fromMorrone,Ross,&Burr,2005.)

-400 -200 0 200 4000

50

100

Time (ms)

App

aren

t dur

atio

n (m

s)

-500 0 500 1,000 1,500 2,000

Time (ms)

ProbeTest

A B

2


Y


2003;McConkie&Zola,1979;Tatler,2001). Inpractice,however,itisstillnecessaryforthebraintoknowwheretolookfortheinformationthatitneeds,sinceeyemovementsarenotrandomandarerarelywastedinnaturaltasks(Land,Mennie,&Rusted,1999;Najemnik&Geisler,2003).Thussomeinformationaboutthelayoutofthesceneandtheposi-tionofimportantobjectsmustsomehowberepresentedandaccumulatedacrosssaccades.Thereisclearevidenceshowingthatatleastthreeorfourobjectsaretransferredsuccessfullyacrosssaccadesevenintheabsenceofallocentriccues(Prime,Niemeier,&Crawford,2006).

Recently, Melcher and Morrone (2003) showed thattranssaccadicintegrationoccursformotionsignalsthatareindividuallybelow threshold (andhencearenotperceivedwhenpresentedalone).Twoperiodsofcoherenthorizontalmotion(150mseach)wereshownsuccessively,separatedbysufficienttimetoallowforasaccadiceyemovementbetweenthem.Onsomeblocksoftrials,subjectssaccadedacrossthestimulusbetweenthetwomotionintervals;onothers,theymaintainedfixationaboveorbelowthestimulus.Thresholdsweresimilarinthetwoconditions,showingthatthemotionsignalsweretemporallyintegratedacrossthesaccade—butonlywhenthetwomotionsignalswereinthesamepositionin space, indicating that the brain must use a mechanismthatisanchoredtoexternalratherthanretinalcoordinates.Importantly,themethodologyexcludedcognitivestrategiesorverbalrecoding,sincethemotionsignalspresentedbeforeand after the saccade were each well below the consciousdetection threshold; only by summating the two signalscouldmotionbecorrectlydiscriminated.

Anotherexampleofcraniotopicmechanismsisthedem-onstration of spatially specific adaptation of event-time(Burr,Tozzi,&Morrone,2007),showingthatadaptationtoa fast-moving (20Hz) spatially localized grating decreasestheapparentdurationofgratingsthatarepresentedtothatpart of the visualfield (in external space)butnot tootherspatiallocations.

Becauseofthespatialselectivityofindividualneurons,theresponseofprimaryandsecondaryvisualcortexformsamap (Morgan,2003), similar inprinciple to that imagedontheretinae(exceptfordistortionsduetomagnificationofcentralvision). This retinotopic representation, which changes com-pletely each time the eyes move, forms the input for allfurtherrepresentationsinthebrain.Soamajorquestionishow this retinotopic representation becomes transformedintothespatiotopicrepresentationthatweperceive,anchoredinstablereal-worldcoordinates.

Electrophysiological studies have shown that neuronsin specific areas of associative visual cortex, including V6(Galletti, Battaglini, & Fattori, 1993) and VIP (Duhamel,Bremmer, BenHamed, & Graf, 1997), do show the spati-otopicselectivitythatwewouldexpecttoexist;theirtuningis invariantofgaze,unlikeareasV1andV2 (thatprovide

their input). Unfortunately, the exact transformation fromretinaltospatiotopiccoordinatesisnotyetfullyunderstood,althoughthesuggestionhasbeenmadethatBayesianfusionoftheretinalsignalwitheyepositionsignalsissufficientinprincipletogeneratespatiotopicmaps,probablyactingviaeyeposition–dependentmodulationoftheneuralresponse,also referred toas gain fields (Pouget,Deneve,&Duhamel,2002;Snyder,Grieve,Brotchie,&Andersen,1998;Zipser&Andersen,1988).

Functional magnetic resonance imaging has also indi-catedtheexistenceof spatiotopiccoding inhumancortex,bothinLO(McKyton&Zohary,2006),anareadeputedtotheanalysis ofobjects and inMT+ (d’Avossa et al., 2007;Goossens,Dukelow,Menon,Vilis,&vandenBerg,2006).UsingstimulisimilartothoseusedbyMelcherandMorrone,our group has reported that the response of a portion ofhumanMTcomplexvarieswithgazepositioninawaythatis consistent with spatiotopic coding. The results are illus-trated in figure 35.7. With gaze fixed in the centre of thescreen, both areas V1 and MT show spatial selectivity,respondingonlywhenthestimuliarepresentedtothecon-tralateral field (figures 35.7A and 35.7B). However, if thestimulus is fixed (in the center) and its retinal projectionvaried by varying gaze, the results are different. V1 stillrespondsonlytothecontralateralstimulus,butMTrespondstobothipsilateralandcontralateralstimuli,equallystrongly.Further experiments suggested that MT actually shifts itsreceptivefieldstocausespatiotopiccoding.

However, it must be pointed out that this result is cur-rentlycontroversial,andcontraryresultshavebeenreported.Gardner, Merriman, Movshon, and Heeger (2008) reportthatundertheconditionsoftheirexperiment,theresponseofMTisretinotopicratherthanspatiotopic.Atthisstage,itisdifficulttounderstandwhatwerethemaindifferencesinexperimentalprocedurethat leadto thesedifferentresults.One interesting possibility is that attention is important,whichwasdirectedtowardtheretinotopicfoveainGardnerand colleagues’ experiment but not in d’Avossa and col-leagues’ study. Or perhaps the coordinate space is object-basedrather thanspatiotopic. Indeed, furtherexperimentsby the Heeger group have provided strong evidence formodulation of BOLD signal response by eye movements,but the modulation seems to reflect more an amplitudemodulation(gainfield)ratherthanashifttopreservespati-otopicity (E.P.Merrian,J.L.Gardner,D.J.Heeger,andJ. A. Movshon, unpublished observations). It is clear thatmorestudiesareneededunderdifferentconditionstodisen-tangle the exact nature of the way in which eye positionmodulations the activity of MT and other cortical areas.Thisshouldprovidearicharea forresearchover thenextfewyears.

The fact that spatiotopic (orat leastcraniotopic)codingis more common in the dorsal area might suggest that it


Y


couldbeusedfortheactionsystem.Aswasdiscussedabove,theactionsystemseemstoupdatespatialmapsmuchlaterthan the perceptual system does. Perhaps the updating ofcraniotopicmapstakestimebutleadstomorerobustcodingof information, explaining the resistance of this system tosaccadic mislocalization. The perceptual system, on theotherhand,mightoperatenotwithacompletemapanchoredinexternalcoordinatesbutwithensemblesofneuronswithreceptivefieldsanchoredinretinotopiccoordinatesbuttran-siently shifted just before each saccade (like the neuron offigure35.5).Thistransientupdatingoneachsaccademightbesufficienttomaintainausefulperceptualrepresentationthat is not actually in spatiotopic coordinates that takeslongertodevelop.

Theimportantconclusionfromtheseandrelatedstudiesisthatthevisualsystemdoescombineinformationfromonefixationtothenextbutthatthisprocessisnotlikestickingpostage stamps on a tailor’s dummy: detailed “snapshots”are not integrated within a transsaccadic buffer that pre-servestheexternalmetric(Jonidesetal.,1982).Indeed,such

a scheme couldbeproblematic, as scenesdo change con-tinuouslyasobjectsmoveandrotate.Inappropriatepixelwiseintegrationcouldleadtoveryweirdpercepts,likecubistart.Transsaccadicintegrationdoesnotoccuratthepixellevel,butafteracertainamountofvisualprocessing,soattributessuchasform,orientation,motion,andevencomplexentitiessuch as faces are integrated across fixations. This in itselfdoes not solve the problem of visual stability, but it couldprovide a basis for visual continuity with ever-changingretinal input. It might well be that the two processes—dynamicreceptivefieldupdatingandcraniotopiccoding—collaborateintheselectionoftheimportantinformationtobe integrate. The remapping neurons are primed beforethe eye movement actually occurs, so they can determinewhether the information of successive fixations should beintegrated. Perhaps if activation during remapping wereconstant, then craniotopic receptive fields could receive aswitchsignalallowinginformationtobeintegratedtranssac-cadicaly. If the remapping neurons do not respond, theswitch could open, vetoing the integration of craniotopicreceptivefields,sotheyaccumulatenewinformationstartingafresh. Within this schema, both the integration acrosssaccades and perisaccadic mislocalization might involvethe same mechanisms to obtain a stable vision acrossseparate glances without fusion of local, pixel-like visualdetails.

Conclusion

Seeing is usually believing. For about two-thirds of ourwaking lives,weperceiveobjectswherevisiontellsus theyare,which,moreoftenthannot,coincideswiththeiractualposition.Intheremainingtime, thevisualsystemsendsuserroneous spatial information, presumably because it isengagedincorrectingthetroublesomeconsequencesofeyemovements on retinal afferences. When this happens, wedisbelievevisualinformation.Ifavailable,spatialcuesfromothersensesbecomedominant;ifwehavetoact,weusetherobust representation of the craniotopic system withoutattempting to update it dynamically. If vision is the onlysignalthat ispresent,wedeformourconceptofspaceandoftimetomakesenseofitandtonotmissvisualinformationformorethanone-thirdofourwakingtime.

acknowledgments ThisworkwassupportedbyEuropeanUnionPF6NEXT(MEMORY)andPF7IDEAS:STANIB.

REFERENCES

Alais, D., & Burr, D. (2004). The ventriloquist effect resultsfrom near-optimal bimodal integration. Curr. Biol., 14(3),257–262.

Alhazen,I.(1083).Bookofoptics.InA.I.Sabra(Ed.),The optics of Ibn al-Haytham.London:WarburgInstitute,1989.

Figure 35.7 The dependence of MT response on gaze. (A, B)BOLDresponsesforareasV1(A)andMT(B)toipsilateral(solidsquares)andcontralateral(opencircles)stimulation,keepinggazefixedat screen center (see icons). Inall conditions, subjectswererequiredtodiscriminatethedirectionofmotiontokeepsustainedattentiononthestimulus.Asexpected,bothareasrespondstronglyonly to contralateral stimulation. (C, D)Responseof V1 (C ) andMT (D) to ipsilateral and contralateral stimulation varying gaze;thestimuluswasinscreencenter,andthesubjectfixatedtotheleftorrightofit(seeicon).TheresponsepatternofV1wasunchanged,suggestingthatV1isselectiveonlytoretinalposition,irrespectiveofgaze.However,theresponsepatternofMTwasquitedifferent,nowbeingstrongforbothipsilateralandcontralateralstimulation,suggestingthatitisselectivetothepositionofthestimulusonthescreen, not on the retina. (Reproduced with permission fromd’Avossaetal.,2007.)

0 5 10 15

0

1

0 5 10 15

0

1

0 5 10 15

0

1

0 5 10 15

0

1

Time (s)

BO

LD C

hang

e (%

) V1

IpsiContra

MT

DC

BA


Y


Awater, H., & Lappe, M. (2006). Mislocalization of perceivedsaccade targetposition inducedbyperisaccadicvisual stimula-tion.J. Neurosci.,26(1),12–20.

Binda, P., Bruno, A., Burr, D. C., & Morrone, M. C. (2007).Fusionofvisualandauditorystimuliduringsaccades:ABaye-sianexplanationforperisaccadicdistortions.J. Neurosci.,27(32),8525–8532.

Binda, P., Burr, D. C., & Morrone, M. C. (2008). Spatio-temporaldistortionsofvisualperceptionduringsaccades.Percep-tion 36(ECVPAbstractSupplement).

Bockisch, C., & Miller, J. (1999). Different motor systems usesimilardampedextraretinaleyeposition information.Vis. Res.,39,1025–1038.

Bodis-Wollner,I.,Bucher,S.F.,&Seelos,K.C.(1999).Corticalactivation patterns during voluntary blinks and voluntary sac-cades.Neurology,53,1800–1805.

Bridgeman,B.,Hendry,D.,&Stark,L.(1975).Failuretodetectdisplacement of visual world during saccadic eye movements.Vis. Res.,15,719–722.

Bridgeman,B.,Lewis,S.,Heit,G.,&Nagle,M.(1979).Relationbetweencognitiveandmotor-orientedsystemsofvisualpositionperception.J. Exp. Psychol. [Hum. Percept.],5(4),692–700.

Bristow,D.,Haynes,J.D.,Sylvester,R.,Frith,C.D.,&Rees,G.(2005).Blinkingsuppressestheneuralresponsetounchangingretinalstimulation.Curr. Biol.,15(14),1296–1300.

Bruno, A., Brambati, S. M., Perani, D., & Morrone, M. C.(2006). Development of saccadic suppression in children.J. Neurophysiol.,96(3),1011–1017.

Burr, D., & Morrone, M. C. (2005). Eye movements: Buildinga stable world from glance to glance. Curr. Biol., 15(20),R839–R840.

Burr,D.,Tozzi,A.,&Morrone,M.C. (2007).Neuralmecha-nismsfortimingvisualeventsarespatiallyselectiveinreal-worldcoordinates.Nature Neurosci.,10(4),423–425.

Burr,D.C.,Holt,J.,Johnstone,J.R.,&Ross,J.(1982).Selectivedepressionofmotionsensitivityduringsaccades.J. Physiol.,333,1–15.

Burr,D.C.,Morgan,M.J.,&Morrone,M.C.(1999).Saccadicsuppression precedes visual motion analysis. Curr. Biol., 9,1207–1209.

Burr, D. C., & Morrone, M. C. (1996). Temporal impulseresponse functions for luminance and colour during saccades.Vis. Res.,36,2069–2078.

Burr,D.C.,Morrone,M.C.,&Ross, J. (1994).Selective sup-pressionofthemagnocellularvisualpathwayduringsaccadiceyemovements.Nature,371,511–513.

Burr,D.C.,Morrone,M.C.,&Ross,J.(2001).Separatevisualrepresentations forperceptionandactionrevealedby saccadiceyemovements.Curr. Biol.,11(10),798–802.

Burr,D.C.,&Ross,J.(1982).Contrastsensitivityathighveloci-ties.Vis. Res.,23,3567–3569.

Cai, R. H., Pouget, A., Schlag-Rey, M., & Schlag, J. (1997).Perceived geometrical relationships affected by eye-movementsignals.Nature,386,601–604.

Campbell, F. W., & Wurtz, R. H. (1978). Saccadic ommission:Whywedonotseeagreyoutduringasaccadiceyemovement.Vis. Res.,18,1297–1303.

d’Avossa,G.,Tosetti,M.,Crespi,S.,Biagi,L.,Burr,D.C.,&Morrone, M. C. (2007). Spatiotopic selectivity of BOLDresponsestovisualmotioninhumanareaMT.Nature Neurosci.,10(2),249–255.

Dassonville,P.,Schlag,J.,&Schlag-Rey,M.(1992).Oculomo-torlocalizationreliesonadampedrepresentationofsaccadiceye

movement displacement in human and nonhuman primates.Visual Neurosci.,9,261–269.

Dassonville,P.,Schlag,J.,&Schlag-Rey,M. (1995).Theuseofegocentricandexocentriclocationcuesinsaccadicprogram-ming.Vis. Res.,35,2191–2199.

Deubel,H.,Schneider,W.X.,&Bridgeman,B.(1996).Postsac-cadic target blanking prevents saccadic suppression of imagedisplacement.Vis. Res.,36,985–996.

Deubel,H.,Schneider,W.X.,&Bridgeman,B. (2002).Trans-saccadic memory of position and form. Prog. Brain Res., 140,165–180.

Diamond, M. R., Ross, J., & Morrone, M. C. (2000). Extrare-tinal control of saccadic suppression. J. Neurosci., 20, 3442–3448.

Dodge, R. (1900). Visual perception during eye movements.Psychol. Rev.,7,454–465.

Duhamel, J., Bremmer, F., BenHamed, S., & Graf, W. (1997).Spatial invariance of visual receptive fields in parietal cortexneurons.Nature,389,845–848.

Duhamel, J.R.,Colby,C.L.,&Goldberg,M.E. (1992).Theupdatingoftherepresentationofvisualspaceinparietalcortexbyintendedeyemovements.Science,255(5040),90–92.

Eagleman,D.M.,&Sejnowski,T.J. (2000).Motionintegrationand postdiction in visual awareness. Science, 287(5460),2036–2038.

Findlay, J. M., & Gilchrist, I. D. (2003). Active vision: The psychology of looking and seeing. Oxford, U.K.: Oxford UniversityPress.

Fischer,B.,Biscaldi,M.,&Gezeck,S.(1997).Onthedevelop-mentofvoluntaryandreflexivecomponentsinhumansaccadegeneration.Brain Res.,754(1–2),285–297.

Galletti, C., Battaglini, P. P., & Fattori, P. (1993). Parietalneurons encoding spatial locations in craniotopic coordinates.Exp. Brain Res.,96,221–229.

Galletti, C., & Fattori, P. (2003). Neuronal mechanisms fordetectionofmotioninthefieldofview.Neuropsychologia,41(13),1717–1727.

Gardner,J.L.,Merriam,E.P.,Movshon,J.A.,&Heeger,D.J.(2008).Mapsofvisualspaceinhumanoccipitalcortexarereti-notopic,notspatiotopic.J. Neurosci.,28(15),3988–3999.

Goodale,M.A.,&Milner,A.D.(1992).Separatepathwaysforperceptionandaction.Trends Neurosci.,15,20–25.

Goossens,J.,Dukelow,S.P.,Menon,R.S.,Vilis,T.,&vandenBerg,A.V. (2006).Representationofhead-centricflowinthehumanmotioncomplex.J. Neurosci.,26(21),5616–5627.

Hallett,P.E.,&Lightstone,A.D.(1976a).Saccadiceyemove-mentstowardsstimulitriggeredbypriorsaccades.Vis. Res.,16(1),99–106.

Hallett, P. E., & Lightstone, D. (1976b). Saccadic eye move-mentstoflashedtargets.Vis. Res.,16,107–114.

Hansen, R. M., & Skavenski, A. A. (1977). Accuracy of eyeposition information for motor control. Vis. Res., 17(8),919–926.

Hansen, R. M., & Skavenski, A. A. (1985). Accuracy of spatiallocationsnearthetimeofsaccadiceyemovments.Vis. Res.,25,1077–1082.

Harris,L.R.,&Lieberman,L.(1996).Auditorystimulusdetectionis not suppressed during saccadic eye movements. Perception,25(8),999–1004.

Holt, E. B. (1903). Eye movements and central anaesthesia.Psychol. Rev.,4,3–45.

Honda,H.(1989).Perceptuallocalizationofvisualstimuliflashedduringsaccades.Percept. Psychophys.,46,162–174.

3


Y


Honda, H. (1991). The time courses of visual mislocalizationand of extra-retinal eye position signals at the time of verticalsaccades.Vis. Res.,31,1915–1921.

Ibbotson,M.,Crowder,N.,Cloherty,S.,Price,N.,&Mustari,M. (2008). Neural correlates of saccadic suppression, post-saccadic enhancement and time compression. J. Neurosci.,inpress.

Jonides, J., Irwin, D. E., & Yantis, S. (1982). Integrating visualinformation from successive fixations. Science, 215(4529),192–194.

Kaiser, M., & Lappe, M. (2004). Perisaccadic mislocalizationorthogonaltosaccadedirection.Neuron,41(2),293–300.

Kleiser, R., Seitz, R. J., & Krekelberg, B. (2004). Neuralcorrelatesof saccadic suppression inhumans.Curr. Biol.,14(5),386–390.

Kusunoki, M., & Goldberg, M. E. (2003). The time course ofperisaccadicreceptivefieldshiftsinthelateralintraparietalareaofthemonkey.J. Neurophysiol.,89(3),1519–1527.

Land, M., Mennie, N., & Rusted, J. (1999). The roles of visionand eye movements in the control of activities of daily living.Perception,28(11),1311–1328.

Lappe,M.,Awater,H.,&Krekelberg,B. (2000).Postsaccadicvisual references generate presaccadic compression of space.Nature,403,892–895.

Lappe,M.,Kuhlmann,S.,Oerke,B.,&Kaiser,M.(2006).Thefateofobjectfeaturesduringperisaccadicmislocalization.J. Vis.,6(11),1282–1293.

Leung, J., Alais, D., & Carlile, S. (2008). Compression ofauditoryspaceduringrapidheadturns.Proc. Natl. Acad. Sci. USA,105,6492–6497.

MacKay,D.M.(1973).Visualstabilityandvoluntaryeyemove-ments.InR.Jung(Ed.),Handbook of sensory physiology(Vol.VII/3,pp.307–331).Berlin:Springer-Verlag.

Mateeff,S. (1978).Saccadiceyemovementsand localizationofvisualstimuli.Percept. Psychophys.,24(3),215–224.

Matin,L.,&Pearce,D.G.(1965).Visualperceptionofdirectionfor stimuli flashed during voluntary saccadic eye movements.Science,148,1485–1487.

Matsumiya, K., & Uchikawa, K. (2001). Apparent size of anobject remains uncompressed during presaccadic compressionofvisualspace.Vis. Res.,41(23),3039–3050.

Maurer, D., Lewis, T. L., & Mondloch, C. J. (2005). Missingsights: Consequences for visual cognitive development. Trends Cogn. Sci.,9(3),144–151.

McConkie,G.W.,&Zola,D.(1979).Isvisualinformationinte-gratedacross succesivefixations in reading?Percept. Psychophys.,25,221–224.

McKyton,A.,&Zohary,E.(2006).Beyondretinotopicmapping:Thespatialrepresentationofobjectsinthehumanlateraloccipi-talcomplex.Cerebral Cortex,17,1164–1172.

Melcher, D. (2005). Spatiotopic transfer of visual-form adap-tation across saccadic eye movements. Curr. Biol., 15(19),1745–1748.

Melcher, D. (2007). Predictive remapping of visual featuresprecedes saccadic eye movements. Nature Neurosci., 10(7),903–907.

Melcher, D., & Morrone, M. C. (2003). Spatiotopic temporalintegration of visual motion across saccadic eye movements.Nature Neurosci.,6(8),877–881.

Miller,J.(1996).Egocentriclocalizationofaperisaccadicflashbymanualpointing.Vis. Res.,36,837–851.

Morgan,M.J.(2003).The space between your ears: How the brain rep-resents visual space.London:Weidenfeld&Nicolson.

Morrone, M. C., Ma-Wyatt, A., & Ross, J. (2005). Seeingand ballistic pointing at perisaccadic targets. J. Vis., 5(9),741–754.

Morrone,M.C.,Ross,J.,&Burr,D.(2005).Saccadiceyemove-mentscausecompressionoftimeaswellasspace.Nature Neurosci.,8(7),950–954.

Morrone,M.C.,Ross,J.,&Burr,D.C.(1997).Apparentposi-tion of visual targets during real and simulated saccadic eyemovements.J. Neurosci.,17,7941–7953.

Morrone,M.C.,Ross,J.,&Burr,D.C.(2008).Keepingvisionstable:rapidupdatingofspatiotopicreceptivefieldsmaycauserelativistic-likeeffects.InR.Nijhawan(Ed.),Problems of space and time in perception and action.Cambridge,UK:CambridgeUniver-sityPress.

Najemnik, J., & Geisler, W. S. (2003). Optimal visual search.J. Vis.,3(9),624–624.

Nakamura, K., & Colby, C. L. (2002). Updating of the visualrepresentationinmonkeystriateandextrastriatecortexduringsaccades.Proc. Natl. Acad. Sci. USA,99(6),4026–4031.

Niemeier,M.,Crawford,J.D.,&Tweed,D.B.(2003).Optimaltranssaccadic integration explains distorted spatial perception.Nature,422(6927),76–80.

O’Regan, J. K., & Noe, A. (2001). A sensorimotor account ofvisionandvisualconsciousness.Behav. Brain Sci.,24(5),939–973;discussion973–1031.

Parrish, E. E., Giaschi, D. E., Boden, C., & Dougherty, R.(2005).Thematurationofformandmotionperceptioninschoolagechildren.Vis. Res.,45(7),827–837.

Pola,J.(2007).Amodelofthemechanismfortheperceivedloca-tionofasingleflashandtwosuccessiveflashespresentedaroundthetimeofasaccade.Vis. Res.,47(21),2798–2813.

Pouget,A.,Deneve,S.,&Duhamel,J.R.(2002).Acomputationalperspectiveontheneuralbasisofmultisensoryspatialrepresen-tations.Nature Rev. Neurosci.,3(9),741–747.

Price,N.S.,Ibbotson,M.R.,Ono,S.,&Mustari,M.J.(2005).RapidprocessingofretinalslipduringsaccadesinmacaqueareaMT.J. Neurophysiol.,94(1),235–246.

Prime,S.L.,Niemeier,M.,&Crawford,J.D.(2006).Transsac-cadicintegrationofvisualfeaturesinalineintersectiontask.Exp. Brain Res.,169(4),532–548.

Reppas,J.B.,Usrey,W.M.,&Reid,R.C.(2002).Saccadiceyemovementsmodulate visual responses in the lateral geniculatenucleus.Neuron,35(5),961–974.

Riggs,L.A.,Merton,P.A.,&Morton,H.B.(1974).Suppressionofvisualphosphenesduringsaccadiceyemovements.Vis. Res.,14,997–1011.

Ross,J.,Morrone,M.C.,&Burr,D.C.(1997).Compressionofvisualspacebeforesaccades.Nature,384,598–601.

Santoro, L., Burr, D., & Morrone, M. C. (2002). SaccadiccompressioncanimprovedetectionofGlasspatterns.Vis. Res.,42(11),1361–1366.

Schlag, J., & Schlag-Rey, M. (1995). Illusory localization ofstimuli flashed in the dark before saccades. Vis. Res., 35,2347–2357.

Sclar,G.,Maunsell,J.H.,&Lennie,P.(1990).Codingofimagecontrastincentralvisualpathwaysofthemacaquemonkey.Vis. Res.,30(1),1–10.

Shapley,R.,&Enroth-Cugell,C.(1984).Visualadaptationandretinalgaincontrols. In J.G.Osborn&N.N.Chadler (Eds.),Progress in retinal research(Vol.3).Oxford,UK:PergamonPress.

Shapley,R.M.,&Victor, J.D. (1981).How the contrast gaincontrolmodifiesthefrequencyresponsesofcatretinalganglioncells.J. Physiol. (Lond.),318,161–179.

4

5


Y


Snyder,L.H.,Grieve,K.L.,Brotchie,P.,&Andersen,R.A.(1998).Separatebody-andworld-referencedrepresentationsofvisualspaceinparietalcortex.Nature,394(6696),887–891.

Sogo,H.,&Osaka,N.(2002).Effectsofinter-stimulusintervalonperceived locationsof successivelyflashedperisaccadic stimuli.Vis. Res.,42(7),899–908.

Sommer, M. A., & Wurtz, R. H. (2002). A pathway in primatebrainforinternalmonitoringofmovements.Science,296(5572),1480–1482.

Sommer,M.A.,&Wurtz,R.H.(2006).Influenceofthethalamusonspatialvisualprocessinginfrontalcortex.Nature,444(7117),374–377.

Sperry,R.W.(1950).Neuralbasisofthespontaneousoptokineticresponseproducedbyvisualinversion.J. Comp. Physiol. Psychol.,43,482–489.

Sylvester,R.,Haynes,J.D.,&Rees,G.(2005).Saccadesdiffer-entiallymodulatehumanLGNandV1responsesinthepresenceandabsenceofvisualstimulation.Curr. Biol.,15(1),37–41.

Tatler,B.W.(2001).Characterisingthevisualbuffer:Real-worldevidenceforoverwritingearlyineachfixation.Perception,30(8),993–1006.

Thiele, A., Henning, P., Kubischik, M., & Hoffmann, K. P.(2002). Neural mechanisms of saccadic suppression. Science,295(5564),2460–2462.

Thilo,K.V.,Santoro,L.,Walsh,V.,&Blakemore,C.(2003).Thesiteofsaccadicsuppression.Nature Neurosci.,7,13–14.

Tolias, A. S., Moore, T., Smirnakis, S. M., Tehovnik, E. J.,Siapas,A.G.,&Schiller,P.H.(2001).Eyemovementsmodu-late visual receptive fields of V4 neurons. Neuron, 29(3),757–767.

Trevarthen,C.B.(1968).Twomechanismsofvisioninprimates.Psychol. Forsch.,31,299–348.

Volkmann,F.C.,Riggs,L.A.,White,K.D.,&Moore,R.K.(1978). Contrast sensitivity during saccadic eye movements.Vis. Res.,18,1193–1199.

von Helmholtz, H. (1866). Handbuch der Physiologischen Optik.(ReprintedinJ.P.C.Southall(Ed.),A treatise on physiological optics.NewYork:Dover,1963.)

vonHolst,E.,&Mittelstdt,H.(1954).DasReafferenzprinzip.Naturwissenschaften,37,464–476.

Walker, M. F., Fitzgibbon, J., & Goldberg, M. E. (1995).Neurons of the monkey superior colliculus predict the visualresultofimpendingsaccadiceyemovements.J. Neurophysiol.,73,1988–2003.

Woodworth, R. S. (1906). Vision and localization during eyemovements.Psychol. Bull.,3,68–70.

Wurtz, R. H. (2008). Neuronal mechanisms of visual stability.Vis. Res.,48(20),2070–2089.

Zipser, D., & Andersen, R. A. (1988). A back-propagation pro-grammednetworkthatsimulatesresponsepropertiesofasubsetofposteriorparietalneurons.Nature,331(6158),679–684.


Y

AUTHOR QUeRy FORMDear Author

During the preparation of your manuscript for publication, the questions listed below have arisen. Please attend to these matters and return this form with your proof.

Many thanks for your assistance.

Query Query RemarksReferences

1. AUTHOR:OKasedited?Ifnot,pls.clarify“inconsistent.”

2. AUTHOR:Idon’tunderstandthisreference.Wasthisdone inanexperiment?

3. AUTHOR:Pls.providepagenumber.

4. AUTHOR:Pls.updateifpossible.

5. AUTHOR:Pls.addpagerange.

GAZ_35


35 visual stability during saccadic eye movements...nant stimuli (varying in color but not...

Documents