Eye-Based Interaction in Graphical Systems: Eye-Based Interaction in Graphical Systems: Theory & PracticeTheory & Practice

Eye-Based Interaction in Graphical Systems: Eye-Based Interaction in Graphical Systems: Theory & PracticeTheory & Practice

Part IPart I

Introduction to the Human Visual SystemIntroduction to the Human Visual System

A: Visual AttentionA: Visual AttentionA: Visual AttentionA: Visual Attention

• Latin translation: Latin translation: “Many filtered into few “Many filtered into few for perception”for perception”

• Visual scene inspection is performed Visual scene inspection is performed minutatimminutatim (piecemeal), not (piecemeal), not in totoin toto

““When the things are apprehended by the When the things are apprehended by the sensessenses, , the number of them that can be attended to at the number of them that can be attended to at once is small, once is small, `Pluribus intentus, minor est ad `Pluribus intentus, minor est ad singula sensus' singula sensus' ””

— — William JamesWilliam James

A.1: Visual Attention—chronological A.1: Visual Attention—chronological reviewreviewA.1: Visual Attention—chronological A.1: Visual Attention—chronological reviewreview

• Qualitative historical background: a Qualitative historical background: a dichotomous theory of attention—the dichotomous theory of attention—the “what” and “where” of (visual) attention“what” and “where” of (visual) attention• Von Helmholtz (ca. 1900): mainly concerned with eye Von Helmholtz (ca. 1900): mainly concerned with eye

movements to spatial locations, the “where”, I.e., movements to spatial locations, the “where”, I.e., attention as overt mechanism (eye movements)attention as overt mechanism (eye movements)

• James (ca. 1900): defined attention mainly in terms of James (ca. 1900): defined attention mainly in terms of the “what”, i.e., attention as a more internally covert the “what”, i.e., attention as a more internally covert mechanismmechanism

A.1: Visual Attention—chronological A.1: Visual Attention—chronological review review (cont’d)(cont’d)

A.1: Visual Attention—chronological A.1: Visual Attention—chronological review review (cont’d)(cont’d)

• Broadbent (ca. 1950): defined attention as “selective Broadbent (ca. 1950): defined attention as “selective filter” from auditory experiments; generally agreeing filter” from auditory experiments; generally agreeing with Von Helmholtz’s “where”with Von Helmholtz’s “where”

• Deutsch and Deutsch (ca. 1960): rejected “selective Deutsch and Deutsch (ca. 1960): rejected “selective filter” in favor of “importance weightings”; generally filter” in favor of “importance weightings”; generally corresponding to James’ “what”corresponding to James’ “what”

• Treisman (ca. 1960): proposed unified theory of Treisman (ca. 1960): proposed unified theory of attention—attenuation filter (the “where”) followed by attention—attenuation filter (the “where”) followed by “dictionary units” (the “what”)“dictionary units” (the “what”)

A.1: Visual Attention—chronological A.1: Visual Attention—chronological review review (cont’d)(cont’d)

A.1: Visual Attention—chronological A.1: Visual Attention—chronological review review (cont’d)(cont’d)

• Main debate at this point: is attention Main debate at this point: is attention parallel (the “where”) or serial (the “what”) parallel (the “where”) or serial (the “what”) in nature?in nature?

• Gestalt view: recognition is a wholistic Gestalt view: recognition is a wholistic process (e.g., Kanizsa figure)process (e.g., Kanizsa figure)

• Theories advanced through early Theories advanced through early recordings of eye movementsrecordings of eye movements

A.1: Visual Attention—chronological A.1: Visual Attention—chronological review review (cont’d)(cont’d)

A.1: Visual Attention—chronological A.1: Visual Attention—chronological review review (cont’d)(cont’d)

• Yarbus (ca. 1967): demonstrated sequential, but Yarbus (ca. 1967): demonstrated sequential, but variable, viewing patterns over particular image variable, viewing patterns over particular image regions (akin to the “what”)regions (akin to the “what”)

• Noton and Stark (ca. 1970): showed that subjects Noton and Stark (ca. 1970): showed that subjects tend to fixate identifiable regions of interest, tend to fixate identifiable regions of interest, containing “informative details”; coined term containing “informative details”; coined term “scanpath” describing eye movement patterns“scanpath” describing eye movement patterns

• Scanpaths helped cast doubt on the Gestalt Scanpaths helped cast doubt on the Gestalt hypothesishypothesis

A.1: Visual Attention—chronological A.1: Visual Attention—chronological review review (cont’d)(cont’d)

A.1: Visual Attention—chronological A.1: Visual Attention—chronological review review (cont’d)(cont’d)

Fig.2: Yarbus’ early scanpath Fig.2: Yarbus’ early scanpath recording:recording:

• trace 1: examine at willtrace 1: examine at will

• trace 2: estimate wealthtrace 2: estimate wealth

• trace 3: estimate agestrace 3: estimate ages

• trace 4: guess previous activitytrace 4: guess previous activity

• trace 5: remember clothingtrace 5: remember clothing

• trace 6: remember positiontrace 6: remember position

• trace 7: time since last visittrace 7: time since last visit

A.1: Visual Attention—chronological A.1: Visual Attention—chronological review review (cont’d)(cont’d)

A.1: Visual Attention—chronological A.1: Visual Attention—chronological review review (cont’d)(cont’d)

• Posner (ca. 1980): proposed attentional “spotlight”, Posner (ca. 1980): proposed attentional “spotlight”, an overt mechanism independent from eye an overt mechanism independent from eye movements (akin to the “where”)movements (akin to the “where”)

• Treisman (ca. 1986): once again unified “what” and Treisman (ca. 1986): once again unified “what” and “where” dichotomy by proposing the Feature “where” dichotomy by proposing the Feature Integration Theory (FIT), describing attention as a Integration Theory (FIT), describing attention as a “glue” which integrates features at particular “glue” which integrates features at particular locations to allow wholistic perceptionlocations to allow wholistic perception

A.1: Visual Attention—chronological A.1: Visual Attention—chronological review review (cont’d)(cont’d)

A.1: Visual Attention—chronological A.1: Visual Attention—chronological review review (cont’d)(cont’d)

• Summary: the “what” and “where” Summary: the “what” and “where” dichotomy provides an intuitive sense of dichotomy provides an intuitive sense of attentional, foveo-peripheral visual attentional, foveo-peripheral visual mechanismmechanism

• Caution: the “what/where” account is Caution: the “what/where” account is probably overly simplistic and is but one probably overly simplistic and is but one theory of visual attentiontheory of visual attention

B: Neurological Substrate of the B: Neurological Substrate of the Human Visual System (HVS)Human Visual System (HVS)B: Neurological Substrate of the B: Neurological Substrate of the Human Visual System (HVS)Human Visual System (HVS)

• Any theory of visual attention must Any theory of visual attention must address the fundamental properties of address the fundamental properties of early visual mechanismsearly visual mechanisms

• Examination of the neurological substrate Examination of the neurological substrate provides evidence of limited information provides evidence of limited information capacity of the visual system—a capacity of the visual system—a physiological reason for an attentional physiological reason for an attentional mechanismmechanism

B.1: The EyeB.1: The EyeB.1: The EyeB.1: The Eye

Fig. 3: The eye—“the world’s Fig. 3: The eye—“the world’s worst camera”worst camera”

• suffers from numerous suffers from numerous optical imperfections...optical imperfections...

• ...endowed with several ...endowed with several compensatory compensatory mechanismsmechanisms

B.1: The Eye B.1: The Eye (cont’d)(cont’d)B.1: The Eye B.1: The Eye (cont’d)(cont’d)

Fig. 4: Ocular opticsFig. 4: Ocular optics

B.1: The Eye B.1: The Eye (cont’d)(cont’d)B.1: The Eye B.1: The Eye (cont’d)(cont’d)

• Imperfections:Imperfections:• spherical abberationsspherical abberations

• chromatic abberationschromatic abberations

• curvature of fieldcurvature of field

• Compensations:Compensations:• iris—acts as a stopiris—acts as a stop

• focal lens—sharp focusfocal lens—sharp focus

• curved retina—matches curved retina—matches curvature of fieldcurvature of field

B.2: The RetinaB.2: The RetinaB.2: The RetinaB.2: The Retina

• Retinal photoreceptors constitute first Retinal photoreceptors constitute first stage of visual perceptionstage of visual perception

• Photoreceptors Photoreceptors transducers converting transducers converting light energy to electrical impulses (neural light energy to electrical impulses (neural signals)signals)

• Photoreceptors are functionally classified Photoreceptors are functionally classified into two types: into two types: rodsrods and and conescones

B.2: The Retina—rods and conesB.2: The Retina—rods and conesB.2: The Retina—rods and conesB.2: The Retina—rods and cones

• Rods: sensitive to dim and achromatic Rods: sensitive to dim and achromatic light (night vision)light (night vision)

• Cones: respond to brighter, chromatic Cones: respond to brighter, chromatic light (day vision)light (day vision)

• Retinal construction: 120M rods, 7M cones Retinal construction: 120M rods, 7M cones arranged concentricallyarranged concentrically

B.2: The Retina—cellular makeupB.2: The Retina—cellular makeupB.2: The Retina—cellular makeupB.2: The Retina—cellular makeup

• The retina is composed of 3 main layers of The retina is composed of 3 main layers of different cell types (a 3-layer “sandwich”)different cell types (a 3-layer “sandwich”)

• Surprising fact: the retina is “inverted”— Surprising fact: the retina is “inverted”— photoreceptors are found in the bottom photoreceptors are found in the bottom layer (furthest away from incoming light)layer (furthest away from incoming light)

• Connection bundles between layers are Connection bundles between layers are called called plexiformplexiform or or synaptic layerssynaptic layers

B.2: The Retina—cellular makeup B.2: The Retina—cellular makeup (cont’d)(cont’d)B.2: The Retina—cellular makeup B.2: The Retina—cellular makeup (cont’d)(cont’d)

Fig.5: The retinocellular Fig.5: The retinocellular layers (w.r.t. incoming layers (w.r.t. incoming light):light):

• ganglion layerganglion layer

• inner synaptic inner synaptic plexiform layerplexiform layer

• inner nuclear layerinner nuclear layer

• outer synaptic outer synaptic plexiform layerplexiform layer

• outer layerouter layer

B.2: The Retina—cellular makeup B.2: The Retina—cellular makeup (cont’d)(cont’d)B.2: The Retina—cellular makeup B.2: The Retina—cellular makeup (cont’d)(cont’d)

Fig.5 (cont’d): The neuron:Fig.5 (cont’d): The neuron:

• all retinal cells are types all retinal cells are types of neuronsof neurons

• certain neurons mimic a certain neurons mimic a “digital gate”, firing when “digital gate”, firing when activation level exceeds a activation level exceeds a thresholdthreshold

• rods and cones are rods and cones are specific types of specific types of dendritesdendrites

B.2: The Retina—retinogeniculate B.2: The Retina—retinogeniculate organization organization (from outside in, w.r.t. cortex)(from outside in, w.r.t. cortex)

B.2: The Retina—retinogeniculate B.2: The Retina—retinogeniculate organization organization (from outside in, w.r.t. cortex)(from outside in, w.r.t. cortex)

• Outer layerOuter layer: rods and cones: rods and cones

• Inner layerInner layer: horizontal cells, laterally : horizontal cells, laterally connected to photoreceptorsconnected to photoreceptors

• Ganglion layerGanglion layer: ganglion cells, connected : ganglion cells, connected (indirectly) to horizontal cells, project via (indirectly) to horizontal cells, project via the myelinated pathways, to the Lateral the myelinated pathways, to the Lateral Geniculate Nuclei (LGN) in the cortexGeniculate Nuclei (LGN) in the cortex

B.2: The Retina—receptive fieldsB.2: The Retina—receptive fieldsB.2: The Retina—receptive fieldsB.2: The Retina—receptive fields

• Receptive fields: collections of Receptive fields: collections of interconnected cells within the inner and interconnected cells within the inner and ganglion layersganglion layers

• Field organization determines impulse Field organization determines impulse signature of cells, based on cell typessignature of cells, based on cell types

• Cells may depolarize due to light Cells may depolarize due to light increments (+) or decrements (-)increments (+) or decrements (-)

B.2: The Retina—receptive fields B.2: The Retina—receptive fields (cont’d)(cont’d)B.2: The Retina—receptive fields B.2: The Retina—receptive fields (cont’d)(cont’d)

Fig.6: Receptive fields:Fig.6: Receptive fields:

• signal profile signal profile resembles a resembles a “Mexican hat”“Mexican hat”

• receptive field receptive field sizes vary sizes vary concentricallyconcentrically

• color-opposing color-opposing fields also existfields also exist

B.3: Visual PathwaysB.3: Visual PathwaysB.3: Visual PathwaysB.3: Visual Pathways

• Retinal ganglion cells project to the LGN Retinal ganglion cells project to the LGN along two major pathways, distinguished along two major pathways, distinguished by morphological cell types: by morphological cell types: and and cells cells cells project to the cells project to the magnocellularmagnocellular (M-) layers (M-) layers

cells project to the cells project to the parvocellularparvocellular (P-) layers (P-) layers

• Ganglion cells are functionally classified Ganglion cells are functionally classified by three types: X, Y, and W cellsby three types: X, Y, and W cells

B.3: Visual Pathways—functional B.3: Visual Pathways—functional response of ganglion cellsresponse of ganglion cellsB.3: Visual Pathways—functional B.3: Visual Pathways—functional response of ganglion cellsresponse of ganglion cells

• X cells: sustained stimulus, location, and X cells: sustained stimulus, location, and fine detailfine detail• nervate along both M- and P- projectionsnervate along both M- and P- projections

• Y cells: transient stimulus, coarse features, Y cells: transient stimulus, coarse features, and motionand motion• nervate along only the M-projectionnervate along only the M-projection

• W cells: coarse features and motionW cells: coarse features and motion• project to the Superior Colliculus (SC)project to the Superior Colliculus (SC)

B.3: Visual Pathways B.3: Visual Pathways (cont’d)(cont’d)B.3: Visual Pathways B.3: Visual Pathways (cont’d)(cont’d)

Fig.7: Optic tract and radiations Fig.7: Optic tract and radiations (visual pathways):(visual pathways):

• The LGN is of particular The LGN is of particular clinical importanceclinical importance

• M- and P-cellular M- and P-cellular projections are clearly projections are clearly visible under microscopevisible under microscope

• Axons from M- and P-layers Axons from M- and P-layers of the LGN terminate in of the LGN terminate in area V1area V1

B.3: Visual Pathways B.3: Visual Pathways (cont’d)(cont’d)B.3: Visual Pathways B.3: Visual Pathways (cont’d)(cont’d)

Table.1: Functional characteristics of ganglionic Table.1: Functional characteristics of ganglionic projectionsprojections

Characteristics Magno Parvoganglion size large smalltransmission time fast slowreceptive fields large smallsensitivity to small objects poor goodsensitivity to change in light levels large smallsensitivity to contrast low highsensitivity to motion high lowcolor discrimination no yes

B.4: The Occipital Cortex and B.4: The Occipital Cortex and BeyondBeyondB.4: The Occipital Cortex and B.4: The Occipital Cortex and BeyondBeyond

Fig.8: The brain Fig.8: The brain and visual and visual pathways:pathways:

• the cerebral the cerebral cortex is cortex is composed of composed of numerous numerous regions regions classified by classified by their functiontheir function

B.4: The Occipital Cortex and B.4: The Occipital Cortex and Beyond Beyond (cont’d)(cont’d)

B.4: The Occipital Cortex and B.4: The Occipital Cortex and Beyond Beyond (cont’d)(cont’d)

• M- and P- pathways terminate in distinct M- and P- pathways terminate in distinct layers of cortical area V1layers of cortical area V1

• Cortical cells (unlike center-surround Cortical cells (unlike center-surround ganglion receptive fields) respond to ganglion receptive fields) respond to orientation-specific stimulusorientation-specific stimulus

• Pathways emanating from V1 joining Pathways emanating from V1 joining multiple cortical areas involved in vision multiple cortical areas involved in vision are called are called streamsstreams

B.4: The Occipital Cortex and B.4: The Occipital Cortex and BeyondBeyond——directional selectivitydirectional selectivityB.4: The Occipital Cortex and B.4: The Occipital Cortex and BeyondBeyond——directional selectivitydirectional selectivity

• Cortical Directional Selectivity (CDS) of cells Cortical Directional Selectivity (CDS) of cells in V1 contributes to motion perception and in V1 contributes to motion perception and control of eye movementscontrol of eye movements

• CDS cells establish a motion pathway from CDS cells establish a motion pathway from V1 projecting to areas V2 and MT (V5)V1 projecting to areas V2 and MT (V5)

• In contrast, Retinal Directional Selectivity In contrast, Retinal Directional Selectivity (RDS) may not contribute to motion (RDS) may not contribute to motion perception, but is involved in eye movementsperception, but is involved in eye movements

B.4: The Occipital Cortex and B.4: The Occipital Cortex and BeyondBeyond——cortical cellscortical cellsB.4: The Occipital Cortex and B.4: The Occipital Cortex and BeyondBeyond——cortical cellscortical cells

• Two consequences of visual system’s Two consequences of visual system’s motion-sensitive, single-cell organization:motion-sensitive, single-cell organization:• due to motion sensitivity, eye movements are never due to motion sensitivity, eye movements are never

perfectly still (instead tiny jitter is observed, termed perfectly still (instead tiny jitter is observed, termed microsaccademicrosaccade)—if eyes were stabilized, image would )—if eyes were stabilized, image would fade!fade!

• due to single-cell organization, representation of due to single-cell organization, representation of natural images is quite abstract: there is no “retinal natural images is quite abstract: there is no “retinal buffer”buffer”

B.4: The Occipital Cortex and B.4: The Occipital Cortex and BeyondBeyond——2 attentional streams2 attentional streamsB.4: The Occipital Cortex and B.4: The Occipital Cortex and BeyondBeyond——2 attentional streams2 attentional streams

• Dorsal stream:Dorsal stream:• V1, V2, MT (V5), MST, Posterior Parietal CortexV1, V2, MT (V5), MST, Posterior Parietal Cortex

• sensorimotor (motion, location) processingsensorimotor (motion, location) processing

• the attentional “where”?the attentional “where”?

• Ventral (temporal) stream:Ventral (temporal) stream:• V1, V2, V4, Inferotemporal CortexV1, V2, V4, Inferotemporal Cortex

• cognitive processingcognitive processing

• the attentional “what”?the attentional “what”?

B.4: The Occipital Cortex and B.4: The Occipital Cortex and BeyondBeyond——3 attentional regions3 attentional regionsB.4: The Occipital Cortex and B.4: The Occipital Cortex and BeyondBeyond——3 attentional regions3 attentional regions

• Posterior Parietal Cortex (dorsal stream):Posterior Parietal Cortex (dorsal stream):• disengages attentiondisengages attention

• Superior Colliculus (midbrain):Superior Colliculus (midbrain):• relocates attentionrelocates attention

• Pulvinar (thalamus; colocated with LGN):Pulvinar (thalamus; colocated with LGN):• engages, or enhances, attentionengages, or enhances, attention

C: Visual Perception C: Visual Perception (with emphasis on foveo-(with emphasis on foveo-

peripheral distinction)peripheral distinction)C: Visual Perception C: Visual Perception (with emphasis on foveo-(with emphasis on foveo-

peripheral distinction)peripheral distinction)

• Measurable performance parameters may Measurable performance parameters may often (but not always!) fall within ranges often (but not always!) fall within ranges predicted by known limitations of the predicted by known limitations of the neurological substrateneurological substrate

• Example: visual acuity may be estimated by Example: visual acuity may be estimated by knowledge of density and distribution of the knowledge of density and distribution of the retinal photoreceptorsretinal photoreceptors

• In general, performance parameters are In general, performance parameters are obtained empiricallyobtained empirically

• Main parameters sought: visual acuity, Main parameters sought: visual acuity, contrast sensitivitycontrast sensitivity

• Dimensions of retinal features are measured Dimensions of retinal features are measured in terms of projected scene onto retina in in terms of projected scene onto retina in units of degrees visual angle,units of degrees visual angle,

where where SS is the object size and is the object size and DD is distance is distance

C.1: Spatial VisionC.1: Spatial VisionC.1: Spatial VisionC.1: Spatial Vision

D

SA

22arctan

C.1: Spatial Vision—visual angleC.1: Spatial Vision—visual angleC.1: Spatial Vision—visual angleC.1: Spatial Vision—visual angle

Fig.9: Visual angleFig.9: Visual angle

C.1: Spatial Vision—common visual C.1: Spatial Vision—common visual anglesanglesC.1: Spatial Vision—common visual C.1: Spatial Vision—common visual anglesangles

Table 2: Common visual anglesTable 2: Common visual angles

Object Distance Angle subtended

thumbnail arm’s length 1.5-2 degsun or moon - .5 degUS quarter coin arm’s length 2 degUS quarter coin 85 m 1 minUS quarter coin 5 km 1 sec

C.1: Spatial Vision—retinal regionsC.1: Spatial Vision—retinal regionsC.1: Spatial Vision—retinal regionsC.1: Spatial Vision—retinal regions

• Visual field: 180° horiz. Visual field: 180° horiz. 130° vert. 130° vert.• Fovea Centralis (foveola): highest acuityFovea Centralis (foveola): highest acuity

• 1.3° visual angle; 25,000 cones1.3° visual angle; 25,000 cones

• Fovea: high acuity (at 5°, acuity drops to 50%)Fovea: high acuity (at 5°, acuity drops to 50%)

• 5° visual angle; 100,000 cones5° visual angle; 100,000 cones

• Macula: within “useful” acuity region (to about 30°)Macula: within “useful” acuity region (to about 30°)

• 16.7° visual angle; 650,000 cones16.7° visual angle; 650,000 cones

• Hardly any rods in the foveal regionHardly any rods in the foveal region

C.1: Spatial Vision—visual angle and C.1: Spatial Vision—visual angle and receptor distributionreceptor distributionC.1: Spatial Vision—visual angle and C.1: Spatial Vision—visual angle and receptor distributionreceptor distribution

Fig.10: Retinotopic receptor distributionFig.10: Retinotopic receptor distribution

C.1: Spatial Vision—visual acuityC.1: Spatial Vision—visual acuityC.1: Spatial Vision—visual acuityC.1: Spatial Vision—visual acuity

Fig.11: Visual acuity at Fig.11: Visual acuity at eccentricities and light levels:eccentricities and light levels:

• at photopic (day) light levels, at photopic (day) light levels, acuity is fairly constant acuity is fairly constant within central 2°within central 2°

• acuity drops of linearly to 5°; acuity drops of linearly to 5°; drops sharply (exp.) beyonddrops sharply (exp.) beyond

• at scotopic (night) light at scotopic (night) light levels, acuity is poor at all levels, acuity is poor at all eccentricitieseccentricities

C.1: Spatial Vision—measuring C.1: Spatial Vision—measuring visual acuityvisual acuityC.1: Spatial Vision—measuring C.1: Spatial Vision—measuring visual acuityvisual acuity

• Acuity roughly corresponds to foveal Acuity roughly corresponds to foveal receptor distribution in the fovea, but not receptor distribution in the fovea, but not necessarily in the peripherynecessarily in the periphery

• Due to various contributing factors Due to various contributing factors (synaptic organization and later-stage (synaptic organization and later-stage neural elements), effective relative visual neural elements), effective relative visual acuity is generally measured by acuity is generally measured by psychophysical experimentationpsychophysical experimentation

C.2: Temporal VisionC.2: Temporal VisionC.2: Temporal VisionC.2: Temporal Vision

• Visual response to motion is characterized by Visual response to motion is characterized by two distinct facts: two distinct facts: persistence of visionpersistence of vision (POV) (POV) and the and the phi phenomenonphi phenomenon

• POV: essentially describes human temporal POV: essentially describes human temporal sampling ratesampling rate

• Phi: describes threshold above which humans Phi: describes threshold above which humans detect detect apparent movementapparent movement

• Both facts exploited in media to elicit motion Both facts exploited in media to elicit motion perceptionperception

C.2: Temporal Vision—persistence of C.2: Temporal Vision—persistence of visionvisionC.2: Temporal Vision—persistence of C.2: Temporal Vision—persistence of visionvision

Fig.12: Critical Fusion Frequency:Fig.12: Critical Fusion Frequency:

• stimulus flashing at about stimulus flashing at about 50-60Hz appears steady50-60Hz appears steady

• CFF explains why flicker is CFF explains why flicker is not seen when viewing not seen when viewing sequence of still imagessequence of still images

• cinema: 24 fps cinema: 24 fps 3 = 72Hz 3 = 72Hz due to 3-bladed shutterdue to 3-bladed shutter

• TV: 60 fields/sec, interlacedTV: 60 fields/sec, interlaced

C.2: Temporal Vision—phi C.2: Temporal Vision—phi phenomenonphenomenonC.2: Temporal Vision—phi C.2: Temporal Vision—phi phenomenonphenomenon

• Phi phenomenon explains why motion is Phi phenomenon explains why motion is perceived in cinema, TV, graphicsperceived in cinema, TV, graphics

• Besides necessary flicker rate (60Hz), Besides necessary flicker rate (60Hz), illusion of apparent, or stroboscopic, illusion of apparent, or stroboscopic, motion must be maintainedmotion must be maintained

• Similar to old-fashioned neon signs with Similar to old-fashioned neon signs with stationary bulbsstationary bulbs

• Minimum rate: 16 frames per secondMinimum rate: 16 frames per second

C.2: Temporal Vision—peripheral C.2: Temporal Vision—peripheral motion perceptionmotion perceptionC.2: Temporal Vision—peripheral C.2: Temporal Vision—peripheral motion perceptionmotion perception

• Motion perception is not homogeneous across Motion perception is not homogeneous across visual fieldvisual field

• Sensitivity to target motion decreases with Sensitivity to target motion decreases with retinal eccentricity for slow motion...retinal eccentricity for slow motion...• higher rate of target motion (e.g., spinning disk) is higher rate of target motion (e.g., spinning disk) is

needed to match apparent velocity in foveaneeded to match apparent velocity in fovea

• ……but, motion is more salient in periphery than but, motion is more salient in periphery than in fovea (easier to detect moving targets than in fovea (easier to detect moving targets than stationary ones)stationary ones)

C.2: Temporal Vision—peripheral C.2: Temporal Vision—peripheral sensitivity to direction of motionsensitivity to direction of motionC.2: Temporal Vision—peripheral C.2: Temporal Vision—peripheral sensitivity to direction of motionsensitivity to direction of motion

Fig.13: Threshold isograms for Fig.13: Threshold isograms for peripheral rotary movement:peripheral rotary movement:

• periphery is twice as periphery is twice as sensitive to horizontal-sensitive to horizontal-axis movement as to axis movement as to vertical-axis movementvertical-axis movement

• (numbers in diagram (numbers in diagram are rates of pointer are rates of pointer movement in rev./min.)movement in rev./min.)

C.3: Color Vision—cone typesC.3: Color Vision—cone typesC.3: Color Vision—cone typesC.3: Color Vision—cone types

Fig.14: Spectral sensitivity curves Fig.14: Spectral sensitivity curves of cone photoreceptorsof cone photoreceptors

• foveal color vision is foveal color vision is facilitated by three types of facilitated by three types of cone photorecptorscone photorecptors

• a good deal is known about a good deal is known about foveal color vision, relatively foveal color vision, relatively little is known about little is known about peripheral color visionperipheral color vision

• of the 7,000,000 cones, of the 7,000,000 cones, most are packed tightly into most are packed tightly into the central 30° foveal regionthe central 30° foveal region

C.3: Color Vision—peripheral color C.3: Color Vision—peripheral color perception fieldsperception fieldsC.3: Color Vision—peripheral color C.3: Color Vision—peripheral color perception fieldsperception fields

Fig.15: Visual fields for monocular Fig.15: Visual fields for monocular color vision (right eye)color vision (right eye)

• blueblue and and yellowyellow fields are fields are larger than larger than redred and and greengreen fieldsfields

• most sensitive to most sensitive to blueblue, up , up to 83°; to 83°; redred up to 76°; up to 76°; greengreen up to 74° up to 74°

• chromatic fields do not chromatic fields do not have definite borders, have definite borders, sensitivity gradually and sensitivity gradually and irregularly drops off over irregularly drops off over 15-30° range15-30° range

C.4: Implications for Design of C.4: Implications for Design of Attentional DisplaysAttentional DisplaysC.4: Implications for Design of C.4: Implications for Design of Attentional DisplaysAttentional Displays

• Need to consider distinct characteristics of Need to consider distinct characteristics of foveal and peripheral vision, in particular:foveal and peripheral vision, in particular:• spatial resolutionspatial resolution

• temporal resolutiontemporal resolution

• luminance / chrominanceluminance / chrominance

• Furthermore, Furthermore, gaze-contingentgaze-contingent systems must systems must match dynamics of human eye movementmatch dynamics of human eye movement

D: Taxonomy and Models of Eye D: Taxonomy and Models of Eye MovementsMovementsD: Taxonomy and Models of Eye D: Taxonomy and Models of Eye MovementsMovements

• Eye movements are mainly used to Eye movements are mainly used to reposition the foveareposition the fovea

• Five main classes of eye movements:Five main classes of eye movements:

• saccadicsaccadic

• smooth pursuitsmooth pursuit

• vergencevergence

• vestibularvestibular

• physiological nystagmusphysiological nystagmus

• (fixations)(fixations)

• Other types of movements are non-Other types of movements are non-positional positional (adaptation, accommodation)(adaptation, accommodation)

D.1: Extra-Ocular MusclesD.1: Extra-Ocular MusclesD.1: Extra-Ocular MusclesD.1: Extra-Ocular Muscles

Fig.16: Extrinsic muscles of the eyes:Fig.16: Extrinsic muscles of the eyes:

• in general, eyes move within 6 degrees of freedom (6 in general, eyes move within 6 degrees of freedom (6 muscles)muscles)

D.1: Oculomotor PlantD.1: Oculomotor PlantD.1: Oculomotor PlantD.1: Oculomotor Plant

Fig.17: Oculomotor system:Fig.17: Oculomotor system:

• eye movement signals eye movement signals emanate from three emanate from three main distinct regions:main distinct regions:

• occipital cortex (areas occipital cortex (areas 17, 18, 19, 22)17, 18, 19, 22)

• superior colliculus (SC)superior colliculus (SC)

• semicircular canals semicircular canals (SCC)(SCC)

D.1: Oculomotor Plant D.1: Oculomotor Plant (cont’d)(cont’d)D.1: Oculomotor Plant D.1: Oculomotor Plant (cont’d)(cont’d)

• Two pertinent observations:Two pertinent observations:1 eye movement system is, to a large extent, a eye movement system is, to a large extent, a

feedback circuitfeedback circuit

2 controlling cortical regions can be functionally controlling cortical regions can be functionally characterized as:characterized as:

• voluntary (occipital cortex—areas 17, 18, 19, 22)voluntary (occipital cortex—areas 17, 18, 19, 22)

• involuntary (superior colliculus, SC)involuntary (superior colliculus, SC)

• reflexive (semicircular canals, SCC)reflexive (semicircular canals, SCC)

D.2: SaccadesD.2: SaccadesD.2: SaccadesD.2: Saccades

• Rapid eye movements used to reposition Rapid eye movements used to reposition foveafovea

• Voluntary and reflexiveVoluntary and reflexive

• Range in duration from 10ms - 100msRange in duration from 10ms - 100ms

• Effectively blind during transitionEffectively blind during transition

• Deemed Deemed ballisticballistic (pre-programmed) and (pre-programmed) and stereotyped stereotyped (reproducible)(reproducible)

D.2: Saccades—modelingD.2: Saccades—modelingD.2: Saccades—modelingD.2: Saccades—modeling

Fig.18: Linear moving average filter model:Fig.18: Linear moving average filter model:

• sstt = input (pulse), = input (pulse), xxtt = output (step), = output (step), ggkk = filter coefficients = filter coefficients

• e.g., Haar filter {1,-1}e.g., Haar filter {1,-1}

0

110

kktk

ttt

sg

sgsgx

D.3: Smooth PursuitsD.3: Smooth PursuitsD.3: Smooth PursuitsD.3: Smooth Pursuits

• Involved when visually tracking a moving Involved when visually tracking a moving targettarget

• Depending on range of target motion, eyes Depending on range of target motion, eyes are capable of matching target velocityare capable of matching target velocity

• Pursuit movements are an example of a Pursuit movements are an example of a control system with built-in negative control system with built-in negative feedbackfeedback

D.3: Smooth Pursuits—modelingD.3: Smooth Pursuits—modelingD.3: Smooth Pursuits—modelingD.3: Smooth Pursuits—modeling

Fig.19: Linear, time-invariant filter model:Fig.19: Linear, time-invariant filter model:

• sstt = target position, = target position, xxtt = (desired) eye position, = (desired) eye position, hh = filter = filter

• retinal receptors give additive velocity errorretinal receptors give additive velocity error

1 ttt xxsh )(

D.4: NystagmusD.4: NystagmusD.4: NystagmusD.4: Nystagmus

• Conjugate eye movements characterized by Conjugate eye movements characterized by sawtooth-like time course pattern (pursuits sawtooth-like time course pattern (pursuits interspersed with saccades)interspersed with saccades)

• Two types (virtually indistinguishable):Two types (virtually indistinguishable):• Optokinetic: compensation for retinal movement of targetOptokinetic: compensation for retinal movement of target

• Vestibular: compensation for head movementVestibular: compensation for head movement

• May be possible to model with combination May be possible to model with combination of saccade/pursuit filtersof saccade/pursuit filters

D.5: FixationsD.5: FixationsD.5: FixationsD.5: Fixations

• Possibly the most important type of eye Possibly the most important type of eye movement for attentional applicationsmovement for attentional applications• 90% viewing time is devoted to fixations90% viewing time is devoted to fixations

• duration: 150ms - 600msduration: 150ms - 600ms

• Not technically eye movements in their Not technically eye movements in their own right, rather characterized by own right, rather characterized by miniature eye movements:miniature eye movements:• tremor, drift, microsaccadestremor, drift, microsaccades

D.6: Eye Movement AnalysisD.6: Eye Movement AnalysisD.6: Eye Movement AnalysisD.6: Eye Movement Analysis

• Two significant observations:Two significant observations:1 only three types of eye movements are mainly only three types of eye movements are mainly

needed to gain insight into overt localization of visual needed to gain insight into overt localization of visual attention:attention:

• fixationsfixations

• saccadessaccades

• smooth pursuits (to a lesser extent)smooth pursuits (to a lesser extent)

2 all three signals may be approximated by linear, all three signals may be approximated by linear, time-invariant (LTI) filter systemstime-invariant (LTI) filter systems

D.6: Eye Movement Analysis—D.6: Eye Movement Analysis—assumptionsassumptionsD.6: Eye Movement Analysis—D.6: Eye Movement Analysis—assumptionsassumptions

• Important point: it is assumed observed eye Important point: it is assumed observed eye movements disclose evidence of movements disclose evidence of overtovert visual visual attentionattention• it is possible to attend to objects it is possible to attend to objects covertlycovertly (without (without

moving eyes)moving eyes)

• Linearity: although practical, this assumption Linearity: although practical, this assumption is an operational oversimplification of is an operational oversimplification of neuronal (non-linear) systemsneuronal (non-linear) systems

D.6: Eye Movement Analysis—goalsD.6: Eye Movement Analysis—goalsD.6: Eye Movement Analysis—goalsD.6: Eye Movement Analysis—goals

• goal of analysis is to locate goal of analysis is to locate regions where signal regions where signal average changes abruptlyaverage changes abruptly• fixation end, saccade startfixation end, saccade start• saccade end, fixation startsaccade end, fixation start

• two main approaches:two main approaches:• summation-basedsummation-based• differentiation-baseddifferentiation-based

• both approaches rely on both approaches rely on empirical thresholdsempirical thresholds

Fig.20: Hypothetical eye movement signalFig.20: Hypothetical eye movement signal

D.6: Eye Movement Analysis—D.6: Eye Movement Analysis—denoisingdenoisingD.6: Eye Movement Analysis—D.6: Eye Movement Analysis—denoisingdenoising

Fig.21: Signal denoising—reduce noise due to:Fig.21: Signal denoising—reduce noise due to:• eye instability (jitter), or worse, blinkseye instability (jitter), or worse, blinks• removal possible based on device characteristics (e.g., blink = removal possible based on device characteristics (e.g., blink =

[0,0])[0,0])

D.6: Eye Movement Analysis—D.6: Eye Movement Analysis—summation basedsummation basedD.6: Eye Movement Analysis—D.6: Eye Movement Analysis—summation basedsummation based

• Dwell-time fixation detection depends on:Dwell-time fixation detection depends on:• identification of a stationary signal (fixation), andidentification of a stationary signal (fixation), and

• size of time window specifying range of duration size of time window specifying range of duration (and hence temporal threshold)(and hence temporal threshold)

• Example: Example: position-varianceposition-variance method: method:• determine whether determine whether MM of of NN points lie within a certain points lie within a certain

distance distance DD of the mean ( of the mean () of the signal) of the signal

• values values MM, , NN, and , and DD are determined empirically are determined empirically

D.6: Eye Movement Analysis—D.6: Eye Movement Analysis—differentiation baseddifferentiation basedD.6: Eye Movement Analysis—D.6: Eye Movement Analysis—differentiation baseddifferentiation based

• Velocity-based saccade/fixation detection:Velocity-based saccade/fixation detection:• calculated velocity (over signal window) is compared calculated velocity (over signal window) is compared

to thresholdto threshold

• if velocity > threshold then saccade, else fixationif velocity > threshold then saccade, else fixation

• Example: Example: velocity detectionvelocity detection method: method:• use short Finite Impulse Response (FIR) filters to use short Finite Impulse Response (FIR) filters to

detect saccade (may be possible in real-time)detect saccade (may be possible in real-time)

• assuming symmetrical velocity profile, can extend to assuming symmetrical velocity profile, can extend to velocity-based predictionvelocity-based prediction

D.6: Eye Movement Analysis D.6: Eye Movement Analysis (cont’d)(cont’d)D.6: Eye Movement Analysis D.6: Eye Movement Analysis (cont’d)(cont’d)

Fig.22: Saccade/fixation detectionFig.22: Saccade/fixation detection

(a) position-variance(a) position-variance (b) velocity-detection(b) velocity-detection

D.6: Eye Movement Analysis—D.6: Eye Movement Analysis—exampleexampleD.6: Eye Movement Analysis—D.6: Eye Movement Analysis—exampleexample

Fig.23: FIR filter velocity-detection Fig.23: FIR filter velocity-detection method based on idealized method based on idealized saccade detection:saccade detection:

• 4 conditions on measured 4 conditions on measured acceleration:acceleration:

maxmin

)()(

||

||

TIIT

ISgnISgn

BI

AI

12

12

2

1 • acc. > thresh. Aacc. > thresh. A

• acc. > thresh. Bacc. > thresh. B

• sign changesign change

• duration thresh.duration thresh.

• thresholds derived from thresholds derived from empirical valuesempirical values

D.6: Eye Movement Analysis—D.6: Eye Movement Analysis—example example (cont’d)(cont’d)

D.6: Eye Movement Analysis—D.6: Eye Movement Analysis—example example (cont’d)(cont’d)

• Amplitude thresholds Amplitude thresholds AA, , BB: derived from expected peak : derived from expected peak saccade velocities: 600°/ssaccade velocities: 600°/s

• Duration thresholds Duration thresholds TTminmin, , TTmaxmax: derived from expected : derived from expected

saccade duration: 120ms - 300mssaccade duration: 120ms - 300ms

Fig.24: FIR filters for saccade detectionFig.24: FIR filters for saccade detection