insights into high-level visual perception
TRANSCRIPT
![Page 1: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/1.jpg)
Jeff B. PelzVisual Perception Laboratory
Carlson Center for Imaging ScienceRochester Institute of Technology
Insights into High-level Visual PerceptionInsights into High-level Visual Perception
or “Where You Look is What You Get”or “Where You Look is What You Get”
![Page 2: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/2.jpg)
StudentsStudents
Roxanne Canosa (Ph.D. Imaging Science)
Jason Babcock (MS Color Science)
Eric Knappenberger (MS Imaging Science)
Dan Lerner (BS Imaging Science)
Marianne Lipps (BS Imaging Science)
![Page 3: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/3.jpg)
“Optical Illusions”
Reveal the shortcomings of the visual system, and our
best effort to make sense from incomplete information
![Page 4: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/4.jpg)
OutlineOutline
1. What are the fundamental limitations
of the visual system?
![Page 5: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/5.jpg)
OutlineOutline
2. What strategies are employed to
compensate for those limitations?
1. Fundamental limitations
![Page 6: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/6.jpg)
OutlineOutline
2. Strategies to compensate for limitations
3. Can we build tools that take advantage of
those strategies to inform the design and
evaluation of imaging systems?
1. Fundamental limitations
![Page 7: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/7.jpg)
OutlineOutline
2. Strategies to compensate for limitations
3. Build design and evaluation tools
1. Fundamental limitations
4. Can we use our understanding of the human visual system to aid design of next-generation computer vision systems?
![Page 8: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/8.jpg)
u Visual perception is a complex process thatunfolds over time, typically occurring at alevel below conscious awareness.
u People are often unaware of the details of howthey perform many tasks, including gatheringvisual information from the environment.
u By monitoring the eye movement patterns ofobservers as they perform a task, we can learnabout task strategy and performance.
IntroductionIntroduction
![Page 9: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/9.jpg)
Fundamental LimitationsFundamental Limitations
1. What are the fundamental limitations
of the visual system?
![Page 10: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/10.jpg)
![Page 11: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/11.jpg)
![Page 12: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/12.jpg)
There were evolutionary pressures for high-acuity vision (human as predator), and a widefield-of-view (human as prey).
The Design of the Visual SystemThe Design of the Visual System
![Page 13: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/13.jpg)
There were evolutionary pressures for high-acuity vision (human as predator), and a widefield-of-view (human as prey).
Even if the entire cortex were devoted to vision,there are not sufficient resources to represent alarge visual field at high acuity.
The Design of the Visual SystemThe Design of the Visual System
![Page 14: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/14.jpg)
The solution favored by nature representeda compromise between the two demands.
The foveal compromise makes use of:
A. Anisotropic sampling of the scene
B. Serial execution (task switching)
C. Limited internal representations
D. Focused attention
The The Foveal CompromiseFoveal Compromise
![Page 15: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/15.jpg)
The foveal compromise
High-acuity central fovea
Limited-acuity periphery
A. Anisotropic Sampling of the Visual FieldA. Anisotropic Sampling of the Visual Field
periphery center periphery
phot
orec
epto
r de
nsity
![Page 16: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/16.jpg)
If you can read this you must be cheating.+
Anisotropic Sampling of the Visual FieldAnisotropic Sampling of the Visual Field
![Page 17: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/17.jpg)
The visual field must be sampled by thehigh-acuity fovea:
If you can read this you must be cheating
The foveal compromise requires a mechanismfor moving the eyes about the scene.
Anisotropic Sampling of the Visual FieldAnisotropic Sampling of the Visual Field
![Page 18: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/18.jpg)
OutlineOutline
2. What strategies are employed to
compensate for those limitations?
1. Fundamental limitations
![Page 19: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/19.jpg)
Each eye has three agonist-antagonist muscle pairs torotate the eye horizontally,vertically, and about theoptical axis.
Foveal Compromise: Eye MovementsFoveal Compromise: Eye Movements
![Page 20: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/20.jpg)
Types of Eye MovementsTypes of Eye Movements
Smooth pursuit: match object motion
Vestibular-ocular response: compensate for self-motion
Vergence: merge images at different distances
Saccades: move fovea to new location
![Page 21: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/21.jpg)
Background: Eye Movement TypesBackground: Eye Movement Types
Smooth pursuit
Vestibular-ocular response
Vergence
Saccades - Image destabilization: shifts fovea to new image region
Imagestabilization
![Page 22: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/22.jpg)
u SaccadesAmplitude: < 1° → > 45° visual angle
Velocity: > 600°/secondFrequency: ~ 3-4/second (>150,000/day)
Saccades are made to targets requiring high spatial resolution and to the locus of attention.
Destabilizing Eye MovementsDestabilizing Eye Movements
![Page 23: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/23.jpg)
B. Serial Execution: Sequential SamplingB. Serial Execution: Sequential Sampling
![Page 24: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/24.jpg)
Serial Execution: Sequential SamplingSerial Execution: Sequential Sampling
![Page 25: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/25.jpg)
Serial Execution: Sequential SamplingSerial Execution: Sequential Sampling
![Page 26: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/26.jpg)
Serial Execution: Sequential SamplingSerial Execution: Sequential Sampling
![Page 27: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/27.jpg)
Serial Execution: Sequential SamplingSerial Execution: Sequential Sampling
![Page 28: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/28.jpg)
Serial Execution: FoveationsSerial Execution: Foveations
With each eye movement, the fovea ‘slidesunder’ a new portion of the retinal image.
A new portion of the image is sampled, buteach new sample is centered on the fovea
![Page 29: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/29.jpg)
Serial Execution: FoveationsSerial Execution: Foveations
![Page 30: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/30.jpg)
Serial Execution: FoveationsSerial Execution: Foveations
![Page 31: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/31.jpg)
Serial Execution: FoveationsSerial Execution: Foveations
![Page 32: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/32.jpg)
Serial Execution: FoveationsSerial Execution: Foveations
![Page 33: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/33.jpg)
C. Internal RepresentationC. Internal Representation
![Page 34: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/34.jpg)
B
A
If a high-acuity internal representation is built
up over multiple fixations, it should be easy to
detect even small differences between images.
Internal RepresentationInternal Representation
![Page 35: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/35.jpg)
Following are two versions of the school
children, separated by a blank slide.
There is a difference between the two;
your task is to identify the difference.
View them in alternation, trying to find
the difference. The difference is clearly
visible in the slide at the end.
Internal RepresentationInternal Representation
![Page 36: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/36.jpg)
A
View ~3 sec, then advanceView ~3 sec, then advance
![Page 37: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/37.jpg)
View ~1/2 sec, then continueView ~1/2 sec, then continue
![Page 38: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/38.jpg)
B
View ~3 sec, then REVERSEView ~3 sec, then REVERSE
![Page 39: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/39.jpg)
A
Compare to previous slideCompare to previous slide
![Page 40: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/40.jpg)
Something beyond variable acuity is responsible.
Deploying attention to different areas insequence conserves limited resources.
Changes to the scene can be made to unattendedregions without affecting conscious perception.
In nature, such changes usually induce apparentmotion, drawing attention to the region.
Limited Neural ResourcesLimited Neural Resources
![Page 41: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/41.jpg)
The limited acuity periphery must besampled by the high-acuity fovea,resulting in serial data acquisition.
The eye movements guiding thatacquisition are externally-observablemarkers of acuity demands, deploymentof attention, and perceptual strategies.
Serial Execution: Eye MovementsSerial Execution: Eye Movements
![Page 42: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/42.jpg)
Serial Execution; Image PreferenceSerial Execution; Image Preference
3 sec viewing
![Page 43: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/43.jpg)
OutlineOutline
2. Strategies to compensate for limitations
3. Can we build tools that take advantage of
those strategies to inform the design and
evaluation of imaging systems?
1. Fundamental limitations
![Page 44: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/44.jpg)
Measuring eye movementsMeasuring eye movements
The Problem:
“After all, the eye is sitting in a bag of fat in ahole in your head, and there are six big musclespulling on it.”
Cornsweet, 1976
![Page 45: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/45.jpg)
The Solution:
“Barlow photographed a droplet of mercury placedon the limbus. Translations of the head wereminimized by having subjects lie on a stone slabwith their heads wedged tightly inside a rigid ironframe”
Kowler, 1990
Measuring eye movementsMeasuring eye movements
![Page 46: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/46.jpg)
Measuring eye movementsMeasuring eye movements
![Page 47: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/47.jpg)
Measuring eye movementsMeasuring eye movements
Video-based eyetrackerLimbus eyetracker
![Page 48: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/48.jpg)
Measuring eye movementsMeasuring eye movements
Scleral eye-coils Dual Purkinje eyetracker
![Page 49: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/49.jpg)
Infrared / VideoHeadband-mounted eyetracker
Head-mounted Head-mounted eyetrackereyetracker
![Page 50: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/50.jpg)
Infrared, Video-based Infrared, Video-based EyetrackersEyetrackers
u Bright Pupil; On-axis Illumination
IRED
IRcamera
![Page 51: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/51.jpg)
Remote eyetrackerRemote eyetracker
Infrared / VideoRemote-head eyetracker
![Page 52: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/52.jpg)
Change BlindnessChange Blindness
![Page 53: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/53.jpg)
Human Computer InterfaceHuman Computer Interface
![Page 54: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/54.jpg)
= 250 ms
VisualizationVisualization
![Page 55: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/55.jpg)
![Page 56: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/56.jpg)
Image & Subject DependenceImage & Subject Dependence
![Page 57: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/57.jpg)
Radiographic Search: ScanpathRadiographic Search: Scanpath
![Page 58: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/58.jpg)
Radiographic Search: Fixation DensityRadiographic Search: Fixation Density
![Page 59: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/59.jpg)
Measuring eye movementsMeasuring eye movements
These commercially available eyetrackersare restricted to laboratory use.
The ability to monitor perception as peopleperform real tasks in the real world wouldallow us to ask new kinds of questions.
![Page 60: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/60.jpg)
RIT Wearable EyetrackerRIT Wearable Eyetracker
![Page 61: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/61.jpg)
color CMOS scene camera
calibration LASER
hot mirror
folding mirror
IR illuminator/optics module
monochrome CMOS eye camera
![Page 62: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/62.jpg)
RIT Wearable EyetrackerRIT Wearable Eyetracker
![Page 63: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/63.jpg)
![Page 64: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/64.jpg)
Fixation Sequence Before Image CaptureFixation Sequence Before Image Capture
![Page 65: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/65.jpg)
Complex, Familiar TasksComplex, Familiar Tasks
![Page 66: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/66.jpg)
OutlineOutline
2. Strategies to compensate for limitations
3. Build design and evaluation tools
1. Fundamental limitations
4. Can we use our understanding of the human
visual system to aid design of next-generation computer vision systems?
![Page 67: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/67.jpg)
Because vision is effortless for humans, computervision was chosen as an early research domain.
Early attempts at computer vision systems attackedthe problem by brute force with limited success:
Tried Image Understanding on static 2D images(“From Pixels to Predicates”)
MotivationMotivation
![Page 68: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/68.jpg)
Even in the face of Moore’s Law, computerswill not have sufficient power in the foreseeablefuture to solve “vision” by brute force.
LimitedLimited ComputationalComputational ResourcesResources
![Page 69: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/69.jpg)
Even in the face of Moore’s Law, computerswill not have sufficient power in the foreseeablefuture to solve “vision” by brute force.
Computer-based perception faces the samefundamental challenge that human perceptiondid during evolution:
limited computational resources
LimitedLimited ComputationalComputational ResourcesResources
![Page 70: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/70.jpg)
The solution favored by nature:
A. Anisotropic sampling of the scene
B. Serial execution (task switching)
C. Limited internal representations
D. Focused attention
The The Foveal CompromiseFoveal Compromise
![Page 71: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/71.jpg)
Sensorial Experience
High-level Visual Perception
Attentional Mechanisms
Eye Movements
MotivationMotivation: : CognitiveCognitive ScienceScience
Human Cognition
![Page 72: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/72.jpg)
Attentional Mechanisms
Eye Movements
Motivation: Cognitive ScienceMotivation: Cognitive Science
Artificial Intelligence
Computer Vision
“Active Vision”
Human Cognition
Sensorial Experience
High-level Visual Perception
![Page 73: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/73.jpg)
Inspiration - Inspiration - Active VisionActive Vision
Active vision was the first step. Unliketraditional approaches to computer vision,active vision systems focused on extractinginformation from dynamic, 3D scenes.
CS @ U PennVision & robotics @ UR
Aloimonos, 1987 Bajcsy, 1988
Ballard, 1989 Brooks, 1991
![Page 74: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/74.jpg)
Active VisionActive Vision
Inspired by anisotropic, binocular vision inhumans, researchers built neuromorphicvision systems that took advantage of‘active’ cameras.
Humanoid robotics @ MITVision & robotics @ UR
![Page 75: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/75.jpg)
InspirationInspiration - “ - “ActiveActive VisionVision””
Visual routines were an important component
of the Active Vision approach. Pre-defined
routines are scheduled and run to extract
information when and where it is needed.
![Page 76: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/76.jpg)
Limited representation + task-switching
Deploying attention and eye movements arecontrolled below conscious awareness; theremust be mechanisms (strategies) that protectus from the constraints of visual perception inthe real world - that help us make sense fromthe incomplete data available.
PerceptualPerceptual StrategiesStrategies
![Page 77: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/77.jpg)
Beyond the mechanics of how the eyesmove during real tasks, we are interested instrategies that may support the consciousperception that is continuous temporally aswell as spatially.
PerceptualPerceptual strategiesstrategies
![Page 78: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/78.jpg)
GoalGoal - “ - “StrategicStrategic VisionVision””
Strategic Vision can use high-level, top-down strategies for extracting informationfrom complex environments.
![Page 79: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/79.jpg)
GoalGoal - “ - “StrategicStrategic VisionVision””
Strategic Vision can use high-level, top-down strategies for extracting informationfrom complex environments.
One goal of our research is to study humanbehavior in natural, complex tasks to searchfor visual routines that emerge under real-world constraints.
![Page 80: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/80.jpg)
Perceptual StrategiesPerceptual Strategies
![Page 81: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/81.jpg)
LimitedLimited representationsrepresentations: Successive Foveations: Successive Foveations
![Page 82: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/82.jpg)
LimitedLimited representationsrepresentations: Successive Foveations: Successive Foveations
0 msec
![Page 83: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/83.jpg)
LimitedLimited representationsrepresentations: Successive Foveations: Successive Foveations
770 msec
![Page 84: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/84.jpg)
LimitedLimited representationsrepresentations: Successive Foveations: Successive Foveations
1400 msec
![Page 85: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/85.jpg)
LimitedLimited representationsrepresentations: Successive Foveations: Successive Foveations
2000 msec
![Page 86: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/86.jpg)
LimitedLimited representationsrepresentations: Successive Foveations: Successive Foveations
2700 msec
![Page 87: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/87.jpg)
LimitedLimited representationsrepresentations: Successive Foveations: Successive Foveations
2800 msec
![Page 88: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/88.jpg)
guiding fixation look-ahead fixation interaction
2000 msec 800 msec
Perceptual Strategies: Perceptual Strategies: Look-ahead Look-ahead fixationsfixations
. . .
Intervening tasks
![Page 89: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/89.jpg)
0 5000
Sub-tasks
Fixations
milliseconds
Interposed look-ahead
2000 7000milliseconds
Sequenced look-ahead
Sub-tasks
Fixations
Perceptual Strategies: Perceptual Strategies: Look-ahead Look-ahead fixationsfixations
![Page 90: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/90.jpg)
Humans employ strategies to ease thecomputational and memory loads inherent incomplex tasks. Look-ahead fixationsrepresent one such strategy:
Opportunistic execution of information-gathering visual routines to pre-fetchinformation needed for future subtasks.
Perceptual Strategies: Perceptual Strategies: Look-ahead Look-ahead fixationsfixations
![Page 91: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/91.jpg)
u Monitoring eye movements gives us a windowinto perception and cognition that can revealdetails not available even to the observer.
u Visual Strategies observed can help usunderstand how people use vision in theirinteraction with the world, and perhaps aid inthe design of artificial systems that takeadvantage of this knowledge.
ConclusionsConclusions
![Page 92: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/92.jpg)
ConclusionsConclusions
Tools that monitor subjects’ eye movementscan aid in the design and evaluation of imagingsystems.
The design of next-generation computer visionsystems may be aided by implementing algorithmsderived by understanding the strategies employedby the human visual system to compensate forlimited computational resources.
![Page 93: Insights into High-level Visual Perception](https://reader031.vdocument.in/reader031/viewer/2022012113/61dcd6c7cb2ac97b1e2e6faf/html5/thumbnails/93.jpg)
Questions?Questions?