computer vision aids for the blind and low-vision patients itai segall & ron merom advanced...
TRANSCRIPT
Computer Vision Aids for the Blind and Low-Vision Patients
Itai Segall & Ron Merom
Advanced Topics in Computer Vision Seminar
April 3rd, 2005
Introduction
180 Million people worldwide, who are visually disabled. 45 Million legally blind. [Vision 2020, 2000]
This number is expected to double by the year 2020. [Vision 2020, 2000]
Efforts are made in various fields to help people with visual impairments.
Types of Visual Impairments
Scotomas
Types of Visual Impairments
Scotomas CFL (Central Field Loss)
Types of Visual Impairments
Scotomas CFL (Central Field Loss) PFL (Peripheral Field Loss)
Types of Visual Impairments
Scotomas CFL (Central Field Loss) PFL (Peripheral Field Loss) Hemianopia
Types of Visual Impairments
Scotomas CFL (Central Field Loss) PFL (Peripheral Field Loss) Hemianopia Total Blindness
Lecture Outline
Studying the problem Suggested Solutions
– Eyewear– Enhancement of TV images– Navigation Aids
Studying the Problem
Example:
How Does the Visual System Deal with Scotomas ?
[D. Zur, S. Ullman, 2002]
What is a Scotoma?
Retinal scotomas can be caused by various diseases such as age-related macular degeneration (AMD)
“Visual scientists sometimes pass their time during a boring lecture by staring at a light on the ceiling until it produces a vivid afterimage. The afterimage can be used to blot out the lecturer’s head.”1
1 Morgan, M. “Making holes in the visual world”, 1999
Filling-in of Visual Patterns
Patients with small enough scotomas perceive the world as uninterrupted
Question: how does the visual system deal with missing information?– eye movements – ignored– filled in
Filling-in of Visual Patterns – cont.
Why study it?– Better understanding of the visual system– Study can lead to developing visual aids
Blind Spot – Extensively studied
Experiment
Subjects: patients with scotomas Show various visual patterns
– Short period of time (400ms)
Patients were asked to:– 1. Rate uniformity– 2. When designated as non-uniform,
choose: Blur Straightness Contrast
Results
Pattern Report
Results
Vs.
Results
Conclusions
• Missing information is filled-in, not ignored
• Higher density Better filling-in
• Higher regularity of stimulus Better filling-in
Lecture Outline
Studying the problem Suggested Solutions
– Eyewear– Enhancement of TV images– Navigation Aids
Eyewear – classical solutions
• Hemianopia – Binocular sector prisms• PFL-Minifying Devices
• CFL-Magnifying Devices
Eyewear
Problem: these solutions correct one problem while creating another one
Multiplexing approach: [Peli, 2001]– Combine a few information streams– But make sure they can be separated by the visual system
Types of multiplexing: – Temporal– Spatial– Bi-ocular– Composite
Temporal Multiplexing
Different signals at different times Healthy people use temporal multiplexing
Bioptic Telescope (for CFL)
Spatial Multiplexing
Show different information in different parts of the field of view
Micro-Telescope (for CFL)
Bi-Ocular Multiplexing
Expose each eye to different information May seem too confusing, but experiments
show patients adapt
Implantable Miniaturized Telescope (for CFL)
Composite Multiplexing
Devices that implement more than one type of multiplexing
Peripheral Monocular Prism (for Hemianopia)
Composite Multiplexing - cont
Composite Multiplexing - cont
Peripheral Monocular Prism combine:– Bi-ocular multiplexing– Spatial multiplexing– Spectral multiplexing
Composite Multiplexing 2
Minified Contours Augmented ViewA computer-aided device for PFL
Composite Multiplexing 2 - cont
Lecture Outline
Studying the problem Suggested Solutions
– Eyewear– Enhancement of TV images– Navigation Aids
Enhancement of TV Images
TV serves as an important medium for retrieving information, entertainment and education
Visual impairments make watching TV difficult
Enhancement of TV Images – cont.
Previous experiments: enhance high frequencies
But, studies show that the periphery is more sensitive to wideband enhancements CFL patients need a different solution
Idea: explicitly emphasize edges and bars in the image domain [Peli et al, 2004]
Enhancement of TV Images - cont.
First – detect edge & bars [Peli, 2002]: Use a visual system-based algorithm Morrone, Burr ’88: edges and bars are where
Fourier components come into phase with each other.
In order to find edges and bars, look for phase congruency =
Enhancement of TV Images – cont.
Simplified feature detection algorithm:– Find congruent polarities instead of congruent
phases of Fourier components
Algorithm for edge & bar detection
= + +
Apply bandpass filters
Binarize results
Algorithm for edge & bar detection
= + +
Apply bandpass filters
Binarize results
Find congruencies
Algorithm for edge & bar detection
=
Apply bandpass filters
Binarize results
Find congruencies
Algorithm for edge & bar detection
=
Apply bandpass filters
Binarize results
Find congruencies
Enhancement of TV Images – cont.
A more interesting example:
Wideband enhancement algorithm
Create feature map Substitute/Add map to original image Features can be weighted according to their
magnitude
Low Enhancement Level
Medium Enhancement Level
High Enhancement Level
Medium Enhancement Level
High Enhancement Level
Enhancement of TV Images – Experimental Results
Most CFL patients selected a slightly enhanced image
But… when asked to compare it to the original image, they didn’t find it to be much better
Why? – Any enhancement necessarily distorts the image– High contrast features were enhanced much more
than moderate ones
Lecture Outline
Studying the problem Suggested Solutions
– Eyewear– Enhancement of TV images– Navigation Aids
Navigation Aids
Classics: a cane & a guide dog Will discuss two solutions
– Specific – locate & recognize signs– General – first steps towards an “inter-sensory”
solution
Sign finding
“Talking Signs” Obvious problem: should be installed Suggested solution: Signfinder [Yuille et al., 1999]
– as an example, we’ll discuss (American) stop signs
What does it take to be a stop sign?
Being red and white? Being octagonal?
Then how to find stop signs?
Assumptions:– Two-colored– Stereotypically shaped– There exists a set of typical illuminants
Preprocessing – find this set
Multiplicative model: Observed color = true color X illuminant
Use a database of labeled signs to find typical illuminants
How?
Preprocessing: find set of typical illuminants
80 40 30
R G B
4 4 3
R G B
20 10 10
R G BX=
Preprocessing: find set of typical illuminants – cont.
Manually mark signs 2-means, for each marked sign
(Rr1, Gr
1, Br1) ; (Rw
1, Gw1, Bw
1)
(Rrn, Gr
n, Brn) ; (Rw
n, Gwn, Bw
n)
Preprocessing: find set of typical illuminants – cont.
Energy function:
Minimize using SVD Get a set of typical illuminants and “true red”, “true white”
R G BE E E E 2 2
, ,G r w r r w wE G G G G G G
- Green component of the illuminant in image α
rG - Green component of observed red in image α
rG - Green component of “true” red color
Where:
wG - Green component of “true” white color
Remember: Observed = True X Illuminant
Algorithm
Now that we have typical illuminants: Algorithm
– Find seed candidates in the image– Find the boundary of the sign– Align it to be fronto-parallel– Recognize it as a “stop sign”
Goal: find red and white windows
Finding seed candidates
(Observed) / (True white) = (Illuminant)
Finding seed candidates – cont.
Illuminance for re
d
Illuminance for white
N N N N N N N N
N N N N N N N N
N N N N N N N N
T T T N N N N N
T T T N N N N N
T T T N N N N N
T T T N N N N N
T T T N N N N N
T T T T T T T T
T T T T T T T T
T T T T T T T T
T T T T T T T T
N N N N T T T T
N N N N T T T T
N N N N T T T T
N N N N T T T T
T T T
T T T
T T T
T T T
T T T
T T T T T T T T
T T T T T T T T
T T T T T T T T
T T T T T T T T
T T T T
T T T T
T T T T
T T T T
Mean
Mean
(R1,G1,B1)
(R2,G2,B2)
≈?
Remember: Observed = True X Illuminant
R,G,B
Finding seed candidates – cont.
Seeds - results
Boundary detection
OK, so we have seeds – now we want to grow them and find the boundaries…
New pixel: close to red, white or neither
Boundary detection – cont.
What if the sign is partly shadowed?
Define a standard red as: R>128, G,B<0.8R
Start with seeds for which the red is non-standard
Then add pixels which are red or white with standard illuminant
Boundary detection – cont.
Problem: results do not yield straight and exact boundaries.
Idea: use a variant of Hough transform to find the edges
Boundary detection – cont.
1. Find center of mass of red pixels. Use this as the center of the image.
2.At each pixel, vote for s which split the neighborhood
3.Find the most popular edges
,d
Boundary detection – cont.
Boundary detection – cont.
Optimization: send rays from the center out, and look only at locations where these rays last contain red pixels
Aligning the sign
Usually the sign takes up a small part of the image a narrow field of view
affine transformation relates the sign and the
fronto-parallel prototype.
Aligning the sign – cont.
Define unknowns: – A, – The affine transformation– Vi,a – Does data corner i relate to model corner a?
And an energy function:
And minimize using EM algorithm Get affine transformation params and corner
matchings
b
i aia
ia
ta
diia VxbxAVbAVE )1(],,[
2
Matched corners transformedclosely to the model corners
Unmatched corners pay a penalty
Results
Recognizing the sign
Now, that’s trivial
Signfinder
Handles partly occluded and partly shadowed signs
What about different signs? Currently manufactured by Blindsight corp.
Navigation Aids 2 – An Inter-sensory Solution
An idea: when you can’t use your eyes, use your ears instead…
How would one transform an image to sound?
Grayscale Image
Left-Right
Up-Down
Brightness
Sound
Time
Pitch
Volume
[Stoerig et al., 2004]
Seeing through the ears
Example 1:
Example 2: What is this? Answer:
The last one: Answer:
But this is cacophony, can one really learn this?
“vOICe” experiment
Blindfolded BlindfoldedPracticesPractices
Geometric Natural
“vOICe” experiment – cont.
Results: – Geometric Images – no improvement
– Natural images – big difference
Blindfolded
Blindfolded, PracticesPractices
“vOICe” experiment – cont.
fMRI results:
Differences between blindfolded subjects
“vOICe” experiment – cont.
And one very interesting result:Day 8 Day 15 Day 21 Day 21
Could be a natural image, no idea what. Ominous, planes intermingle
Could be anything. Very heterogeneous; reminds me most of a plant
Plant A plant, no doubt. And a bar at the bottom
“I shall never forget the shock and joy of first glimpsing down my hallway and seeing blinds hanging on the window.”
Pat Fletcher
vOICe
Summary
Example of vision impairment research Solutions
– Eyewear devices that use multiplexing– Electronic image enhancement– Sign-finding and recognizing– Turning images to sounds