computer vision aids for the blind and low-vision patients itai segall & ron merom advanced...

Computer Vision Aids for the Blind and Low-Vision Patients

Itai Segall & Ron Merom

Advanced Topics in Computer Vision Seminar

April 3rd, 2005

Introduction

180 Million people worldwide, who are visually disabled. 45 Million legally blind. [Vision 2020, 2000]

This number is expected to double by the year 2020. [Vision 2020, 2000]

Efforts are made in various fields to help people with visual impairments.

Types of Visual Impairments

Scotomas

Types of Visual Impairments

Scotomas CFL (Central Field Loss)

Types of Visual Impairments

Scotomas CFL (Central Field Loss) PFL (Peripheral Field Loss)

Types of Visual Impairments

Scotomas CFL (Central Field Loss) PFL (Peripheral Field Loss) Hemianopia

Types of Visual Impairments

Scotomas CFL (Central Field Loss) PFL (Peripheral Field Loss) Hemianopia Total Blindness

Lecture Outline

Studying the problem Suggested Solutions

– Eyewear– Enhancement of TV images– Navigation Aids

Studying the Problem

Example:

How Does the Visual System Deal with Scotomas ?

[D. Zur, S. Ullman, 2002]

What is a Scotoma?

Retinal scotomas can be caused by various diseases such as age-related macular degeneration (AMD)

“Visual scientists sometimes pass their time during a boring lecture by staring at a light on the ceiling until it produces a vivid afterimage. The afterimage can be used to blot out the lecturer’s head.”1

1 Morgan, M. “Making holes in the visual world”, 1999

Filling-in of Visual Patterns

Patients with small enough scotomas perceive the world as uninterrupted

Question: how does the visual system deal with missing information?– eye movements – ignored– filled in

Filling-in of Visual Patterns – cont.

Why study it?– Better understanding of the visual system– Study can lead to developing visual aids

Blind Spot – Extensively studied

Experiment

Subjects: patients with scotomas Show various visual patterns

– Short period of time (400ms)

Patients were asked to:– 1. Rate uniformity– 2. When designated as non-uniform,

choose: Blur Straightness Contrast

Results

Pattern Report

Results

Vs.

Results

Conclusions

• Missing information is filled-in, not ignored

• Higher density Better filling-in

• Higher regularity of stimulus Better filling-in

Lecture Outline

Studying the problem Suggested Solutions

– Eyewear– Enhancement of TV images– Navigation Aids

Eyewear – classical solutions

• Hemianopia – Binocular sector prisms• PFL-Minifying Devices

• CFL-Magnifying Devices

Eyewear

Problem: these solutions correct one problem while creating another one

Multiplexing approach: [Peli, 2001]– Combine a few information streams– But make sure they can be separated by the visual system

Types of multiplexing: – Temporal– Spatial– Bi-ocular– Composite

Temporal Multiplexing

Different signals at different times Healthy people use temporal multiplexing

Bioptic Telescope (for CFL)

Spatial Multiplexing

Show different information in different parts of the field of view

Micro-Telescope (for CFL)

Bi-Ocular Multiplexing

Expose each eye to different information May seem too confusing, but experiments

show patients adapt

Implantable Miniaturized Telescope (for CFL)

Composite Multiplexing

Devices that implement more than one type of multiplexing

Peripheral Monocular Prism (for Hemianopia)

Composite Multiplexing - cont

Composite Multiplexing - cont

Peripheral Monocular Prism combine:– Bi-ocular multiplexing– Spatial multiplexing– Spectral multiplexing

Composite Multiplexing 2

Minified Contours Augmented ViewA computer-aided device for PFL

Composite Multiplexing 2 - cont

Lecture Outline

Studying the problem Suggested Solutions

– Eyewear– Enhancement of TV images– Navigation Aids

Enhancement of TV Images

TV serves as an important medium for retrieving information, entertainment and education

Visual impairments make watching TV difficult

Enhancement of TV Images – cont.

Previous experiments: enhance high frequencies

But, studies show that the periphery is more sensitive to wideband enhancements CFL patients need a different solution

Idea: explicitly emphasize edges and bars in the image domain [Peli et al, 2004]

Enhancement of TV Images - cont.

First – detect edge & bars [Peli, 2002]: Use a visual system-based algorithm Morrone, Burr ’88: edges and bars are where

Fourier components come into phase with each other.

In order to find edges and bars, look for phase congruency =

Enhancement of TV Images – cont.

Simplified feature detection algorithm:– Find congruent polarities instead of congruent

phases of Fourier components

Algorithm for edge & bar detection

= + +

Apply bandpass filters

Binarize results

Algorithm for edge & bar detection

= + +

Apply bandpass filters

Binarize results

Find congruencies

Algorithm for edge & bar detection

=

Apply bandpass filters

Binarize results

Find congruencies

Algorithm for edge & bar detection

=

Apply bandpass filters

Binarize results

Find congruencies

Enhancement of TV Images – cont.

A more interesting example:

Wideband enhancement algorithm

Create feature map Substitute/Add map to original image Features can be weighted according to their

magnitude

Low Enhancement Level

Medium Enhancement Level

High Enhancement Level

Medium Enhancement Level

High Enhancement Level

Enhancement of TV Images – Experimental Results

Most CFL patients selected a slightly enhanced image

But… when asked to compare it to the original image, they didn’t find it to be much better

Why? – Any enhancement necessarily distorts the image– High contrast features were enhanced much more

than moderate ones

Lecture Outline

Studying the problem Suggested Solutions

– Eyewear– Enhancement of TV images– Navigation Aids

Navigation Aids

Classics: a cane & a guide dog Will discuss two solutions

– Specific – locate & recognize signs– General – first steps towards an “inter-sensory”

solution

Sign finding

“Talking Signs” Obvious problem: should be installed Suggested solution: Signfinder [Yuille et al., 1999]

– as an example, we’ll discuss (American) stop signs

What does it take to be a stop sign?

Being red and white? Being octagonal?

Then how to find stop signs?

Assumptions:– Two-colored– Stereotypically shaped– There exists a set of typical illuminants

Preprocessing – find this set

Multiplicative model: Observed color = true color X illuminant

Use a database of labeled signs to find typical illuminants

How?

Preprocessing: find set of typical illuminants

80 40 30

R G B

4 4 3

R G B

20 10 10

R G BX=

Preprocessing: find set of typical illuminants – cont.

Manually mark signs 2-means, for each marked sign

(Rr1, Gr

1, Br1) ; (Rw

1, Gw1, Bw

1)

(Rrn, Gr

n, Brn) ; (Rw

n, Gwn, Bw

n)

Preprocessing: find set of typical illuminants – cont.

Energy function:

Minimize using SVD Get a set of typical illuminants and “true red”, “true white”

R G BE E E E 2 2

, ,G r w r r w wE G G G G G G

- Green component of the illuminant in image α

rG - Green component of observed red in image α

rG - Green component of “true” red color

Where:

wG - Green component of “true” white color

Remember: Observed = True X Illuminant

Algorithm

Now that we have typical illuminants: Algorithm

– Find seed candidates in the image– Find the boundary of the sign– Align it to be fronto-parallel– Recognize it as a “stop sign”

Goal: find red and white windows

Finding seed candidates

(Observed) / (True white) = (Illuminant)

Finding seed candidates – cont.

Illuminance for re

d

Illuminance for white

N N N N N N N N

N N N N N N N N

N N N N N N N N

T T T N N N N N

T T T N N N N N

T T T N N N N N

T T T N N N N N

T T T N N N N N

T T T T T T T T

T T T T T T T T

T T T T T T T T

T T T T T T T T

N N N N T T T T

N N N N T T T T

N N N N T T T T

N N N N T T T T

T T T

T T T

T T T

T T T

T T T

T T T T T T T T

T T T T T T T T

T T T T T T T T

T T T T T T T T

T T T T

T T T T

T T T T

T T T T

Mean

Mean

(R1,G1,B1)

(R2,G2,B2)

≈?

Remember: Observed = True X Illuminant

R,G,B

Finding seed candidates – cont.

Seeds - results

Boundary detection

OK, so we have seeds – now we want to grow them and find the boundaries…

New pixel: close to red, white or neither

Boundary detection – cont.

What if the sign is partly shadowed?

Define a standard red as: R>128, G,B<0.8R

Start with seeds for which the red is non-standard

Then add pixels which are red or white with standard illuminant

Boundary detection – cont.

Problem: results do not yield straight and exact boundaries.

Idea: use a variant of Hough transform to find the edges

Boundary detection – cont.

1. Find center of mass of red pixels. Use this as the center of the image.

2.At each pixel, vote for s which split the neighborhood

3.Find the most popular edges

,d

Boundary detection – cont.

Boundary detection – cont.

Optimization: send rays from the center out, and look only at locations where these rays last contain red pixels

Aligning the sign

Usually the sign takes up a small part of the image a narrow field of view

affine transformation relates the sign and the

fronto-parallel prototype.

Aligning the sign – cont.

Define unknowns: – A, – The affine transformation– Vi,a – Does data corner i relate to model corner a?

And an energy function:

And minimize using EM algorithm Get affine transformation params and corner

matchings

b

i aia

ia

ta

diia VxbxAVbAVE )1(],,[

2

Matched corners transformedclosely to the model corners

Unmatched corners pay a penalty

Results

Recognizing the sign

Now, that’s trivial

Signfinder

Handles partly occluded and partly shadowed signs

What about different signs? Currently manufactured by Blindsight corp.

Navigation Aids 2 – An Inter-sensory Solution

An idea: when you can’t use your eyes, use your ears instead…

How would one transform an image to sound?

Grayscale Image

Left-Right

Up-Down

Brightness

Sound

Time

Pitch

Volume

[Stoerig et al., 2004]

Seeing through the ears

Example 1:

Example 2: What is this? Answer:

The last one: Answer:

But this is cacophony, can one really learn this?

“vOICe” experiment

Blindfolded BlindfoldedPracticesPractices

Geometric Natural

“vOICe” experiment – cont.

Results: – Geometric Images – no improvement

– Natural images – big difference

Blindfolded

Blindfolded, PracticesPractices

“vOICe” experiment – cont.

fMRI results:

Differences between blindfolded subjects

“vOICe” experiment – cont.

And one very interesting result:Day 8 Day 15 Day 21 Day 21

Could be a natural image, no idea what. Ominous, planes intermingle

Could be anything. Very heterogeneous; reminds me most of a plant

Plant A plant, no doubt. And a bar at the bottom

“I shall never forget the shock and joy of first glimpsing down my hallway and seeing blinds hanging on the window.”

Pat Fletcher

vOICe

Summary

Example of vision impairment research Solutions

– Eyewear devices that use multiplexing– Electronic image enhancement– Sign-finding and recognizing– Turning images to sounds