Finding Glass

Kenton McHenry

Jean Ponce

David Forsyth

Background

Layer Seperation (Szleski, Avidan, and Aniandan, CVPR'00),

(Levin, Zomet, and Weiss, CVPR'04)

3D Structure (Hata, Saitoh, Kumamura and Kaida, ICPR'96)

(Ben-Ezra and Nayar, ICCV'03)

(Miyazaki, Kagesawa and Ikeuchi, ICCV'03)

(Murase, ICCV'90)

Recognition (Osadchy, Jacobs, and Ramamoorthi, ICCV'03)

Segmentation (Singh and Huang, CVPR'03)

(Adelson and Anandan, AAAI'90)

I = IB+ e

0 < ≤ 1e ≥ 0

Classifying Junctions

Non-Reversing: transparency, ambiguous depth ordering

Double-Reversing: no transparency

Single-Reversing: transparency

(Singh and Huang, CVPR'03)

(Singh and Huang, CVPR'03)

Our Goal

The Background

The appearance of a glass object changes with

the background (i.e. the scene w/o any

transparent objects)We have seen how knowledge of the

background can be extremeley useful in

reconstructing transparent surfacesIdeal situation: know the background, use

background subtraction

Glass Objects and their Edges

Why?HighlightsMirrorsHysteresis

Adelson et al Revisited

Though they focus on junctions they are

classifying edgesThe proposed rules are binary cues between a

transparent object and its background

Proposed Method

Break edges into small segments and classify them

based on the information from the two sidesProperties of glass: transparency, refraction and

reflection

Cues

Transparency Color Similarity Overlay Consistency

Refraction Texture Distortion Blurring

Reflection Highlights

Color Similarity

(HSV) Hue(HSV) Saturation

Overlay Consistency

Texture Distortion

Filer Bank: 2

scales, 6

orientations (0,)

Blurring

DCTShift in mean in

frequency space

Highlights

Highlights on smooth

shiny surfaces tend to

have a profile with a

sharp spike (Healey and Binford, '87),

(Nayar, Ikeuchi and Kanade, '91)

Highlights

Iteratively fit a line to

perimeter (starting

from threshold of 1.0)Plot line fit errors

Highlights

Single Classifier

5 cues provide 6 valuesSVM with Gaussian kernelMust be conservative with false positives

Classifier can achieve high accuracy on training

data Move hyperplane until true positives < 30%

Multiple Classifiers

glass ⇐ similar_color ∧ high_alpha ∧ (low_emmission ∨ highlight ∨ smoother ∨ distortion)

If we were to consider the 6 values as logical

propositions we could write:

glass ⇐ similar_color ∧ high_alpha ∧ low_emmission

glass ⇐ similar_color ∧ high_alpha ∧highlight

glass ⇐ similar_color ∧ high_alpha ∧smoother

glass ⇐ similar_color ∧ high_alpha ∧distortion

Multiple Classifiers

We can re-write the previous statement as four

different statements of three propositions:

Multiple Classifiers

Each proposition is a seperatley trained

classifier of lower dimensionCombining the sub-classifiers

Logical OR Weighted Sum Exponential Model

Global Integration

Due to conservativeley built classifiers we will

have few positivesHysteresis: connect positves along a common

edgeSnakes

(Kass, Witkin, Terzopoulos, '87)

Experiments

Training Set: 15 images, 6 with glass objects in

front of various backgrounds, 9 with no glass

objects 333 positive examples 4581 negative examples

Test Set: 50 images, 35 with glass objects, 15

with no glass objects at all

Experiments

Single SVM

Multiple SVM's + OR

Multiple SVM's + Weighted Sum

Multiple SVM's + Exponential Model

Multiple SVM's + Weighted Sum (sampled)

Precision

68.76%

56.04%

58.78%

56.04%

73.70%

Results

Results

Results

Results

Classifying Regions as Glass

We need not restrict ourselves to regions

around edgesGiven two regions we ask the question “is one

region a glass covered version of the other?”

Over Segmentation

We want regions of similar material (Felzenszwalb and Huttenlocher, '04)

Can adjust size of super-pixels (degree of over-

segmentation) with smaller k valuesUse color, texture, and edgels to set weights

Discrepency

We use our previous classifier as a measure of

how much two regions don't belong two the

same material (i.e. glass and not glass) Use distance from seperating hyperplane (Platt, '00)

Large values: far on the postive glass side Small values (negative): far on the not glass side Reasonable if data takes a normal distribution

Drop blur cue since DCT can't be done on non-

rectangular regions.

Ambiguities

Discrepency is high for a material and a glass

covered version of that material, but also for two

completley different materialsAbove example has two possible segmentations

Affinity

Aij = 1 – a

ij /

Affinity

Because of refraction most straight background

edges that pass through the glass will appear

brokenEdges from glass contour ussually the longest

smoothest edges in the area

Affinity

Certainty of Discrepency/Affinity

High discrepency: likely different materialsLow discrepency: cannot ascertain whether

one regions is glass and the other is

backgroundHigh affinity: likely same materialLow affinity: not very informative, edge path

may just have been broken

Objective Function

We wish to maximize our measuresFirst term: maximize discrepency between

glass and other stuffSecond term: maximize affinity in the glassThird term: minimize affinities between glass

and otherCombinatorial problem!

Relaxed Objective Function

Relax region constraints Treat pixels as a sampling of an underlying

continuous function

Geodesic Active Contours

Curve Evolution

Experiments

Single SVM

Multiple SVM's + OR

Multiple SVM's + Weighted Sum

Multiple SVM's + Exponential Model

Multiple SVM's + Weighted Sum (sampled)

Proposed Method

Precision

68.76%

56.04%

58.78%

56.04%

73.70%

77.03%

Results

Results

Results

Results

Results