stanford cs223b computer vision, winter 2007 lecture 6 advanced stereo professors sebastian thrun...

Stanford CS223B Computer Vision, Winter 2007

Lecture 6 Advanced Stereo

Professors Sebastian Thrun and Jana Košecká

CAs: Vaibhav Vaish and David Stavens

StereoStereo

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007

Summary Stereo Epipolar Geometry Fundamental/Essential Matrix

plp

r

P

Ol Orel er

Pl Pr

Epipolar Plane

Epipolar Lines

Epipoles

0lTr Epp


Correspondence: Where to search?

Search window?


Stereo Vision 2: Outline Image Rectification Correspondence Active Stereo Dense and Layered Stereo Smoothing With Markov Random Fields


Rectification Problem: Epipolar lines not parallel to scan lines

plp

r

P

Ol Orel er

Pl Pr

Epipolar Plane

Epipolar Lines

Epipoles


Rectification Problem: Epipolar lines not parallel to scan lines

plp

r

P

Ol Or

Pl Pr

Epipolar Plane

Epipolar Lines

Epipoles at ininity

Rectified Images


Epipolar Rectified Stereo Images

Epipolar line


Epipolar Rectified Images

Source: A. Fusiello, Verona, 2000]


Example Rectification


Final Step: Image Normalization

Even when the cameras are identical models, there can be differences in gain and sensitivity.

The cameras do not see exactly the same surfaces, so their overall light levels can differ.

For these reasons and more, it is a good idea to normalize the pixels in each window:

pixel Normalized ),(

),(ˆ

magnitude Window )],([

pixel Average ),(

),(

),(),(

2

),(

),(),(),(

1

yxW

yxWvuyxW

yxWvuyxW

m

mm

m

m

II

IyxIyxI

vuII

vuII


Correspondence

1P1Oy

x

z

f

2Oy

x

z

1.lp

1,rp

1P

Phantom points


Correspondence via Correlation

Rectified images

Left Right

scanline

SSD error

disparity

(Same as max-correlation / max-cosine for normalized image patch)


Images as Vectors

Left Right

LwRw

Each window is a vectorin an m2 dimensionalvector space.Normalization makesthem unit length.


Image Metrics

Lw)(dwR

2

),(),(

2SSD

)(

)],(ˆ),(ˆ[)(

dww

vduIvuIdC

RL

yxWvuRL

m

(Normalized) Sum of Squared Differences

Normalized Correlation

cos)(

),(ˆ),(ˆ)(),(),(

NC

dww

vduIvuIdC

RL

yxWvuRL

m

)(maxarg)(minarg2* dwwdwwd RLdRLd


Correspondence Using Correlation

Left Disparity Map

Images courtesy of Point Grey Research


LEFT IMAGE

corner line

structure

Correspondence By Features


Correspondence By Features

RIGHT IMAGE

corner line

structure

Search in the right image… the disparity (dx, dy) is the displacement when the similarity measure is maximum


Stereo Correspondences

… …Left scan line Right scan line


Stereo Correspondences

… …Left scanline Right scanline

Match

Match

MatchOcclusion Disocclusion


Search Over Correspondences

Three cases:–Sequential – cost of match–Occluded – cost of no match–Disoccluded – cost of no match

Left scanline

Right scanline

Occluded Pixels

Disoccluded Pixels


Scan across grid computing optimal cost for each node given its upper-left neighbors.Backtrack from the terminal to get the optimal path.

Occluded Pixels

Left scanline

Dis-occluded Pixels

Right scanline

Terminal

Stereo Matching with Dynamic Programming


DP Algorithm:

V[0,0] = 0

V[i,k] = min { V[i-1,k-1] + m(i,j), c+V[i, k-1], c+V[i-1,k] }d[i,k] = argmin { … }

Occluded Pixels

Left scanline

Dis-occluded Pixels

Right scanline

Terminal



Occluded Pixels

Left scanline

Dis-occluded Pixels

Right scanline

Terminal


DP Algorithm:

V[0,0] = 0

V[i,k] = min { V[i-1,k-1] + m(i,j), c+V[i, k-1], c+V[i-1,k] }d[i,k] = argmin { … }


Find Stereo Alignment

D=[X,Y]repeat until D=[1,1] add D to alignment D = d[D]end

Occluded Pixels

Left scanline

Dis-occluded Pixels

Right scanline

Terminal



Commercial-Grade Stereo Tyzx, a leading stereo camera manufacturer (here strapped on our DARPA Grand Challenge vehicle)

Disparity map


Dense Stereo Matching: Examples

View extrapolation results

input depth image novel view [Matthies,Szeliski,Kanade’88]


Dense Stereo Matching Some other view extrapolation results

input depth image novel view


Dense Stereo Matching Compute certainty map from correlations

input depth map certainty map


DP for Correspondence Does this always work? When would it fail?

– Failure Example 1– Failure Example 2– Failure Example 3– Failure Example 4


Correspondence Problem 1 Ambiguities


Correspondence Problem 2 Multiple occluding objects

Figure fromForsyth & Ponce


Correspondence Problem 3 Correspondence fail for smooth surfaces (edge =

occlusion boundary, poorly localized)

There is currently no good solution to this correspondence problem


Correspondence Problem 4

Regions without texture Highly Specular surfaces Translucent objects


Some Stereo Results

Side view

Top view


More Stereo Results

Side view


A True Challenge!

http://www.well.com/user/jimg/stereo/stereo_list.html


How can We Improve Stereo?

Space-time stereo scanneruses unstructured light to aidin correspondence

Result: Dense 3D mesh (noisy)


Active Stereo: Adding Texture to Scene

By James Davis, Honda Research,

Now UCSC


rect

ified

Active Stereo (Structured Light)


Structured Light: 3-D Result

3D Snapshot

By James Davis, Honda Research


Time of Flight Sensor: Shutter

http://www.3dvsystems.com


Scanning Laser Range Finders


Scanning Laser Results


Layered Stereo Assign pixel to different “layers” (objects, sprites)


Layered Stereo

Track each layer from frame to frame, compute plane eqn. and composite mosaic

Re-compute pixel assignment by comparing original images to sprites


Layered Stereo Re-synthesize original or novel images from

collection of sprites


Layered Stereo Advantages:

– can represent occluded regions– can represent transparent and border (mixed) pixels

(sprites have alpha value per pixel)– works on texture-less interior regions

Limitations:– fails for high depth-complexity scenes


Fitting Planar Surfaces (with EM)

******


Expectation Maximization 3D Model:

},,,{ 21 J

3, jjj Planar surface in 3D

jijij zz ),dist(

Distance point-surfacesurface

surface normal a

y

x

z

displacement b


Mixture Measurement Model Case 1: Measurement zi caused by plane j

2

2)(

2

1

22

1)|(

jij z

ji ezp

2

2max

2ln

2

1

2max

*2

11)|(

z

i ez

zp

§ Case 2: Measurement zi caused by something else


Measurement Model with Correspondences

J

j

jijj

zc

zc

Ji eccczp 12

2

2

2max

*

)(

2ln

2

1

2*12

1),,,,|(

correspondence variables C:

}

J

jj

j

cc

cc

1*

*

1

}1,0{,

1

)(

2ln

2

1

2

12

2

2

2max

*

2

1),|(

i

zc

zc

J

j

jijiji

eCZp


Expected Log-Likelihood Function

1

)(

2ln

2

1

2

12

2

2

2max

*

2

1),|(

i

zc

zc

J

j

jijiji

eCZp

i

J

j

jijij

ic

zcE

zcE

J

CZpE

12

2

2

2max

*

2

)(][

2

1

2ln][

2

1

2)1(

1ln

)|,(ln

…after some simple math

mapping with known data association

probabilistic data association


The EM Algorithm

E-step: given plane params, compute

M-step: given expectations, compute

i

J

j

jijijc

zcECZpE

12

2)(][const)|,(ln

][ ijcE

},{ jja


Choosing the “Right” Number of Planes: AIC

J=2 J=3 J=5J=0 J=1 J=4

increased data likelihood

increased prior probability

)(log)|(log)|(log JpJdpconstdJp


Determining Number of Surfaces

J =1

First model component

*

*

J =1

E-Step

*

*

J =3

Add model components

J =3

E-Step

J =3

M-step

J =1

Prune model

J =3

Add model components

J =3

E/M Steps

*J =2

Prune model


Layered Stereo Resulting sprite collection


Layered Stereo

Estimated depth map


Example (here with laser range finder)


Example (here with laser range finder)

Another Example


Motivation and Goals

James Diebel


Network of Constraints (Markov Random Field)

Vertex Node

Edge Node

Face Node

Vertex Node

Edge Node

Face Node

Vertex Node

Edge Node

Face Node

DirectionsDirectionsDirections

James Diebel


MRF Approach to Smoothing Potential function: contains a sensor-model term

and a surface prior

The edge potential is important! Minimize by conjugate gradient

– Optimize systems with tens of thousands of parameters in just a couple seconds

– Time to converge is O(N), between 0.7 sec (25,000 nodes in the MRF) and 25 sec (900,000 nodes)

j

ji

iiiT

ii nnxxxx 2100 1

Diebel/Thrun, 2006


Possible Edge Potential Functions


Results: Smoothing

James Diebel


Movies…

Movies in Windows Media Player


Summary Image Rectification Correspondence Active Stereo Dense and Layered Stereo Smoothing With Markov Random Fields

stanford cs223b computer vision, winter 2007 lecture 6 advanced stereo professors sebastian thrun...

Documents

stereo vision

correspondencesebastian

maximumsebastian thrun

featuressebastian thrun

jana koeckcas

image normalizationeven

search window

layered stereosmoothing