stanford cs223b computer vision, winter 2007 lecture 6 advanced stereo professors sebastian thrun...

Stanford CS223B Computer Vision, Winter 2007

Lecture 6 Advanced Stereo

Professors Sebastian Thrun and Jana Košecká

CAs: Vaibhav Vaish and David Stavens

StereoStereo

Sebastian Thrun and Jana Košecká CS223B Computer Vision, Winter 2007

Summary Stereo Epipolar Geometry Fundamental/Essential Matrix

Ol Orel er

Epipolar Plane

Epipolar Lines

Epipoles

0lTr Epp

Correspondence: Where to search?

Search window?

Stereo Vision 2: Outline Image Rectification Correspondence Active Stereo Dense and Layered Stereo Smoothing With Markov Random Fields

Rectification Problem: Epipolar lines not parallel to scan lines

Ol Orel er

Epipolar Plane

Epipolar Lines

Epipoles

Rectification Problem: Epipolar lines not parallel to scan lines

Epipolar Plane

Epipolar Lines

Epipoles at ininity

Rectified Images

Epipolar Rectified Stereo Images

Epipolar line

Epipolar Rectified Images

Source: A. Fusiello, Verona, 2000]

Example Rectification

Final Step: Image Normalization

Even when the cameras are identical models, there can be differences in gain and sensitivity.

The cameras do not see exactly the same surfaces, so their overall light levels can differ.

For these reasons and more, it is a good idea to normalize the pixels in each window:

pixel Normalized ),(

magnitude Window )],([

pixel Average ),(

),(),(

),(),(),(

yxWvuyxW

IyxIyxI

Correspondence

Phantom points

Correspondence via Correlation

Rectified images

Left Right

scanline

SSD error

disparity

(Same as max-correlation / max-cosine for normalized image patch)

Images as Vectors

Left Right

Each window is a vectorin an m2 dimensionalvector space.Normalization makesthem unit length.

Image Metrics

Lw)(dwR

),(),(

)],(ˆ),(ˆ[)(

vduIvuIdC

yxWvuRL

(Normalized) Sum of Squared Differences

Normalized Correlation

),(ˆ),(ˆ)(),(),(

vduIvuIdC

yxWvuRL

)(maxarg)(minarg2* dwwdwwd RLdRLd

Correspondence Using Correlation

Left Disparity Map

Images courtesy of Point Grey Research

LEFT IMAGE

corner line

structure

Correspondence By Features

RIGHT IMAGE

corner line

structure

Search in the right image… the disparity (dx, dy) is the displacement when the similarity measure is maximum

Stereo Correspondences

… …Left scan line Right scan line

Stereo Correspondences

… …Left scanline Right scanline

MatchOcclusion Disocclusion

Search Over Correspondences

Three cases:–Sequential – cost of match–Occluded – cost of no match–Disoccluded – cost of no match

Left scanline

Right scanline

Occluded Pixels

Disoccluded Pixels

Scan across grid computing optimal cost for each node given its upper-left neighbors.Backtrack from the terminal to get the optimal path.

Occluded Pixels

Left scanline

Dis-occluded Pixels

Right scanline

Terminal

Stereo Matching with Dynamic Programming

DP Algorithm:

V[0,0] = 0

V[i,k] = min { V[i-1,k-1] + m(i,j), c+V[i, k-1], c+V[i-1,k] }d[i,k] = argmin { … }

Occluded Pixels

Left scanline

Dis-occluded Pixels

Right scanline

Terminal

Occluded Pixels

Left scanline

Dis-occluded Pixels

Right scanline

Terminal

DP Algorithm:

V[0,0] = 0

V[i,k] = min { V[i-1,k-1] + m(i,j), c+V[i, k-1], c+V[i-1,k] }d[i,k] = argmin { … }

Find Stereo Alignment

D=[X,Y]repeat until D=[1,1] add D to alignment D = d[D]end

Occluded Pixels

Left scanline

Dis-occluded Pixels

Right scanline

Terminal

Commercial-Grade Stereo Tyzx, a leading stereo camera manufacturer (here strapped on our DARPA Grand Challenge vehicle)

Disparity map

Dense Stereo Matching: Examples

View extrapolation results

input depth image novel view [Matthies,Szeliski,Kanade’88]

Dense Stereo Matching Some other view extrapolation results

input depth image novel view

Dense Stereo Matching Compute certainty map from correlations

input depth map certainty map

DP for Correspondence Does this always work? When would it fail?

– Failure Example 1– Failure Example 2– Failure Example 3– Failure Example 4

Correspondence Problem 1 Ambiguities

Correspondence Problem 2 Multiple occluding objects

Figure fromForsyth & Ponce

Correspondence Problem 3 Correspondence fail for smooth surfaces (edge =

occlusion boundary, poorly localized)

There is currently no good solution to this correspondence problem

Correspondence Problem 4

Regions without texture Highly Specular surfaces Translucent objects

Some Stereo Results

Side view

Top view

More Stereo Results

Side view

A True Challenge!

http://www.well.com/user/jimg/stereo/stereo_list.html

How can We Improve Stereo?

Space-time stereo scanneruses unstructured light to aidin correspondence

Result: Dense 3D mesh (noisy)

Active Stereo: Adding Texture to Scene

By James Davis, Honda Research,

Now UCSC

Active Stereo (Structured Light)

Structured Light: 3-D Result

3D Snapshot

By James Davis, Honda Research

Time of Flight Sensor: Shutter

http://www.3dvsystems.com

Scanning Laser Range Finders

Scanning Laser Results

Layered Stereo Assign pixel to different “layers” (objects, sprites)

Layered Stereo

Track each layer from frame to frame, compute plane eqn. and composite mosaic

Re-compute pixel assignment by comparing original images to sprites

Layered Stereo Re-synthesize original or novel images from

collection of sprites

Layered Stereo Advantages:

– can represent occluded regions– can represent transparent and border (mixed) pixels

(sprites have alpha value per pixel)– works on texture-less interior regions

Limitations:– fails for high depth-complexity scenes

Fitting Planar Surfaces (with EM)

******

Expectation Maximization 3D Model:

},,,{ 21 J

3, jjj Planar surface in 3D

jijij zz ),dist(

Distance point-surfacesurface

surface normal a

displacement b

Mixture Measurement Model Case 1: Measurement zi caused by plane j

ji ezp

§ Case 2: Measurement zi caused by something else

Measurement Model with Correspondences

Ji eccczp 12

1),,,,|(

correspondence variables C:

}1,0{,

jijiji

Expected Log-Likelihood Function

jijiji

)|,(ln

…after some simple math

mapping with known data association

probabilistic data association

The EM Algorithm

E-step: given plane params, compute

M-step: given expectations, compute

jijijc

zcECZpE

2)(][const)|,(ln

][ ijcE

},{ jja

Choosing the “Right” Number of Planes: AIC

J=2 J=3 J=5J=0 J=1 J=4

increased data likelihood

increased prior probability

)(log)|(log)|(log JpJdpconstdJp

Determining Number of Surfaces

First model component

E-Step

Add model components

E-Step

M-step

Prune model

Add model components

E/M Steps

Prune model

Layered Stereo Resulting sprite collection

Layered Stereo

Estimated depth map

Example (here with laser range finder)

Another Example

Motivation and Goals

James Diebel

Motivation and Goals

James Diebel

Network of Constraints (Markov Random Field)

Vertex Node

Edge Node

Face Node

Vertex Node

Edge Node

Face Node

Vertex Node

Edge Node

Face Node

DirectionsDirectionsDirections

James Diebel

MRF Approach to Smoothing Potential function: contains a sensor-model term

and a surface prior

The edge potential is important! Minimize by conjugate gradient

– Optimize systems with tens of thousands of parameters in just a couple seconds

– Time to converge is O(N), between 0.7 sec (25,000 nodes in the MRF) and 25 sec (900,000 nodes)

ii nnxxxx 2100 1

Diebel/Thrun, 2006

Possible Edge Potential Functions

Results: Smoothing

James Diebel

Results: Smoothing

James Diebel

Results: Smoothing

James Diebel

Results: Smoothing

James Diebel

Movies…

Movies in Windows Media Player

Summary Image Rectification Correspondence Active Stereo Dense and Layered Stereo Smoothing With Markov Random Fields

stanford cs223b computer vision, winter 2007 lecture 6 advanced stereo professors sebastian thrun...

stereo vision

correspondencesebastian

maximumsebastian thrun

featuressebastian thrun

jana koeckcas

image normalizationeven

search window

layered stereosmoothing

Documents

submitted by: bhaskar vaish - computer...

taxation of charitable trusts mr. rohit jain vaish...

(aqar) - vaish college of education | vaish b.ed. | vce ·...

vaish college of education, rohtak 2008-2009.pdfvaish...

stanford cs223b computer vision, winter 2007 lecture 12...

satwinder singh partner, vaish associates advocates ... ·...

stanford cs223b computer vision, winter 2005 lecture 3 ...

documents.pdfvaish college + vaish college of education...

sebastian thrun cs223b computer vision, winter 2005 1...

structure from motion introduction to computer vision...

stereo sebastian thrun, gary bradski, daniel russakoff...

relevance = revenue - pk vaish, copernica

healthy eating sheetal vaish and jessica leigh fleming

cs223b, jana kosecka rigid body motion and image formation

1 integrating vision models for holistic scene understanding...

sebastian thrun and jana kosecha cs223b computer vision,...

image stitching on a camera cell phone t. brow, s....

introduction to opencv - sebastian...

image stitching and panoramas stanford cs223b computer...

active contours / planes sebastian thrun, gary bradski,...