lecture 8: stereo. depth from stereo goal: recover depth by finding image coordinate x’ that...
TRANSCRIPT
![Page 1: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/1.jpg)
Lecture 8: Stereo
![Page 2: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/2.jpg)
Depth from Stereo
• Goal: recover depth by finding image coordinate x’ that corresponds to x
f
x x’
BaselineB
z
C C’
X
f
X
x
x'
![Page 3: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/3.jpg)
Depth from Stereo• Goal: recover depth by finding image
coordinate x’ that corresponds to x• Problems
– Calibration: How do we recover the relation of the cameras (if not already known)?
– Correspondence: How do we search for the matching point x’?
f
x x’
BaselineB
z
C C’
X
f
X
x
x'
![Page 4: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/4.jpg)
Correspondence Problem
• We have two images taken from cameras at different positions
• How do we match a point in the first image to a point in the second? What constraints do we have?
![Page 5: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/5.jpg)
A pre-sequitor
• Fun with vectors.
Let a and b be two vectors.• What is a (dot) b?• What is a (cross) b?Quiz:• What is a (dot) a?
– (what is the data type?)
• What is a (cross) a?
![Page 6: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/6.jpg)
Computer Vision, Robert Pless
Recap, Camera Calibration.
• Projection from the world to the image:
1100
0
10
0
world
world
world
z
y
x
y
x
Z
Y
X
T
T
T
Ryf
xsf
y
x
Calibration Rotation Translation
Homogeneous equal, “equal up to a scale factor”
TPRKp
![Page 7: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/7.jpg)
Computer Vision, Robert Pless
Idea of today…
• Today we are going to characterize the geometry of how two cameras look at a scene… but perhaps a more complicated.
• And by scene, we mean, at first, just one point.
![Page 8: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/8.jpg)
Computer Vision, Robert Pless
Epipolar Constraint
There are 3 degrees of freedom in the position of a point in space; there are four DOF for image points in two planes… Where does that fourth DOF go?
![Page 9: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/9.jpg)
Computer Vision, Robert Pless
Epipolar Lines
Each point in one image corresponds to a line of possibilities in the other.
Red point - fixed
=> Blue point lies on a line
“Epipolar Line”
Potential 3d points
![Page 10: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/10.jpg)
Computer Vision, Robert Pless
Epipolar Geometry
An epipole is the image point of the other camera’s center.
=> Red point lies on a line
Blue point - fixed baseline
All epipolar lines meet at the epipoles. Epipoles lie on the cameras’ baseline.
Is there other structure available among epipolar lines?
![Page 11: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/11.jpg)
Computer Vision, Robert Pless
Another look (with math).
• We have two images, with a point in one and the epi-polar line in the other. Lets take away the image plane, and just leave the image centers.
![Page 12: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/12.jpg)
Computer Vision, Robert Pless
Another look (with math).
![Page 13: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/13.jpg)
Computer Vision, Robert Pless
Another look (with math).
a translation vector defining where one camera is relative to the other.
![Page 14: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/14.jpg)
Computer Vision, Robert Pless
Another look (with math).
a translation vector defining where one camera is relative to the other.
![Page 15: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/15.jpg)
Computer Vision, Robert Pless
Another look (with math).
A point on one image lies on ray in space with direction
![Page 16: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/16.jpg)
Computer Vision, Robert Pless
Another look (with math).
• Which rays from, the second camera center might intersect ray p?
![Page 17: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/17.jpg)
Computer Vision, Robert Pless
Another look (with math).
• Those rays lie in the plane defined by the ray in space and the second camera center.
![Page 18: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/18.jpg)
Computer Vision, Robert Pless
Another look (with math).
• normal of the plane is perpendicular to both p and t.
• Math fact: a x b is a vector perpendicular to a and b.
-
![Page 19: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/19.jpg)
Computer Vision, Robert Pless
Another look (with math).
• All lines in the plane are perpendicular to the normal to normal to the plane.
• Math fact. aTb = 0 if a is perpendicular to b
![Page 20: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/20.jpg)
Computer Vision, Robert Pless
Putting it all together.
• Cameras separated by translation t• Ray from one camera center in direction p• Ray from second camera center q
Fact:
![Page 21: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/21.jpg)
Computer Vision, Robert Pless
Lets put the images back in.
x
y
P is relative to some coordinate system.
![Page 22: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/22.jpg)
Computer Vision, Robert Pless
• q is relative to some coordinate system, but that camera may have rotated.• So the q in the first coordinate system is some rotation times the q measured in the
second coordinate system
y
x
x
y
![Page 23: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/23.jpg)
Computer Vision, Robert Pless
• All three vectors in the same plane:
yx
x
y
![Page 24: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/24.jpg)
Computer Vision, Robert Pless
Put images even more back in.
• K maps normalized coordinates onto pixel coordinates. Given pixel coordinates (x,y), K-1 remaps those to a direction from the camera center.
(x,y)
![Page 25: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/25.jpg)
Computer Vision, Robert PlessF is the “fundamental matrix”.
Normalized camera system, epipolar equation.
“Uncalibrated” Case, epipolar equation:
![Page 26: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/26.jpg)
Computer Vision, Robert Pless
Using the equation…
• Click on “the same world point” in the left and right image, to get a set of point correspondences:
(x,y) that correspond to (x’,y’).
• Need at least 8 points (each point gives one constraint, F is 3x3, but scale invariant, so there are 8 degrees of freedom in F).
![Page 27: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/27.jpg)
Computer Vision, Robert Pless
So what… how to use F
Red point - fixed
=> Blue point lies on a line
Given a point (x,y) on the left image, F defines the “Epipolar Line” and tells where the corresponding points must lie. How is that line defined? Only easy
in homogenous coordinates!
Potential 3d points
![Page 28: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/28.jpg)
Computer Vision, Robert Pless
Examples
http://www-sop.inria.fr/robotvis/personnel/sbougnou/Meta3DViewer/EpipolarGeo.html
![Page 29: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/29.jpg)
Computer Vision, Robert Pless
Examples
![Page 30: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/30.jpg)
Computer Vision, Robert Pless
Examples
Geometrically, why do all epipolar lines intersect?
![Page 31: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/31.jpg)
Estimating the Fundamental Matrix
• 8-point algorithm– Least squares solution using SVD on equations
from 8 pairs of correspondences– Enforce det(F)=0 constraint using SVD on F
• Minimize reprojection error– Non-linear least squares
![Page 32: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/32.jpg)
8-point algorithm
1. Solve a system of homogeneous linear equations
a. Write down the system of equations
0xx FT
![Page 33: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/33.jpg)
8-point algorithm
1. Solve a system of homogeneous linear equations
a. Write down the system of equationsb. Solve f from Af=0 using SVD
Matlab: [U, S, V] = svd(A);f = V(:, end);F = reshape(f, [3 3])’;
![Page 34: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/34.jpg)
Need to enforce singularity constraint
![Page 35: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/35.jpg)
8-point algorithm
1. Solve a system of homogeneous linear equations
a. Write down the system of equationsb. Solve f from Af=0 using SVD
2. Resolve det(F) = 0 constraint by SVD
Matlab: [U, S, V] = svd(A);f = V(:, end);F = reshape(f, [3 3])’;
Matlab: [U, S, V] = svd(F);S(3,3) = 0;F = U*S*V’;
![Page 36: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/36.jpg)
8-point algorithm1. Solve a system of homogeneous linear equations
a. Write down the system of equationsb. Solve f from Af=0 using SVD
2. Resolve det(F) = 0 constraint by SVD
Notes:• Use RANSAC to deal with outliers (sample 8 points)• Solve in normalized coordinates
– mean=0– RMS distance = (1,1,1)– This also help estimating the homography for stitching
![Page 37: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/37.jpg)
Comparison of homography estimation and the 8-point algorithm
Homography (No Translation) Fundamental Matrix (Translation)
Assume we have matched points x x’ with outliers
![Page 38: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/38.jpg)
Homography (No Translation)
• Correspondence Relation
• RANSAC with 4 points
0HxxHxx ''
Fundamental Matrix (Translation)
Comparison of homography estimation and the 8-point algorithm
Assume we have matched points x x’ with outliers
![Page 39: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/39.jpg)
Comparison of homography estimation and the 8-point algorithm
Homography (No Translation) Fundamental Matrix (Translation)
• Correspondence Relation
• RANSAC with 8 points• Enforce by
SVD
• Correspondence Relation
• RANSAC with 4 points
Assume we have matched points x x’ with outliers
0HxxHxx ''
0~
det F
0 Fxx T
![Page 40: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/40.jpg)
So
• Given 2 images. Can find the relative translation and rotations of the cameras. How do we find depth?
![Page 41: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/41.jpg)
Simplest Case: Parallel images• Image planes of cameras are
parallel to each other and to the baseline
• Camera centers are at same height
• Focal lengths are the same• Then, epipolar lines fall along
the horizontal scan lines of the images
![Page 42: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/42.jpg)
Depth from disparity
f
x’
BaselineB
z
O O’
X
f
z
fBxxdisparity
Disparity is inversely proportional to depth.
xz
f
OO
xx
![Page 43: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/43.jpg)
Stereo image rectification
![Page 44: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/44.jpg)
Stereo image rectification
• Reproject image planes onto a common plane parallel to the line between camera centers
• Pixel motion is horizontal after this transformation
• Two homographies (3x3 transform), one for each input image reprojection
C. Loop and Z. Zhang. Computing Rectifying Homographies for Stereo Vision. IEEE Conf. Computer Vision and Pattern Recognition, 1999.
![Page 45: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/45.jpg)
Rectification example
![Page 46: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/46.jpg)
Basic stereo matching algorithm
• If necessary, rectify the two stereo images to transform epipolar lines into scanlines
• For each pixel x in the first image– Find corresponding epipolar scanline in the right image– Examine all pixels on the scanline and pick the best match x’– Compute disparity x-x’ and set depth(x) = fB/(x-x’)
![Page 47: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/47.jpg)
Matching cost
disparity
Left Right
scanline
Correspondence search
• Slide a window along the right scanline and compare contents of that window with the reference window in the left image
• Matching cost: SSD or normalized correlation
![Page 48: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/48.jpg)
Left Right
scanline
Correspondence search
SSD
![Page 49: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/49.jpg)
Left Right
scanline
Correspondence search
Norm. corr
![Page 50: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/50.jpg)
Effect of window size
W = 3 W = 20
• Smaller window+ More detail– More noise
• Larger window+ Smoother disparity maps– Less detail
![Page 51: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/51.jpg)
Failures of correspondence search
Textureless surfaces Occlusions, repetition
Non-Lambertian surfaces, specularities
![Page 52: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/52.jpg)
Results with window search
Window-based matching Ground truth
Data
![Page 53: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/53.jpg)
How can we improve window-based matching?
• So far, matches are independent for each point
• What constraints or priors can we add?
![Page 54: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/54.jpg)
Stereo constraints/priors• Uniqueness
– For any point in one image, there should be at most one matching point in the other image
![Page 55: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/55.jpg)
Stereo constraints/priors• Uniqueness
– For any point in one image, there should be at most one matching point in the other image
• Ordering– Corresponding points should be in the same order in both
views
![Page 56: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/56.jpg)
Stereo constraints/priors• Uniqueness
– For any point in one image, there should be at most one matching point in the other image
• Ordering– Corresponding points should be in the same order in both
views
Ordering constraint doesn’t hold
![Page 57: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/57.jpg)
Priors and constraints• Uniqueness
– For any point in one image, there should be at most one matching point in the other image
• Ordering– Corresponding points should be in the same order in both
views• Smoothness
– We expect disparity values to change slowly (for the most part)
![Page 58: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/58.jpg)
Stereo matching as energy minimization
I1 I2 D
• Energy functions of this form can be minimized using graph cuts
Y. Boykov, O. Veksler, and R. Zabih, Fast Approximate Energy Minimization via Graph Cuts, PAMI 2001
W1(i ) W2(i+D(i )) D(i )
)(),;( smooth21data DEIIDEE 2
,neighborssmooth )()(
ji
jDiDE 2
21data ))(()( i
iDiWiWE
![Page 59: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/59.jpg)
Many of these constraints can be encoded in an energy function and solved using graph cuts
Graph cuts Ground truth
For the latest and greatest: http://www.middlebury.edu/stereo/
Y. Boykov, O. Veksler, and R. Zabih, Fast Approximate Energy Minimization via Graph Cuts, PAMI 2001
Before
![Page 60: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/60.jpg)
Summary
• Epipolar geometry– Epipoles are intersection of baseline with image planes– Matching point in second image is on a line passing through its
epipole– Fundamental matrix maps from a point in one image to a line (its
epipolar line) in the other– Can solve for F given corresponding points (e.g., interest points)– Can recover canonical camera matrices from F (with projective
ambiguity)
• Stereo depth estimation– Estimate disparity by finding corresponding points along scanlines– Depth is inverse to disparity
![Page 61: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/61.jpg)
Next class: structure from motion
![Page 62: Lecture 8: Stereo. Depth from Stereo Goal: recover depth by finding image coordinate x’ that corresponds to x f xx’ Baseline B z CC’ X f X x x](https://reader038.vdocument.in/reader038/viewer/2022102906/56649d155503460f949e9c4a/html5/thumbnails/62.jpg)
1. But first, Project 2.