9 – stereo reconstruction
DESCRIPTION
9 – Stereo Reconstruction. Slides from A. Zisserman & S. Lazebnik. Overview. Single camera geometry Recap of Homogenous coordinates Perspective projection model Camera calibration Stereo Reconstruction Epipolar geometry Stereo correspondence Triangulation. Single camera geometry. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/1.jpg)
9 – Stereo Reconstruction
Slides from A. Zisserman & S. Lazebnik
![Page 2: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/2.jpg)
Overview
• Single camera geometry• Recap of Homogenous coordinates
• Perspective projection model
• Camera calibration
• Stereo Reconstruction• Epipolar geometry
• Stereo correspondence
• Triangulation
![Page 3: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/3.jpg)
Single camera geometry
![Page 4: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/4.jpg)
Projection
![Page 5: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/5.jpg)
Projection
![Page 6: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/6.jpg)
Projective Geometry
• Recovery of structure from one image is inherently ambiguous
• Today focus on geometry that maps world to camera image
x
X? X? X?
![Page 7: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/7.jpg)
Recall: Pinhole camera model
• Principal axis: line from the camera center perpendicular to the image plane
• Normalized (camera) coordinate system: camera center is at the origin and the principal axis is the z-axis
![Page 8: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/8.jpg)
)/,/(),,( ZYfZXfZYX
101
0
0
1
Z
Y
X
f
f
Z
Yf
Xf
Z
Y
X
Recall: Pinhole camera model
PXx
![Page 9: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/9.jpg)
Recap: Homogeneous coordinates
• Is this a linear transformation?
Trick: add one more coordinate:
homogeneous image coordinates
homogeneous scene coordinates
Converting from homogeneous coordinates
• no—division by z is nonlinear
Slide by Steve Seitz
),(),,(z
yf
z
xfzyx
![Page 10: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/10.jpg)
Principal point
• Principal point (p): point where principal axis intersects the image plane (origin of normalized coordinate system)
• Normalized coordinate system: origin is at the principal point• Image coordinate system: origin is in the corner• How to go from normalized coordinate system to image
coordinate system?
![Page 11: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/11.jpg)
)/,/(),,( yx pZYfpZXfZYX
101
0
0
1
Z
Y
X
pf
pf
Z
pZYf
pZXf
Z
Y
X
y
x
y
x
Principal point offset
principal point: ),( yx pp
![Page 12: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/12.jpg)
101
01
01
1Z
Y
X
pf
pf
Z
ZpYf
ZpXf
y
x
y
x
Principal point offset
1y
x
pf
pf
K calibration matrix 0|IKP
principal point: ),( yx pp
![Page 13: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/13.jpg)
111yy
xx
y
x
y
x
pf
pf
m
m
K
Pixel coordinates
• mx pixels per meter in horizontal direction, my pixels per meter in vertical direction
Pixel size: yx mm
11
pixels/m m pixels
![Page 14: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/14.jpg)
C~
-X~
RX~
cam
Camera rotation and translation
• In general, the camera coordinate frame will be related to the world coordinate frame by a rotation and a translation
coords. of point in camera frame
coords. of camera center in world frame
coords. of a pointin world frame (nonhomogeneous)
![Page 15: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/15.jpg)
C~
-X~
RX~
cam
X10
C~
RR
1
X~
10
C~
RRXcam
XC~
R|RKX0|IKx cam ,t|RKP C~
Rt
Camera rotation and translation
In non-homogeneouscoordinates:
Note: C is the null space of the camera projection matrix (PC=0)
![Page 16: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/16.jpg)
Camera parameters• Intrinsic parameters
• Principal point coordinates
• Focal length
• Pixel magnification factors
• Skew (non-rectangular pixels)
• Radial distortion
111yy
xx
y
x
y
x
pf
pf
m
m
K
![Page 17: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/17.jpg)
Camera parameters• Intrinsic parameters
• Principal point coordinates
• Focal length
• Pixel magnification factors
• Skew (non-rectangular pixels)
• Radial distortion
• Extrinsic parameters• Rotation and translation relative to world coordinate
system
![Page 18: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/18.jpg)
Camera calibration
• Given n points with known 3D coordinates Xi and known image projections xi, estimate the camera parameters
? P
Xi
xi
![Page 19: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/19.jpg)
ii PXx
Camera calibration
0PXx ii0
XP
XP
XP
1 3
2
1
iT
iT
iT
i
i
y
x
0
P
P
P
0XX
X0X
XX0
3
2
1
Tii
Tii
Tii
Ti
Tii
Ti
xy
x
y
Two linearly independent equations
![Page 20: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/20.jpg)
Camera calibration
• P has 11 degrees of freedom (12 parameters, but scale is arbitrary)
• One 2D/3D correspondence gives us two linearly independent equations
• Homogeneous least squares• 6 correspondences needed for a minimal solution
0pA 0
P
P
P
X0X
XX0
X0X
XX0
3
2
1111
111
Tnn
TTn
Tnn
Tn
T
TTT
TTT
x
y
x
y
![Page 21: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/21.jpg)
Camera calibration
• Note: for coplanar points that satisfy ΠTX=0,we will get degenerate solutions (Π,0,0), (0,Π,0), or (0,0,Π)
0Ap 0
P
P
P
X0X
XX0
X0X
XX0
3
2
1111
111
Tnn
TTn
Tnn
Tn
T
TTT
TTT
x
y
x
y
![Page 22: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/22.jpg)
Camera calibration
• Once we’ve recovered the numerical form of the camera matrix, we still have to figure out the intrinsic and extrinsic parameters
• This is a matrix decomposition problem, not an estimation problem (see F&P sec. 3.2, 3.3)
![Page 23: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/23.jpg)
CSE 576, Spring 2008 Projective Geometry 23
Alternative: multi-plane calibration
Images courtesy Jean-Yves Bouguet, Intel Corp.
Advantage• Only requires a plane• Don’t have to know positions/orientations• Good code available online!
– Intel’s OpenCV library: http://www.intel.com/research/mrl/research/opencv/
– Matlab version by Jean-Yves Bouget: http://www.vision.caltech.edu/bouguetj/calib_doc/index.html
– Zhengyou Zhang’s web site: http://research.microsoft.com/~zhang/Calib/
![Page 24: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/24.jpg)
Stereo Reconstruction
knownknowncameracamera
viewpointsviewpoints
Shape (3D) from two (or more) images
![Page 25: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/25.jpg)
Example
images
shape
surface reflectance
![Page 26: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/26.jpg)
Scenarios
The two images can arise from
• A stereo rig consisting of two cameras• the two images are acquired simultaneously
or
• A single moving camera (static scene)• the two images are acquired sequentially
The two scenarios are geometrically equivalent
![Page 27: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/27.jpg)
Stereo head
Camera on a mobile vehicle
![Page 28: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/28.jpg)
The objective
Given two images of a scene acquired by known cameras compute the 3D position of the scene (structure recovery)
Basic principle: triangulate from corresponding image points
• Determine 3D point at intersection of two back-projected rays
![Page 29: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/29.jpg)
Corresponding points are images of the same scene point
Triangulation
C C /
The back-projected points generate rays which intersect at the
3D scene point
![Page 30: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/30.jpg)
An algorithm for stereo reconstruction
1. For each point in the first image determine the corresponding point in the second image
(this is a search problem)
2. For each pair of matched points determine the 3D point by triangulation
(this is an estimation problem)
![Page 31: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/31.jpg)
The correspondence problem
Given a point x in one image find the corresponding point in the other image
This appears to be a 2D search problem, but it is reduced to a 1D search
by the epipolar constraint
![Page 32: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/32.jpg)
1. Epipolar geometry
• the geometry of two cameras• reduces the correspondence problem to a line search
2. Stereo correspondence algorithms
3. Triangulation
Outline
![Page 33: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/33.jpg)
Notation
x x /
X
C C /
The two cameras are P and P/, and a 3D point X is imaged as
for equations involving homogeneous quantities ‘=’ means ‘equal up to scale’
P P/
Warning
![Page 34: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/34.jpg)
Epipolar geometry
![Page 35: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/35.jpg)
Epipolar geometry
Given an image point in one view, where is the corresponding point in the other view?
epipolar line
?
baseline
• A point in one view “generates” an epipolar line in the other view
• The corresponding point lies on this line
epipole C /C
![Page 36: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/36.jpg)
Epipolar line
Epipolar constraint• Reduces correspondence problem to 1D search along an
epipolar line
![Page 37: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/37.jpg)
Epipolar geometry continued
Epipolar geometry is a consequence of the coplanarity of the camera centres and scene point
x x /
X
C C /
The camera centres, corresponding points and scene point lie in a single plane, known as the epipolar plane
![Page 38: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/38.jpg)
Nomenclature
• The epipolar line l/ is the image of the ray through x
• The epipole e is the point of intersection of the line joining the camera centres with the image plane
this line is the baseline for a stereo rig, and
the translation vector for a moving camera
• The epipole is the image of the centre of the other camera: e = PC/ , e/ = P/C
xx /
X
C C /
e
left epipolar lineright epipolar line
e /
l/
![Page 39: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/39.jpg)
The epipolar pencil
e e /
baseline
X
As the position of the 3D point X varies, the epipolar planes “rotate” about the baseline. This family of planes is known as an epipolar pencil. All epipolar lines intersect at the epipole.
(a pencil is a one parameter family)
![Page 40: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/40.jpg)
The epipolar pencil
e e /
baseline
X
As the position of the 3D point X varies, the epipolar planes “rotate” about the baseline. This family of planes is known as an epipolar pencil. All epipolar lines intersect at the epipole.
(a pencil is a one parameter family)
![Page 41: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/41.jpg)
Epipolar geometry example I: parallel cameras
Epipolar geometry depends only on the relative pose (position and orientation) and internal parameters of the two cameras, i.e. the position of the camera centres and image planes. It does not depend on the scene structure (3D points external to the camera).
![Page 42: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/42.jpg)
Epipolar geometry example II: converging cameras
Note, epipolar lines are in general not parallel
e e /
![Page 43: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/43.jpg)
Homogeneous notation for lines
![Page 44: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/44.jpg)
• The line l through the two points p and q is l = p x q
Example: compute the point of intersection of the two lines l and m in the figure below
Proof
y
x
1
2
• The intersection of two lines l and m is the point x = l x m
l
m
which is the point (2,1)
![Page 45: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/45.jpg)
Matrix representation of the vector cross product
![Page 46: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/46.jpg)
Example: compute the cross product of l and m
![Page 47: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/47.jpg)
Algebraic representation of epipolar geometry
We know that the epipolar geometry defines a mapping
x l/
point in first image
epipolar line in second image
![Page 48: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/48.jpg)
P
Derivation of the algebraic expression
Outline
Step 1: for a point x in the first image back project a ray with camera P
Step 2: choose two points on the ray and project into the second image with camera P/
Step 3: compute the line through the two image points using the relation l/ = p x q
P/
![Page 49: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/49.jpg)
• choose camera matrices
internal calibration
rotation translation
from world to camera coordinate frame
• first camera
world coordinate frame aligned with first camera
• second camera
![Page 50: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/50.jpg)
Step 1: for a point x in the first image back project a ray with camera
P
A point x back projects to a ray
where Z is the point’s depth, since
satisfies
![Page 51: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/51.jpg)
Step 2: choose two points on the ray and project into the second image with camera P/
P/
Consider two points on the ray
• Z = 0 is the camera centre
• Z = is the point at infinity
Project these two points into the second view
![Page 52: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/52.jpg)
Using the identity
Compute the line through the points
F
F is the fundamental matrix
Step 3: compute the line through the two image points using the relation l/ = p x q
![Page 53: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/53.jpg)
Example I: compute the fundamental matrix for a parallel camera stereo rig
• reduces to y = y/ , i.e. raster correspondence (horizontal scan-lines)
f
f
X Y
Z
![Page 54: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/54.jpg)
f
f
X Y
Z
Geometric interpretation ?
![Page 55: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/55.jpg)
Example II: compute F for a forward translating camera
f
f
X Y
Z
![Page 56: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/56.jpg)
f
f
X YZ
first image second image
![Page 57: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/57.jpg)
![Page 58: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/58.jpg)
![Page 59: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/59.jpg)
Summary: Properties of the Fundamental matrix
![Page 60: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/60.jpg)
Admin Interlude
• Assignment 1 due this time next week
• So come to office hours right after class!
• Meet Chaitanya Rudra ([email protected]) who is the class TA
![Page 61: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/61.jpg)
Stereo correspondence algorithms
![Page 62: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/62.jpg)
Problem statement
Given: two images and their associated cameras compute
corresponding image points.
Algorithms may be classified into two types:
1. Dense: compute a correspondence at every pixel
2. Sparse: compute correspondences only for features
The methods may be top down or bottom up
![Page 63: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/63.jpg)
Top down matching
1. Group model (house, windows, etc) independently in each image
2. Match points (vertices) between images
![Page 64: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/64.jpg)
Bottom up matching
• epipolar geometry reduces the correspondence search from 2D to a 1D search on corresponding epipolar lines
• 1D correspondence problem
b/
a/
bca
CBA
c/
![Page 65: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/65.jpg)
![Page 66: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/66.jpg)
Stereograms• Invented by Sir Charles Wheatstone, 1838
![Page 67: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/67.jpg)
Red/green stereograms
![Page 68: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/68.jpg)
Random dot stereograms
![Page 69: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/69.jpg)
![Page 70: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/70.jpg)
Autostereograms
Autostereograms: www.magiceye.com
![Page 71: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/71.jpg)
Autostereograms
Autostereograms: www.magiceye.com
![Page 72: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/72.jpg)
Correspondence algorithms
Algorithms may be top down or bottom up – random dot stereograms are an existence proof that bottom up algorithms are possible
From here on only consider bottom up algorithms
Algorithms may be classified into two types:
1. Dense: compute a correspondence at every pixel
2. Sparse: compute correspondences only for features
![Page 73: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/73.jpg)
Example image pair – parallel cameras
![Page 74: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/74.jpg)
First image
![Page 75: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/75.jpg)
Second image
![Page 76: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/76.jpg)
Dense correspondence algorithm
Search problem (geometric constraint): for each point in the left image, the corresponding point in the right image lies on the epipolar line (1D ambiguity)
Disambiguating assumption (photometric constraint): the intensity neighbourhood of corresponding points are similar across images
Measure similarity of neighbourhood intensity by cross-correlation
Parallel camera example – epipolar lines are corresponding rasters
epipolar line
![Page 77: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/77.jpg)
Intensity profiles
• Clear correspondence between intensities, but also noise and ambiguity
![Page 78: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/78.jpg)
region A
Normalized Cross Correlation
region B
vector a vector b
write regions as vectors
a
b
![Page 79: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/79.jpg)
Cross-correlation of neighbourhood regions
epipolar line
translate so that mean is zero
(exercise)
![Page 80: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/80.jpg)
left image band
right image band
cross correlation
1
0
0.5
x
![Page 81: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/81.jpg)
left image band
right image band
cross correlation
1
0
x
0.5
target region
![Page 82: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/82.jpg)
Why is cross-correlation such a poor measure in the second case?
1. The neighbourhood region does not have a “distinctive” spatial intensity distribution
2. Foreshortening effects
fronto-parallel surface
imaged length the same
slanting surface
imaged lengths differ
![Page 83: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/83.jpg)
Limitations of similarity constraint
Textureless surfaces Occlusions, repetition
Non-Lambertian surfaces, specularities
![Page 84: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/84.jpg)
Results with window search
Window-based matching Ground truth
Data
![Page 85: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/85.jpg)
Sketch of a dense correspondence algorithm
For each pixel in the left image• compute the neighbourhood cross correlation along the
corresponding epipolar line in the right image• the corresponding pixel is the one with the highest cross
correlation
Parameters• size (scale) of neighbourhood• search disparity
Other constraints• uniqueness• ordering• smoothness of disparity field
Applicability• textured scene, largely fronto-parallel
![Page 86: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/86.jpg)
Stereo matching as energy minimization
MAP estimate of disparity image D: )()|,(),|( 2121 DPDIIPIIDP
I1I2 D
W1(i ) W2(i+D(i )) D(i )
)(),,( smooth21data DEDIIEE
)(log)|,(log),|(log 2121 DPDIIPIIDP
ji
jDiDE,neighbors
smooth )()( 2
21data ))(()( i
iDiWiWE
![Page 87: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/87.jpg)
Stereo matching as energy minimization
I1I2 D
• Energy functions of this form can be minimized using graph cuts
Y. Boykov, O. Veksler, and R. Zabih, Fast Approximate Energy Minimization via Graph Cuts, PAMI 2001
W1(i ) W2(i+D(i )) D(i )
)(),,( smooth21data DEDIIEE
ji
jDiDE,neighbors
smooth )()( 2
21data ))(()( i
iDiWiWE
![Page 88: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/88.jpg)
Graph cuts solution
Graph cuts Ground truth
For the latest and greatest: http://www.middlebury.edu/stereo/
Y. Boykov, O. Veksler, and R. Zabih, Fast Approximate Energy Minimization via Graph Cuts, PAMI 2001
![Page 89: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/89.jpg)
Example dense correspondence algorithm
left image right image
![Page 90: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/90.jpg)
right image depth map
3D reconstruction
intensity = depth
![Page 91: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/91.jpg)
Texture mapped 3D triangulation
![Page 92: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/92.jpg)
range map
Pentagon example
left image right image
![Page 93: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/93.jpg)
Rectification
e e /
For converging cameras
• epipolar lines are not parallel
![Page 94: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/94.jpg)
Project images onto plane parallel to baseline
epipolar plane
![Page 95: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/95.jpg)
Rectification continued
Convert converging cameras to parallel camera geometry by an image mapping
Image mapping is a 2D homography (projective transformation)
(exercise)
![Page 96: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/96.jpg)
Rectification continued
Convert converging cameras to parallel camera geometry by an image mapping
Image mapping is a 2D homography (projective transformation)
(exercise)
![Page 97: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/97.jpg)
Exampleoriginal stereo pair
rectified stereo pair
![Page 98: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/98.jpg)
Note• image movement (disparity) is inversely proportional to depth Z
• depth is inversely proportional to disparity
Example: depth and disparity for a parallel camera stereo rig
Then, y/ = y, and the disparity
Derivation
x
x /
d
![Page 99: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/99.jpg)
Triangulation
![Page 100: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/100.jpg)
1. Vector solution
C C /
Compute the mid-point of the shortest line between the two rays
![Page 101: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/101.jpg)
2. Linear triangulation (algebraic solution)
![Page 102: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/102.jpg)
Problem: does not minimize anything meaningful
Advantage: extends to more than two views
![Page 103: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/103.jpg)
3. Minimizing a geometric/statistical error
![Page 104: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/104.jpg)
• It can be shown that if the measurement noise is Gaussian mean zero, , then minimizing geometric error is the Maximum Likelihood Estimate of X
• The minimization appears to be over three parameters (the position X), but the problem can be reduced to a minimization over one parameter
![Page 105: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/105.jpg)
Different formulation of the problem
![Page 106: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/106.jpg)
Minimization method
• Parametrize the pencil of epipolar lines in the first image by t, such that the epipolar line is l(t)
• Using F compute the corresponding epipolar line in the second image l/ (t)
• Express the distance function explicitly as a function of t
• Find the value of t that minimizes the distance function
• Solution is a 6th degree polynomial in t
![Page 107: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/107.jpg)
Other approaches to obtaining 3D
structure
![Page 108: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/108.jpg)
Active stereo with structured light
• Project “structured” light patterns onto the object• simplifies the correspondence problem
• Allows us to use only one camera
camera
projector
L. Zhang, B. Curless, and S. M. Seitz. Rapid Shape Acquisition Using Color Structured Light and Multi-pass Dynamic Programming. 3DPVT 2002
![Page 109: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/109.jpg)
Active stereo with structured light
L. Zhang, B. Curless, and S. M. Seitz. Rapid Shape Acquisition Using Color Structured Light and Multi-pass Dynamic Programming. 3DPVT 2002
![Page 110: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/110.jpg)
Microsoft Kinect
![Page 111: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/111.jpg)
Laser scanning
• Optical triangulation• Project a single stripe of laser light• Scan it across the surface of the object• This is a very precise version of structured light scanning
Digital Michelangelo Projecthttp://graphics.stanford.edu/projects/mich/
Source: S. Seitz
![Page 112: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/112.jpg)
Laser scanned models
The Digital Michelangelo Project, Levoy et al.
Source: S. Seitz
![Page 113: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/113.jpg)
Laser scanned models
The Digital Michelangelo Project, Levoy et al.
Source: S. Seitz
![Page 114: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/114.jpg)
Laser scanned models
The Digital Michelangelo Project, Levoy et al.
Source: S. Seitz
![Page 115: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/115.jpg)
Laser scanned models
The Digital Michelangelo Project, Levoy et al.
Source: S. Seitz
![Page 116: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/116.jpg)
Laser scanned models
The Digital Michelangelo Project, Levoy et al.
Source: S. Seitz
![Page 117: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/117.jpg)
Aligning range images
• A single range scan is not sufficient to describe a complex surface
• Need techniques to register multiple range images
B. Curless and M. Levoy, A Volumetric Method for Building Complex Models from Range Images, SIGGRAPH 1996
![Page 118: 9 – Stereo Reconstruction](https://reader036.vdocument.in/reader036/viewer/2022062422/56813f64550346895daa3767/html5/thumbnails/118.jpg)
Aligning range images
• A single range scan is not sufficient to describe a complex surface
• Need techniques to register multiple range images
• … which brings us to multi-view stereo