The slides are from several sources through James Hays (Brown); Silvio Savarese (U. of Michigan); Svetlana Lazebnik (U. Illinois); Bill Freeman and Antonio Torralba (MIT), including their own slides.
EPIPOLAR GEOMETRY
Pinhole perspective projection Recovering structure from a single view
C
Ow
P p
Calibration rig
Scene
Camera K
Why is it so difficult?
Intrinsic ambiguity of the mapping from 3D to image in 2D. see example...
Is this an illusion of 3D to 2D?
Courtesy slide S. Lazebnik
Why multiple views? • Structure and depth are inherently ambiguous from
single views.
Optical center
P1 P2
P1’=P2’
Geometry of two views constrains where the corresponding pixel for some image point in the first view must occur
in the second view. It must be on the line carved out by a plane connecting the
world point and the optical centers.
Epipolar constraint
• Epipolar Plane
Epipole
Epipolar Line
Baseline
Epipolar geometry
Epipole
Epipolar Line
• Baseline: line joining the camera centers. • Epipole: point of intersection of baseline with image plane. • Epipolar plane: plane containing baseline and world point. • Epipolar line: intersection of epipolar plane with the image
plane.
• All epipolar lines in an image intersect at the epipole... or, an epipolar plane intersects the left and right image planes
in epipolar lines.
Epipolar geometry: terms
Why is the epipolar constraint useful?
Epipolar constraint
Reduces the correspondence problem to a 1D searchin the second image along an epipolar line.
Image from Andrew Zisserman
Two examples:
Slide credit: Kristen Grauman
Converging cameras have finite epipoles.
Figure from Hartley & Zisserman
O1 O2
X
e2 x1 x2
e
Parallel cameras have epipoles at infinity.
• Baseline intersects the image plane at infinity. • Epipoles are at infinity. • Epipolar lines are parallel to x axis.
at infinityat infinity
1
Figure from Hartley & Zisserman
In parallel cameras search is only along x coord.
Slide credit: Kristen Grauman
Motion perpendicular to image plane
Motion perpendicular to image plane
forward
Forward translation
• The epipoles have same position in both images. • Epipole here is called FOE (focus of expansion).
O2
e1
e2
O 1
first
second
A 3x3 matrix connects the two 2D images.This matrix is called
• the “Essential Matrix”, E– when image intrinsic parameters are known
• the “Fundamental Matrix”, F – more the general uncalibrated case
- Two views of the same object - Suppose we know the camera positions and camera matrices ==> E matrix - Given a point on left image, how can we find the corresponding point on right image?
Essential matrix: E
x E x = lT
2
1vu
PMP
0IKM
O O’
p p’
P
R, T
Epipolar Constraint - E matrix
TRK'M
1vu
PMPhomogeous coordinates
'
0IKM
O O’
p p’
P
R, T
TRK'M K and K are known (calibrated cameras)
0IM TRM '
2''
O O’
p p’
P
R, T
0)pR(TpT
In the epipolar plane we have
Perpendicular to epipolar plane
)( pRT first camera coordinates
Cross products can be written as matrix multiplication.
...verify itThe matrix derived from a is skew-symmetric.
The matrix is rank 2. The null vector is along the vector a.
O O’
p p’
P
R, T
0)pR(TpT 0pRTpT
E = essential matrix (Longuet-Higgins, 1981)
Essential matrixE is from p' to p p' E p = 0
T
T T
some denote E as from image 2 to 1
second cameracoordinate
then R and t are from image 1 to 2
E is a rank 2 matrix!
X
• E x2 is the epipolar line associated with x2 (l1 = E x2) • ET x1 is the epipolar line associated with x1 (l2 = ET x1) • E is singular (rank two) -- two equal singular values are one.• E e2 = 0 and ET e1 = 0 • E is 3x3 matrix with 5 DOF: 3(R) + 3(t) -1(scale)
Essential matrix properties
O1 O2
x2
X
x1
e1 e2
l1 l2
(x_1) E x_2 = 0 T
calibrated
l e = (E x ) e = 0 valid for any x1
T1 2 1
T2
Fundamental matrix: F
- Uncalibrated cameras.- No additional information about the scene and camera is given ==> F matrix- Given a point on left image, how can I find the corresponding point on right image?
l’ = FT x x
PMP
vu
p 0IKM
Epipolar Constraint - F matrix
unknown
O O’
p p’
P uncalibrated camera
M' = K'[R t]
MR t
uncalibrated camera
(3x4)
1 homogeous coord.
O O’
p p’
P
F matrix derived from E matrix
0pRTpT
pKp 1 pKp 1'
0pKRT)pK( 1T1
0pKRTKp 1TT
0pFpT
rank 2
[T] !!x
O O’
p p’
P
Fundamental matrix
0pFpT (Faugeras and Luong, 1992)
F is from p' to pp' F p = 0second cameracoordinate
T
T Tsome denote F as from image 2 to 1
then R and t are from image 1 to 2
R t
The fundamental matrix has a projective ambiguity. Two pairsof camera matrices (P, P') and (P, P') give the same F ifP = PH and P' = P'H where H is a 4x4 nonsingular matrix.
~ ~
~ ~
Fundamental matrix properties
O1 O2
x2
X
x1
e1 e2
• F x2 is the epipolar line associated with x2 (l1 = F x2) • FT x1 is the epipolar line associated with x1 (l2 = FT x1) • F is singular (rank two)• F e2 = 0 and FT e1 = 0 • F is 3x3 matrix with 7 DOF: 9 - 1(rank 2) - 1(scale)
(x_1) F x_2 = 0T
uncalibratedfundamental matrixis much more used
The eight-point algorithm of F (linear)
x = (u, v, 1)T, x’ = (u’, v’, 1)T
Minimize:
2)(N
T F ′∑under the constraint
1)( i
ii xFx∑
=
F33 = 1
(Hartley, 1995)
can be an other F_ij constraint also
Estimating F
Lsq. solution by SVD • Rank 8 A non-zero solution exists (unique)
• Homogeneous system
• If N>8 F̂
f
1f
W 0f
W
rank 3 solution
0pF̂pT
The estimated F have full rank (det(F) ≠0) but F should have rank=2 instead.
0F̂F Find F that minimizes
subject to det(F)=0
Taking into account the rank-2 constraint.
Frobenius norm (*)
Taking the first two s.v. and the three equal zero.
^ ^
(*) Sqrt root of the sum of squares of all entries
Data courtesy of R. Mohr and B. Boufama. Example of F recovery
Mean errors: 10.0pixel and 9.1pixel
This are large errors...
The problem with eight-point algorithm
Poor numerical conditioning.Can be fixed by rescaling the data before estimation. Can be used for any DLT type algorithm.
More sophisticated nonlinear methods after the 8-point algorithm exist, but we will not cover.
RESCALING BY NORMALIZATION
You have i = 1, . . . , n points xi. The mean of thesepoints is
x̄ =1
n
n∑
i=1
xi
which is translated to the origin (0, 0) by the vector −x̄.The new coordinated of a point are x̃i = xi − x̄.
Compute the mean squared distance of the points fromthe center
a2 =1
n
n∑
i=1
x̃⊤i x̃i
and move the square norm equal to 2 by multiplying thecomponents of the original point
√2/a. This is the scal-
ing. The translation are the mean coordinates with oppo-site sign multiplied with
√2/a.
In the homogeneous 2D coordinates
T =
√2
a
1 0 −x̄1
0 1 −x̄2
0 0 a√2
Txi =
√2
a (xi1 − x̄1)√2
a (xi2 − x̄2)1
a 2D similarity transformation.
The normalized eight-point algorithm
• Center the image data at the origin, and scale it so
(Hartley, 1995)
the mean squared distance between the origin and the data points is 2 pixels.
• Use the eight point algorithm to compute F from the• Use the eight-point algorithm to compute F from the normalized points, n_1 and n_2.
• Enforce the rank-2 constraint. For example, take SVD of F and throw out the smallest singular value.
• Transform fundamental matrix back to original units: if T and T’ are the normalizing transformations in theif T and T are the normalizing transformations in the two images, than the fundamental matrix in original coordinates is TT F T’.
Isotropic translation (mean to origin) and scale in each image separately.
x_1 = T n_1 x_2 = T' n_2 (n_1) T F T' n_2 = 0 => final F
-1 -1
T -T -1
given
With
tran
sfor
mat
ion
With
out t
rans
form
atio
n Mean errors: 10.0pixel and 9.1pixel
Mean errors: 1.0pixel and 0.9pixel
Comparison of estimation algorithms
8-point Normalized 8-point Nonlinear least squaresp p q
Av. Dist. 1 2.33 pixels 0.92 pixel 0.86 pixel
Av. Dist. 2 2.18 pixels 0.85 pixel 0.80 pixel
From epipolar geometry to camera calibration• Estimating the fundamental matrix is known
as “weak calibration”.• If we know the calibration matrices of the two
cameras, we can estimate the essential matrix: E = KTFK’matrix: E = KTFK
• The essential matrix can give us the relative rotation and translation between the camerasrotation and translation between the cameras, with 5 point pairs.
(see F from E slide)
O1 O2
X
e2 x1 x2
e
Example: Parallel calibrated images
E=? K1=K2 = known Hint :
x
y
z
x parallel to O1O2 R = I t = (T, 0, 0)
t1
O1 O2
X
e2 x1 x2
e
E=? K1=K2 = known
x
y
z
x parallel to O1O2
0000
000][
T
TRtE
see cross-productas matrix, before
1
O1 O2
X
e2 x1 x2
e
x
y
z
Epipolar constraint reduces to y = y’ In stereo vision that will be a big help.
1