epipolar geometry - rutgers school of engineeringsoe.rutgers.edu/~meer/ugrad/cv13epipolar.pdfan...

The slides are from several sources through James Hays (Brown); Silvio Savarese (U. of Michigan); Svetlana Lazebnik (U. Illinois); Bill Freeman and Antonio Torralba (MIT), including their own slides.

EPIPOLAR GEOMETRY

Pinhole perspective projection Recovering structure from a single view

C

Ow

P p

Calibration rig

Scene

Camera K

Why is it so difficult?

Intrinsic ambiguity of the mapping from 3D to image in 2D. see example...

Is this an illusion of 3D to 2D?

Courtesy slide S. Lazebnik

Why multiple views? • Structure and depth are inherently ambiguous from

single views.

Optical center

P1 P2

P1’=P2’

Geometry of two views constrains where the corresponding pixel for some image point in the first view must occur

in the second view. It must be on the line carved out by a plane connecting the

world point and the optical centers.

Epipolar constraint

• Epipolar Plane

Epipole

Epipolar Line

Baseline

Epipolar geometry

Epipole

Epipolar Line

• Baseline: line joining the camera centers. • Epipole: point of intersection of baseline with image plane. • Epipolar plane: plane containing baseline and world point. • Epipolar line: intersection of epipolar plane with the image

plane.

• All epipolar lines in an image intersect at the epipole... or, an epipolar plane intersects the left and right image planes

in epipolar lines.

Epipolar geometry: terms

Why is the epipolar constraint useful?

Epipolar constraint

Reduces the correspondence problem to a 1D searchin the second image along an epipolar line.

Image from Andrew Zisserman

Two examples:

Slide credit: Kristen Grauman

Converging cameras have finite epipoles.

Figure from Hartley & Zisserman

O1 O2

X

e2 x1 x2

e

Parallel cameras have epipoles at infinity.

• Baseline intersects the image plane at infinity. • Epipoles are at infinity. • Epipolar lines are parallel to x axis.

at infinityat infinity

1

Figure from Hartley & Zisserman

In parallel cameras search is only along x coord.

Slide credit: Kristen Grauman

Motion perpendicular to image plane

Motion perpendicular to image plane

forward

Forward translation

• The epipoles have same position in both images. • Epipole here is called FOE (focus of expansion).

O2

e1

e2

O 1

first

second

A 3x3 matrix connects the two 2D images.This matrix is called

• the “Essential Matrix”, E– when image intrinsic parameters are known

• the “Fundamental Matrix”, F – more the general uncalibrated case

- Two views of the same object - Suppose we know the camera positions and camera matrices ==> E matrix - Given a point on left image, how can we find the corresponding point on right image?

Essential matrix: E

x E x = lT

2

1vu

PMP

0IKM

O O’

p p’

P

R, T

Epipolar Constraint - E matrix

TRK'M

1vu

PMPhomogeous coordinates

'

0IKM

O O’

p p’

P

R, T

TRK'M K and K are known (calibrated cameras)

0IM TRM '

2''

O O’

p p’

P

R, T

0)pR(TpT

In the epipolar plane we have

Perpendicular to epipolar plane

)( pRT first camera coordinates

Cross products can be written as matrix multiplication.

...verify itThe matrix derived from a is skew-symmetric.

The matrix is rank 2. The null vector is along the vector a.

O O’

p p’

P

R, T

0)pR(TpT 0pRTpT

E = essential matrix (Longuet-Higgins, 1981)

Essential matrixE is from p' to p p' E p = 0

T

T T

some denote E as from image 2 to 1

second cameracoordinate

then R and t are from image 1 to 2

E is a rank 2 matrix!

X

• E x2 is the epipolar line associated with x2 (l1 = E x2) • ET x1 is the epipolar line associated with x1 (l2 = ET x1) • E is singular (rank two) -- two equal singular values are one.• E e2 = 0 and ET e1 = 0 • E is 3x3 matrix with 5 DOF: 3(R) + 3(t) -1(scale)

Essential matrix properties

O1 O2

x2

X

x1

e1 e2

l1 l2

(x_1) E x_2 = 0 T

calibrated

l e = (E x ) e = 0 valid for any x1

T1 2 1

T2

Fundamental matrix: F

- Uncalibrated cameras.- No additional information about the scene and camera is given ==> F matrix- Given a point on left image, how can I find the corresponding point on right image?

l’ = FT x x

PMP

vu

p 0IKM

Epipolar Constraint - F matrix

unknown

O O’

p p’

P uncalibrated camera

M' = K'[R t]

MR t

uncalibrated camera

(3x4)

1 homogeous coord.

O O’

p p’

P

F matrix derived from E matrix

0pRTpT

pKp 1 pKp 1'

0pKRT)pK( 1T1

0pKRTKp 1TT

0pFpT

rank 2

[T] !!x

O O’

p p’

P

Fundamental matrix

0pFpT (Faugeras and Luong, 1992)

F is from p' to pp' F p = 0second cameracoordinate

T

T Tsome denote F as from image 2 to 1

then R and t are from image 1 to 2

R t

The fundamental matrix has a projective ambiguity. Two pairsof camera matrices (P, P') and (P, P') give the same F ifP = PH and P' = P'H where H is a 4x4 nonsingular matrix.

~ ~

~ ~

Fundamental matrix properties

O1 O2

x2

X

x1

e1 e2

• F x2 is the epipolar line associated with x2 (l1 = F x2) • FT x1 is the epipolar line associated with x1 (l2 = FT x1) • F is singular (rank two)• F e2 = 0 and FT e1 = 0 • F is 3x3 matrix with 7 DOF: 9 - 1(rank 2) - 1(scale)

(x_1) F x_2 = 0T

uncalibratedfundamental matrixis much more used

The eight-point algorithm of F (linear)

x = (u, v, 1)T, x’ = (u’, v’, 1)T

Minimize:

2)(N

T F ′∑under the constraint

1)( i

ii xFx∑

=

F33 = 1

(Hartley, 1995)

can be an other F_ij constraint also

Estimating F

Lsq. solution by SVD • Rank 8 A non-zero solution exists (unique)

• Homogeneous system

• If N>8 F̂

f

1f

W 0f

W

rank 3 solution

0pF̂pT

The estimated F have full rank (det(F) ≠0) but F should have rank=2 instead.

0F̂F Find F that minimizes

subject to det(F)=0

Taking into account the rank-2 constraint.

Frobenius norm (*)

Taking the first two s.v. and the three equal zero.

^ ^

(*) Sqrt root of the sum of squares of all entries

Data courtesy of R. Mohr and B. Boufama. Example of F recovery

Mean errors: 10.0pixel and 9.1pixel

This are large errors...

The problem with eight-point algorithm

Poor numerical conditioning.Can be fixed by rescaling the data before estimation. Can be used for any DLT type algorithm.

More sophisticated nonlinear methods after the 8-point algorithm exist, but we will not cover.

RESCALING BY NORMALIZATION

You have i = 1, . . . , n points xi. The mean of thesepoints is

x̄ =1

n

n∑

i=1

xi

which is translated to the origin (0, 0) by the vector −x̄.The new coordinated of a point are x̃i = xi − x̄.

Compute the mean squared distance of the points fromthe center

a2 =1

n

n∑

i=1

x̃⊤i x̃i

and move the square norm equal to 2 by multiplying thecomponents of the original point

√2/a. This is the scal-

ing. The translation are the mean coordinates with oppo-site sign multiplied with

√2/a.

In the homogeneous 2D coordinates

T =

√2

a

1 0 −x̄1

0 1 −x̄2

0 0 a√2

Txi =

√2

a (xi1 − x̄1)√2

a (xi2 − x̄2)1

a 2D similarity transformation.

The normalized eight-point algorithm

• Center the image data at the origin, and scale it so

(Hartley, 1995)

the mean squared distance between the origin and the data points is 2 pixels.

• Use the eight point algorithm to compute F from the• Use the eight-point algorithm to compute F from the normalized points, n_1 and n_2.

• Enforce the rank-2 constraint. For example, take SVD of F and throw out the smallest singular value.

• Transform fundamental matrix back to original units: if T and T’ are the normalizing transformations in theif T and T are the normalizing transformations in the two images, than the fundamental matrix in original coordinates is TT F T’.

Isotropic translation (mean to origin) and scale in each image separately.

x_1 = T n_1 x_2 = T' n_2 (n_1) T F T' n_2 = 0 => final F

-1 -1

T -T -1

given

With

tran

sfor

mat

ion

With

out t

rans

form

atio

n Mean errors: 10.0pixel and 9.1pixel

Mean errors: 1.0pixel and 0.9pixel

Comparison of estimation algorithms

8-point Normalized 8-point Nonlinear least squaresp p q

Av. Dist. 1 2.33 pixels 0.92 pixel 0.86 pixel

Av. Dist. 2 2.18 pixels 0.85 pixel 0.80 pixel

From epipolar geometry to camera calibration• Estimating the fundamental matrix is known

as “weak calibration”.• If we know the calibration matrices of the two

cameras, we can estimate the essential matrix: E = KTFK’matrix: E = KTFK

• The essential matrix can give us the relative rotation and translation between the camerasrotation and translation between the cameras, with 5 point pairs.

(see F from E slide)

O1 O2

X

e2 x1 x2

e

Example: Parallel calibrated images

E=? K1=K2 = known Hint :

x

y

z

x parallel to O1O2 R = I t = (T, 0, 0)

t1

O1 O2

X

e2 x1 x2

e

E=? K1=K2 = known

x

y

z

x parallel to O1O2

0000

000][

T

TRtE

see cross-productas matrix, before

1

O1 O2

X

e2 x1 x2

e

x

y

z

Epipolar constraint reduces to y = y’ In stereo vision that will be a big help.

1