two-views geometry outline background: camera, projection necessary tools: a taste of projective...

Two-views geometryOutline

Background: Camera, Projection Necessary tools: A taste of projective geometry Two view geometry:

Homography Epipolar geometry, the essential matrix Camera calibration, the fundamental matrix

3D reconstruction (Stereo algorithms) next week.

Many of the slides are courtesy of Prof. Ronen Basri

3-D Scene

u

u’

What can 2 images tell us about ….Faugeras et. al. ECCV 92

Objective

3-D Scene

u

u’

Study the mathematical relations between corresponding image points.

“Corresponding” means originated from the same 3D point.

Objective

World Cup 66: England-Germany

World Cup 66: Second View

World Cup 66: England-Germany

Conclusion: no goal (missing 3 inches)

(Reid and Zisserman, “Goal-directed video metrology”)

Camera Obscura

"Reinerus Gemma-Frisius, observed an eclipse of the sun at Louvain on January 24, 1544, and later he used this illustration of the event in his book De Radio Astronomica et Geometrica, 1545. It is thought to be the first published illustration of a camera obscura..." Hammond, John H., The Camera Obscura, A Chronicle

A few words about Cameras

Camera obscura dates from 15th century First photograph on record shown in the book - 1822 Basic abstraction is the pinhole camera Current cameras contain a lens and a recording device

(film, CCD, CMOS) The human eye functions very much like a camera

Ideal LensesLens acts as a pinhole (for 3D points at the focal depth).

Regular LensesE.g., the cameras in our lab.

To learn more on lens-distortion see Hartley & Zisserman Sec. 7.4 p.189.Not part of this class.

Pinhole Camera

Single View Geometry

f

X

P Y

Z

x

p y

f

∏x

p y

f

Notation

O – Focal center π – Image plane Z – Optical axis f – Focal length

Projection

x y f

X Y Z

f

x

y

Z

X

Y

Perspective Projection

f Xx

Zf Y

yZ

Origin (0,0,0) is the Focal center X,Y (x,y) axis are along the image axis (height / width). Z is depth = distance along the Optical axis f – Focal length

Orthographic Projection

•Projection rays are parallel•Image plane is fronto-parallel(orthogonal to rays)

•Focal center at infinity

x X

y Y

Scaled Orthographic ProjectionAlso called “weak perspective”

x sX

y sY

0

fs

Z

Pros and Cons of Projection Models Weak perspective has simpler math.

Accurate when object is small and distant. Useful for object recognition.

Pinhole perspective much more accurate. Used in structure from motion.

When accuracy really matters (SFM), we must model the real camera (exact imaging processes): Perspective projection, calibration parameters (later), and

all other issues (radial distortion).

Two-views geometryOutline

Background: Camera, Projection Necessary tools: A taste of projective geometry Two view geometry:

Homography Epipolar geometry, the essential matrix Camera calibration, the fundamental matrix

3D reconstruction from two views (Stereo algorithms)

Hartley & Zisserman: Sec. 2 Proj. Geom. of 2D.Sec. 3 Proj. Geom. of 3D.

Reading

Hartley & Zisserman:

Sec. 2 Proj. Geo. of 2D:• 2.1- 2.2.3 point lines in 2D• 2.3 -2.4 transformations • 2.7 line at infinity

Sec. 3 Proj. Geo. of 3D. • 3.1 – 3.2 point planes & lines. • 3.4 transformations

Euclidean Geometry is good for

questions like:

what objects have the same shape (= congruent)

Same shapes are related by rotation and translation

Why projective Geometry (Motivation)

Why Projective Geometry (Motivation) Answers the question what appearances

(projections) represent the same shape

Same shapes are related by a projective transformation

Where do parallel lines meet?

Parallel lines meet at the horizon (“vanishing line”)

Why Projective Geometry (Motivation)

Coordinates in Euclidean Space

0 1 2 3 ∞

Not in space

Coordinates in Projective Line P1

-1 0 1 2 ∞

k(0,1)

k(1,0)

k(2,1)k(1,1)k(-1,1)

Points on a line P1 are represented as rays from origin in 2D,Origin is excluded from space

“Ideal point”

Coordinates in Projective Plane P2

k(0,0,1)

k(x,y,0)

k(1,1,1)

k(1,0,1)

k(0,1,1)

“Ideal point”

Take R3 –{0,0,0} and look at scale equivalence class (rays/lines trough the origin).

z

y

x

z

y

x

Projective Line vs. the Real Line

-1 0 1 2 ∞

k(0,1)

k(1,0)

k(2,1)k(1,1)k(-1,1)

“Ideal point”

Symbol R P1

Space The real line R^2 – {0,0}

Objects (points) points Equivalence classes (2D “rays”)

Realization Intersection with line y=1

Projective Plane vs Euclidian plane

k(0,0,1)

k(x,y,0)

k(1,1,1)

k(1,0,1)

k(0,1,1)“Ideal line”

Symbol R2 P2

Space The real plane R3 – {0,0,0}

Objects (points) point Equivalence classes (3D rays)

Realization Intersection with plane z=1

2D Projective Geometry: Basics A point:

A line:

we denote a line with a 3-vector

Line coordinates are homogenous

Points and lines are dual: p is on l if

Intersection of two lines/points

2 2( , , ) ( , )T Tx yx y z P

z z

0 ( ) ( ) 0x y

ax by cz a b cz z

0Tl p

1 2 ,l l 1 2p p

( , , )Ta b c

ll

Cross Product

1 2 1 2 1 2

1 2 1 2 1 2

1 2 1 2 1 2

x x y z z y

y y z x x z

z z x y y x

0T Tw u v w u w v

Every entry is a determinant of the two other entries

w Area of parallelogram bounded by u and v

Hartley & Zisserman p. 581

Cross Product in matrix notation [ ]x

0

0

0

xy

xz

yz

x

tt

tt

tt

t1 2 1 2 1 2

1 2 1 2 1 2

1 2 1 2 1 2

x x y z z y

y y z x x z

z z x y y x

0

0

0

x y z z y

y z x z x

z x y y x

t x t z t y t t x

t y t x t z t t y

t z t y t x t t z

Hartley & Zisserman p. 581

ptpt x

Example: Intersection of parallel lines

00

)(

0

)(

)(

2122

21

21 a

b

a

b

cccca

ccb

ll

1 2 1 2 1 2

1 2 1 2 1 2

1 2 1 2 1 2

x x y z z y

y y z x x z

z z x y y x

Q: How many ideal points are there in P2?A: 1 degree of freedom family – the line at infinity

),,( ),,( 2211 cbalcbal

Projective Transformations

u

u’

Transformations of the projective line

dycx

byax

y

xG dc

ba

:

1/

//

dc

dbda

dc

ba

11'

''

1 xc

bxax G

Given a 2D linear transformation G:R2 R2 Study the induced transformation on the Equivalents classes.

1'

''

xc

bxax G

On the realization y=1 we get

Properties:1'

''

xc

bxax T

dc

baT

1. Invertible (T-1 exists) 2. Composable (To G is a projective transformation)3. Closed under composition

• Has 4 parameters • 3 degrees of freedom • Defined by 3 points

TT Every point defines 1 constraint

Transformations of the projective line

1P

Pencil of raysPerspective mapping

A perspective mapping is a projective transformation T:P1 P1

Perceptivity is a special projective mapping. Hartley & Zisserman p. 632Lines connecting corresponding points are “concurrent”

Ideal points and projective transformations

Projective transformation can map ∞ to a real point

Plane Perspective

2P

2D Projective Transformation

Projectivity: An invertible mapping h:P2 P2

S.T:

Homography. A 3x3 (non singular) invertible matrix acting on homogenous 3-vectores.

Collineation A transformations that map lines to lines

Hartley & Zisserman p. 32

line aon lie )(),(),( line aon lie ,, 321321 xhxhxhxxx

4 names 3 definitions

2D Projective Transformation

H is defined up to scale

9 parameters 8 degrees of freedom Determined by 4 corresponding points

how does H operate on lines?

0

1: 0 ( )( ) 0T T Tl H l l p l H Hp

Hartley & Zisserman p. 32

HH

Plane Perspective

2P

This mapping clearly maps lines to lines

Plane Perspective acting on conics

2P

Hartley & Zisserman p. 30 & 36Not part of this class

Rotation:Translation:

Hierarchy of Transformations

Rigid (Isometry)

Similarity

Affine

Projective

Scale

Hartley & Zisserman p. Sec. 2.4

cos sin, , det 1

sin cosTR R R I R

Rotation:

Translation:x

y

tt

t

2 2, 1, (2)a b

R a b R SOb a

Euclidean Transformations (Isometries)

q Rp t

Hierarchy of Transformations

Isometry (Euclidean),

Similarity,

Affine, general linear

Projective,

0 1

R t

,0 1

a bsR tsR

b a

, (2)0 1

A tA GL

(3) : , 0H GL q Hp

Invariants

Length Area Angles Parallelism

Isometry √ √ √ √

Similarity ××

(Scale)√ √

Affine × × × √

Projective × × × ×

Two-views geometryOutline

Background: Camera, Projection Necessary tools: A taste of projective geometry Two view geometry:

Homography Epipolar geometry, the essential matrix Camera calibration, the fundamental matrix

3D reconstruction from two views (Stereo algorithms)

Two View Geometry When a camera changes position and

orientation, the scene moves rigidly relative to the camera

3-D Scene

u

u’

X

Y

Z

d

p

Rotation + translation

3-D Scene

Rotation + translation

u

u’

X

Y

Z

d

p

Objective:

find formulas that links corresponding points

Two View Geometry (simple cases) In two cases this results in homography:

1. Camera rotates around its focal point

2. The scene is planar

Then: Point correspondence forms 1:1mapping depth cannot be recovered

Camera Rotation

' , 0

( )

'' ' ( ' ')

' ( ' )'

P RP t

Zp P P p

f

Zp P P p

f

Zp Rp p Rp

Z

(R is 3x3 non-singular)

Planar Scenes

IntuitivelyA sequence of two perspectivities

Algebraically

Need to show:

( )

1'

1, '

' ,'

T

TT

T

n P d aX bY cZ d

n PP RP t RP t R tn P

d d

H R tn P HPd

Zp Hp

Z

Scene

Camera 1

Camera 2

Hpp '

Summary: Two Views Related by HomographyTwo images are related by homography:

One to one mapping from p to p’ H contains 8 degrees of freedom Given correspondences, each point determines

2 equations 4 points are required to recover H Depth cannot be recovered

' ,'

Zp Hp

Z

The General Case: Epipolar Lines

epipolar lineepipolar line

Epipolar Plane

epipolar plane

epipolar lineepipolar lineepipolar lineepipolar line

BaselineBaseline

PP

OO O’O’

Epipole Every plane through the baseline is an epipolar

plane It determines a pair of epipolar lines (one in each image)

Two systems of epipolar lines are obtained Each system intersects in a point, the epipole The epipole is the projection of the center of the

other camera

epipolar planeepipolar linesepipolar linesepipolar linesepipolar lines

BaselineBaselineOO O’O’

Example

Epipolar Lines

epipolar plane

epipolar lineepipolar lineepipolar lineepipolar line

BaselineBaseline

PP

OO O’O’

To define an epipolar plane, we define the plane through the two camera centers O and O’ and some point P. This can be written algebraically (in some worldcoordinates as follows:

' ' 0T

OP OO O P

Essential Matrix (algebraic constraint between corresponding image points) Set world coordinates around the first camera

What to do with O’P? Every rotation changes the observed coordinate in the second image

We need to de-rotate to make the second image plane parallel to the first

Replacing by image points

' ' 0T

OP OO O P

' 0TP t RP

, 'P OP t OO

' 0Tp t Rp Other derivations Hartley & Zisserman p. 241

Essential Matrix (cont.)

Denote this by:

Then

Define

E is called the “essential matrix”

t p t p

' ' 0T Tp t Rp p t Rp

E t R

' 0Tp Ep

' 0Tp t Rp

Properties of the Essential Matrix E is homogeneous Its (right and left) null spaces are the two epipoles 9 parameters Is linear E, E can be recovered up to scale using 8 points. Has rank 2.

The constraint detE=0 7 points suffices In fact, there are only 5 degrees of freedom in E,

3 for rotation 2 for translation (up to scale), determined by epipole

0 ': l plpE t

' 0Tp Ep

e) trough lines ( : : 12 all PPEThus

BackgroundThe lens optical axis does not coincide with

the sensor

We model this using a 3x3 matrix the Calibration matrix

Camera Internal Parameters or Calibration matrix

Camera Calibration matrix

The difference between ideal sensor ant the real one is modeled by a 3x3 matrix K:

(cx,cy) camera center, (ax,ay) pixel dimensions, b skew

We end with

0

0 0 1

x x

y y

a b c

K a c

q Kp

Fundamental Matrix

F, is the fundamental matrix.

1 1

1

1

' 0 ( ) ( ') 0

( ) ' 0

T T

T T

T

p Ep K q E K q

q K EK q

F K EK

Properties of the Fundamental Matrix F is homogeneous Its (right and left) null spaces are the two epipoles 9 parameters Is linear F, F can be recovered up to scale using 8 points. Has rank 2.

The constraint detF=0 7 points suffices

e) trough lines ( : 12 all PPF

0'Fpp t

Homography Epipolar

Form

Shape One-to-one map Concentric epipolar lines

D.o.f. 8 8/5 F/E

Eqs/pnt 2 1

Minimal configuration

4 5+ (8, linear)

Depth No Yes, up to scale

Scene Planar

(or no translation)

3D scene

Two-views geometry Summary:

0'Fpp tHpp '