computer vision calibration marc pollefeys comp 256 many slides and illustrations from j. ponce

ComputerVision

Calibration

Marc PollefeysCOMP 256

Many slides and illustrations from J. Ponce

ComputerVision

Aug 26/28 - Introduction

Sep 2/4 Cameras Radiometry

Sep 9/11 Sources & Shadows Color

Sep 16/18 Linear filters & edges

(hurricane Isabel)

Sep 23/25 Pyramids & Texture Multi-View Geometry

Sep30/Oct2 Stereo Project proposals

Oct 7/9 Tracking (Welch) Optical flow

Oct 14/16 - -

Oct 21/23 Silhouettes/carving (Fall break)

Oct 28/30 - Structure from motion

Nov 4/6 Project update Proj. SfM

Nov 11/13 Camera calibration Segmentation

Nov 18/20 Fitting Prob. segm.&fit.

Nov 25/27 Matching templates (Thanksgiving)

Dec 2/4 Matching relations Range data

Dec ? Final project

Tentative class schedule

ComputerVision GEOMETRIC CAMERA MODELS

• Elements of Euclidean Geometry

• The Intrinsic Parameters of a Camera

• The Extrinsic Parameters of a Camera

• The General Form of the Perspective Projection Equation

• Line Geometry

Reading: Chapter 2.

ComputerVision Quantitative Measurements and Calibration

Euclidean Geometry

ComputerVision Euclidean Coordinate Systems

z

y

x

zyxOP

OPz

OPy

OPx

Pkji

k

j

i

.

.

.

ComputerVision Planes

1

and where

0.00.

zyx

dcba

dczbyaxAP

PΠ

PΠn

ComputerVision Coordinate Changes: Pure Translations

OBP = OBOA + OAP , BP = AP + BOA

ComputerVision

Coordinate Changes: Pure Rotations

BABABA

BABABA

BABABABA R

kkkjki

jkjjji

ikijii

...

...

...

AB

AB

AB kji

TB

A

TB

A

TB

A

k

j

i

ComputerVision

Coordinate Changes: Rotations about the z Axis

100

0cossin

0sincos

RBA

ComputerVision A rotation matrix is characterized by the following properties:

• Its inverse is equal to its transpose, and

• its determinant is equal to 1.

Or equivalently:

• Its rows (or columns) form a right-handedorthonormal coordinate system.

ComputerVision Coordinate Changes:

Pure Rotations

PRP

z

y

x

z

y

x

OP

ABA

B

B

B

B

BBBA

A

A

AAA

kjikji

ComputerVision Coordinate Changes: Rigid Transformations

ABAB

AB OPRP

ComputerVision

Block Matrix Multiplication

2221

1211

2221

1211

BB

BBB

AA

AAA

What is AB ?

2222122121221121

2212121121121111

BABABABA

BABABABAAB

Homogeneous Representation of Rigid Transformations

11111

PT

OPRPORP ABA

ABAB

AA

TA

BBA

B

0

ComputerVision Rigid Transformations as Mappings

ComputerVision

Rigid Transformations as Mappings: Rotation about the k Axis

ComputerVision Pinhole Perspective Equation

z

yfy

z

xfx

''

''

ComputerVision

The Intrinsic Parameters of a Camera

Normalized ImageCoordinates

Physical Image Coordinates

Units:

k,l : pixel/m

f : m

: pixel

ComputerVision

The Intrinsic Parameters of a Camera

Calibration Matrix

The PerspectiveProjection Equation

1

0 .. 2

2h

w

ff

Kge

ComputerVision

Extrinsic Parameters

ComputerVision

Explicit Form of the Projection Matrix

Note:

M is only defined up to scale in this setting!!

ComputerVision

Theorem (Faugeras, 1993)

ComputerVision GEOMETRIC CAMERA CALIBRATION

• The Calibration Problem

• Least-Squares Techniques

• Linear Calibration from Points

• Linear Calibration from Lines

• Analytical Photogrammetry

Reading: Chapter 3

ComputerVision

Calibration Problem

ComputerVision Linear Systems

A

A

x

x b

b=

=

Square system:

• unique solution

• Gaussian elimination

Rectangular system ??

• underconstrained: infinity of solutions

Minimize |Ax-b|2

• overconstrained: no solution

ComputerVision

How do you solve overconstrained linear equations ??

ComputerVision

Homogeneous Linear Systems

A

A

x

x 0

0=

=

Square system:

• unique solution: 0

• unless Det(A)=0

Rectangular system ??

• 0 is always a solution

Minimize |Ax| under the constraint |x| =12

2

ComputerVision

How do you solve overconstrained homogeneous linear equations ??

The solution is e .1

remember: EIG(ATA)=SVD(A), i.e. solution is Vn

ComputerVision Linear Camera Calibration

ComputerVision Once M is known, you still got to recover the intrinsic and

extrinsic parameters !!!

This is a decomposition problem, not an estimationproblem.

• Intrinsic parameters

• Extrinsic parameters

ComputerVision

Degenerate Point Configurations

Are there other solutions besides M ??

• Coplanar points: ()=() or () or ()

• Points lying on the intersection curve of two quadricsurfaces = straight line + twisted cubic

Does not happen for 6 or more random points!

ComputerVision

Analytical Photogrammetry

Non-Linear Least-Squares Methods

• Newton• Gauss-Newton• Levenberg-Marquardt

Iterative, quadratically convergent in favorable situations

ComputerVision Mobile Robot Localization (Devy et al., 1997)

ComputerVision

Absolute scale cannot be recovered! The Euclidean shape(defined up to an arbitrary similitude) is the best that can berecovered.

From Projective to Euclidean Images

If z , P , R and t are solutions, so are z , P , R and t .

ComputerVision From uncalibrated to calibrated cameras

Perspective camera:

Calibrated camera:

Problem: what is Q ?

ComputerVision From uncalibrated to calibrated cameras II

Perspective camera:

Calibrated camera:

Problem: what is Q ?

Example: known image center

ComputerVision Sequential SfM

• Initialize motion from two images• Initialize structure• For each additional view

– Determine pose of camera– Refine and extend structure

• Refine structure and motion

ComputerVision

Initial projective camera motion

• Choose P and P´compatible with F

Reconstruction up to projective ambiguity

(reference plane;arbitrary)

•Initialize motion•Initialize structure•For each additional view

•Determine pose of camera•Refine and extend structure

•Refine structure and motion

Same for more views?

different projective basis

ComputerVision Initializing projective structure

• Reconstruct matches in projective frame by minimizing the reprojection error

Non-iterative optimal solution •Initialize motion•Initialize structure•For each additional view

•Determine pose of camera•Refine and extend structure

•Refine structure and motion

ComputerVision Projective pose estimation

• Infere 2D-3D matches from 2D-2D matches

• Compute pose from (RANSAC,6pts)

F

X

x

Inliers: inx,X x X DD iii P

•Initialize motion•Initialize structure•For each additional view

•Determine pose of camera•Refine and extend structure

•Refine structure and motion

ComputerVision

• Refining structure

• Extending structure2-view triangulation

X~

P

1

3

(Iterative linear)

•Initialize motion•Initialize structure•For each additional view

•Determine pose of camera•Refine and extend structure

•Refine structure and motion

Refining and extending structure

ComputerVision Refining structure and motion

• use bundle adjustment

Also model radial distortion to avoid bias!

ComputerVision Metric structure and motion

Note that a fundamental problem of the uncalibrated approach is that it fails if a purely planar scene is observed (in one or more views)

(solution possible based on model selection)

use self-calibration (see next class)

ComputerVision Dealing with dominant planes

ComputerVision

PPPgric

HHgric

ComputerVision Farmhouse 3D models

(note: reconstruction much larger than camera field-of-view)

ComputerVision Application: video augmentation

ComputerVision Next class:

Segmentation

Reading: Chapter 14