Parameter estimationclass 6

Multiple View GeometryComp 290-089Marc Pollefeys

Content

• Background: Projective geometry (2D, 3D), Parameter estimation, Algorithm evaluation.

• Single View: Camera model, Calibration, Single View Geometry.

• Two Views: Epipolar Geometry, 3D reconstruction, Computing F, Computing structure, Plane and homographies.

• Three Views: Trifocal Tensor, Computing T.• More Views: N-Linearities, Multiple view

reconstruction, Bundle adjustment, auto-calibration, Dynamic SfM, Cheirality, Duality

Multiple View Geometry course schedule(subject to change)

Jan. 7, 9 Intro & motivation Projective 2D Geometry

Jan. 14, 16

(no class) Projective 2D Geometry

Jan. 21, 23

Projective 3D Geometry (no class)

Jan. 28, 30

Parameter Estimation Parameter Estimation

Feb. 4, 6 Algorithm Evaluation Camera Models

Feb. 11, 13

Camera Calibration Single View Geometry

Feb. 18, 20

Epipolar Geometry 3D reconstruction

Feb. 25, 27

Fund. Matrix Comp. Structure Comp.

Mar. 4, 6 Planes & Homographies Trifocal Tensor

Mar. 18, 20

Three View Reconstruction

Multiple View Geometry

Mar. 25, 27

MultipleView Reconstruction

Bundle adjustment

Apr. 1, 3 Auto-Calibration Papers

Apr. 8, 10

Dynamic SfM Papers

Apr. 15, 17

Cheirality Papers

Apr. 22, 24

Duality Project Demos

Parameter estimation

• 2D homographyGiven a set of (xi,xi’), compute H (xi’=Hxi)

• 3D to 2D camera projectionGiven a set of (Xi,xi), compute P (xi=PXi)

• Fundamental matrixGiven a set of (xi,xi’), compute F (xi’TFxi=0)

• Trifocal tensor Given a set of (xi,xi’,xi”), compute T

DLT algorithmObjective

Given n≥4 2D to 2D point correspondences {xi↔xi’}, determine the 2D homography matrix H such that xi’=Hxi

Algorithm

(i) For each correspondence xi ↔xi’ compute Ai. Usually only two first rows needed.

(ii) Assemble n 2x9 matrices Ai into a single 2nx9 matrix A

(iii) Obtain SVD of A. Solution for h is last column of V

(iv) Determine H from h

Geometric distancemeasured coordinatesestimated coordinatestrue coordinates

xxx

2H

xH,xargminH iii

d Error in one image

e.g. calibration pattern

221-

HHx,xxH,xargminH iiii

i

dd Symmetric transfer error

d(.,.) Euclidean distance (in image)

Reprojection error

22

x,xH,x,xx,xargminx,x,H iiii

iii dd

ii

ii xHx subject to

Geometric interpretation of reprojection error

Estimating homography~fit surface to points X=(x,y,x’,y’)T in 4Hν

2H22 ,Xx,xx,x iiiii ddd

Analog to conic fitting CxxC,x T2

alg d

2C,xd

eedSampson

1TT2 JJCx,

Statistical cost function and Maximum Likelihood

Estimation• Optimal cost function related to noise

model• Assume zero-mean isotropic Gaussian

noise (assume outliers removed) 22 2/xx,2πσ2

1xPr de

22i 2xH,x /

2iπσ2

1H|xPr ide

i

constantxH,xH|xPrlog 2i2i

σ2

1id

Error in one image

Maximum Likelihood Estimate

2i xH,x id

Statistical cost function and Maximum Likelihood

Estimation• Optimal cost function related to noise

model• Assume zero-mean isotropic Gaussian

noise (assume outliers removed) 22 2/xx,2πσ2

1xPr de

22i

2i 2xH,xx,x /

2iπσ2

1H|xPr

ii dd

ei

Error in both images

Maximum Likelihood Estimate

2i2

i x,xx,x ii dd

Mahalanobis distance

• General Gaussian caseMeasurement X with covariance

matrix Σ

XXXXXX 1T2

22XXXX

Error in two images (independent)

22XXXX

iiii

iii

Varying covariances

Invariance to transforms ?

xTx~ Txx~ Hxx x~H

~x~

TH~

TH 1-?

TxH~

xT TxH

~Tx -1

will result change? for which algorithms? for which transformations?

Non-invariance of DLT

Given and H computed by DLT,

and

Does the DLT algorithm applied to yield ?

iiii xTx~,Txx~ ii xx

ii x~x~ -1HTTH

~

iiiiie TxHTTxTx~H~

x~~ -1 iii e** THxxT

hA,R~,~h~

A~

i2121i seesee TT

Effect of change of coordinates on algebraic error

10

tsRT

ss

Rt-

0RT T

*

for similarities

so

iiii sdd x~H~

,x~Hx,x algalg

Non-invariance of DLT

Given and H computed by DLT,

and

Does the DLT algorithm applied to yield ?

iiii xTx~,Txx~ ii xx

ii x~x~ -1HTTH

~

1H

~ subject tox~H

~,x~minimize

1H subject tox~H~

,x~minimize

1H subject toHx,xminimize

2

alg

2

alg

2alg

iii

iii

iii

d

d

d

Invariance of geometric error

ii

iiiiii

sd

ddd

Hx,x

HxT,xTTxHTT,xTx~H~

,x~ -1

Given and H,

and

Assume T’ is a similarity transformations

,xTx~,Txx~ iiii ii xx

,x~x~ ii -1HTTH~

Normalizing transformations

• Since DLT is not invariant,what is a good choice of coordinates?e.g.• Translate centroid to origin• Scale to a average distance to the

origin• Independently on both images

2

1

norm

100

2/0

2/0

T

hhw

whwOr

Importance of normalization

0

h

h

h

0001

1000

3

2

1

iiiiiii

iiiiiii

xyxxxyx

yyyxyyx

~102 ~102 ~102 ~102 ~104 ~104 ~10211

orders of magnitude difference!

Normalized DLT algorithmObjective

Given n≥4 2D to 2D point correspondences {xi↔xi’}, determine the 2D homography matrix H such that xi’=Hxi

Algorithm

(i) Normalize points

(ii) Apply DLT algorithm to

(iii) Denormalize solution

,x~x~ ii inormiinormi xTx~,xTx~

norm-1

norm TH~

TH

Iterative minimization metods

Required to minimize geometric error (i) Often slower than DLT(ii) Require initialization(iii) No guaranteed convergence, local minima(iv) Stopping criterion requiredTherefore, careful implementation required:(i) Cost function(ii) Parameterization (minimal or not)(iii) Cost function ( parameters )(iv) Initialization(v) Iterations

Parameterization

Parameters should cover complete space and allow efficient estimation of cost

• Minimal or over-parameterized? e.g. 8 or 9 (minimal often more complex, also cost surface)

(good algorithms can deal with over-parameterization)

(sometimes also local parameterization)

• Parametrization can also be used to restrict transformation to particular class, e.g. affine

Function specifications

(i) Measurement vector XN with covariance Σ(ii) Set of parameters represented by vector P N

(iii) Mapping f : M →N. Range of mapping is surface S representing allowable measurements

(iv) Cost function: squared Mahalanobis distance

Goal is to achieve , or get as close as possible in terms of Mahalanobis distance

PXPXPX 1T2fff

XP f

Error in one image

2i xH,x id

nf Hx,...,Hx,Hxh: 21

hX f

221- Hx,xxH,x iiiii

dd Symmetric transfer error

nnf Hx,...,Hx,Hx,xH,...,xH,xHh: 21-1

2-1

1-1

hX f

Reprojection error

2i2

i x,xx,x ii dd nnf x,...,x,xx,...,x,xh,: 2121

hX f

Initialization

• Typically, use linear solution• If outliers, use robust algorithm

• Alternative, sample parameter space

Iteration methods

Many algorithms exist• Newton’s method• Levenberg-Marquardt

• Powell’s method• Simplex method

Gold Standard algorithmObjective

Given n≥4 2D to 2D point correspondences {xi↔xi’}, determine the Maximum Likelyhood Estimation of H

(this also implies computing optimal xi’=Hxi)

Algorithm

(i) Initialization: compute an initial estimate using normalized DLT or RANSAC

(ii) Geometric minimization of -Either Sampson error:

● Minimize the Sampson error

● Minimize using Levenberg-Marquardt over 9 entries of h

or Gold Standard error:

● compute initial estimate for optimal {xi}

● minimize cost over {H,x1,x2,…,xn}

● if many points, use sparse method

2i2

i x,xx,x ii dd

Robust estimation

• What if set of matches contains gross outliers?

RANSACObjective

Robust fit of model to data set S which contains outliers

Algorithm

(i) Randomly select a sample of s data points from S and instantiate the model from this subset.

(ii) Determine the set of data points Si which are within a distance threshold t of the model. The set Si is the consensus set of samples and defines the inliers of S.

(iii) If the subset of Si is greater than some threshold T, re-estimate the model using all the points in Si and terminate

(iv) If the size of Si is less than T, select a new subset and repeat the above.

(v) After N trials the largest consensus set Si is selected, and the model is re-estimated using all the points in the subset Si

Distance threshold

Choose t so probability for inlier is α (e.g. 0.95)

• Often empirically• Zero-mean Gaussian noise σ then

follows distribution with m=codimension of model

2d

2m

(dimension+codimension=dimension space)

Codimension

Model t 2

1 l,F 3.84σ2

2 H,P 5.99σ2

3 T 7.81σ2

How many samples?

Choose N so that, with probability p, at least one random sample is free from outliers. e.g. p=0.99

sepN 11log/1log

peNs 111

proportion of outliers es 5% 10% 20% 25% 30% 40% 50%2 2 3 5 6 7 11 173 3 4 7 9 11 19 354 3 5 9 13 17 34 725 4 6 12 17 26 57 1466 4 7 16 24 37 97 2937 4 8 20 33 54 163 5888 5 9 26 44 78 272 117

7

Acceptable consensus set?

• Typically, terminate when inlier ratio reaches expected ratio of inliers

neT 1

Adaptively determining the number of samples

e is often unknown a priori, so pick worst case, e.g. 50%, and adapt if more inliers are found, e.g. 80% would yield e=0.2

• N=∞, sample_count =0• While N >sample_count repeat

• Choose a sample and count the number of inliers• Set e=1-(number of inliers)/(total number of

points)• Recompute N from e• Increment the sample_count by 1

• Terminate

sepN 11log/1log

Robust Maximum Likelyhood Estimation

Previous MLE algorithm considers fixed set of inliers

Better, robust cost function (reclassifies)

outlier

inlier ρ with ρ

222

222

i tet

teeed iR

Other robust algorithms

• RANSAC maximizes number of inliers

• LMedS minimizes median error

• Not recommended: case deletion, iterative least-squares, etc.

Automatic computation of HObjective

Compute homography between two imagesAlgorithm

(i) Interest points: Compute interest points in each image

(ii) Putative correspondences: Compute a set of interest point matches based on some similarity measure

(iii) RANSAC robust estimation: Repeat for N samples

(a) Select 4 correspondences and compute H

(b) Calculate the distance d for each putative match

(c) Compute the number of inliers consistent with H (d<t)

Choose H with most inliers

(iv) Optimal estimation: re-estimate H from all inliers by minimizing ML cost function with Levenberg-Marquardt

(v) Guided matching: Determine more matches using prediction by computed H

Optionally iterate last two steps until convergence

Determine putative correspondences

• Compare interest pointsSimilarity measure:• SAD, SSD, ZNCC on small neighborhood

• If motion is limited, only consider interest points with similar coordinates

• More advanced approaches exist, based on invariance…

Example: robust computation

Interest points(500/image)

Putative correspondences (268)

Outliers (117)

Inliers (151)

Final inliers (262)

Assignment

• Take two or more photographs taken from a single viewpoint

• Compute panorama• Use different measures DLT, MLE

• Use Matlab • Due Feb. 13

Next class: Algorithm evaluation and error analysis

• Bounds on performance• Covariance propagation• Monte Carlo covariance

estimation