estimation of transformations

Estimation of Transformations

簡韶逸 Shao-Yi Chien

Department of Electrical Engineering

National Taiwan University

Spring 2021

1

Outline

• Estimation – 2D Projective Transformation

2

[Slides credit: Marc Pollefeys]

Parameter Estimation

• 2D homographyGiven a set of (xi,xi’), compute H (xi’=Hxi)

• 3D to 2D camera projectionGiven a set of (Xi,xi), compute P (xi=PXi)

• Fundamental matrixGiven a set of (xi,xi’), compute F (xi’TFxi=0)

• Trifocal tensor Given a set of (xi,xi’,xi”), compute T

Number of Measurements Required• At least as many independent equations as degrees

of freedom required

• Example:

Hxx'=

=

11

λ

333231

232221

131211

y

x

hhh

hhh

hhh

y

x

2 independent equations / point8 degrees of freedom

4x2≥8

Approximate Solutions

• Minimal solution• 4 points yield an exact solution for H

• More points• Robust estimation algorithms, such as RANSAC

• No exact solution, because measurements are inexact (“noise”)

• Search for “best” according to some cost function• Algebraic or geometric/statistical cost

Gold Standard Algorithm

• Cost function that is optimal for some assumptions

• Computational algorithm that minimizes it is called “Gold Standard” algorithm

• Other algorithms can then be compared to it

Direct Linear Transformation(DLT)

ii Hxx = 0Hxx = ii

=

i

i

i

i

xh

xh

xh

Hx3

2

1

T

T

T

−

−

−

=

iiii

iiii

iiii

ii

yx

xw

wy

xhxh

xhxh

xhxh

Hxx12

31

23

TT

TT

TT

0

h

h

h

0xx

x0x

xx0

3

2

1

=

−

−

−

TTT

TTT

TTT

iiii

iiii

iiii

xy

xw

yw

( )Tiiii wyx = ,,x

0hA =i

→

Direct Linear Transformation(DLT)

• Equations are linear in h

0

h

h

h

0xx

x0x

xx0

3

2

1

=

−

−

−

TTT

TTT

TTT

iiii

iiii

iiii

xy

xw

yw

0hA =i

• Only 2 out of 3 are linearly independent

(indeed, 2 eq/pt)

Direct Linear Transformation(DLT)

• Equations are linear in h0hA =i

• Only 2 out of 3 are linearly independent

(indeed, 2 eq/pt)

0

h

h

h

x0x

xx0

3

2

1

=

−

−TTT

TTT

iiii

iiii

xw

yw

• Holds for any homogeneous

representation, e.g. (xi’,yi’,1)

Direct Linear Transformation(DLT)

• Solving for H

0Ah =0h

A

A

A

A

4

3

2

1

=

size A is 8x9 or 12x9, but rank 8

Trivial solution is h=09T is not interesting

1-D null-space yields solution of interest

pick for example the one with 1h =

Direct Linear Transformation(DLT)

• Over-determined solution

No exact solution because of inexact measurement

i.e. “noise”

0Ah = 0h

A

A

A

n

2

1

=

Find approximate solution

- Additional constraint needed to avoid 0, e.g.

- not possible, so minimize

1h =

Ah0Ah =

DLT Algorithm

Objective

Given n≥4 2D to 2D point correspondences {xi↔xi’},

determine the 2D homography matrix H such that xi’=Hxi

Algorithm

(i) For each correspondence xi↔xi’ compute Ai. Usually

only the first two rows are needed.

(ii) Assemble n 2x9 matrices Ai into a single 2nx9 matrix A

(iii) Obtain SVD of A. Solution for h is the last column of V

(iv) Determine H from h

Inhomogeneous Solution

−=

−−−

'

'h~

''000'''

'''''000

ii

ii

iiiiiiiiii

iiiiiiiiii

xw

yw

xyxxwwwywx

yyyxwwwywx

Since h can only be computed up to scale,

pick hj=1, e.g. h9=1, and solve for 8-vector h~

Solve using Gaussian elimination (4 points) or using linear least-squares (more than 4 points)

However, if h9=0 this approach fails also poor results if h9 close to zero Therefore, not recommended

Note h9=H33=0 if origin is mapped to infinity

0

1

0

0

H100Hxl 0 =

=

T

Degenerate Configurations

x4

x1

x3

x2

x4

x1

x3

x2H? H’?

x1

x3

x2

x4

0Hxx = iiConstraints: i=1,2,3,4

TlxH 4

* =Define:

( ) 4444

* xxlxxH == kT

( ) 3,2,1 ,0xlxxH 4

* === iii

TThen,

H* is rank-1 matrix and thus not a homography

(case A) (case B)

If H* is unique solution, then no homography mapping xi→xi’(case B)If further solution H exist, then also αH*+βH (case A) (2-D null-space in stead of 1-D null-space)

Solutions from Lines

ii lHl T= 0Ah =

2D homographies from 2D lines

Minimum of 4 lines

Minimum of 5 points or 5 planes

3D Homographies (15 dof)

2D affinities (6 dof)

Minimum of 3 points or lines

Conic provides 5 constraints

Solutions from Mixed Type

• 2D homography• cannot be determined uniquely from the

correspondence of 2 points and 2 line

• can from 3 points and 1 line or 1 point and 3 lines

16

Cost Functions

• Algebraic distance

• Geometric distance

• Reprojection error

• Comparison

• Geometric interpretation

• Sampson error

Algebraic Distance

AhDLT minimizes

Ah=e residual vector

ie partial vector for each (xi↔xi’)

algebraic error vector

( )2

22

alg hx0x

xx0Hx,x

−−

−−==

TTT

TTT

iiii

iiii

iiixw

ywed

algebraic distance

( ) 2

2

2

1

2

21alg x,x aad += where ( ) 21

T

321 xx,,a == aaa

( ) 2222

alg AhHx,x eedi

i

i

ii === Not geometrically/statistically meaningfull, but given good normalization it works fine and is very fast (use for initialization for non-linear minimization)

Geometric Distancemeasured coordinates

estimated coordinates

true coordinates

xxx

( )2H

xH,xargminH ii

i

d =

Error in one image e.g. calibration pattern

( ) ( )221-

H

Hx,xxH,xargminH iiii

i

dd +=

Symmetric transfer error

d(.,.) Euclidean distance (in image)

( ) ( ) ( )

ii

iiii

i

ii ddii

xHx subject to

x,xx,xargminx,x,H22

x,xH,

=

+=

Reprojection error

Symmetric Transfer Error v.s. Reprojection Error

( ) ( )221- Hx,xxHx, + dd

( ) ( )22x,xxx, + dd

Symmetric Transfer Error

Reprojection Error

Comparison of Geometric and Algebraic Distances

−

−==

iiii

iiii

iiwxxw

ywwye

ˆˆ

ˆˆhA

Error in one image

( )Tiiii wyx = ,,x ( ) xHˆ,ˆ,ˆx ==T

iiii wyx

−

−

3

2

1

h

h

h

x0x

xx0TTT

TTT

iiii

iiii

xw

yw

( ) ( ) ( )222

iialgˆˆˆˆx,x iiiiiiii wxxwywwyd −+−=

( ) ( ) ( )( )( ) ii

iiiiiiii

wwd

wxwxwywyd

=

−+−=

ˆ/x,x

/ˆ/ˆˆ/ˆ/x,x

iialg

2/1222

ii

typical, but not , except for affinities 1=iw i3xhˆ =iw

➔ For affinities, DLT can minimize geometric distance

these two distance metrics are related, but not identical

Sampson Error

2

XX −

HνVector that minimizes the geometric error is the closest point on the variety to the measurement

XX

between algebraic and geometric error

XSampson error: 1st order approximation of

( ) 0XAh H ==C

( ) ( ) XH

HXH δX

XδX

+=+

CCC XXδX −= ( ) 0XH =C

( ) 0δX

X XH

H =

+

CC e−=XJδ

Find the vector that minimizes subject to Xδ e−=XJδXδ

XJ H

with

=

C

Find the vector that minimizes subject to Xδ e−=XJδXδ

Use Lagrange multipliers:

minimize

derivatives

( ) 0Jδ2λ-δδ XXX =+ eT

TT 0J2λ-δ2 X =

( ) 0Jδ2 X =+ e

λJδ X

T=

0λJJ T =+ e

( ) e1TJJλ −

−=

( ) e1TT

X JJJδ −

−=

XδXX += ( ) ee1TT

X

T

X

2

X JJδδδ −

==

Sampson Error

Sampson Error

2

XX −

HνVector that minimizes the geometric error is the closest point on the variety to the measurement

XX

between algebraic and geometric error

XSampson error: 1st order approximation of

( ) 0XAh H ==C

( ) ( ) XH

HXH δX

XδX

+=+

CCC XXδX −= ( ) 0XH =C

( ) 0δX

X XH

H =

+

CC e−=XJδ

Find the vector that minimizes subject to Xδ e−=XJδXδ

XJ H

with

=

C

( ) ee1TT

X

T

X

2

X JJδδδ −

== (Sampson error)

Sampson Approximation

A few points

(i) For a 2D homography X=(x,y,x’,y’)

(ii) is the algebraic error vector

(iii) is a 2x4 matrix, e.g.

(iv) Similar to algebraic error in fact, same as Mahalanobis distance

(v) Sampson error independent of linear reparametrization

(cancels out in between e and J)

(vi) Must be summed for all points

(vii) Close to geometric error, but much fewer unknowns

( ) ee1TT2

X JJδ −

=

eee T2 =

2

JJT e

( ) 3121

3T2T

11 /hxhx hyhwxywJ iiiiii+−=+−=X

J H

=

C

( )XHCe =

( )−

ee1TT JJ

Statistical Cost Function and Maximum Likelihood Estimation

• Optimal cost function related to noise model of measurement

• Assume zero-mean isotropic Gaussian noise (assume outliers removed)

( ) ( ) ( )222/xx,

2πσ2

1xPr de−=

( ) ( ) ( )22

i 2xH,x /

2iπσ2

1H|xPr

ide

i

−=

( ) ( ) +−= constantxH,xH|xPrlog2

i2iσ2

1id

Error in one image

Maximum Likelihood Estimate

( ) 2

i xH,x id Equivalent to minimizing the geometric error function

Statistical Cost Function and Maximum Likelihood Estimation

• Optimal cost function related to noise model of measurement

• Assume zero-mean isotropic Gaussian noise (assume outliers removed)

( ) ( ) ( )222/xx,

2πσ2

1xPr de−=

( )( ) ( ) ( )22

i2

i 2xH,xx,x /

2iπσ2

1H|xPr

+−

=ii dd

ei

Error in both images

Maximum Likelihood Estimate

( ) ( )2i

2

i x,xx,x ii dd +Equivalent to minimizing the reprojection error function

Mahalanobis Distance

• General Gaussian case

Measurement X with covariance matrix Σ

( ) ( )XXXXXX 1T2

−−=− −

22

XXXX−+−

Error in two images (independent)

22

XXXXii

ii

i

ii −+−

Varying covariances

Invariance to Transforms ?

xTx~ =

Txx~ =Hxx = x~H

~x~ =

TH~

TH 1-?

=

TxH~

xT =

TxH~

Tx -1=

will result change?

for which algorithms? for which transformations?

Non-invariance of DLT

Given and H computed by DLT,

and

Does the DLT algorithm applied to

yield ?

iiii xTx~,Txx~ ==

ii xx

ii x~x~ -1HTTH

~=

( ) iiiiie TxHTTxTx~H~

x~~ -1== ( ) iii e** THxxT ==

( ) ( ) hA,R~,~h~

A~

i2121i seesee ===TT

Effect of change of coordinates on algebraic error

=

10

tsRT

=

ss

Rt-

0RT

T

*

for similarities

so

( ) ( )iiii sdd x~H~

,x~Hx,x algalg=

Non-invariance of DLT

(T*: cofactor matrix)

Non-invariance of DLT

Given and H computed by DLT,

and

Does the DLT algorithm applied to

yield ?

iiii xTx~,Txx~ ==

ii xx

ii x~x~ -1HTTH

~=

( )

( )

( ) 1H~

subject tox~H~

,x~minimize

1H subject tox~H~

,x~minimize

1H subject toHx,xminimize

2

alg

2

alg

2

alg

=

=

=

i

ii

i

ii

i

ii

d

d

d

Invariance of Geometric Error

( ) ( ) ( )

( )ii

iiiiii

sd

ddd

Hx,x

HxT,xTTxHTT,xTx~H~

,x~ -1

=

==

Given and H,

and

Assume T’ is a similarity transformations

,xTx~,Txx~ iiii==

ii xx

,x~x~ ii -1HTTH

~=

Normalizing Transformations

• Since DLT is not invariant,

what is a good choice of coordinates?e.g. Isotropic scaling

• Translate centroid to origin

• Scale to a average distance to the origin

• Independently on both images2

1

norm

100

2/0

2/0

T

−

+

+

= hhw

whwOr

Importance of Normalization

0

h

h

h

0001

1000

3

2

1

=

−−−

−−−

iiiiiii

iiiiiii

xyxxxyx

yyyxyyx

~102 ~102 ~102 ~102 ~104 ~104 ~10211

orders of magnitude difference!

Without normalization With normalization

Normalized DLT AlgorithmObjective

Given n≥4 2D to 2D point correspondences {xi↔xi’},

determine the 2D homography matrix H such that xi’=Hxi

Algorithm

(i) Normalize points

(ii) Apply DLT algorithm to

(iii) Denormalize solution

,x~x~ ii

inormiinormi xTx~,xTx~ ==

norm

-1

norm TH~

TH =

Employ this algorithm instead of the original DLT algorithm!• More accurate• Invariant to arbitrary choices of the scale and coordinate origin

Normalization is also called pre-conditioning

Iterative Minimization MethodsRequired to minimize geometric error

(i) Often slower than DLT

(ii) Require initialization

(iii) No guaranteed convergence, local minima

(iv) Stopping criterion required

Therefore, careful implementation required:

(i) Cost function

(ii) Parameterization (minimal or not)

(iii) Cost function ( parameters )

(iv) Initialization

(v) Iterations

Parameterization

Parameters should cover complete space and allow efficient estimation of cost

• Minimal or over-parameterized? e.g. 8 or 9

(minimal often more complex, also cost surface)

(good algorithms can deal with over-parameterization)

(sometimes also local parameterization)

• Parametrization can also be used to restrict transformation to particular class, e.g. affine

Function Specifications

(i) Measurement vector X N with covariance Σ

(ii) Set of parameters represented by vector P M

(iii) Mapping f : M → N. Range of mapping is surface S representing allowable measurements

(iv) Cost function: squared Mahalanobis distance

Goal is to achieve , or get as close as

possible in terms of Mahalanobis distance

( ) ( )( ) ( )( )PXPXPX 1T2fff −−=− −

( ) XP =f

Error in one image

( ) 2

i xH,x id

( )nf Hx,...,Hx,Hxh: 21→

( )hX f−

( ) ( )221- Hx,xxH,x iiii

i

dd +

Symmetric transfer error

( )nnf Hx,...,Hx,Hx,xH,...,xH,xHh: 21

-1

2

-1

1

-1 →

( )hX f−

Reprojection error

( ) ( )2i

2

i x,xx,x ii dd +

( )hX f−

X composed of 2n inhomogeneous coordinates of the points 𝑥𝑖

′

X composed of 4n-vector inhomogeneous

coordinates of the points𝑥𝑖 and 𝑥𝑖′

X composed of 4n-vector

Initialization

• Typically, use linear solution

• If outliers, use robust algorithm

• Alternative, sample parameter space

Iteration Methods

Many algorithms exist

• Newton’s method

• Levenberg-Marquardt

• Powell’s method

• Simplex method

Levenberg-Marquardt Algorithm

For a mapping function f with parameter vector

To an estimated measurement vector

We want to find p that can minimize , where

f(p) can be approximated as

with small and J=

➔Find to minimize

44

Levenberg-Marquardt Algorithm

Find to minimize

The least-square solution:

Augmented normal equation (with damping term ):

45

Hessian

Gold Standard AlgorithmObjective

Given n≥4 2D to 2D point correspondences {xi↔xi’}, determine

the Maximum Likelyhood Estimation of H

(this also implies computing optimal xi’=Hxi)

Algorithm

(i) Initialization: compute an initial estimate using normalized DLT

or RANSAC

(ii) Geometric minimization of -Either Sampson error:

● Minimize the Sampson error

● Minimize using Levenberg-Marquardt over 9 entries of h

or Gold Standard error:

● compute initial estimate for optimal {xi}

● minimize cost over {H,x1,x2,…,xn}

● if many points, use sparse method

( ) ( )2i

2

i x,xx,x ii dd +

Robust Estimation

• What if set of matches contains gross outliers?

RANSAC: RANdom SAmple Consensus

Objective

Robust fit of model to data set S which contains outliers

Algorithm

(i) Randomly select a sample of s data points from S and

instantiate the model from this subset.

(ii) Determine the set of data points Si which are within a

distance threshold t of the model. The set Si is the

consensus set of samples and defines the inliers of S.

(iii) If the subset of Si is greater than some threshold T, re-

estimate the model using all the points in Si and terminate

(iv) If the size of Si is less than T, select a new subset and

repeat the above.

(v) After N trials the largest consensus set Si is selected, and

the model is re-estimated using all the points in the

subset Si

Distance Threshold

Choose t so probability for inlier is α (e.g. 0.95)

• Often empirically

• Zero-mean Gaussian noise σ then follows

distribution with m=codimension of model

2

⊥d2

m

(dimension+codimension=dimension space)

Codimension Model t 2

1 Line (l), Fundamental matrix (F) 3.84σ2

2 Homography (H), Camera Matrix (P) 5.99σ2

3 Trifocal tensor (T) 7.81σ2

How Many Samples?

Choose N so that, with probability p, at least one random sample is free from outliers. e.g. p=0.99

( ) ( )( )sepN −−−= 11log/1log

( )( ) peNs

−=−− 111

proportion of outliers e

s 5% 10% 20% 25% 30% 40% 50%2 2 3 5 6 7 11 173 3 4 7 9 11 19 354 3 5 9 13 17 34 725 4 6 12 17 26 57 1466 4 7 16 24 37 97 2937 4 8 20 33 54 163 5888 5 9 26 44 78 272 1177

Acceptable Consensus Set?

• Typically, terminate when inlier ratio reaches expected ratio of inliers

( )neT −= 1

Adaptively Determining the Number of Samples

e is often unknown a priori, so pick worst case, e.g. 50%, and adapt if more inliers are found, e.g. 80% would yield e=0.2

• N=∞, sample_count =0

• While N >sample_count repeat• Choose a sample and count the number of inliers

• Set e=1-(number of inliers)/(total number of points)

• Recompute N from e

• Increment the sample_count by 1

• Terminate( ) ( )( )( )s

epN −−−= 11log/1log

Robust Maximum Likelyhood Estimation

Previous MLE algorithm considers fixed set of inliers

Better, robust cost function (reclassifies)

( ) ( )

== ⊥

outlier

inlier ρ with ρ

222

222

i tet

teeed iR

Other Robust Algorithms

• RANSAC maximizes number of inliers

• LMedS minimizes median error

Automatic Computation of H

Objective

Compute homography between two images

Algorithm

(i) Interest points: Compute interest points in each image

(ii) Putative correspondences: Compute a set of interest point

matches based on some similarity measure

(iii) RANSAC robust estimation: Repeat for N samples

(a) Select 4 correspondences and compute H

(b) Calculate the distance d⊥ for each putative match

(c) Compute the number of inliers consistent with H (d⊥<t)

Choose H with most inliers

(iv) Optimal estimation: re-estimate H from all inliers by minimizing ML

cost function with Levenberg-Marquardt

(v) Guided matching: Determine more matches using prediction by

computed H

Optionally iterate last two steps until convergence

Determine Putative Correspondences

• Compare interest points

Similarity measure:

• SAD, SSD, ZNCC on small neighborhood

• If motion is limited, only consider interest points with similar coordinates

• More advanced approaches exist, based on invariance…• Such as SIFT

Example: robust computation

Interest points

(500/image)

Putative correspondences (268)

Outliers (117)

Inliers (151)

Final inliers (262)