singular value decomposition

Singular Value Decomposition

Tnnnmmmnm VΣUA

IUU T

021 n

IVV T

nm

XXVT XVΣ T XVUΣ T

TTTnnn VUVUVUA 222111

000

00

00

00

Σ

2

1

n


• Homogeneous least-squares

• Span and null-space

• Closest rank r approximation

• Pseudo inverse

1X AXmin subject to nVX solution

0 ,, 0 ,,,,~

21 rdiag TVΣ~

UA~ TVUΣA

0000

0000

000

000

Σ 2

1

4321 UU;UU LL NS 4321 VV;VV RR NS

TUVΣA 0 ,, 0 ,,,, 112

11 rdiag

TVUΣA


XXVT XVΣ T XVUΣ T

Homogeneous least-squares

TVUΣA

1X AXmin subject to nVX solution

Parameter estimation

• 2D homographyGiven a set of (xi,xi’), compute H (xi’=Hxi)

• 3D to 2D camera projectionGiven a set of (Xi,xi), compute P (xi=PXi)

• Fundamental matrixGiven a set of (xi,xi’), compute F (xi’TFxi=0)

• Trifocal tensor Given a set of (xi,xi’,xi”), compute T

Number of measurements required

• At least as many independent equations as degrees of freedom required

• Example:

Hxx'

11

λ

333231

232221

131211

y

x

hhh

hhh

hhh

y

x

2 independent equations / point8 degrees of freedom

4x2≥8

Approximate solutions

• Minimal solution4 points yield an exact solution for H

• More points• No exact solution, because

measurements are inexact (“noise”)• Search for “best” according to some

cost function• Algebraic or geometric/statistical cost

Gold Standard algorithm

• Cost function that is optimal for some assumptions

• Computational algorithm that minimizes it is called “Gold Standard” algorithm

• Other algorithms can then be compared to it

Direct Linear Transformation(DLT)

ii Hxx 0Hxx ii

i

i

i

i

xh

xh

xh

Hx3

2

1

T

T

T

iiii

iiii

iiii

ii

yx

xw

wy

xhxh

xhxh

xhxh

Hxx12

31

23

TT

TT

TT

0

h

h

h

0xx

x0x

xx0

3

2

1

TTT

TTT

TTT

iiii

iiii

iiii

xy

xw

yw

Tiiii wyx ,,x

0hA i


• Equations are linear in h

0

h

h

h

0xx

x0x

xx0

3

2

1

TTT

TTT

TTT

iiii

iiii

iiii

xy

xw

yw

0AAA 321 iiiiii wyx

0hA i• Only 2 out of 3 are linearly independent

(indeed, 2 eq/pt)

•Holds for any homogeneous representation,

e.g. (xi’,yi’,1)

(only drop third row if wi’≠0)

0

h

h

h

x0x

xx0

3

2

1

TTT

TTT

iiii

iiii

xw

yw


• Solving for H

0Ah

0h

A

A

A

A

4

3

2

1

size A is 8x9 or 12x9, but rank 8

Trivial solution is h=09T is not interesting

1-D null-space yields solution of interestpick for example the one with 1h


• Over-determined solution

No exact solution because of inexact measurementi.e. “noise”

0Ah 0h

A

A

A

n

2

1

Find approximate solution- Additional constraint needed to avoid 0, e.g.

- not possible, so minimize

1h Ah0Ah

DLT algorithmObjective

Given n≥4 2D to 2D point correspondences {xi↔xi’}, determine the 2D homography matrix H such that xi’=Hxi

Algorithm

(i) For each correspondence xi ↔xi’ compute Ai. Usually only two first rows needed.

(ii) Assemble n 2x9 matrices Ai into a single 2nx9 matrix A

(iii) Obtain SVD of A. Solution for h is last column of V

(iv) Determine H from h

Inhomogeneous solution

'

'h~

''000'''

'''''000

ii

ii

iiiiiiiiii

iiiiiiiiii

xw

yw

xyxxwwwywx

yyyxwwwywx

Since h can only be computed up to scale, pick hj=1, e.g. h9=1, and solve for 8-vector

h~

Solve using Gaussian elimination (4 points) or using linear least-squares (more than 4 points)

However, if h9=0 this approach fails also poor results if h9 close to zeroTherefore, not recommendedNote h9=H33=0 if origin is mapped to infinity

0

1

0

0

H100Hxl 0

T

Degenerate configurations

x4

x1

x3

x2

x4

x1

x3

x2H? H’?

x1

x3

x2

x4

0Hxx iiConstraints: i=1,2,3,4

TlxH 4* Define:

4444* xxlxxH kT

3,2,1 ,0xlxxH 4* iii

TThen,

H* is rank-1 matrix and thus not a homography

(case A) (case B)

If H* is unique solution, then no homography mapping xi→xi’(case B)

If further solution H exist, then also αH*+βH (case A) (2-D null-space in stead of 1-D null-space)

Solutions from lines, etc.

ii lHl T 0Ah

2D homographies from 2D lines

Minimum of 4 lines

Minimum of 5 points or 5 planes

3D Homographies (15 dof)

2D affinities (6 dof)

Minimum of 3 points or lines

Conic provides 5 constraints

Mixed configurations?

Cost functions

• Algebraic distance• Geometric distance• Reprojection error

• Comparison• Geometric interpretation• Sampson error

Algebraic distanceAhDLT minimizes

Ahe residual vector

ie partial vector for each (xi↔xi’)algebraic error vector

2

22alg h

x0x

xx0Hx,x

TTT

TTT

iiii

iiiiiii

xw

ywed

algebraic distance

22

21

221alg x,x aad where 21

T321 xx,,a aaa

2222alg AhHx,x eed

ii

iii

Not geometrically/statistically meaningfull, but given good normalization it works fine and is very fast (use for initialization)

Geometric distancemeasured coordinatesestimated coordinatestrue coordinates

xxx

2H

xH,xargminH iii

d Error in one image

e.g. calibration pattern

221-

HHx,xxH,xargminH iiii

i

dd Symmetric transfer error

d(.,.) Euclidean distance (in image)

Reprojection error

22

x,xH,x,xx,xargminx,x,H iiii

iii dd

ii

ii xHx subject to

Reprojection error

221- Hx,xxHx, dd

22 x,xxx, dd

Comparison of geometric and algebraic distances

iiii

iiiiii wxxw

ywwye

ˆˆ

ˆˆhA

Error in one image

Tiiii wyx ,,x xHˆ,ˆ,ˆx Tiiii wyx

3

2

1

h

h

h

x0x

xx0TTT

TTT

iiii

iiii

xw

yw

222iialg ˆˆˆˆx,x iiiiiiii wxxwywwyd

ii

iiiiiiii

wwd

wxwxwywyd

ˆ/x,x

/ˆ/ˆˆ/ˆ/x,x

iialg

2/1222ii

typical, but not , except for affinities 1iw i3xhˆ iw

For affinities DLT can minimize geometric distance

Possibility for iterative algorithm

represents 2 quadrics in 4 (quadratic in X)

Geometric interpretation of reprojection error

Estimating homography~fit surface to points X=(x,y,x’,y’)T in 4

0Hxx ii

Hν

22

22222

ii

x,xx,x

ˆˆˆˆXX

iiii

iiiiiiii

dd

yyxxyyxx

2H22 ,Xx,xx,x iiiii ddd

Analog to conic fitting

CxxC,x T2alg d

2C,xd

Sampson error

2XX

HνVector that minimizes the geometric error is the closest point on the variety to the measurement

XX

between algebraic and geometric error

XSampson error: 1st order approximation of 0XAh H C

XH

HXH δX

XδX

C

CC XXδX 0XH C

0δX

X XH

H

C

C eXJδ

Find the vector that minimizes subject to

Xδ eXJδXδ

XJ Hwith

C


Xδ eXJδXδ

Use Lagrange multipliers:

minimize

derivatives

0Jδ2λ-δδ XXX eT

TT 0J2λ-δ2 X 0Jδ2 X e

λJδ XT

0λJJ T e

e1TJJλ

e1TT

X JJJδ

XδXX ee1TT

XTX

2

X JJδδδ

Sampson error

2XX

HνVector that minimizes the geometric error is the closest point on the variety to the measurement

XX

between algebraic and geometric error

XSampson error: 1st order approximation of 0XAh H C

XH

HXH δX

XδX

C

CC XXδX 0XH C

0δX

X XH

H

C

C eXJδ


Xδ eXJδXδ

ee1TT

XTX

2

X JJδδδ

(Sampson error)

Sampson approximation

A few points

(i) For a 2D homography X=(x,y,x’,y’) (ii) is the algebraic error vector(iii) is a 2x4 matrix,

e.g.

(iv) Similar to algebraic error in fact, same as Mahalanobis distance

(v) Sampson error independent of linear reparametrization (cancels out in between e and J)

(vi) Must be summed for all points(vii) Close to geometric error, but much fewer

unknowns

ee1TT2

X JJδ

eee T2

2

JJT e

31213T2T

11 /hxhx hyhwxywJ iiiiii XJ H

C XHCe

ee

1TT JJ

Statistical cost function and Maximum Likelihood

Estimation• Optimal cost function related to noise

model• Assume zero-mean isotropic Gaussian

noise (assume outliers removed) 22 2/xx,2πσ2

1xPr de

22i 2xH,x /

2iπσ2

1H|xPr ide

i

constantxH,xH|xPrlog 2i2i

σ2

1id

Error in one image

Maximum Likelihood Estimate

2i xH,x id

Statistical cost function and Maximum Likelihood

Estimation• Optimal cost function related to noise

model• Assume zero-mean isotropic Gaussian

noise (assume outliers removed) 22 2/xx,2πσ2

1xPr de

22i

2i 2xH,xx,x /

2iπσ2

1H|xPr

ii dd

ei

Error in both images

Maximum Likelihood Estimate

2i2

i x,xx,x ii dd

Mahalanobis distance

• General Gaussian caseMeasurement X with covariance

matrix Σ

XXXXXX 1T2

22XXXX

Error in two images (independent)

22XXXX

iiii

iii

Varying covariances

Invariance to transforms ?

xTx~ Txx~ Hxx x~H

~x~

TH~

TH 1-?

TxH~

xT TxH

~Tx -1

will result change? for which algorithms? for which transformations?

Non-invariance of DLT

Given and H computed by DLT,

and

Does the DLT algorithm applied to yield ?

iiii xTx~,Txx~ ii xx

ii x~x~ -1HTTH

~

iiiiie TxHTTxTx~H~

x~~ -1 iii e** THxxT

hA,R~,~h~

A~

i2121i seesee TT

Effect of change of coordinates on algebraic error

10

tsRT

ss

Rt-

0RT T

*

for similarities

so

iiii sdd x~H~

,x~Hx,x algalg

Non-invariance of DLT

Given and H computed by DLT,

and

Does the DLT algorithm applied to yield ?

iiii xTx~,Txx~ ii xx

ii x~x~ -1HTTH

~

1H

~ subject tox~H

~,x~minimize

1H subject tox~H~

,x~minimize

1H subject toHx,xminimize

2

alg

2

alg

2alg

iii

iii

iii

d

d

d

Invariance of geometric error

ii

iiiiii

sd

ddd

Hx,x

HxT,xTTxHTT,xTx~H~

,x~ -1

Given and H,

and

Assume T’ is a similarity transformations

,xTx~,Txx~ iiii ii xx

,x~x~ ii -1HTTH~

Normalizing transformations

• Since DLT is not invariant,what is a good choice of coordinates?e.g.• Translate centroid to origin• Scale to a average distance to the

origin• Independently on both images

2

1

norm

100

2/0

2/0

T

hhw

whwOr

Importance of normalization

0

h

h

h

0001

1000

3

2

1

iiiiiii

iiiiiii

xyxxxyx

yyyxyyx

~102 ~102 ~102 ~102 ~104 ~104 ~10211

orders of magnitude difference!

Normalized DLT algorithmObjective

Given n≥4 2D to 2D point correspondences {xi↔xi’}, determine the 2D homography matrix H such that xi’=Hxi

Algorithm

(i) Normalize points

(ii) Apply DLT algorithm to

(iii) Denormalize solution

,x~x~ ii inormiinormi xTx~,xTx~

norm-1

norm TH~

TH

Iterative minimization metods

Required to minimize geometric error (i) Often slower than DLT(ii) Require initialization(iii) No guaranteed convergence, local minima(iv) Stopping criterion requiredTherefore, careful implementation required:(i) Cost function(ii) Parameterization (minimal or not)(iii) Cost function ( parameters )(iv) Initialization(v) Iterations

Parameterization

Parameters should cover complete space and allow efficient estimation of cost

• Minimal or over-parameterized? e.g. 8 or 9 (minimal often more complex, also cost surface)

(good algorithms can deal with over-parameterization)

(sometimes also local parameterization)

• Parametrization can also be used to restrict transformation to particular class, e.g. affine

Function specifications

(i) Measurement vector XN with covariance Σ(ii) Set of parameters represented by vector P N

(iii) Mapping f : M →N. Range of mapping is surface S representing allowable measurements

(iv) Cost function: squared Mahalanobis distance

Goal is to achieve , or get as close as possible in terms of Mahalanobis distance

PXPXPX 1T2fff

XP f

Error in one image

2i xH,x id

nf Hx,...,Hx,Hxh: 21

hX f

221- Hx,xxH,x iiiii

dd Symmetric transfer error

nnf Hx,...,Hx,Hx,xH,...,xH,xHh: 21-1

2-1

1-1

hX f

Reprojection error

2i2

i x,xx,x ii dd nnf x,...,x,xx,...,x,xh,: 2121

hX f

Initialization

• Typically, use linear solution• If outliers, use robust algorithm

• Alternative, sample parameter space

Iteration methods

Many algorithms exist• Newton’s method• Levenberg-Marquardt

• Powell’s method• Simplex method

Gold Standard algorithmObjective

Given n≥4 2D to 2D point correspondences {xi↔xi’}, determine the Maximum Likelyhood Estimation of H

(this also implies computing optimal xi’=Hxi)

Algorithm

(i) Initialization: compute an initial estimate using normalized DLT or RANSAC

(ii) Geometric minimization of -Either Sampson error:

● Minimize the Sampson error

● Minimize using Levenberg-Marquardt over 9 entries of h

or Gold Standard error:

● compute initial estimate for optimal {xi}

● minimize cost over {H,x1,x2,…,xn}

● if many points, use sparse method

2i2

i x,xx,x ii dd

Robust estimation

• What if set of matches contains gross outliers?

RANSACObjective

Robust fit of model to data set S which contains outliers

Algorithm

(i) Randomly select a sample of s data points from S and instantiate the model from this subset.

(ii) Determine the set of data points Si which are within a distance threshold t of the model. The set Si is the consensus set of samples and defines the inliers of S.

(iii) If the subset of Si is greater than some threshold T, re-estimate the model using all the points in Si and terminate

(iv) If the size of Si is less than T, select a new subset and repeat the above.

(v) After N trials the largest consensus set Si is selected, and the model is re-estimated using all the points in the subset Si

Distance threshold

Choose t so probability for inlier is α (e.g. 0.95)

• Often empirically• Zero-mean Gaussian noise σ then

follows distribution with m=codimension of model

2d

2m

(dimension+codimension=dimension space)

Codimension

Model t 2

1 l,F 3.84σ2

2 H,P 5.99σ2

3 T 7.81σ2

How many samples?

Choose N so that, with probability p, at least one random sample is free from outliers. e.g. p=0.99

sepN 11log/1log

peNs 111

proportion of outliers es 5% 10% 20% 25% 30% 40% 50%2 2 3 5 6 7 11 173 3 4 7 9 11 19 354 3 5 9 13 17 34 725 4 6 12 17 26 57 1466 4 7 16 24 37 97 2937 4 8 20 33 54 163 5888 5 9 26 44 78 272 117

7

Acceptable consensus set?

• Typically, terminate when inlier ratio reaches expected ratio of inliers

neT 1

Adaptively determining the number of samples

e is often unknown a priori, so pick worst case, e.g. 50%, and adapt if more inliers are found, e.g. 80% would yield e=0.2

• N=∞, sample_count =0• While N >sample_count repeat

• Choose a sample and count the number of inliers• Set e=1-(number of inliers)/(total number of

points)• Recompute N from e• Increment the sample_count by 1

• Terminate

sepN 11log/1log

Robust Maximum Likelyhood Estimation

Previous MLE algorithm considers fixed set of inliers

Better, robust cost function (reclassifies)

outlier

inlier ρ with ρ

222

222

i tet

teeed iR

Other robust algorithms

• RANSAC maximizes number of inliers

• LMedS minimizes median error

• Not recommended: case deletion, iterative least-squares, etc.

Automatic computation of HObjective

Compute homography between two imagesAlgorithm

(i) Interest points: Compute interest points in each image

(ii) Putative correspondences: Compute a set of interest point matches based on some similarity measure

(iii) RANSAC robust estimation: Repeat for N samples

(a) Select 4 correspondences and compute H

(b) Calculate the distance d for each putative match

(c) Compute the number of inliers consistent with H (d<t)

Choose H with most inliers

(iv) Optimal estimation: re-estimate H from all inliers by minimizing ML cost function with Levenberg-Marquardt

(v) Guided matching: Determine more matches using prediction by computed H

Optionally iterate last two steps until convergence

Determine putative correspondences

• Compare interest pointsSimilarity measure:• SAD, SSD, ZNCC on small neighborhood

• If motion is limited, only consider interest points with similar coordinates

• More advanced approaches exist, based on invariance…

Example: robust computation

Interest points(500/image)

Putative correspondences (268)

Outliers (117)

Inliers (151)

Final inliers (262)

• Maximum Likelihood Estimation

• DLT not invariant normalization• Geometric minimization invariant

• Iterative minimization• Cost function• Parameterization• Initialization• Minimization algorithm

22XXXX

iiii

iii

Automatic computation of HObjective

Compute homography between two imagesAlgorithm

(i) Interest points: Compute interest points in each image

(ii) Putative correspondences: Compute a set of interest point matches based on some similarity measure

(iii) RANSAC robust estimation: Repeat for N samples

(a) Select 4 correspondences and compute H

(b) Calculate the distance d for each putative match

(c) Compute the number of inliers consistent with H (d<t)

Choose H with most inliers

(iv) Optimal estimation: re-estimate H from all inliers by minimizing ML cost function with Levenberg-Marquardt

(v) Guided matching: Determine more matches using prediction by computed H

Optionally iterate last two steps until convergence

Algorithm Evaluation and Error Analysis

• Bounds on performance• Covariance estimation

?

? residual error

uncertainty

Algorithm evaluation

measured coordinates

estimated quantities

true coordinates

xH ,x

H ,x

Test on real data ortest on synthetic data

ii xx • Generate synthetic correspondences

• Add Gaussian noise, yielding ii xx

• Estimate from using algorithmii xx Hmaybe also ii x,x

• Verify how well or ii xHx ii xHx • Repeat many times (different noise, same )

• Error in one image

• Error in two images

ii xx )σ,0(xx Gii

2/1

1

2x,x2

1

n

iires dn

e

Estimate , thenH

Note: residual error ≠ absolute measure of quality of He.g. estimation from 4 points yields eres=0

more points better results, but eres will increase

2/1

1

2

1

2 x,xx,x4

1

n

ii

n

iires ddn

e

)σ,0(xx 2Gii ii xx )σ,0(xx 2Gii Estimate so that , thenii x,x,H ii xHx

Optimal estimators (MLE)

f

XP

Estimate expected residual error of MLE,Other algorithms can then be judged to this standard

f : M → N (parameter space to measurement space)

X N P M XP f

M NSM

dimension of submanifold SM = #essential parameters

nX

X

X

SM

2σ,XX NG

)X(X MLE

Assume SM locally planar around X

projection of isotropic Gaussian distribution on N with total variance N2 onto a subspace of dimension s is an isotropic Gaussian distribution with total variance s2

N measurements (independent Gaussian noise ) model with d essential parameters(use s=d and s=(N-d))

(i) RMS residual error for ML estimator

(ii) RMS estimation error for ML estimator

2/12/12

/1/XX NdNEeres

2/12/12

//XX NdNEeest

nX

X

X

SM

Error in one image Error in two images

2/1/1 Nderes

2/1/ Ndeest

2/1/41 neres

2/1/4 neest

2/1

2

4

n

neres

2/1

2

4

n

neest

nNd 2 and 8 nNnd 4 and 28

Covariance of estimated model

• Previous question: how close is the error to smallest possible error?

• Independent of point configuration

• Real question: how close is estimated model to real model?

• Dependent on point configuration (e.g. 4 points close to a line)

Forward propagation of covariance

Let v be a random vector in M with mean v and covariance matrix , and suppose that f: M →N is an affine mapping defined by f(v)=f(v)+A(v-v). Then f(v) is a random variable with mean f(v) and covariance matrix AAT.

Note: does not assume A is a square matrix

Example:

22 2,0,1,0 GyGx

723),( yxyxfx

7)0,0( fx

40

01 23A

252

3

40

0123AA T

5)( xstd

Example:

22 2,0,1,0 GyGx

yxy

yxx

23

23

40

01

23

23A

257

725

22

33

40

01

23

23AA T

;7;25;25 yxEyyExxE

Non-linear propagation of covariance

Let v be a random vector in M with mean v and covariance matrix , and suppose that f: M →N

differentiable in the neighborhood of v. Then, up to a first order approximation, f(v) is a random variable with mean f(v) and covariance matrix JJT, where J is the Jacobian matrix evaluated at v

)vv(J)v()v( ffj

iij

fJ

v

Note: good approximation if f close to linear within variability of v

Example:

40

01,

0

0),(x T Gyx

523),( 2 yxxyxfx

dxdyyxfyxPx ),(),(

dxdyxyxfyxP 22 ),(),(

222 2/)4/(

π8

1),( yxeyxP

2σ5x422 σ2σ25σ x

5x 222 σ25

2

3

4

123σσ

x

23J

Example:

feydxcybxyaxyxf 22),(

fca 2y

2x σσmean

2y

2x

2y

22y

2x

4x

2 eσσσ2σσσ2variance dcba

fmean est.2y

2x eσσ varianceest. d

Backward propagation of covariance

f : M → N

X N P M

PX1 f

X

f -1

P

X

XX η

PX1 f


PX1 f

PPJPP -ff XP f

PXXX f

PPXX J

XXJJJPP 1T-11T

XP 1 f XP 1 f

PX1 f XXXJJJ 11T-11T f

assume f is affineX

f -1PX

what about f -1o ?

solution:

minimize:

XXXJJJ 11T-11T f


PX1 fX

f -1PX

1T-11T JJJA

T1T-11T1T-11T JJJJJJ1 f

-T1TT1T-11T JJJJJJ

-11T JJ


PX1 fX

f -1PX

-11x

TP JJ If f is affine, then

non-linear case, obtain first order approximations by using Jacobian

Over-parameterization

In this case f is not one-to-one and rank J<M

-11x

TP JJ so can not hold

e.g. scale ambiguity infinite variance!

However, if constraints are imposed, then ok.

T11x

TTA1x

TP AJAJAAJJ

A

Invert d xd in stead of MxM

Over-parameterization

When constraint surface is locally orthogonal to the null space of J

JJ 1x

TP

e.g. usual constraint ||P||=1

nullspace||P||=1

(pseudo-inverse)

Example: error in one image

(i) Estimate the transformation from the data(ii) Compute Jacobian , evaluated at(iii) The covariance matrix of the estimated is

given by JJ 1

xT

h

h/XJ f

Hh

h

TT

TT

x~x~0

x~0x~1h/xJ

iii

iii

iii

y

x

w

iiii JJJJ 1T1

xT

h

Example: error in both images

BBAB

BAAAJJ

1x

T1x

T

1x

T1x

T1

xT

B|AJ

separate in homography and point parameters

Using covariance matrix in point transfer

Thhhx JJ

Txxx

Thhhx JJJJ

Error in one image

Error in two images

(if h and x independent, i.e. new points)

=1 pixel =0.5cm (Crimisi’97)

Example:

(Crimisi’97)

Example:

Monte Carlo estimation of covariance

• To be used when previous assumptions do not hold (e.g. non-flat within variance) or to complicate to compute.

• Simple and general, but expensive

• Generate samples according to assumed noise distribution, carry out computations, observe distribution of result

singular value decomposition

Documents