estimation- 2d projective transformations
TRANSCRIPT
1
Estimation-2D 1
2008-1
Multi View Geometry (Spring '08)
Prof. Kyoung Mu LeeSoEECS, Seoul National University
Estimation- 2D Projective Transformations
Estimation-2D 2
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
• Any real mxn matrix A can be decomposed uniquely by
where
SVD (Singular Value Decomposition)
mm×= IUUT
021 ≥≥≥≥ nddd L
nn×= IVVT
⎥⎥⎥⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢⎢⎢⎢
⎣
⎡
=
000
00
0000
2
1
L
MMM
L
MOMM
L
L
nd
dd
D
Tnnnmmmnm ×××× = VDUA nm ≥
[ ][ ]n
mn
vvVuuuU
K
LK
,,
1
1
== orthonormal column vectors
2
Estimation-2D 3
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
SVD (Singular Value Decomposition)
• If we consider only the non-zero terms, we can write it by
• Properties:
•
•
∑=
=+++=
+++==n
iiiinnn
nnnT
dddd
ddd
1222111
222111
TTTT
TTT
vuvuvuvu
vuvuvuUDVA
L
L
2
2
⎢⎣
⎡
==
⇒
==
iiiT
iiiT
TTTT
dd
vAvAuuAA
UUDVDUUDVAA 2
Tnnnnnmnm ×××× = VDUA nm ≥
Estimation-2D 4
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
SVD (Singular Value Decomposition)
•
• For square, non-singular matrices,
• Closest rank r approximation
( ) 0 ,, 0 ,,,,~21 LL rddddiag=D
TTT
T
rrrddd vuvuvuVDUA
+++=
=
L222111
~~
=Ax
DVAx
xDVxVVDxAxAxAx
and by affectedonly is
222
⇒
=== TTTTT
TUVDA 11 −− = ( ) /1,,/1 11
ndddiag L=−D
xxVTxDVTxUDVT
3
Estimation-2D 5
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
SVD (Singular Value Decomposition)
• Solution of linear equations: • Least squares solution: Find x that minimize
i) if rank Amxn = n (full column rank)bAx−
⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢
⎣
⎡
′
′′
′′
=
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢
⎣
⎡
⇒
=′=′−⇒
−=−=−
+
m
n
n
n
n
TT
TTT
b
bb
bb
y
yy
d
dd
y
M
M
M
O
1
2
1
2
1
2
1
0
, ,min
since
bUbxVbDy
bUxDVbxUDVbAx
Vyx =
=′
=⇒
And
,1 , nidby
i
ii L
min
bxA =×nm
(unique solution)
Estimation-2D 6
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
SVD (Singular Value Decomposition)
ii) if rank Amxn = r < n (not full column rank)
• Pseudo inverse
nnrr vvVyx λλ +++= ++ K11
TUVDA ++ = ( ) 0 ,, 0 ,,,, 112
11 LL −−−+ = rddddiagD
bAxbxA +× =→=nm
homogeneous solution
4
Estimation-2D 7
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
SVD (Singular Value Decomposition)
• Solution of homogeneous equations • Least squares solution:
Find x that minimize
i) if rank Amxn = n (full col. rank)
ii) if rank Amxn = r < n (x = null-vector of A)
0=× xA nm
1 subject to =xAx
,1 subject to min
and , since
xVyyDy
xVxxDVxUDVAxT
TTT
==⇒
===
nvVyxy
==⇒= T,0,1)(0,0,K
nnrr
Tnr
λλ),λ,λ,,(
vvVyxy
++==⇒=
++
+
K
KK
11
1,00
(the last e-vector)
Estimation-2D 8
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
SVD (Singular Value Decomposition)
• Another interpretation
Which x makes ? i) if
ii) if
xvuxvuxvuxUDVAx TTTnnn
T ddd +++== L222111
1 subject to minimum be to =xAx,n,idi K1 ,0 =≠
n
n
d is minimum thevector)-elast (the ,
⇒=⇒ vx
,n,ridi K1 ,0 +==
0 ) of space nullor vectors,-esingular theofspan (the
,11
=⇒
++==⇒ ++
AxA
vvVyx nnrr λλ K
5
Estimation-2D 9
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Parameter Estimation
• We are interested in the following problems:
2D homographyGiven a set of (xi,xi’), how to determine H (xi’=Hxi)
3D to 2D camera projectionGiven a set of (Xi,xi), how to determine P (xi=PXi)
Fundamental matrixGiven a set of (xi,xi’), how to determine F (xi’TFxi=0)
Trifocal tensor Given a set of (xi,xi’,xi”), how to determine T
Estimation-2D 10
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Number of measurements required• To estimate the parameters, we need
# of independent equations ≥ degrees of freedom• Example:
To estimate H:
H has 8 DOF2 independent equations / a point correspondence ( )4x2≥8So, at least 4 point correspondences are required
Hxx ='
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡=
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡′′
11λ
333231
232221
131211
yx
hhhhhhhhh
yx
ii xx ′↔
6
Estimation-2D 11
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Approximate solutions
• Minimal solution4 points an exact solution for H
• More pointsNo exact solution, because measurements are inexact (“noise”)Find the optimal solution according to some cost functionAlgebraic or geometric/statistical cost
Estimation-2D 12
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Gold Standard algorithm
• Cost function that is optimal for some assumptions• Computational algorithm that minimizes it is called “Gold
Standard” algorithm • Other algorithms can then be compared to it
7
Estimation-2D 13
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
The Direct Linear Transformation (DLT) algorithm
• Given a set of (noisy) point correspondences , how to compute H robustly for ?
• Note that H must satisfy
• Let and
• Then
ii xx ′↔
ii Hxx =′
0=×′ ii Hxx
),,( iiii yx ω′′′=′x
0=hAi
⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜
⎝
⎛
=
i
i
i
i
xhxhxh
HxT
T
T
3
2
1
0hhh
0xxx0x
xx0=
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
′′−
′−′′′−
3
2
1
TTii
Tii
Tii
TTii
Tii
Tii
T
xyx
yω
ω
0xhxhxhxhxhxh
Hxx =⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
′−′′−′′−′
=×′
iT
iiT
i
iT
iiT
i
iT
iiT
i
ii
yxx
y
12
31
23
ωω
Estimation-2D 14
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
The DLT algorithm (cont.)
• Since 3rd row is redundant, we have
Two lin. independent Eqs. for each point correspondence.• Thus, 4 point correspondences are enough.• For more than 4 point correspondences
(Over-determined System)
0hA =i for noisy measurementii ehA =
0hhh
x0xxx0
=⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
⎥⎦
⎤⎢⎣
⎡
−−
3
2
1
~~
Tii
TTi
Tii
Ti
T
xy )~,~ (where
i
ii
i
ii
yyxxωω ′′
=′′
=
inhomogeneous mapped point
0hhh
x0xxx0
=⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
⎥⎦
⎤⎢⎣
⎡′−′′′−
3
2
1
Tii
TTii
Tii
Tii
T
xy
ωω
8
Estimation-2D 15
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
The DLT algorithm (cont.)
• 4 point case:Solve the exact solution H satisfying
The size of A is 8x9 or 12x9 (rank is 8)1-D null-space h is the nontrivial solutionsChoose the one with
0Ah =0h
AAAA
=
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
4
3
2
1
1=h
Estimation-2D 16
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
The DLT algorithm (cont.)
• More points case:Over-determined solution satisfying
No exact solution due to the “noise”So, find the approximate solution
Additional constraint needed for nontrivial solution e.g. 1=h
0Ah =0h
A
AA
=
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
n
2
1
M
Ahhh
min=∗
9
Estimation-2D 17
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
ObjectiveGiven n≥4 2D to 2D point correspondences {xi↔xi’}, determine the 2D homography matrix H such that xi’=Hxi
Algorithm(i) For each correspondence xi ↔xi’ compute Ai. Usually only
two first rows needed.
(ii) Assemble n 2x9 matrices Ai into a single 2nx9 matrix A(iii) Obtain SVD of A. Solution for h is last column of V(iv) Determine H from h
The DLT algorithm (cont.)
Estimation-2D 18
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
• By setting hj=1 (e.g. h9=1), and solve for 8-vector
• It can be solved by Gaussian elimination (4 points) or linear least-squares (more than 4 points)
• However, if h9=0 this approach fails, and also gives poor results if h9≈ 0
• Therefore, this approach is not recommended• Note h9=H33=0 if origin is mapped to infinity
Inhomogeneous solution
⎟⎟⎠
⎞⎜⎜⎝
⎛−=⎥
⎦
⎤⎢⎣
⎡ −−−''~
''000''''''''000
ii
ii
iiiiiiiiii
iiiiiiiiii
xwyw
xyxxwwwywxyyyxwwwywx
h
h~
( ) [ ] 0100
1000 =⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡=∞ HHxlT
10
Estimation-2D 19
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
• 2D homographies (8dof)Minimum of 4 points or lines
• 3D homographies (15 dof)Minimum of 5 points or 5 planes
• 2D affinities (6 dof)Minimum of 3 points or lines
• How about mixed configurations?2 point and 2 lines (x)3 lines and 1 point (o)
Solutions from lines, etc.
ii lHl T=′ 0Ah =• in 2D
Point/line 2 constraintsConic 5 constraints
• in 3DPoint/plane 3 constraints
Estimation-2D 20
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Cost functions
• Which cost function can we employ?
• Algebraic distance• Geometric distance
Transfer error, Symmetric transfer error, Reprojection error
• Sampson error
11
Estimation-2D 21
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Algebraic distance error
• Residual (algebraic error) vector:Error associated with each correspondence
• The total algebraic distance error:
• Has no geometrical and statistical meaning• However, with good normalization it works satisfactory, so it
can be used for initialization
hAe ii =
ii xx ′↔
( )2
22alg x0x
xx0, heHxx ⎥
⎦
⎤⎢⎣
⎡′−′−
′−′−==′
TTT
TTT
iiii
iiiiiii xw
ywd
algebraic distance
( ) 2222alg , eAheHxx ===′ ∑∑
ii
iiid
( ) 22
21
221alg , aad +=xx where ( ) 21
T321 ,, xxa ×== aaa
Estimation-2D 22
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Geometric distance error
• Transfer error (error in one (second) image)
• Symmetric transfer error (error in both images)
• Reprojection errorFind and pairs of perfectly matched points and simultaneously
x′ˆ
: measured value: estimated value: true value
xxx
H
( )2,argminˆii
i
d xHxHH
′= ∑
( ) ( )221- ,,argminˆiiii
i
dd HxxxHxHH
′+′= ∑
( ) ( ) ( )
ii
iiiii
ii ddii
xHx
xxxxxxHxxH
ˆˆˆ subject to
ˆ,ˆ,argminˆ,ˆ,ˆ 22
ˆ,ˆ,
=′
′′+=′ ∑′
x
e.g. calibration patterndistanceEuclidean : ),( yxd
12
Estimation-2D 23
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Comparison of symmetric transfer error and reprojection error
( ) ( )221- ,, HxxxHx ′+′ dd
( ) ( )22 ˆ,ˆ, xxxx ′′+ dd
Estimation-2D 24
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Comparison of geometric and algebraic distances
• For one image case,
• The algebraic error vector:
• The geometric distance:
• For 2D affine transform case;
Thus geometric distance is equal to algebraic distance
⎟⎟⎠
⎞⎜⎜⎝
⎛′′−′′′′−′′
=⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
⎥⎦
⎤⎢⎣
⎡′−′′′−
==iiii
iiii
iiii
iiiiii xx
yyx
yωω
ωωω
ωˆˆˆˆ
00
3
2
1
hhh
xxxx
ehA TTT
TTT
1ˆ then ,1 if =′=′= iii ωωω
⇒′′′==′′′′=′ Tiiii
Tiiii yxyx )ˆ,ˆ,ˆ(ˆ and ,),,(Let ωω xHxx
( ) ( ) ( )222iialg ˆˆˆˆˆ, iiiiiiii xxyyd ωωωω ′′−′′+′′−′′=′′ xx
( ) ( ) ( )( )( ) ii
iiiiiiii
dxxyyd
ωωωωωω
ˆ/ˆ,/ˆ/ˆˆ/ˆ/ˆ,
iialg
2/1222ii
′′′′=
′′−′′+′′−′′=′′
xxxx
13
Estimation-2D 25
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Geometric interpretation of reprojection error
• Estimating H is to fit surface VH to the measurement points, in R4
• Let be the closest point, then
• Conic analogue:
Tiiiii yxyx ),,,( ′′=X
Tiiiii yxyx )ˆ,ˆ,ˆ,ˆ(ˆ ′′=X
( ) ( ) ( ) ( )( ) ( )22
22222
ˆ,ˆ,
ˆˆˆˆˆ
iiii
iiiiiiiiii
dd
yyxxyyxx
xxxx
XX
′′+=
′−′+′−′+−+−=−
( ) ( ) ( )2H22 ,ˆ,ˆ, νXxxxx iiiii ddd ⊥=′′+
Estimation-2D 26
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Sampson error
• Close approximation to geometric error (between algebraic and geometric error)
• 1st order approximation of • For given H and any point on VH satisfies
• Linearize it by a Taylor series expansion
• Find lying on VH so that
•
XTyxyx ),,,( ′′=X
0X0Ah =⇔= )(HC
XδXX +=ˆ
εJδ
0δX
X0X
X
X
−=⇒
=∂∂
+⇒=
)()ˆ( HHH
CCC
X
( ) ( ) XH
HXH δX
XδX∂∂
+=+CCC
εJδδδ XXX −= subject to minimizes that vector theFind
14
Estimation-2D 27
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Sampson error (cont.)
• Use of Lagrange method:minimize
Then The Sampson error: Overall error:
( )εJδδδ += XXX 2λ-TE
εJJJδεJJλελJJ
0εJδ0λ
λJδ0Jλδ0δ
X
X
XXX
1
1
)( And)( Thus,
22
−
−
−=
−=⇒−=
=+⇒=∂∂
=⇒=−⇒=∂∂
TT
TT
TTTTT
E
E
XδXX +=ˆ
( ) εJJεδδδ 1TTX
TX
2X −
==
Estimation-2D 28
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Maximum Likelihood estimation (MLE)
• Optimal estimator in statistical sense• Assume iid mean-zero isotropic Gaussian measurement error
(outlier removed):
• Error in one image:
MLE:• Error in both images
MLE: minimize
( ) ( ) ( )22 2/,2πσ2
1Pr σxxx de−=
2),( Minimize)|}Pr({log Maximize iii
i d xHxHx ′⇒′ ∑
{ }( ) ( ) ( )22i 2, /
2i πσ2
1|Pr σidei
xHxHx ′−Π=′
{ }( ) ( ) ( )( ) ( )222i 2(, /
2iii πσ2
1}{,|,Pr σiii ddei
xHxx XxHxx ′+−Π=′
( ) ( ) iiii dd xHxxxxx ˆˆˆ with ,ˆ,ˆ, 2i
2i =′′′+∑
15
Estimation-2D 29
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Mahalanobis distance
• Consider non-isotropic Gaussian model• Density normalized distance.
Measurement X with covariance matrix Σ
Error in two images (independent)
Varying covariances
( ) ( )XXΣXXXX −−=− −
Σ
1T2
22
Σ′Σ′−′+− XXXX
22
iiii
iii Σ′Σ
′−′+−∑ XXXX
Estimation-2D 30
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Non-invariance of DLT
• Let
eg. Coordinate transformation
• Does the DLT algorithm applied to the correspondence setyield the transformation ?
1
~
~ and
~ ,~ where,~~ and ,
−′=
′′=′=′′ ↔↔HTTH
xTxTxxxxxxHH
iiiiiiii
ii xx ~~ ′↔ 1~ −′= HTTH
)~~~~( 1iiii xHxHTTHxTxTx =′=′=′′=′ −
16
Estimation-2D 31
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Non-invariance of DLT
• Effect of change of coordinates on algebraic error
• For similarity transform
( ) iiiii TxHTTxTxHxe -1~~~~ ′×′′=×′= ( ) iii eTHxxT ** ′=×′′=
( ) ( ) ( ) hAhA iiiiiiii seeseese,e ==== TTT212121 ,,R~~~~
⎥⎦
⎤⎢⎣
⎡=′
1sR0
tT ⎥
⎦
⎤⎢⎣
⎡=′
ss
Rt0R
T T*
-
( ) ( )iiii sdd HxxxHx ,~~,~algalg ′=′
cofactor matrix of T’
Estimation-2D 32
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Non-invariance of DLT
• However
( )
( )( ) 1~ subject to~~,~minimize
1 subject to~~,~minimize
1 subject to,minimize
2alg
2alg
2alg
=′⇔
=′⇔
=′
∑
∑
∑
HxHx
HxHx
HHxx
iii
iii
iii
d
d
d
17
Estimation-2D 33
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Invariance of geometric error
• Given
• If T and T’ are Euclidean transforms
• If T and T’ are Similarity transforms, then
• Thus, geometric error is invariant to Euclidean and similarity transformations
1
~
~ and
~ ,~ where,~~ and ,
−′=
′′=′=′′ ↔↔HTTH
xTxTxxxxxxHH
iiiiiiii
( ) ( ) ( )( )ii
iiiiii
sdddd
HxxHxTxTTxHTTxTxHx
,,,~~,~ -1
′=
′′′=′′′=′
( ) ( ) ( )( )ii
iiiiii
dddd
HxxHxTxTTxHTTxTxHx
,,,~~,~ -1
′=
′′′=′′′=′
Estimation-2D 34
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Normalizing transformations
• How to make DLT invariant? Choice of coordinates?• Data normalization: (Essential step in the DLT algorithm)
Translate so that the centroid to be the originScale so that the average distance to the origin to bePerform independently on both images
– Improves accuracy– Invariant to scale and coordinate system
• Or apply
2
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡++++
=⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡+
+=
−
100)(2/)/(10)(2/0)/(1
1002/02/0
T
1
norm hwhhwhwwhw
hhwwhw
w,h: width and height of the image
18
Estimation-2D 35
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Importance of normalization
0hhh
00011000
3
2
1
=⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
⎥⎦
⎤⎢⎣
⎡′−′−′−
′′′−′−′−
iiiiiii
iiiiiii
xyxxxyxyyyxyyx
~102 ~102 ~102 ~102 ~104 ~104 ~10211
Big difference in orders of magnitude unstable
Estimation-2D 36
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Normalized DLT algorithm
ObjectiveGiven n≥4 2D to 2D point correspondences {xi↔xi’}, determine the 2D homography matrix H such that xi’=Hxi
Algorithm(i) Normalize points (ii) Apply DLT algorithm to and find (iii) Denormalize solution
,~~ii xx ′↔
inormiinormi~ ,~ xTxxTx ′′=′=
norm-1
norm~THTH ′=
H~
• What about the rotation?• Don’t we need normalize the rotation effect? Why?
HHRRH =′= −1~
19
Estimation-2D 37
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Iterative minimization methods
• For minimization of geometric errorSlower than direct methods (DLT)Initialization and stopping criterion are neededNo guaranteed convergence, local minima
• Steps:Defining cost functionParameterizationFunction specificationInitializationIteration
Estimation-2D 38
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Parameterization
• Parameters should cover complete space and allow the estimation to be efficient
• Minimal or over-parameterization? e.g. 8 or 9 minimal parameterization gives often more complicate functionsgood algorithms can deal with over-parameterizationSo, it is not necessary to use minimal parameterization
• Parameterization can also be used to restrict transformation to particular class, e.g. affine
20
Estimation-2D 39
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Function Specification
• Function specification:
(i) Measurement vector X∈RN with covariance Σ(ii) Set of parameters represented by vector P ∈RM
(iii) Mapping f : RM →RN. Range of mapping is surface S representing allowable measurements
(iv) Cost function: squared Mahalanobis distance
• The goal is to achieve P such that , or get as close as possible in terms of Mahalanobis distance
( ) ( )( ) ( )( )PXΣPXPX fff −−=− −Σ
1T2
( ) XP =f
Estimation-2D 40
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Iterative minimization methods (cont.)
• Error in one image:Let Then
• Symmetric transfer error:Let
Then
• Reprojection error:Let
Then
Tnn yxyxyx ),,,,,,( 2211 ′′′′′′= KX
22 ),()( ii
idf xHxhX ∑ ′⇔−
Tnnnn yxyxyxyx ),,,,,,,,,( 1111 ′′′′= KKX
2212 ),(),()( ii
iii ddf HxxxHxhX ∑ ′+′⇔− −
Tnnnn yxyxyxyx ),,,,,,,,,( 1111 ′′′′= KKX
222 )ˆ,()ˆ,()( ii
iii ddf xxxxPX ′′+⇔− ∑
( )nf Hx,...,Hx,Hxh: 21→
( )nnf HxHxHxxHxHxHh ,...,,,,...,,: 21-1
2-1
1-1 ′′′→
( ) ( )nnnf x,x,...,x,x,x,xx,...,x,xh,: 221121 ′′′→
21
Estimation-2D 41
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Iterative minimization methods (cont.)
• Sampson approximation:Since it determines the 2n variables for each particular choice h, the minimization only requires the 9 parameters of h.
• Initialization:Good initialization is very important.Use linear solution (Normalized DLT) or random sampling (robust algorithm), etc.
• Iteration:Newton methodLM (Levenberg-Marquardt) methodPowell’s MethodSimplex method, etc.
}ˆ{ ix
Estimation-2D 42
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
The Gold Standard Algorithm
ObjectiveGiven n≥4 2D to 2D point correspondences {xi↔xi’}, determine the Maximum Likelyhood Estimation of H(this also implies computing optimal xi’=Hxi)
Algorithm(i) Initialization: compute an initial estimate using normalized
DLT or RANSAC (ii) Geometric minimization of -Either Sampson error:
• Minimize the Sampson error• Minimize using Levenberg-Marquardt over 9 entries of h
or Gold Standard error:• compute initial estimate for optimal {xi}• minimize cost over {H,x1,x2,…,xn}• if many points, use sparse method
( ) ( )22 ˆ,ˆ, iiii dd xxxx ′′+∑
22
Estimation-2D 43
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Experimental Comparison
•
Estimation-2D 44
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Robust Estimation-RANSAC
• How to determine inliers (discard outliers or mismatched correspondences) to estimate the homography robustly?
• RANSAC (RAndom SAmple Consensus):• Consider a 2D line-fitting example
Select two points randomly – determine a lineCheck the support points that lie within a distance thresholdThe line with most support is the robust fit.
baxx +=′ Estimation of 1-D affine transform
23
Estimation-2D 45
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
RANSAC
Estimation-2D 46
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
RANSAC (cont.)
ObjectiveRobust fit of model to data set S which contains outliers
Algorithm(i) Randomly select a sample of s data points from S and
instantiate the model from this subset.(ii) Determine the set of data points Si which are within a distance
threshold t of the model. The set Si is the consensus set of samples and defines the inliers of S.
(iii) If the subset of Si is greater than some threshold T, re-estimate the model using all the points in Si and terminate
(iv) If the size of Si is less than T, select a new subset and repeat the above.
(v) After N trials the largest consensus set Si is selected, and the model is re-estimated using all the points in the subset Si
24
Estimation-2D 47
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
RANSAC (cont.)
• What is the distance threshold?• The square of the point distance is a sum of squared
Gaussian variables, i.e., Chi-distribution with m degree of freedom (co-dimension m).
•• Thus, for a given probability α,
• And, for Gaussian with N(0,σ) we can decide
2mχ
2⊥d
∫==<2
0
2222 )()()Pr(k
mmm dkFk ζζχχ
)(
)()Pr()Pr(12
22222
α
αχα−
⊥
=⇒
==<⇒=<
m
mm
Ft
tFttd
(cdf)
chi distribution
chi-square distribution
21222
22
)( with outlier inlier
σα−
⊥
⊥ =⎩⎨⎧
<<
mFttdtd
Estimation-2D 48
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
RANSAC (cont.)
• How many samples?• Let p be the probability that at least one of the random
samples of s points among N selections is free from outliers, and ω be the probability that any selected point is an inlier.
• Then,
εωω −=−−=
−−=
−=
1 ,)1(1inliers)) are points s (allPr 1(1
outliers)includeselectionsN(allPr 1
Ns
N
p
7.81σ2T3
5.99σ2H,P2
3.84σ2l,F1
t 2ModelCodimension
95.0for )(12 == − ααmFt
25
Estimation-2D 49
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
RANSAC (cont.)
• Ex) line fitting case; n=12, ε=2/12=1/6, s=2. Thus N=5.
( ) ( )( )spN ε−−−= 11log/1log
11772727844269585881635433208472939737241674614657261712645723417139534351911974331711765322
50%40%30%25%20%10%5%sproportion of outliers e
Example of N for p=0.99 for a given s and ε
Estimation-2D 50
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
RANSAC (cont.)
• How large is an acceptable consensus set?• Terminate when the inlier ratio reaches to the expected
ratio of inliersFor n data points, T=(1- ε)nFor the line fitting example, T=(1- 0.2)12=10.
• Adaptive determination of the # of samples
26
Estimation-2D 51
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Robust MLE
• Final step for RANSAC• Re-estimate the model using all the inliers.
A least-squares fit by minimizing
• Iterative estimation:Optimal fit to inliersRe-classify inliersIterate until no change
Robust cost function
∑ ⊥=i
id 2C
( ) ( )⎩⎨⎧
><
== ⊥∑ outlier inlier
with 222
222
i tettee
ed i γγD
Estimation-2D 52
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Other Robust Estimation Algorithms
• RANSACMaximizes the # of inliers
• LMedS (Least Median Squares) estimationMinimizes the median errorScore the model by the median of the distances to all dataNo threshold or prior knowledge of error varianceHowever, fails if more than half of the data is outlyingUse the proportion if outliers to determine the distance
• Case deletion, Iterative least-squares Not recommended !
27
Estimation-2D 53
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Automatic computation of a H using RANSAC
ObjectiveCompute homography between two images
Algorithm(i) Interest points: Compute interest points in each image(ii) Putative correspondences: Compute a set of interest point
matches based on some similarity measure(iii) RANSAC robust estimation: Repeat for N samples
(a) Select 4 correspondences and compute H(b) Calculate the distance d⊥ for each putative match(c) Compute the number of inliers consistent with H (d⊥<t)Choose H with most inliers
(iv) Optimal estimation: re-estimate H from all inliers by minimizing ML cost function with Levenberg-Marquardt
(v) Guided matching: Determine more matches using prediction by computed H
Optionally iterate last two steps until convergence
Estimation-2D 54
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
• Determining putative correspondences:Select corner points in each imagesMatch points using similarity measure (SAD, SSD, NCC, etc) symmetrically within a search region
• Distance measure:Symmetric transfer errorReprojection errorSampson error
• Sample selection:Avoid degenerate samples (collinear ones)Samples with good spatial distribution
• Robust ML estimation (cycle approach)Carry out ML estimation on inliers using Levenberg-Marquardt algorithmRecompute the inliers using new HIterate cycle until converges
Automatic computation of a H using RANSAC
28
Estimation-2D 55
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Example: robust computation
Interest points(500/image)
Putative correspondences (268)
Outliers (117)
Inliers (151)
Final inliers (262)
Estimation-2D 56
Multi View Geometry (Spring '08) K. M. Lee, EECS, SNU
Implementation and run details
• Harris corner detector is used• SSD threshold is set 20 (40 for the guided matching stage)• For 640 x 480 image, ROI of 80 x 80 is sufficient. • t=1.25• N=43, two MLE cycles, 10 iterations for LM algorithm