institut für elektrische meßtechnik und meßsignalverarbeitung professor horst cerjak, 19.12.2005...

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung

Professor Horst Cerjak, 19.12.20051

15.4.2008Augmented Reality VU 3 Algorithms Axel Pinz

Algorithms

• Point correspondences– Salient point detection – Local descriptors

• Matrix decompositions– RQ decomposition– Singular value decomposition - SVD

• Estimation– Systems of linear equations– Solving systems of linear equations

• Direct Linear Transform – DLT• Normalization• Iterative error / cost minimization• Outliers Robustness, RANSAC

– Pose estimation• Perspective n-point problem – PnP




Point Correspondences - Example 1

Structure and motion fromStructure and motion from““natural” landmarks [Schweighofer]natural” landmarks [Schweighofer]

Stereo reconstruction of Harris cornersStereo reconstruction of Harris corners




Point Correspondences - Example 2

[Mikolajczyk+Schmid]

normalization“canonical view”

ellipticalsupport

correspondence




Salient points (corners) based on 1st derivatives

• Autocorrelation of 2D image signal [Moravec]– Approximation by sum of squared differences (SSD)

– Window W

– Differences between grayvalues in W and a window shifted by (Δx,Δy)

– Four different shift directions fi(x,y):

– A corner is detected, when fMoravec>th

Wyxwwww

ywwww

x

ww

ww

yyxxIyxI

yyxxIyxIyxf

),(

2

2

)],(),([

)],(),([),(

4

1Moravec

iiff





• Autocorrelation (second moment) matrix:– Avoids various shift directions

– Approximate I(xw+Δx,yw+Δy) by Taylor expansion:

– Rewrite f(x,y):

y

xyxIyxIyxIyyxxI wwywwxwwww ),(),(),(),(

y

xyx

y

xyxIyxI

yxI

yxIyx

y

xyxIyxIyxf

wwywwxWyx wwy

wwx

Wyxwwywwx

ww

ww

M

),(),(),(

),(

]),(),([),(

),(

),(

2

“second moment matrix M”





• Autocorrelation (second moment) matrix:

– M can be used to derive a measure of “cornerness”– Independent of various displacements (Δx,Δy)– Corner: significant gradients in >1 directions rank M = 2– Edge: significant gradient in 1 direction rank M = 1– Homogeneous region rank M = 0

• Several variants of this corner detector:– KLT corners, Förstner corners

Wyxwwy

Wyxwwywwx

Wyxwwywwx

Wyxwwx

wwww

wwww

yxIyxIyxI

yxIyxIyxI

),(

2

),(

),(),(

2

),(),(),(

),(),(),(

M





• Harris corners – Most popular variant of a detector based on M

– Local derivatives with “derivation scale” σD

– Convolution with a Gaussian with “integration scale” σI

– MHarris for each point x in the image

– Cornerness cHarris does not require to compute eigenvalues

– Corner detection: cHarris > tHarris

),(),(),(

),(),(),()(),,(),,( 2

22

HarrisDyDxDx

DxDxDxIDDIDI III

IIIG

xxx

xxxxxM

MM tracedetHarris c

25000 ,05.0

7.0 ,2

Harris

tDI





• Harris corners




Salient points (corners) based on 2nd derivatives

• Hessian determinant

– Local maxima of det H [Beaudet]

– Zero crossings of det H [Dreschler+Nagel]

– Detectors are related to curvature

– Invariant to rotation

– Similar cornerness measure: local maxima of K [Kitchen+Rosenfeld]

2detdet xyyyxxyyxy

xyxxIII

II

II

H

22

22 2

yx

xyyyxxyyxx

II

IIIIIIIK




Salient points (corners) based on 2nd derivatives

• DoG / LoG [Marr+Hildreth]– Zero crossings

– “Mexican hat”, “Sombrero”

– Edge detector !

• Lowe’s DoG keypoints [Lowe]– Edge zero-crossing

– Blob at corresponding scale: local extremum !

– Low contrast corner suppression: threshold

– Assess curvature distinguish corners from edges

– Keypoint detection:

0 :DoG

0)()( :LoG

21

IGIGD

IGIGL

DDD

DD

yyxy

xyxxD ofmatrix Hessian

H

thD

D H

H

det

trace2




Salient points (corners) without derivatives

• Morphological corner detector [Laganière]– 4 structuring elements:

+, ◊, x, □

– Assymetrical closing




Salient points (corners) without derivatives• SUSAN corners [Smith+Brady]

– Sliding window

– Faster than Harris




Salient points (corners) without derivatives• Kadir/Brady saliency [Kadir+Brady]

– Histograms

– Shannon entropy

– Scale selection

– Used in constellation

model [Fergus et al.]




Salient points (corners) without derivatives• MSER – maximally stable extremal regions [Matas et al.]

– Successive thresholds

– Stability: regions

“survive” over many

thresholds




Affine covariant corner detectors• Locally planar patch affine distortion

• Detect “characteristic scale” – see also [Lindeberg], scale-space

• Recover affine deformation that fits local image data best

[Mikolajczyk+Schmid]

normalization“canonical view”

ellipticalsupport

correspondence




Scaled Harris Corner Detector• “Harris Laplace” [Mikolajczyk+Schmid, Mikolajczyk et al.]




Scaled Hessian Detector• “Hessian Laplace” [Mikolajczyk+Schmid , Mikolajczyk et al.]




Harris Affine Detector• “Harris affine” [Mikolajczyk+Schmid , Mikolajczyk et al.]




Hessian Affine Detector• “Hessian affine” [Mikolajczyk+Schmid , Mikolajczyk et al.]




Qualitative comparison of detectors (1)

Harris Harris affine Harris Laplace

Hessian affine Hessian Laplace




Qualitative comparison of detectors (2)

Kadir/Brady morphological

MSER SUSAN




Descriptors (1)

• Representation of salient regions • “descriptive” features feature vector• There are many possibilities !• Categorization vs. specific OR, matching

– Sufficient descriptive power

– Not too much emphasis on specific individuals

– Performance is often category-specific

Tjnnn ff ),,( ,1, f

feature vector extracted from patch Pn

vs. AR ?




Descriptors (2)

• Grayvalues– Raw pixel values of a patch P

– “local appearance-based description”

– “local affine frame” LAF [Obdržálek+Matas] for MSER

• General moments of order p+q:

• Moment invariants:– Central moments μpq: invariant to translation

Py

qp

Pxpq yxIyxm ),(

),()()( :,00

01

00

10 yxIyyxxm

my

m

mx q

Px Py

ppq




Descriptors (3)

• Moment invariants:– Normalized central moments

– Translation, rotation, scale invariant moments Φ1 ... Φ7 [Hu]

– Geometric/photometric, color invariants [vanGool et al.]

• Filters– “local jets” [Koenderink+VanDoorn]

– Gabor banks, steerable filters, discrete cosine transform DCT

21 with ,

00

qppqpq

211

202202

02201

4)(




Descriptors (4)

• SIFT descriptors [Lowe]– Scale invariant feature transform

– Calculated for local patch P: 8x8 or 16x16 pixels

– Subdivision into 4x4 sample regions

– Weighted histogram of 8 gradient directions: 0º, 45º, …

– SIFT vector dimension: 128 for a 16x16 patch

[Lowe]




Algorithms









PZ

Y

X

pppp

pppp

pppp

y

x

p

P

1

~

1 34333231

24232221

14131211

RQ Decomposition (1)

• Remember camera projection matrix P

• P can be decomposed, e.g. finite projective camera

41|

~| pC

MIMIKRP

10

0

y

xs

y

x

K




RQ Decomposition (2)• Unfortunately: R refers to “upper triangular”, Q to “rotation”…• “Givens rotations”:

• How to decompose a given 3 x 3 matrix (say M) ?– MQx enforcing M32 = 0, first column of M unchanged, last two columns

replaced by linear combinations of themselves– MQxQy enforcing M31 = 0, 2nd column unchanged (M32 remains 0)– MQxQyQz enforcing M21 = 0, first two columns replaced by linear combinations

of themselves, thus M31 and M32 remain 0

MQxQyQz = R, M = RQxTQy

TQzT , where R is upper triangular

• How to enforce

e.g. M21 = 0 ?

)cos()sin(

)sin()cos(

1

rollroll

rollrollxQ

)cos()sin(

1

)sin()cos(

pitchpitch

pitchpitch

yQ

1

)cos()sin(

)sin()cos(

yawyaw

yawyaw

zQ

)sin(),cos(

0: 222121

yawsyawc

smcm

M

cyaw

mmmcsc

arccos

/1 222

22122

22




Singular Value Decomposition - SVD• Given a square matrix A (e.g. 3x3)• A can be decomposed into

• where U and V are orthogonal matrices, and• D is a diagonal matrix with

– non-negative entries,

– entries in descending order.

“the column of V corresponding to the smallest singular value”

↔ “the last column of V”

,TUDVA




SVD (2)• SVD is also possible when A is non-square (e.g. m x n, m≥n)• A can again be decomposed into

• where U is m x n with orthogonal columns (UTU=Inxn),

• D is an n x n diagonal matrix with– non-negative entries,

– entries in descending order,

• V is an n x n orthogonal matrix.

,TUDVA




SVD for Least-Squares Solutions

• Overdetermined system of linear equations• Find least-squares ( algebraic error! ) solution

• Algorithm:

1. Find the SVD

2. Set

3. Find

4. The solution is

nnmbxnm AA rank , , :equations ofset

TUDVA

bb T

U'

D ofentry diagonalth - theis where, /by defined , ' iddbyy iiii

yxV

Even easier for Ax=0:“x is the last column of V”




Algorithms









(2n x 9) / (2n x 12) matrix representing correspondences

Systems of Linear Equations (1)• Estimation of

– A homography H:– The fundamental matrix:– The camera projection matrix:

• By finding n point correspondences– between 2 images– between image and scene

• And solving a system of linear equations– Typical form:

xx

H'0' xx T

FXx

P~

ii xx

'ii Xx

0v

A

9 - vector representing H, F12 - vector representing P




Systems of Linear Equations (2)• How to obtain ?

• Homography H– 3 x 3 matrix, 8 DoF, non-singular– at least 4 point correspondences are required

• Fundamental matrix F– 3 x 3 matrix, 7 DoF, rank 2– at least 7 point correspondences are required

• Camera projection matrix P– 3 x 4 matrix, 11 DoF, decomposition into K, R, t– at least 5-1/2 (6) point correspondences are required

0v

A

why ?




Homography Estimation (1)

iiii xxxx

H '' :

0 0 ' ii xxxx

H Equation defining the computation of H

Point correspondences

9

1

3

2

1

3

2

1

987

654

321

987

654

321

'

'

'

' ,,,

h

h

h

h

h

h

h

h

h

hhh

hhh

hhh

hhh

hhh

hhh

w

y

x

x

w

y

x

xT

T

T

i

i

i

i

i

i

i

i

H

Some notation:

iT

iiT

i

iT

iiT

i

iT

iiT

i

ii

xhyxhx

xhxxhw

xhwxhy

xx

1'2'

3'1'

2'3'

' H 0

0

0

0

:3

2

1

''

''

''

h

h

h

xxxy

xxxw

xyxw

hxxhTT

iiTii

Tii

TTii

Tii

Tii

T

jTii

jT

Simple rewriting:





0

0

0

0

3

2

1

''

''

''

h

h

h

xxxy

xxxw

xyxw

TTii

Tii

Tii

TTii

Tii

Tii

T

0

hiA

Ai is a 3 x 9 matrix , h is a 9-vector

• the system describes 3 equations• the equations are linear in the unknown h• elements of Ai are quadratic in the known point coordinates• only 2 equations are linearly independent• thus, the 3rd equation is usually omitted [Sutherland 63]:

00

0

3

2

1

''

''

h

h

h

xxxw

xyxwTii

TTii

Tii

Tii

T 0

hiA

Ai is a 2 x 9 matrix , h is a 9-vector





0

0

000

000:0

9

1

'11

'11

'11

'11

'11

'11

'11

'11

'11

'11

'11

'11

1

h

h

xwxyxxwwwywx

ywyyyxwwwywxh

A

1 point correspondence defines 2 equationsH has 9 entries, but is defined up to scale 8 degrees of freedom at least 4 point correspondences needed

0

0

000

000:0

0

0

9

1

'44

'44

'44

'44

'44

'44

'44

'44

'44

'44

'44

'44

4

3

2

h

h

xwxyxxwwwywx

ywyyyxwwwywxh

h

h

A

A

A

0

4

3

2

1

hh A

A

A

A

A

4 x 2 equations

General case:• overdetermined• n point correspondences• 2n equations, A is a 2n x 9 matrix 0

hA




Camera Projection Matrix Estimation

3

2

1

333231

232221

131211

3

2

1

'

'

'

'

x

x

x

hhh

hhh

hhh

x

x

x

x

x

H XtXZ

Y

X

pppp

pppp

pppp

y

x

x

|

1

~

1 34333231

24232221

14131211

RKP

homography H: projection matrix P

very similar ! 0 0

pxx A

• n point correspondences 2n equations that are linear in elements of P• A is a 2n x 12 matrix, entries are quadratic in point coordinates• p is a 12-vector• P has only 11 degrees of freedom• a minimum of 11 equations is required 5-1/2 (6) point correspondences




Fundamental (Essential) Matrix Estimation (1)

• solving is different from solving

• each correspondence gives only one equation in the coefficients of F !

• for n point matches we again obtain a set of linear equations (linear in f1-f9)

xx

H' 0' xx T F

xx

'

0''''''' :

1

,

1

'

'

' 333231232221131211

fyfxffyyfyfxyfxyfxxfxy

x

xy

x

x

0

1

1

''''''

11'11

'11

'1

'11

'11

'1

ff

yxyyyxyxyxxx

yxyyyxyxyxxx

nnnnnnnnnnnn

A




Fundamental (Essential) Matrix Estimation (2)

• F is a 3 x 3 matrix, has rank 2, |F| = 0 F has only 7 degrees of freedom• at least 7 point correspondences are required to estimate F

0

fA

Back to the solution of systems of linear equations !

similar systems of equations, but: different constraints

0 0 0

fph AAA




SVD for Least-Squares Solutions• Overdetermined system of linear equations• Find least-squares ( algebraic error! ) solution

• Algorithm:

1. Find the SVD

2. Set

3. Find

4. The solution is

nnmbxnm AA rank , , :equations ofset

TUDVA

bb T

U'

D ofentry diagonalth - theis where, /by defined , ' iddbyy iiii

yxV

Even easier for Ax=0:“x is the last column of V”

This is also called “direct linear transform” – DLT




Relevant Issues in Practice

• Poor condition of A Normalization

• Algebraic error vs.

geometric error, Iterative minimization

nonlinearities (lens dist.)

• Outliers Robust algorithms

(RANSAC)

institut für elektrische meßtechnik und meßsignalverarbeitung professor horst cerjak, 19.12.2005...

Documents

augmented reality vu

frstner corners slide

klt corners

local derivatives

rosenfeld slide

c harris t harris slide

derivatives autocorrelation

y corner