multiple view reconstruction class 23 multiple view geometry comp 290-089 marc pollefeys

Multiple View ReconstructionClass 23

Multiple View GeometryComp 290-089Marc Pollefeys

Administrivia

• No papers to be presented• Probably take home exam

Content

• Background: Projective geometry (2D, 3D), Parameter estimation, Algorithm evaluation.

• Single View: Camera model, Calibration, Single View Geometry.

• Two Views: Epipolar Geometry, 3D reconstruction, Computing F, Computing structure, Plane and homographies.

• Three Views: Trifocal Tensor, Computing T.• More Views: N-Linearities, Self-

Calibration,Multi View Reconstruction, Bundle adjustment, Dynamic SfM, Cheirality, Duality

Multi-view computation

Multiple view computation

• Tensors (2,3,4-views)• Factorization

• Orthographic• Perpective

• Sequential• Bundle adjustment

Orthographic factorizationThe ortographic projection equations are

where njmijiij ,...,1,,...,1,Mm P

All equations can be collected for all i and j

where

n

mmnmm

n

n

M,...,M,M,,

mmm

mmm

mmm

212

1

21

22221

11211

M

P

P

P

Pm

MPm

M ~

~m

j

j

j

jyi

xi

iij

ijij

Z

Y

X

,P

P,

y

xP

(Tomasi Kanade’92)

Orthographic factorizationFactorize m through singular value

decomposition

An affine reconstruction is obtained as follows

TVUm

TVMUP ~,

~

(Tomasi Kanade’92)

nm

mnmm

n

n

M,...,M,M

mmm

mmmmmm

min 212

1

21

22221

11211

P

P

P

Closest rank-3 approximation yields MLE!

Perspective factorizationAll equations can be collected for all i as

wherePMm

mnn P

P

P

P

m

m

m

m...

,...

2

1

22

11

m is known, but i,P and M are unknown

P=the first four columns of UM=the first four rows of Vrefine ij iteratively (initialize with 1 or from tensors)

practical structure and motion recovery from images

• Obtain reliable matches using matching or tracking and 2/3-view relations

• Compute initial structure and motion• Refine structure and motion• Auto-calibrate• Refine metric structure and motion

Computation of initial structure and motion

according to Hartley and Zisserman “this area is still to some extend a

black-art”All features not visible in all images No direct method Build partial reconstructions and assemble (more views is more stable, but less corresp.)

1) Sequential structure and motion recovery

2) Hierarchical structure and motion recovery

Sequential structure and motion recovery

Initialize structure and motion from 2 views

For each additional view• Determine pose• Refine and extend structure

Initial structure and motion

eeaFeP

0IPT

x

2

1

Epipolar geometry Projective calibration

012 FmmT

compatible with F

Yields correct projective camera setup(Faugeras´92,Hartley´92)

Obtain structure through triangulation

Use reprojection error for minimizationAvoid measurements in projective space

Compute Pi+1 using robust approach (6-point RANSAC)

Extend and refine reconstruction

2D-2D

2D-3D 2D-3D

mimi+1

M

new view

Determine pose towards existing structure

Compute P with 6-point RANSAC

• Generate hypothesis using 6 points

• Count inliers • Projection error ?x,x...,,xXP 11 td iii

• Back-projection error ijtd jiij ?,x,xF

• Re-projection error td iiii x,x,x...,,xXP 11

• 3D error ?X,xP 3-1

Dii td

• Projection error with covariance

td iii x,x...,,xXP 11

• Expensive testing? Abort early if not promising• Also verify at random, abort if e.g. P(wrong)>0.95

(Chum and Matas, BMVC’02)

Non-sequential image collections

4.8im/pt64 images

3792

po

ints

Problem:Features are lost and reinitialized as new features

Solution:Match with other close views

For every view iExtract featuresCompute two view geometry i-1/i and matches Compute pose using robust algorithmRefine existing structureInitialize new structure

Relating to more views

Problem: find close views in projective frame

For every view iExtract featuresCompute two view geometry i-1/i and matches Compute pose using robust algorithmFor all close views k

Compute two view geometry k/i and matchesInfer new 2D-3D matches and add to list

Refine pose using all 2D-3D matchesRefine existing structureInitialize new structure

Determining close views

• If viewpoints are close then most image changes can be modelled through a planar homography

• Qualitative distance measure is obtained by looking at the residual error on the best possible planar homography

Distance = m´,mmedian min HD

9.8im/pt

4.8im/pt

64 images

64 images

3792

po

ints

2170

po

ints

Non-sequential image collections (2)

Hierarchical structure and motion recovery

• Compute 2-view• Compute 3-view• Stitch 3-view reconstructions• Merge and refine reconstruction

FT

H

PM

Stitching 3-view reconstructions

Different possibilities1. Align (P2,P3) with (P’1,P’2) -1

23-1

12H

HP',PHP',Pminarg AA dd

2. Align X,X’ (and C’C’) j

jjAd HX',XminargH

3. Minimize reproj. error

jjj

jjj

d

d

x',HXP'

x,X'PHminarg 1-

H

4. MLE (merge) j

jjd x,PXminargXP,

Refining structure and motion

• Minimize reprojection error

• Maximum Likelyhood Estimation (if error zero-mean

Gaussian noise)• Huge problem but can be solved

efficiently (Bundle adjustment)

m

k

n

iikD

ik 1 1

2

kiM̂,P̂

M̂P̂,mmin

Non-linear least-squares

• Newton iteration• Levenberg-Marquardt • Sparse Levenberg-Marquardt

(P)X f (P)X argminP

f

Newton iteration

Taylor approximation

J)(P )(P 00 ffP

XJ

Jacobian

)(PX 1f

JJ)(PX)(PX 001 eff

0T-1T

0TT JJJJJJ ee

i1i PP 0T-1T JJJ e

01-T-11-T JJJ e

normal eq.

Levenberg-Marquardt

0TT JNJJ e

0TJN' e

Augmented normal equations

Normal equations

J)λdiag(JJJN' TT

30 10λ

10/λλ :success 1 ii

ii λ10λ :failure solve again

accept

small ~ Newton (quadratic convergence)

large ~ descent (guaranteed decrease)

Levenberg-Marquardt

Requirements for minimization• Function to compute f

• Start value P0

• Optionally, function to compute J(but numerical ok, too)

Sparse Levenberg-Marquardt

• complexity for solving• prohibitive for large problems

(100 views 10,000 points ~30,000 unknowns)

• Partition parameters• partition A • partition B (only dependent on A and

itself)

0T-1 JN' e3N

Sparse bundle adjustment

residuals:normal equations:

with

note: tie points should be in partition A

Sparse bundle adjustment

normal equations:

modified normal equations:

solve in two parts:

Sparse bundle adjustment

U1

U2

U3

WT

W

V

P1 P2 P3 M

Jacobian of has sparse block structure

J JJN T

12xm 3xn(in general

much larger)

im.pts. view 1

m

k

n

iikD

1 1

2

ki M̂P̂,m

Needed for non-linear minimization

Sparse bundle adjustment• Eliminate dependence of

camera/motion parameters on structure parametersNote in general 3n >> 11m

WT V

U-WV-1WT

NI0WVI 1

11xm 3xn

Allows much more efficient computations

e.g. 100 views,10000 points,

solve 1000x1000, not 30000x30000Often still band diagonaluse sparse linear algebra algorithms

Sparse bundle adjustment

normal equations:

modified normal equations:

solve in two parts:

Sparse bundle adjustment

• Covariance estimation

-1WVY

1a

Tb

1-a

VYY

WWVU

Yaab -

Next class: Dynamic structure from motion