ComputerVision

Projective structure from motion

Marc PollefeysCOMP 256

Some slides and illustrations from J. Ponce, …

ComputerVision

2

Last class: Affine camera

• The affine projection equations are

1

~

~

j

j

j

yi

xi

ij

ij

ZYX

P

Py

x

how to find the origin? or for that matter a 3D reference point?

affine projection preserves center of gravity

i

ijijij xxx~ i

ijijij yyy~

2

1

tt

~

~

j

j

j

yi

xi

ij

ij

Z

Y

X

P

Py

x

ComputerVision

3

Orthographic factorization

• The orthographic projection equations are

• where

njmijiij ,...,1,,...,1,Xx P

• All equations can be collected for all i and j

• where

n

mmnmm

n

n

X,...,X,X , ,

xxx

xxxxxx

212

1

21

22221

11211

X

P

P

P

Px

XPx

X ~

~x

j

j

j

jyi

xi

iij

ijij

Z

Y

X

,P

P,

y

xP

Note that P and X are resp. 2mx3 and 3xn matrices and

therefore the rank of x is at most 3

2

1

tt

~

~

j

j

j

yi

xi

ij

ij

Z

Y

X

P

Py

x

ComputerVision

4

Orthographic factorization

• Factorize m through singular value decomposition

• An affine reconstruction is obtained as follows

TVUx

T~,

~VXUP

n

mmnmm

n

n

X,...,X,X

xxx

xxxxxx

min 212

1

21

22221

11211

P

P

P

Closest rank-3 approximation yields MLE!

ComputerVision

5

Jan 16/18 - Introduction

Jan 23/25 Cameras Radiometry

Jan 30/Feb1 Sources & Shadows Color

Feb 6/8 Linear filters & edges Texture

Feb 13/15 Multi-View Geometry Stereo

Feb 20/22 Optical flow Project proposals

Feb27/Mar1 Affine SfM Projective SfM

Mar 6/8 Camera Calibration Silhouettes and Photoconsistency

Mar 13/15 Springbreak Springbreak

Mar 20/22 Segmentation Fitting

Mar 27/29 Prob. Segmentation Project Update

Apr 3/5 Tracking Tracking

Apr 10/12 Object Recognition Object Recognition

Apr 17/19 Range data Range data

Apr 24/26 Final project Final project

Tentative class schedule

ComputerVision

6

PROJECTIVE STRUCTURE FROM MOTION

Reading: Chapter 13.

• The Projective Structure from Motion Problem• Elements of Projective Geometry• Projective Structure and Motion from Two Images • Projective Motion from Fundamental Matrices• Projective Structure and Motion from Multiple Images

ComputerVision

7

The Projective Structure-from-Motion Problem

Given m perspective images of n fixed points P we can write

Problem: estimate the m 3x4 matrices M andthe n positions P from the mn correspondences p .

i

j ij

2mn equations in 11m+3n unknowns

Overconstrained problem, that can be solvedusing (non-linear) least squares!

j

ComputerVision

8

The Projective Ambiguity of Projective SFM

If M and P are solutions, i j

So are M’ and P’ wherei j

and Q is an arbitrary non-singular 4x4 matrix.

When the intrinsic and extrinsic parameters are unknown

Q is a projective transformation.

ComputerVision

9

Projective Spaces: (Semi-Formal) Definition

ComputerVision

10

A Model of P( R )3

ComputerVision

11

Projective Subspaces and Projective Coordinates

ComputerVision

12

Projective Subspaces and Projective Coordinates

Projectivecoordinates

P

ComputerVision

13

Projective Subspaces

Given a choice of coordinate frame

Line: Plane:

ComputerVision

14

Affine and Projective Spaces

ComputerVision

15

Affine and Projective Coordinates

ComputerVision

16

Cross-Ratios

Collinear points

Pencil of coplanar lines Pencil of planes

{A,B;C,D}=sin(+)sin(+)

sin(++)sin

ComputerVision

17

Cross-Ratios and Projective Coordinates

Along a line equipped with the basis

In a plane equipped with the basis

In 3-space equipped with the basis

*

*

ComputerVision

18

Projective Transformations

Bijective linear map:

Projective transformation:( = homography )

Projective transformations map projective subspaces ontoprojective subspaces and preserve projective coordinates.

Projective transformations map lines onto lines andpreserve cross-ratios.

ComputerVision

19

Perspective Projections induce projective transformationsbetween planes.

ComputerVision

20

Geometric SceneReconstruction

Idea: use (A,O”,O’,B,C)as a projective basis.

ComputerVision

21

Reprinted from “Relative Stereo and Motion Reconstruction,” by J. Ponce, T.A. Cass and D.H. Marimont, Tech. ReportUIUC-BI-AI-RCV-93-07, Beckman Institute, Univ. of Illinois (1993).

ComputerVision

22

Motion estimation from fundamental matrices

Q

Facts:

b’ can be found using LLS.

Once M and M’ are known,P can be computed with LLS.

ComputerVision

23

Projective Structure from Motion and Factorization

Factorization??

Algorithm (Sturm and Triggs, 1996) • Guess the depths;• Factorize D;• Iterate.

Does it converge? (Mahamud and Hebert, 2000)

ComputerVision

24

Bundle adjustment - refining structure and motion

• Minimize reprojection error

– Maximum Likelyhood Estimation (if error zero-mean Gaussian

noise)– Huge problem but can be solved

efficiently (exploit sparseness)

m

i

n

jjiijD

ji 1 1

2

X̂,P̂X̂P̂,xmin

ComputerVision

25

• Developed in photogrammetry in 50´s

Bundle adjustment

ComputerVision

26

Non-linear least squares• Linear approximation of residual• allows quadratic approximation of sum- of-squares

J0e

0T1-T

0TT

JJJ

0J2J2J

e

e

JJ 00 ee T

J)λdiag(JJJN' TT

N

(extra term = descent term)

Minimization corresponds to finding zeros of derivative

Levenberg-Marquardt: extra term to deal with singular N(decrease/increase if success/failure to descent)

ComputerVision

27

Bundle adjustment

U1

U2

U3

WT

W

V

P1 P2 P3 M

• Jacobian of has sparse block structure– cameras independent of other cameras,– points independent of other points

J JJN T

12xm 3xn(in general

much larger)

im.pts. view 1

m

i

n

jjiijD

1 1

2M̂P̂,m

Needed for non-linear minimization

ComputerVision

28

Bundle adjustment

Eliminate dependence of camera/motion parameters on structure parametersNote in general 3n >> 11m

WT V

U-WV-1WT

NI0WVI 1

11xm 3xn

Allows much more efficient computations

e.g. 100 views,10000 points,

solve 1000x1000, not 30000x30000Often still band diagonaluse sparse linear algebra algorithms

ComputerVision

29

Sequential SfM

• Initialize motion from two images• Initialize structure• For each additional view

– Determine pose of camera– Refine and extend structure

• Refine structure and motion

ComputerVision

30

Initial projective camera motion

• Choose P and P´compatible with F

Reconstruction up to projective ambiguity

(reference plane;arbitrary)

•Initialize motion•Initialize structure•For each additional view

•Determine pose of camera•Refine and extend structure

•Refine structure and motion

Same for more views?

different projective basis

ComputerVision

31

Initializing projective structure

• Reconstruct matches in projective frame by minimizing the reprojection error

Non-iterative optimal solution •Initialize motion•Initialize structure•For each additional view

•Determine pose of camera•Refine and extend structure

•Refine structure and motion

ComputerVision

32

Projective pose estimation

• Infere 2D-3D matches from 2D-2D matches

• Compute pose from (RANSAC,6pts)

F

X

x

Inliers: inx,X x X DD iii P

•Initialize motion•Initialize structure•For each additional view

•Determine pose of camera•Refine and extend structure

•Refine structure and motion

ComputerVision

33

• Refining structure

• Extending structure2-view triangulation

X~

P

1

3

(Iterative linear)

•Initialize motion•Initialize structure•For each additional view

•Determine pose of camera•Refine and extend structure

•Refine structure and motion

Refining and extending structure

ComputerVision

34

Refining structure and motion

• use bundle adjustment

Also model radial distortion to avoid bias!

ComputerVision

35

Hierarchical structure and motion recovery

• Compute 2-view• Compute 3-view• Stitch 3-view reconstructions• Merge and refine reconstruction

FT

H

PM

ComputerVision

36

Metric structure and motion

Note that a fundamental problem of the uncalibrated approach is that it fails if a purely planar scene is observed (in one or more views)

(solution possible based on model selection)

use self-calibration (see next class)

ComputerVision

37

Dealing with dominant planes

ComputerVision

38

PPPgric

HHgric

ComputerVision

39

Farmhouse 3D models

(note: reconstruction much larger than camera field-of-view)

ComputerVision

40

Application: video augmentation

ComputerVision

41

Next class: Camera calibration (and self-calibration)

Reading: Chapter 2 and 3