Structure from Motion

Course web page:vision.cis.udel.edu/~cv

April 25, 2003 Lecture 26

Announcements

• Read Hartley & Zisserman Chapter 17.2 (skip 17.2.1) and Forsyth & Ponce Chapter 12.3 on affine structure from motion for Monday

• Homework 4 due on Monday

Outline

• Triangulation• Stratified reconstruction

Computing Structure

• Recall that canonical camera matrices P, P’ can be computed from fundamental matrix F– E.g. P = [Id j 0] and P’ = [[e’]£F j e’],

• Triangulation: Back-projection of rays from image points x, x’ to 3-D point of intersection X such that x = PX and x’ = P’X

from Hartley & Zisserman

Triangulation: Issues

• Errors in points x, x’ ) @ F such that x’T

F x = 0 or X such that x = PX and x’ = P’X

• This means that rays are skew — they don’t intersect

from Hartley & Zisserman

Triangulation with Non-Intersecting Rays

• Define some heuristic for best estimate of X– Idea: Find midpoint of common perpendicular to the two rays

• But this is not invariant to projective transformations – Recall that without calibration the camera matrices are only known

up to projection—i.e., PH, P’H are the “true” camera matrices for some non-singular H—so we will get different answers for X

X

from Hartley & Zisserman

Definition of Projectively Invariant Triangulation

• Suppose we compute a 3-D point X from the image points x, x’ and camera matrices P, P’ by some triangulation method ¿

• We say that ¿ is projective-invariant if for any projective transformation H:

X = ¿ (x, x’, P, P’) = H-1

¿ (x, x’, PH-1, P’H-

1)

Optimal Projective-Invariant Triangulation: Reprojection Error

• Pick that exactly satisfies camera geometry so that and , and which minimizes

• Can use as error

function for nonlinear minimization on two views– Polynomial solution exists

from Hartley & Zisserman

DLT Triangulation

• There is a Direct Linear Transformation method for triangulation (see Hartley & Zisserman Chapter 11.2)– Not projectively invariant– Easily extends to > 2 views (whereas

nonlinear method does not)

Covariance of Structure Recovery

• Bigger angle between rays ) Less uncertainty

• Can’t triangulate points on baseline (epipoles) because rays intersect along entire length

from Hartley & Zisserman

Projective Reconstruction Theorem

• With uncalibrated cameras alone, we can reconstruct a scene (e.g., via triangulation) up to a projective ambiguity

• Calibrated cameras give metric reconstruction

Example: Projective Reconstruction Ambiguity

from Hartley & Zisserman

Reconstructions related by a 4 x 4 projection H

Two views from which F and

hence P, P’ are computed

Hierarchy of Transformations

Less ambiguity

Properties of transformations (2-D)from Hartley & Zisserman

Stratified Reconstruction• Idea: Try to upgrade reconstruction to differ from the truth

by a less ambiguous transformation • Use additional constraints imposed by:

– Scene – Motion– Camera calibration

• “Cheats”– Again: Cameras with known K, K’ ! Metric reconstruction– ¸ 5 known 3-D points (no 4 coplanar) ! Euclidean reconstruction

from Hartley & Zisserman

Projective ! Affine Upgrade

• Identify plane at infinity ¼1 (in the “true”

coord-inate frame, ¼1 = (0, 0, 0, 1)T)

– E.g., intersection points of three sets of parallel lines define a plane

– E.g., if one camera is known to be affine

from Hartley & Zisserman

Projective ! Affine Upgrade• Then apply 4 x 4 transformation:

• This is the 3-D analog of affine image rectification via the line at infinity l1

• Things that can be computed/constructed with only affine ambiguity:– Midpoint of two points– Centroid of group of points– Lines parallel to other lines, planes

Example: Affine Reconstruction

Ambiguity

Affine reconstructionsfrom Hartley & Zisserman

Affine ! Metric Upgrade

• Identify absolute conic 1 on ¼1 via image of absolute conic (IAC) ! – From scene

• E.g., orthogonal lines– From known camera calibration

• Completely constrained: ! = K-T K-1

• Partially constrained:– Zero skew– Square pixels

– Same camera took all images

• 1 and ! are beyond scope of this class—they won’t be on the final

Example: Metric Reconstruction with Texture

Mapping

Only overall scale ambiguity remains—i.e., what are units of length?

from Hartley & Zisserman

Original views Synthesized views of reconstruction