day 2: structure from motion diagram

Day 2: Structure from motion diagram

n-viewgeometry

Calibration

Three-viewgeometry

Two-viewgeometry

Camerageometry

Projectivegeometry

First topic: projective geometry

n-viewgeometry

Calibration

Three-viewgeometry

Two-viewgeometry

Camerageometry

ProjectivegeometryHZ 2,3

But a short introduction to the camera first

n-viewgeometry

Calibration

Three-viewgeometry

Two-viewgeometry

Camerageometry

ProjectivegeometryHZ 2,3

Introduction to the camera

• Faugeras: the camera as ‘projective engine’: “A camera is a particular geometric engine which constructs planar images of the 3-dimensional world through a projection.” [FL 174]

• we want to invert this imaging process to yield a model of the 3-dimensional world

• must understand image formation

Theory of perspective in history• pinhole cameras and the image formation process

were first explored by Renaissance painters, where it was called the theory of perspective

• Leonardo da Vinci (quoting John Pecham): “Perspective ... should be preferred above all man’s discourses and disciplines. In this subject the visual rays are elucidated by means of demonstrations which derive their glory not only from mathematics but also from physics; the one is adorned equally with the flowers of the other.” [Kemp, p. 5]

• Brunelleschi, a Florentine architect (Baptistery, Duomo) developed theory of perspective circa 1413

• theory perspective machines camera obscura daguerrotype camera

– Faugeras cover + FL176 + camera obscura FL177– need a dark room to capture just the light from the

pinhole– used by painters as a portable device (see Vermeer in

The Girl with a Pearl Earring)– lenses added at pinhole to gather light better– Daguerre replaced one wall of camera obscura with

photosensitive film, yielding first photograph (daguerrotype)

– et voila, the camera

Camera obscura

Theory of perspective 2

• behaviour at infinity immediately became of interest, as vanishing points were incorporated into paintings– vanishing point in Bosse: FL205– vanishing line in Houckgeest: FL206

• renewed interest with the invention of cameras, leading to photogrammetry (theory of measurement of 3d world from 2d images): FL174 + Pollefeys CACM02

• projective geometry developed in mathematics: a tool that can be used to understand the image formation process

Pinhole camera model

• geometrically very simple, encoding perspective projection– camera center = optical center C– image plane = retinal plane P– image x of X = intersection of XC with P

• practical setup: – two screens: image plane and focal plane– pinhole in focal plane

Camera terminology

• camera center, image plane– confusion: HZ (153) equate the image and focal planes; but for Faugeras

(F35), the focal plane passes through the optical center• focal length f = dist (C, P)• focal plane = plane through C parallel to P• principal axis = line through C normal to P = optical axis• principal point = intersection of principal axis and P

Perspective projection

• not a linear map in Euclidean geometry!• suppose

– camera center = origin– principal axis = z-axis

• (X,Y,Z) f/Z (X,Y) in image plane Z=f• this is not linear

Projective geometry simplifies

1. perspective projection will be a linear map in projective geometry• what are other motivations for using projective geometry rather than

Euclidean?2. projective geometry is a simpler generalization of Euclidean

geometry• lifting up a dimension simplifies, just as lifting from flatland to 3-space

simplifies interpretation of flatland (e.g., sphere passing through flatland)• so

(a) less special cases• e.g., translation vs. rotation/scaling

(b) algebra is often simplified (homogeneous coordinates)

3. can reason about geometry at infinity• infinity is one of those special cases for Euclidean geometry

4. points and lines are treated equivalently (dual), so they interact well

First topic: projective geometry

n-viewgeometry

Calibration

Three-viewgeometry

Two-viewgeometry

Camerageometry

ProjectivegeometryHZ 2,3

Projective geometry diagram(2-space)

DLT algHZ 4.1

Rectification

HZ 2.7

Hierarchy of maps

InvariantsHZ 2.4

Projectivetransform

HZ 2.3

Behaviourat infinity

Primitivespt/line/conic

HZ 2.2

First topic: primitives

DLT algHZ 4.1

Rectification

HZ 2.7

Hierarchy of maps

InvariantsHZ 2.4

Projectivetransform

HZ 2.3

Behaviourat infinity

Primitivespt/line/conic

HZ 2.2

Point/line diagram

point/pointjoin

P1 x P2

P1P2

L1L2

line/lineintersection

L1 x L2

line(a,b,c)

pointxyw

join

intersection

point on lineP . L = 0

Point-line representation• we want a new data structure (or encoding) for points and lines

– importance of representation and notation: think of pre-20th century mathematics notation (e.g., Descartes)• consider the implicit representation of a line in 2-space ax+by+c=0• the line’s homogeneous representation is (a,b,c)

– coefficients of the line’s implicit equation• for consistency with tensors later, let us treat line as a row vector (although HZ introduce as a

column); OK to treat lines and points as generic vectors at early stages

• consider a point in 2-space (x,y)• its homogeneous representation is (x,y,1)

– coordinates in the implicit equation• a column vector

• what are the ramifications of this encoding?• point/line incidence, intersection and join are all simplified

– point P lies on line L iff P.L = 0– intersection of two lines L1 and L2 is L1 x L2 (recall cross product)– join of two points P1 and P2 is P1 x P2

• HZ 26-27

Proofs

• the point P lies on the line L iff P.L = 0– after all, point satisfies line equation– dually, L intersects P iff L.P = 0

• recall that inner product = 0 encodes orthogonality• intersection of two (distinct) lines L1 and L2 is L1 x L2

– think geometrically: L1 x L2 is normal to L1 and L2– so (L1xL2) . L1 = 0 and (L1xL2) . L2 = 0– so the point L1xL2 lies on L1 and L2– e.g., what is the intersection of x=1 and y=1?

• join of two (distinct) points P1 and P2 is P1xP2– again (P1xP2) . P1 = 0 and (P1xP2) . P2 = 0, so this line contains these two points

• morals– inner product encodes incidence

• this is also simple matrix multiplication if we represent points as columns and lines as rows

– cross product encodes intersection and join

Exercise

• think of inputting lines and computing their intersections using these projective results– points lines intersections

Homogeneous coordinates

• the point and line encodings are vectors in projective space, with homogeneous representations– note that ax+by+c=0 is equivalent to kax + kby + kc = 0– (a,b,c) = k(a,b,c) for any k \neq 0

• projective space P^2 = the set of equivalence classes of vectors in R^3 – (0,0,0) under the equivalence relation (a,b,c) = k(a,b,c) for nonzero k

• in projective space, objects are only known up to a scalar multiple– therefore, both the point and line in 2-space have two degrees of freedom

(dof), not three– the two dof in a line (a,b,c) are the two independent ratios a:b:c– projective space always uses one extra dimension

• can visualize P^2 as lines through the origin in 3-space

• projective geometry is the study of the geometry of projective space

Perspective projection is now linear

• translate from homogeneous coordinates in projective space to Cartesian coordinates in Euclidean space by:– (x,y,w) (x/w, y/w)– after all, (x,y,1) (x,y) and (kx,ky,k) (x,y)– P^2 R^2– this should look familiar: perspective projection

• in projective space, perspective projection is a linear map:– Euclidean: (X,Y,Z) f/Z (X,Y)– (X,Y,Z,1) (fX,fY,Z)– this can be encoded by a matrix

• (X,Y,Z,1) (diag(f,f,1) 0) (X,Y,Z,1)

• aside: translation in 3-space is now encodable by a homogeneous 4x4 matrix

Duality

• points and lines are dual under this representation– think of ax+by+cw = 0– point representation and line representation are

interchangeable in this way• can exchange points and lines with no ramifications

– therefore, all statements about points can be replaced by statements about lines

• e.g., P.L = 0: P lies on L or L intersects P• e.g., P1xP2 maps from two points to a line, and L1xL2 maps

from two lines to a point– in general, points and hyperplanes in n-space are dual