fundamentals of photogrammetry - faridesmfundamentals of photogrammetry fundamentals of...

25
Fundamentals of Photogrammetry Fundamentals of Photogrammetry Niclas B¨ orlin, Ph.D. [email protected] Department of Computing Science Ume˚ a University Sweden April 1, 2014 Fundamentals of Photogrammetry Presentation Ph.D. in Computing Science (2000). Numerical Linear Algebra. Non-linear least squares with non-linear equality constraint. X-ray photogrammetry — Radiostereometry (RSA). Post doc at Harvard Medical School, Boston, MA. Fundamentals of Photogrammetry Radiostereometric analysis (RSA) Developed by Hallert (1960), Selvik (1974), K¨ arrholm (1989), orlin (2002, 2006), Valstar (2005). Procedure Dual X-ray setup Calibration cage Marker measurements Reconstruction of projection geometry Motion analysis Software UmRSA Digital Measure running in Europe, North America, Australia, Asia. Used to produce 150+ scientific papers. Fundamentals of Photogrammetry Introduction Definition Photogrammetry — measuring from photographs photos — “light” gramma — “that which is drawn or written” metron — “to measure” Definition in Manual of Photogrammetry, 1st ed., 1944, American Society for Photogrammetry: Photogrammetry is the science or art of obtaining reliable measurement by means of photographs

Upload: others

Post on 25-Jun-2020

115 views

Category:

Documents


12 download

TRANSCRIPT

Page 1: Fundamentals of Photogrammetry - faridesmFundamentals of Photogrammetry Fundamentals of Photogrammetry Niclas B orlin, Ph.D. ... computations Fundamentals of Photogrammetry History

Fundamentals of Photogrammetry

Fundamentals of Photogrammetry

Niclas Borlin, [email protected]

Department of Computing ScienceUmea University

Sweden

April 1, 2014

Fundamentals of Photogrammetry

Presentation

Ph.D. in Computing Science (2000).

Numerical Linear Algebra.

Non-linear least squares with non-linear equality constraint.

X-ray photogrammetry — Radiostereometry (RSA).

Post doc at Harvard Medical School, Boston, MA.

Fundamentals of Photogrammetry

Radiostereometric analysis (RSA)

Developed by Hallert (1960), Selvik (1974), Karrholm (1989),Borlin (2002, 2006), Valstar (2005).

Procedure

Dual X-ray setupCalibration cageMarkermeasurementsReconstruction ofprojection geometryMotion analysis

Software UmRSA Digital Measure running in Europe, NorthAmerica, Australia, Asia. Used to produce 150+ scientificpapers.

Fundamentals of Photogrammetry

Introduction

Definition

Photogrammetry — measuring from photographs

photos — “light”gramma — “that which is drawn or written”metron — “to measure”

Definition in Manual of Photogrammetry, 1st ed., 1944,American Society for Photogrammetry:

Photogrammetry is the science or art of obtainingreliable measurement by means of photographs

Page 2: Fundamentals of Photogrammetry - faridesmFundamentals of Photogrammetry Fundamentals of Photogrammetry Niclas B orlin, Ph.D. ... computations Fundamentals of Photogrammetry History

Fundamentals of Photogrammetry

Introduction

Overview

Principles

History

Mathematical models

Processing

Applications

Fundamentals of Photogrammetry

Principles

Principles

Non-contact measurements.

(Passive sensor.)

Collinearity.

Triangulation.

Fundamentals of Photogrammetry

Principles

Collinearity

Collinearity

The collinearity principle is the assumption that

the object points Q,the projection center C , andthe projected points q

are collinear, i.e. lie on a straight line.Q

C

q

Fundamentals of Photogrammetry

Principles

Triangulation

Triangulation

One image coordinate measurement (x , y) is too little todetermine the object point coordinates (X ,Y ,Z ).

We need at least two measurements of the same point.

C1

q1?

C1

q1

C2

q2

Q

Page 3: Fundamentals of Photogrammetry - faridesmFundamentals of Photogrammetry Fundamentals of Photogrammetry Niclas B orlin, Ph.D. ... computations Fundamentals of Photogrammetry History

Fundamentals of Photogrammetry

Principles

Triangulation

Triangulation (2)

The position of object points are calculated by triangulation,i.e. by angles, but without any range values.

B

α β

Q

C1 C2

Fundamentals of Photogrammetry

Principles

Triangulation

Other techniques

Trilateration, ranges but noangles (GPS).

C1r1

C2

r2

C3

r3

Q

Tachyometry, angles andranges (surveying, laserscanning)

C

Q

Fundamentals of Photogrammetry

History

Overview

Principles

History

Mathematical models

Processing

Applications

Fundamentals of Photogrammetry

History

Pre-history

Pre-history

Geometry, perspective, pinhole camera model — Euclid (300BC).

Leonardo da Vinci (1480)

Perspective is nothing else than the seeing of anobject behind a sheet of glass, smooth and quitetransparent, on the surface of which all the thingsmay be marked that are behind this glass. All thingstransmit their images to the eye by pyramidal lines,and these pyramids are cut by the said glass. Thenearer to the eye these are intersected, the smallerthe image of their cause will appear.

Page 4: Fundamentals of Photogrammetry - faridesmFundamentals of Photogrammetry Fundamentals of Photogrammetry Niclas B orlin, Ph.D. ... computations Fundamentals of Photogrammetry History

Fundamentals of Photogrammetry

History

Plane table photogrammetry

First generation — Plane table photogrammetry

First photograph — Niepce,1825. Required 8 hour expo-sure.

Glass negative — Hershel,1839.

First use of terrestrial photographs for topographic maps —Laussedat, 1849 “Father of photogrammetry”. City map ofParis (1851).Film — Eastman, 1884.Architectural photogrammetry — Meydenbauer, 1893, coinedthe word “photogrammetry”.Measurements made on a map on a table. Photographs usedto extract angles.

Fundamentals of Photogrammetry

History

Analog photogrammetry

Second generation — Analog photogrammetry

Stereocomparator (Pulfrich, Fourcade, 1901). Requiredcoplanar photographs. Measurements made by floating mark.

Aeroplane (Wright 1903). First aerial imagery from aeroplanein 1909.Aerial survey camera for overlapping vertical photos (Messter1915).

Fundamentals of Photogrammetry

History

Analog photogrammetry

Second generation — Analog photogrammetry (2)

Opto-mechanical stere-oplotters (von Orel,Thompson, 1908, Zeiss1921, Wild 1926). Allowednon-coplanar photographs.

Wild A8 Autograph (1950)

Relative orientation determined by 6 points in overlappingimages — von Gruber points (1924)

Photogrammetry — the art of avoidingcomputations

Fundamentals of Photogrammetry

History

Analytical photogrammetry

Third generation — analytical photogrammetry

Finsterwalder (1899) — equations for analyticalphotogrammetry, intersection of rays, relative and absoluteorientation, least squares theory.von Gruber (1924) — projective equations and theirdifferentials,Computer (Zuse 1941, Turing, Flowers, 1943, Aiken 1944).Schmid, Brown multi-station analytical photogrammetry,bundle block adjustment (1953), adjustment theory.

The [Ballistic Research] laboratory had a virtualglobal monopoly on electronic computing power.This unique circumstance combined with Schmid setthe stage for the rapid transistion from classicalphotogrammetry to the analytic approach (Brown).

Ackermann independent models (1966).

Page 5: Fundamentals of Photogrammetry - faridesmFundamentals of Photogrammetry Fundamentals of Photogrammetry Niclas B orlin, Ph.D. ... computations Fundamentals of Photogrammetry History

Fundamentals of Photogrammetry

History

Analytical photogrammetry

Third generation — analytical photogrammetry (2)

Analytical plotter (Helava 1957) - image-map coordinatetransformation by electronic computation, servocontrol.

Zeiss Planicomp P3

Camera calibration (Brown 1966, 1971).

Direct Linear Transform (DLT) (Abdel-Azis, Karara, 1971).

Fundamentals of Photogrammetry

History

Digital photogrammetry

Digital photogrammetry

Charge-Coupled Device (CCD) (Boyle, Smith 1969).

Landsat (1972)

Digital camera (Sesson (Eastman Kodak) 1975 — 0.01Mpixels).

Flash memory (Masuoka (Toshiba) 1980).

Matching (Forstner 1986, Gruen 1985, Lowe 1999).

Projective Geometry (Klein 1939)

5-point relative orientation (Nister 2004)

Fundamentals of Photogrammetry

Mathematical models

Overview

Principles

History

Mathematical models

Processing

Applications

Fundamentals of Photogrammetry

Mathematical models

Preliminaries

Matrix multiplication

C = AB

a11 a12 a13

a21 a22 a23

a31 a32 a33

b11 b12

b21 b22

b31 b32

c11 c12

c21 c22

c31 c32

a 21× b 12

a 22× b 22

a23× b32

+

+

Page 6: Fundamentals of Photogrammetry - faridesmFundamentals of Photogrammetry Fundamentals of Photogrammetry Niclas B orlin, Ph.D. ... computations Fundamentals of Photogrammetry History

Fundamentals of Photogrammetry

Mathematical models

Preliminaries

Image plane placement

The projected coordinates q will be identicalif a (negative) sensor is placed behind the camera center orif a (positive) sensor is mirrored and placed in front of thecamera center. Q

C

q

q

Fundamentals of Photogrammetry

Mathematical models

The collinearity equations

The collinearity equations

Z

X

Y

C

Q

c

ZX

Y

Z0

X0

Y0

x

y qqp

The collinearity equationsx − xpy − yp−c

= kR

X − X0

Y − Y0

Z − Z0

describe the relationshipbetween the object point(X ,Y ,Z )T , the positionC = (X0,Y0,Z0)T of thecamera center and theorientation R of thecamera.

Fundamentals of Photogrammetry

Mathematical models

The collinearity equations

The collinearity equations (2)

Z

X

Y

C

Q

c

ZX

Y

Z0

X0

Y0

x

y qqp

The distance c is known asthe principal distance orcamera constant.

The point qp = (xp, yp)T iscalled the principal point.

The ray passing throughthe camera center C andthe principal point qp iscalled the principal ray.

Fundamentals of Photogrammetry

Mathematical models

The collinearity equations

The collinearity equations (3)

Fromx − xpy − yp−c

= kR

X − X0

Y − Y0

Z − Z0

, and R =

r11 r12 r13

r21 r22 r23

r31 r32 r33

,

we can solve for k and insert:

x = xp − cr11(X − X0) + r12(Y − Y0) + r13(Z − Z0)

r31(X − X0) + r32(Y − Y0) + r33(Z − Z0),

y = yp − cr21(X − X0) + r22(Y − Y0) + r23(Z − Z0)

r31(X − X0) + r32(Y − Y0) + r33(Z − Z0).

Page 7: Fundamentals of Photogrammetry - faridesmFundamentals of Photogrammetry Fundamentals of Photogrammetry Niclas B orlin, Ph.D. ... computations Fundamentals of Photogrammetry History

Fundamentals of Photogrammetry

Mathematical models

Projective geometry

Homogenous coordinates

In projective geometry, points, lines, etc. are represented byhomogenous coordinates.Any cartesian coordinates (x , y) may be transformed tohomogenous by adding a unit value as an extra coordinate:(

xy

)7→

xy1

.

All homogenous vector multiplied by a non-zero scalar kbelong to the same equivalence class and correspond to thesame object. Thus,x

y1

and k

xy1

=

kxkyk

, k 6= 0

all correspond to the same 2D point (x , y)T .

Fundamentals of Photogrammetry

Mathematical models

Projective geometry

Homogenous coordinates (2)

Any homogenous vector (x1, x2, x3)T , x3 6= 0 may betransformed to cartesian coordinates by dividing by the lastelement x1

x2

x3

7→x1/x3

x2/x3

x3/x3

=

x1/x3

x2/x3

1

.

A homogenous vector (x1, x2, x3)T with x3 = 0 is called anideal point and is “infinitely far away” in the direction of(x1, x2).

The point (0, 0, 0)T is undefined.

The space <3 \ (0, 0, 0)T is called the projective plane P2.

A homogenous point in P2 has 2 degrees of freedom.

Fundamentals of Photogrammetry

Mathematical models

Projective geometry

Interpretation of the projective plane P2

A homogenous vectorx ∈ P2 may beinterpreted as a linethrough the origin in <3.

The intersection with theplane x3 = 1 gives thecorresponding cartesiancoordinates.

x3

x1

x2

xO

idealpoint

1

Fundamentals of Photogrammetry

Mathematical models

Transformations

Transformations

Transformation of homogenous 2D points may be describedby multiplication by a 3×3 matrixu

v1

=

a11 a12 a13

a21 a22 a23

a31 a32 a33

xy1

,

or

q = Ap.

Page 8: Fundamentals of Photogrammetry - faridesmFundamentals of Photogrammetry Fundamentals of Photogrammetry Niclas B orlin, Ph.D. ... computations Fundamentals of Photogrammetry History

Fundamentals of Photogrammetry

Mathematical models

Transformations

Basic transformations — Translation

A translation of points in <2 may be described usinghomogenous coordinates as

q = T (x0, y0)p =

1 0 x0

0 1 y0

0 0 1

xy1

=

x + x0

y + y0

1

.

Fundamentals of Photogrammetry

Mathematical models

Transformations

Basic transformations — Rotation

A rotation may be described using homogenous coordinates as

R(ϕ)p =

cosϕ − sinϕ 0sinϕ cosϕ 0

0 0 1

xy1

=

x cosϕ− y sinϕx sinϕ+ y cosϕ

1

.

Fundamentals of Photogrammetry

Mathematical models

Transformations

Basic transformations — Scaling

Scaling of points in <2 along the coordinate axes may bedescribed using homogenous coordinates as

q = S(k , l)p =

k 0 00 l 00 0 1

xy1

=

kxly1

.

Fundamentals of Photogrammetry

Mathematical models

Transformations

Combination of transformations

Combinations of transformations are constructed by matrixmultiplication:

q = T (x0, y0)R(ϕ)T (−x0,−y0)p

Page 9: Fundamentals of Photogrammetry - faridesmFundamentals of Photogrammetry Fundamentals of Photogrammetry Niclas B orlin, Ph.D. ... computations Fundamentals of Photogrammetry History

Fundamentals of Photogrammetry

Mathematical models

Transformation classes

Transformation classes

Transformation may be classified based on their properties.

The most important transformations are

Similarity (rigid-body transformation).Affinity.Projectivity (homography).

Fundamentals of Photogrammetry

Mathematical models

Transformation classes

Similarity

A similarity transformation consists of a combinationof rotations, isotropic scalings, and translations.s cos θ −s sin θ tx

s sin θ s cos θ ty0 0 1

or (

sR t0 1

),

where the scalar s is the scaling, R is a 2× 2rotation matrix and t is the translation vector.

A 2D similarity has 4 degrees of freedom.

A similarity preserves angles (and “shape”).

Fundamentals of Photogrammetry

Mathematical models

Transformation classes

Affinity

For an affine transformation the rotation and scalingis replaced by any non-singular 2× 2 matrix Aa11 a12 tx

a21 a22 ty0 0 1

or (

A t0 1

).

A 2D affinity has 6 degrees of freedom.

A similarity preserves parallelity but not angles.

Fundamentals of Photogrammetry

Mathematical models

Transformation classes

Projectivity (Homography)

A projectivity or homography consists of anynon-singular 3× 3 matrix Hh11 h12 h13

h21 h22 h23

h31 h32 h33

.

A 2D projectivity has 8 degrees of freedom.

A projectivity preserves neither parallelity nor angles.

Page 10: Fundamentals of Photogrammetry - faridesmFundamentals of Photogrammetry Fundamentals of Photogrammetry Niclas B orlin, Ph.D. ... computations Fundamentals of Photogrammetry History

Fundamentals of Photogrammetry

Mathematical models

Transformation classes

The effect of different transformations

Similarity Affinity Projectivity

Fundamentals of Photogrammetry

Mathematical models

Planar rectification

Planar rectification

If the coordinates for 4 points pi and their mappings qi = Hpi

in the image are known, we may calculate the homography H.

From each point pair pi = (xi , yi , 1)T , qi = (x ′i , y′i , 1)T we get

the following equations:x ′iy ′i1

=

u/wv/w

1

, where

uvw

=

h11 h12 h13

h21 h22 h23

h31 h32 h33

xiyi1

or

x ′i = u/w =h11xi + h12yi + h13

h31xi + h32yi + h33,

y ′i = v/w =h21xi + h22yi + h23

h31xi + h32yi + h33.

Fundamentals of Photogrammetry

Mathematical models

Planar rectification

Planar rectification (2)

Rearranging

x ′i (h31xi + h32yi + h33) = h11xi + h12yi + h13,

y ′i (h31xi + h32yi + h33) = h21xi + h22yi + h23.

This equation is linear in hij .

Given 4 points we get 8 equations, enough to uniquelydetermine H assuming the points are in “standard position”,i.e. no 3 points are collinear.

Fundamentals of Photogrammetry

Mathematical models

Planar rectification

Planar rectification (3)

Given H we may apply H−1 to remove the effect of thehomography.

Page 11: Fundamentals of Photogrammetry - faridesmFundamentals of Photogrammetry Fundamentals of Photogrammetry Niclas B orlin, Ph.D. ... computations Fundamentals of Photogrammetry History

Fundamentals of Photogrammetry

Mathematical models

The camera model

The pinhole camera model

The most commonly used camera model is called the pinholecamera.In the pinhole camera model:

All object points Q are projected via a central projectionthrough the same point C , called the camera center.The object point Q, the camera center C , and the projectedpoint q are collinear.A pinhole camera is straight line-preserving.

Q

C

q

Fundamentals of Photogrammetry

Mathematical models

The camera model

The central projection

If the camera center is at the origin and the image plane is theplane Z = c , the world coordinate (X ,Y ,Z )T is mapped tothe point (cX/Z , cY /Z , c)T in space or (cX/Z , cY /Z ) in theimage plane, i.e.

(X ,Y ,Z )T 7→ (cX/Z , cY /Z )T

Z

X

Y

Q

Oq

x

y

qp

Fundamentals of Photogrammetry

Mathematical models

The camera model

The central projection (2)

If the camera center is at the origin and the image plane is theplane Z = c , the world coordinate (X ,Y ,Z )T is mapped tothe point (cX/Z , cY /Z , c)T in space or (cX/Z , cY /Z ) in theimage plane, i.e.

(X ,Y ,Z )T 7→ (cX/Z , cY /Z )T

O

Y

Z

q

qp

c

cY /Zα

Fundamentals of Photogrammetry

Mathematical models

The camera model

The central projection (3)

The corresponding expression in homogenous coordinates maybe written as

XYZ1

7→

cXcYZ

=

c 0c 0

1 0

XYZ1

.

q P Q

The matrix P is called the camera matrix and maps the worldpoint Q onto the image point q.

In more compact form P may be written as

P = diag(c , c , 1)(I | 0

),

where diag(c , c , 1) is a diagonal matrix and I is the 3× 3identity matrix.

Page 12: Fundamentals of Photogrammetry - faridesmFundamentals of Photogrammetry Fundamentals of Photogrammetry Niclas B orlin, Ph.D. ... computations Fundamentals of Photogrammetry History

Fundamentals of Photogrammetry

Mathematical models

The camera model

The principal point

If the principal point is not at the origin of the imagecoordinate system, the mapping becomes

(X ,Y ,Z )T 7→ (cX/Z + px , cY /Z + py )T ,

where (px , py )T are the image coordinates for the principalpoint qp.

Z

X

Y

Q

Oq

x

y

qp

Fundamentals of Photogrammetry

Mathematical models

The camera model

The principal point (2)

In homogenous coordinatesXYZ

7→ (cX/Z + px

cY /Z + py

)

becomesXYZ1

7→cX + Zpx

cY + Zpy

Z

=

c px 0c py 0

1 0

XYZ1

Fundamentals of Photogrammetry

Mathematical models

The camera model

The camera calibration matrix

If we write

K =

c px

c py1

,

the projection may be written as

q = K(I | 0

)Q.

The matrix K is known as the camera calibration matrix.

Fundamentals of Photogrammetry

Mathematical models

The camera model

The camera position and orientation

Introduce

Q ′ =

X ′

Y ′

Z ′

1

and q = K(I | 0

)Q ′.

to describe coordinates in the camera coordinate system.The camera and world coordinate systems are identical if thecamera center is at the origin, the X and Y axes coincidewith the sensor coordinate system and the Z axes coincidewith the principal ray.

Z

X

Y

O

X ′Y ′

Z ′

Page 13: Fundamentals of Photogrammetry - faridesmFundamentals of Photogrammetry Fundamentals of Photogrammetry Niclas B orlin, Ph.D. ... computations Fundamentals of Photogrammetry History

Fundamentals of Photogrammetry

Mathematical models

The camera model

The camera position and orientation (2)

In the general case, the transformation between thecoordinate systems is usually described asX ′

Y ′

Z ′

= R

XYZ

−X0

Y0

Z0

,

where C = (X0,Y0,Z0)T is the camera center in worldcoordinates and the rotation matrix R describes the rotationfrom world coordinates to camera coordinates.

Z

X

Y

O

CX ′

Y ′

Z ′

Fundamentals of Photogrammetry

Mathematical models

The camera model

The camera position and orientation (3)

In homogenous coordinates, this transformation becomes

Q ′ =

(R 00 1

)(I −C0 1

)XYZ1

=

(R −RC0 1

)Q.

The full projection is given by

q = KR(I | − C

)Q.

The equation

q = PQ = KR(I | − C

)Q,

is sometimes referred to as the camera equation.

The 3× 4 matrix P is known as the camera matrix.

Fundamentals of Photogrammetry

Mathematical models

The camera model

The camera position and orientation (4)

If the transformation from the world to the camera is writtenas

Q ′ =

X ′

Y ′

Z ′

= R

XYZ

−X0

Y0

Z0

,

how does the transformation from the camera to the worldlook like?

Z

X

Y

O

CX ′

Y ′

Z ′

Fundamentals of Photogrammetry

Mathematical models

The camera model

Camera coordinates

What are the (Z) coordinates of points in front of the camera?

Z

X

Y

O

CX ′

Y ′

Z ′

Z

X

Y

C

Q

c

ZX

Y

Z0

X0

Y0

xy q

qp

Page 14: Fundamentals of Photogrammetry - faridesmFundamentals of Photogrammetry Fundamentals of Photogrammetry Niclas B orlin, Ph.D. ... computations Fundamentals of Photogrammetry History

Fundamentals of Photogrammetry

Mathematical models

The camera model

The collinearity equations (revisited)

Given

K =

−c xq−c yp

1

,

the camera equationxy1

= q = K R(I | − C

)Q = K R

(I | − C

)XYZ1

becomes

x = xp − cr11(X − X0) + r12(Y − Y0) + r13(Z − Z0)

r31(X − X0) + r32(Y − Y0) + r33(Z − Z0),

y = yp − cr21(X − X0) + r22(Y − Y0) + r23(Z − Z0)

r31(X − X0) + r32(Y − Y0) + r33(Z − Z0).

Fundamentals of Photogrammetry

Mathematical models

The camera model

Internal and external parameters

The camera equation

q = K R(I | − C

)Q

that describes the general projection for a pinhole camera has9 degrees of freedom: 3 in K (the elements c , px , py ), 3 in R(rotation angles) and 3 for C .The elements of K describes properties internal to the camerawhile the parameters of R and C describe the relationbetween the camera and the world.The parameters are therefore called one of

K R,C

internal parameters external parametersinternal orientation external orientationintrinsic parameters extrinsic parameterssensor model platform model

Fundamentals of Photogrammetry

Mathematical models

The camera model

Aspect ratio

If we have different scale in the x and y directions, i.e. thepixels are not square, we have to include that deformationinto the equation.Let mx and my be the number of pixels per unit in the x andy direction of the image. Then the camera calibration matrixbecomes

K =

mx

my

1

c pxc py

1

=

mxc mxpxmy c mypy

1

=

αx x0

αy y0

1

,

where αx = fmx and αy = fmy is the camera constant inpixels in the x and y directions and(x0, y0)T = (mxpx ,mypy )T is the principal point in pixels.

A camera with unknown aspect ratio has 10 degrees offreedom.

Fundamentals of Photogrammetry

Mathematical models

The camera model

Skew

For an even more general camera model we can add a skewparameter s to describe any non-orthogonality between theimage axis. Then the camera calibration matrix becomes

K =

αx s x0

αy y0

1

.

The complete 3× 4 camera matrix

P = KR(I | − C

)has 11 degrees of freedom, the same as a 3× 4 homogenousmatrix.

Page 15: Fundamentals of Photogrammetry - faridesmFundamentals of Photogrammetry Fundamentals of Photogrammetry Niclas B orlin, Ph.D. ... computations Fundamentals of Photogrammetry History

Fundamentals of Photogrammetry

Mathematical models

Rotations in <3

Rotations in <3

A rotation in <3 is usually described as a sequence of 3elementary rotations, by the so called Euler angles.

Warning: There are many different Euler angles and Eulerrotations!

Each elementary rotation takes place about a cardinal axis, x ,y , or z .

The sequence of axis determines the actual rotation.

A common example is the ω − ϕ− κ (omega-phi-kappa orx-y-z) convention that correspond to sequential rotationsabout the x , y , and z axes, respectively.

Fundamentals of Photogrammetry

Mathematical models

Rotations in <3

Elementary rotations (1)

The first elementary rotation (ω, omega) is about the x-axis.The rotation matrix is defined as

R1(ω) =

1 0 00 cosω − sinω0 sinω cosω

.

Z ′

X ′

Y ′

Z

X

Y

ω

ω

Fundamentals of Photogrammetry

Mathematical models

Rotations in <3

Elementary rotations (2)

The second elementary rotation (ϕ, phi) is about the y -axis.The rotation matrix is defined as

R2(ϕ) =

cosϕ 0 sinϕ0 1 0

− sinϕ 0 cosϕ

.

Z ′

X ′

Y ′

Z

X

Y

ϕ

ϕ

Fundamentals of Photogrammetry

Mathematical models

Rotations in <3

Elementary rotations (3)

The third elementary rotation (κ, kappa) is about the z-axis.The rotation matrix is defined as

R3(κ) =

cosκ − sinκ 0sinκ cosκ 0

0 0 1

.

Z ′

X ′

Y ′

Z

X

Y

κ

κ

Page 16: Fundamentals of Photogrammetry - faridesmFundamentals of Photogrammetry Fundamentals of Photogrammetry Niclas B orlin, Ph.D. ... computations Fundamentals of Photogrammetry History

Fundamentals of Photogrammetry

Mathematical models

Rotations in <3

Combined rotations

The axes follow the rotated object, so the second rotation isabout a once-rotated axis, the third about a twice-rotatedaxis.

A sequential rotation of 20 degrees about each of the axisis. . .

Fundamentals of Photogrammetry

Mathematical models

Rotations in <3

Combined rotations (2)

Z

X

Y

ω

ω

Z ′

X ′

Y ′. . . first a rotation about the x-axis. . .

Fundamentals of Photogrammetry

Mathematical models

Rotations in <3

Combined rotations (3)

Z ′

X ′

Y ′

ϕ

ϕ Z ′′

X ′′

Y ′′. . . followed by a rotation aboutthe once-rotated y -axis. . .

Fundamentals of Photogrammetry

Mathematical models

Rotations in <3

Combined rotations (4)

Z ′′

X ′′

Y ′′

κ

κ

Z ′′′

X ′′′

Y ′′′. . . followed by a final rotation aboutthe twice-rotated z-axis. . .

Page 17: Fundamentals of Photogrammetry - faridesmFundamentals of Photogrammetry Fundamentals of Photogrammetry Niclas B orlin, Ph.D. ... computations Fundamentals of Photogrammetry History

Fundamentals of Photogrammetry

Mathematical models

Rotations in <3

Combined rotations (5)

Z

X

Y

Z ′′′

X ′′′

Y ′′′. . . resulting in a total rotation lookinglike this.

Fundamentals of Photogrammetry

Mathematical models

Rotations in <3

Rotations in <3 (2)

The inverse rotation is about the same axes in reversesequence with angles of opposite sign.

This is sometimes called roll-pitch-yaw, where the κ angle iscalled the roll angle.

Other rotations: azimuth-tilt-swing (z-x-z), axis-and-angle,etc.

Every 3-parameter-description of a rotation has some rotationwithout a unique representation.

x-y -z if the middle rotation is 90 degrees,z-x-z if the middle rotation is 0 degrees,axis-and-angle when the rotation is zero (axis undefined).

However, the rotation is always well defined.

Fundamentals of Photogrammetry

Mathematical models

Rotations in <3

Lens distortion

A lens is designed to bend rays of light to construct a sharpimage.

A side effect is that the collinearity between incoming andoutgoing rays is destroyed.

Fundamentals of Photogrammetry

Mathematical models

Rotations in <3

Lens distortion (2)

Positive radial distortion(pin-cushion)

Negative radial distortion(barrel)

Page 18: Fundamentals of Photogrammetry - faridesmFundamentals of Photogrammetry Fundamentals of Photogrammetry Niclas B orlin, Ph.D. ... computations Fundamentals of Photogrammetry History

Fundamentals of Photogrammetry

Mathematical models

Rotations in <3

Lens distortion (3)

The effect of lens distortion is that the projected point ismoved toward or away from a point of symmetry.

The most common distortion model is due to Brown (1966,1971).

The distortion is separated into a symmetric (radial) andasymmetric (tangential) about the principal point:(

xcyc

)=

(xmym

)+

( (xryr

)+

(xtyt

) ).

corrected measured radial tangential

Warning: Someone’s positive distortion is someone else’snegative!

Fundamentals of Photogrammetry

Mathematical models

Rotations in <3

Lens distortion (4)

The radial distortion is formulated as(xryr

)= (K1r 2 + K2r 4 + . . .)

(xmym

),

for any number of coefficients (usually 1–2), where r is afunction of the distance to the principal point

r 2 = ∆x2 + ∆y 2, and

(∆x∆y

)=

(xm − xpym − yp

).

The tangential distortion is formulated as follows(xtyt

)=

(2P1∆x∆y + P2(r 2 + 2∆x2)2P2∆x∆y + P1(r 2 + 2∆y 2)

),

Fundamentals of Photogrammetry

Mathematical models

Rotations in <3

Lens distortion (5)

The radial distortion follows from that the lens bends rays oflight. It is neglectable only for large focal lengths.

Any tangential distortion is due to de-centering of the opticalaxis for the various lens components. It is neglectable exceptfor high precision measurements.

The lens distortion parameters are usually determined atcamera calibration.

The lens distortion varies with the focal length. To use acalibrated camera, the focal length (and hence any zoom)must be the same as during calibration.

Warning: Some internal parameters are strongly correlated,e.g. the tangential coefficients P1,P2 and the principal point.Any calibration including P1,P2 must have multiple images atroll angles 0 and 90 degrees.

Fundamentals of Photogrammetry

Processing

Overview

Principles

History

Mathematical models

Processing

Applications

Page 19: Fundamentals of Photogrammetry - faridesmFundamentals of Photogrammetry Fundamentals of Photogrammetry Niclas B orlin, Ph.D. ... computations Fundamentals of Photogrammetry History

Fundamentals of Photogrammetry

Processing

Processing

1 Camera calibration

2 Image acquisition

3 Measurements

4 Spatial resection

5 Forward intersection

6 (Bundle adjustment)

1 Camera calibration

2 Image acquisition

3 Measurements

4 Relative orientation

5 Forward intersection

6 (Bundle adjustment)

7 (Absolute orientation)

Fundamentals of Photogrammetry

Processing

Camera calibration

Camera calibration

Special cameras may be calibrated by measuring deviationbetween input/output rays.

Most of the time, camera calibration is performed by imaginga calibration object or scene.

A 3D scene is preferable, but may be expensive.

A 2D object is easier to manufacture and transport.

Fundamentals of Photogrammetry

Processing

Camera calibration

Camera calibration (2)

Ideally, the calibration situation should mimic the actual scene.

With a 2D object, multiple images must be taken.

Remember: use the same focal setting during calibration andimage acquisition!

If possible, include rolled images of the calibration object.

Fundamentals of Photogrammetry

Processing

Image acquisition

Image acquisition

Camera networks

Parallel (stereo)ConvergentAerialOther

Page 20: Fundamentals of Photogrammetry - faridesmFundamentals of Photogrammetry Fundamentals of Photogrammetry Niclas B orlin, Ph.D. ... computations Fundamentals of Photogrammetry History

Fundamentals of Photogrammetry

Processing

Image acquisition

Stereo images

Simplified measurements.

Simplified automation.

May be viewed in “3D”.

C1

q1

C2

q2

Q

Fundamentals of Photogrammetry

Processing

Image acquisition

Convergent networks

Stronger geometry.

More than 2measurements per objectpoint.

Should ideally surroundthe object.

C1

q1

C2

q2

C3

q3

Q

Fundamentals of Photogrammetry

Processing

Image acquisition

Aerial networks

Highly structurized.Typically around 60% overlap (along-track) and 30% sidelap(cross-track).

-� -�

Fundamentals of Photogrammetry

Processing

Measurements

Measurements

Page 21: Fundamentals of Photogrammetry - faridesmFundamentals of Photogrammetry Fundamentals of Photogrammetry Niclas B orlin, Ph.D. ... computations Fundamentals of Photogrammetry History

Fundamentals of Photogrammetry

Processing

Spatial resection

Spatial resection

Determine the external orientation C ,R of the camera frommeasurements and (ground) control points.

Direct method from 3 points — solve 4th order polynomial(Grunert 1841, Haralick 1991, 1994). May have multiplesolutions.

Q1

C

q1

Q2

q2

Q3q3

Fundamentals of Photogrammetry

Processing

Forward intersection

Forward intersection

If the camera external orientations are known, an object pointmay be estimated from measurements in (at least) twoimages.Requires at least two observations.Linear estimation, robust.

C1

q1

C2

q2

Q

Fundamentals of Photogrammetry

Processing

Forward intersection

Forward intersection (2)

From the left camera we know that

Q = C1 + (v1 − C1)α1,

for some value of α1, where v1 are the 3D coordinates of q1.

Similarly, for the right camera

Q = C2 + (v2 − C2)α2.

We have 3+3 equations and 5 un-knowns (Q, α1, α2).

In theory, the point Q is at the in-tersection of the two lines, so wedrop 1 equation and solve the re-maining 5 to get Q.

C1

q1

C2

q2

Q

Fundamentals of Photogrammetry

Processing

Forward intersection

Forward intersection (3)

C1

q1

C2

q2

Q

Q

In reality, the lines may not intersect.

In that case, we may choose to findthe point that is closest to both linesat the same time, i.e. that solves thefollowing minimization problem

minQ,α1,α2

‖Q − l1(α1)‖2 + ‖Q − l2(α2)‖2 ,

or

minQ,α1,α2

∥∥∥∥[Q − (C1 + t1α1)Q − (C2 + t2α2)

]∥∥∥∥2

,

where ti = xi − Ci .

Page 22: Fundamentals of Photogrammetry - faridesmFundamentals of Photogrammetry Fundamentals of Photogrammetry Niclas B orlin, Ph.D. ... computations Fundamentals of Photogrammetry History

Fundamentals of Photogrammetry

Processing

Forward intersection

Forward intersection (4)

This problem is linear in the unknowns and may be rewritten

minx‖[

I3 −t1 0I3 0 −t2

]︸ ︷︷ ︸

A

Qα1

α2

︸ ︷︷ ︸

x

−[

C1

C2

]︸ ︷︷ ︸

b

‖2.

The solution is given by the normal equations

ATAx = ATb.

Fundamentals of Photogrammetry

Processing

Forward intersection

Forward intersection (5)

C1

q1

C2

q2

C3

q3

Q

Given one more camera, we extend the equation system

minx‖

I3 −t1 0 0I3 0 −t2 0I3 0 0 −t3

︸ ︷︷ ︸

A

Qα1

α2

α3

︸ ︷︷ ︸

x

C1

C2

C3

︸ ︷︷ ︸

b

‖2,

with the solution againgiven by the normal equa-tions

ATAx = ATb.

Fundamentals of Photogrammetry

Processing

Forward intersection

Forward intersection (6)

Stereorestitution (“normal case”)

O1 O2X

Z

P = (PX ,PZ )

x xc

B

P1 P2

x1 x2

In photo 1: In photo 2:

X = Zx1

−cY = Z

y1

−cX = B + Z

x2

−cY = Z

y2

−c

Fundamentals of Photogrammetry

Processing

Forward intersection

Forward intersection (7)

If we have non-zero y parallax, i.e. py = y1 − y2 6= 0, we mustapproximate.

Otherwise,

−Zx1

−c= B − Z

x2

−c,

−Z =Bc

x1 − x2=

Bc

px.

Error propagation (first order)

σZ =Bc

p2x

σx =Z

c

Z

Bσx .

The ratio B/Z is the base/object distance.

The ratio Z/c is the scale factor.

Page 23: Fundamentals of Photogrammetry - faridesmFundamentals of Photogrammetry Fundamentals of Photogrammetry Niclas B orlin, Ph.D. ... computations Fundamentals of Photogrammetry History

Fundamentals of Photogrammetry

Processing

Relative orientation

Relative orientation

C1

p1

p2

p3

p4

p5

C2

q1

q2

q3

q4

q5

One camera fixed, determineposition and orientation ofsecond camera.

Need 5 point pairs measuredin both images.

No 3D information isnecessary.

Direct method (Nister2004). Solve 10th orderpolynomial. May havemultiple solutions.

Fundamentals of Photogrammetry

Processing

Absolute orientation

Absolute orientation

A 3D similarity transformation.

7 degrees of freedom (3 translations, 3 rotations, 1 scale).

Direct method based on singular value decomposition (Arun1987) for isotropic errors.

Fundamentals of Photogrammetry

Processing

Bundle adjustment

Bundle adjustment

Simultaneous estimation of camera external orientation andobject points.

Iterative method, needs initial values.

May diverge.

Fundamentals of Photogrammetry

Applications

Applications

Architecture

Forensics

Maps

Industrial

Motion analysis

Movie industry

Orthopaedics

Space science

Microscopy

GIS

Page 24: Fundamentals of Photogrammetry - faridesmFundamentals of Photogrammetry Fundamentals of Photogrammetry Niclas B orlin, Ph.D. ... computations Fundamentals of Photogrammetry History

Fundamentals of Photogrammetry

Epipolar geometry

Epipolar geometry

Let Q be an object point and q1 and q2 its projections in twoimages through the camera centers C1 and C2.

The point Q, the camera centers C1 and C2 and the (3Dpoints corresponding to) the projected points q1 and q2 willlie in the same plane.

This plane is called the epipolar plane for C1, C2 and Q.

C1

q1

C2

q2

Q

Fundamentals of Photogrammetry

Epipolar geometry

Epipolar lines

Given a point q1 in image 1, the epipolar plane is defined bythe ray through q1 and C1 and the baseline through C1 andC2.A corresponding point q2 thus has to lie on the intersectingline l2 between the epipolar plane and image plane 2.The line l2 is the projection of the ray through q1 and C1 inimage 2 and is called the epipolar line to q1.

q1q2

e1 e2C1

C2

Q

Fundamentals of Photogrammetry

Epipolar geometry

Epipoles

The intersection points between the base line and the imageplanes are called epipoles.

The epipole e2 in image 2 is the mapping of the cameracenter C1.

The epipole e1 in image 1 is the mapping of the cameracenter C2.

q1q2

e1 e2C1

C2

Q

Fundamentals of Photogrammetry

Epipolar geometry

Examples

Page 25: Fundamentals of Photogrammetry - faridesmFundamentals of Photogrammetry Fundamentals of Photogrammetry Niclas B orlin, Ph.D. ... computations Fundamentals of Photogrammetry History

Fundamentals of Photogrammetry

Epipolar geometry

RANSAC

Robust estimation — RANSAC

The Random Sample Consensus (RANSAC) algorithm (Fishlerand Bolles, 1981) is an algorithm for handling observationswith large errors (outliers).

Given a model and a data set S containing outliers:

Pick randomly s data points from the set S and calculate themodel from these points. For a line, pick 2 points.Determine the consensus set Si of s, i.e. the set of pointsbeing within t units from the model. The set Si define theinliers in S .If the number of inliers are larger than a threshold T ,recalculate the model based on all points in Si and terminate.Otherwise repeat with a new random subset.After N tries, choose the largest consensus set Si , recalculatethe model based on all points in Si and terminate.

Fundamentals of Photogrammetry

Epipolar geometry

RANSAC

b

da

c

C DB

A

C DB

A