ee 576 - camera & image...

Outline

Image Acquisition

Calibration

EE 576 - Camera & Image Formation

H.I. Bozma

Electric Electronic Engineering

Bogazici University

February 17, 2020

H.I. Bozma EE 576 - Camera & Image Formation

Outline

Image Acquisition

Calibration

Image AcquisitionIntroductionExtrinsic Parameters

CalibrationCalibrationLeast Squares ApproachLagrangian ApproachSolving for Camera ParametersOrthographic Projection


Outline

Image Acquisition

Calibration

Introduction

Extrinsic Parameters

Camera models

◮ Goal: To understand the image acquisition process.

◮ Function of the camera Similar to that of the eye inbiological systems.


Outline

Image Acquisition

Calibration

Introduction


Pinhole camera model

◮ A pinhole Infinitesimally small hole through which lightenters before forming an inverted image on the camerasurface facing the hole

◮ A pinhole camera Modeled by placing the image planebetween the focal point of the camera and the object, so thatthe image is not inverted.

◮ Perspective projection This mapping of three dimensionsonto two

◮ Perspective geometry Fundamental to image formation.


Outline

Image Acquisition

Calibration

Introduction


Pinhole camera model: Perspective projection


Outline

Image Acquisition

Calibration

Introduction


Perspective geometry

◮ Perspective geometry → Mathematical modeling of the visualcomputational processes possible

◮ Perspective projection: projection of a 3D object onto a 2Dsurface by straight lines that pass through a single point.

◮ Simple geometry◮ [X1;X2;X3]

T- object coordinates

◮ f - the distance of the image plane to the center of projection◮ x = [x1 x2]

T- image coordinates

x1 =f

X3

X1 x2 =f

X3

X2

◮ Non-linear equations.


Outline

Image Acquisition

Calibration

Introduction


Perspective geometry

◮ Linear by introducing homogeneous transformations as:

U

V

S

=

f 0 0 00 f 0 00 0 1 0

X1

X2

X3

1

◮ x1 =USand x2 =

VS

if S 6= 0.

◮ In general, a point in Rn → [Sx , S ]T ∈ Rn+1, where S 6= 0.


Outline

Image Acquisition

Calibration

Introduction


Simple Lens Model

◮ Use lenses to focus an image onto the camera’s focal plane.

◮ Incoming light parallel to optical axis - Focused on focal point

◮ Limitation – Only bring into focus those objects that lie onone particular plane that is parallel to the image plane.


Outline

Image Acquisition

Calibration

Introduction


Simple Lens Model(cont)

◮ Assuming◮ Lens relatively thin◮ optical axis perpendicular to the image plane,

◮ Lens law: Image formation

1

x3+

1

X3

=1

f

◮ X3 - Distance of an object point from the plane of the lens,◮ x3 Distance of the focused image from this plane,◮ f - Focal length of the lens


Outline

Image Acquisition

Calibration

Introduction


Simple Lens Model - Blurring and depth of field


Outline

Image Acquisition

Calibration

Introduction


Simple Lens Model(cont)

◮ Ideal lens Brings focus light from points at distance X3.◮ Points at other distances imaged as little circles◮ If objects at distance X3 are correctly focused, an object at

distance X ′

3.

x ′3 − x3 =f 2(X ′

3 − X3)

(X ′

3 − f )(X3 − f )

◮ The blurring will be with circles whose radius is

∝d

x3

∣∣x ′3 − x3∣∣

∝d

x3

∣∣X ′

3 − X3

∣∣

◮ The depth of field = The range of distances |X ′

3 − X3| whichthe objects are focused sufficiently well◮ |x ′3 − x3| < Camera resolution.H.I. Bozma EE 576 - Camera & Image Formation

Outline

Image Acquisition

Calibration

Introduction


Lens Selection - Focal Length

Amount of subject matter that can be viewed relative to thesubject distance.

◮ Length: Short (Wide angle), normal, long (telephoto)

◮ Zoom - Fixed, variable


Outline

Image Acquisition

Calibration

Introduction


Selection of Lens

◮ Aperture: This refers to the size of the opening in the lensfor light (iris).◮ Depth-of-field (the amount of scene in sharp focus): As

aperture ↓, depth-of-field ↑◮ f-stop: The focal length divided by the ”effective” aperture

diameter. As f-stop ↓, light ↑. The amount of lighttransmitted to the film (or sensor) decreases with the f-numbersquared. Doubling the f-number increases the necessaryexposure time by a factor of four.

◮ Speed: Fast (f1.2, f1.4) =⇒ Slow (f3.5)


Outline

Image Acquisition

Calibration

Introduction


Selection of Camera

◮ Format: This refers to the size of the camera pickup device(12,23, 1 inch). As the format decreases, the pickup becomes

darker.

◮ Mount type: C-mount standard

◮ Installation (size & weight)

◮ Vibration (Locking screws)


Outline

Image Acquisition

Calibration

Introduction


Coordinate Systems

◮ Imagine we have a three dimensional coordinate system◮ Origin at the center of projection (also called the optical

center)◮ X3 axis is along the optical axis.

◮ This coordinate system is called the standard coordinatesystem of the camera.

◮ These coordinates are with respect to a coordinate systemwhere◮ The origin is at the intersection of the optical axis and the

image plane,◮ x1 and x2 axes are parallel to the X1 and X2 axes.


Outline

Image Acquisition

Calibration

Introduction


World Image Coordinate Transformations

Figure: Coordinate transformations between world and image coordinatesystems H.I. Bozma EE 576 - Camera & Image Formation

Outline

Image Acquisition

Calibration

Introduction


Image Formation(cont.)

◮ This can be written in homogeneous coordinates as:

∣∣∣∣∣∣

sx1sx2s

∣∣∣∣∣∣=

∣∣∣∣∣∣

f 0 0 00 f 0 00 0 1 0

∣∣∣∣∣∣

∣∣∣∣∣∣∣∣

X1

X2

X3

1

∣∣∣∣∣∣∣∣

◮ s 6= 0 and f is the effective focal length1 of the camera and isoften measured in pixel units.

◮ Not the same as the focal length (e.g., a 9mm lens) markedon the lens of the camera.


Outline

Image Acquisition

Calibration

Introduction



◮ Now, the actual pixel coordinates u = [u1, u2]T are defined

with respect to an origin in the top left hand corner of theimage plane, and will satisfy

u1 = o1 + x1

u2 = o2 + x2

◮ Express the transformation from three dimensional worldcoordinates to image pixel coordinates using a 3× 4 matrix.

X3u1 = X3o1 + X3x1 = X3o1 + X1f

X3u2 = X3o2 + X3x2 = X3o2 + X2f


Outline

Image Acquisition

Calibration

Introduction



◮ Substituting

X3u1 = X3o1 + X1f

X3u2 = X3o2 + X2f

∣∣∣∣∣∣

su1su2s

∣∣∣∣∣∣=

∣∣∣∣∣∣

f 0 o1 00 f o2 00 0 1 0

∣∣∣∣∣∣

∣∣∣∣∣∣∣∣

X1

X2

X3

1

∣∣∣∣∣∣∣∣

where the scaling factor s has value X3.


Outline

Image Acquisition

Calibration

Introduction



◮ In short hand notation, we write this as

u = KX

where◮ u is the homogeneous vector of image pixel coordinates◮ K is the perspective projection matrix◮ X is the homogeneous vector of world coordinates.


Outline

Image Acquisition

Calibration

Introduction


Intrinsic Camera Parameters

There are four intrinsic camera parameters:

◮ Two are for the position of the origin of the image coordinateframe, and

◮ Two are for the scale factors of the axes of this frame.

◮ A camera → A system that performs a linear projectivetransformation from the projective space that is subset ofR3 × S3 into the projective plane X .

◮ There are three camera parameters – :

1. The focal length f,2. o1 and o2

◮ Intrinsic parameters: Do not depend on position andorientation of the camera in space


Outline

Image Acquisition

Calibration

Introduction


Aspect Ratio

◮ Some old-fashioned CCD cameras non-square pixels.◮ Aspect ratio 6= 1 → Different scalings in the u and v-axes

(e.g., a sphere would appear as an ellipse in the image).◮ Two terms f1 and f2 : To describe the effective focal length.

◮ The term f1 - The effective focal length in the u pixel units◮ f2 - The effective focal length in the v pixel units

As all modern cameras have unit aspect ratio, assumef1 = f2 = f .


Outline

Image Acquisition

Calibration

Introduction


Imaging Distortions

◮ Radial distortion: Distortion as a function of r (Fish-eyeefffect)

xc = x(1 + k1r2 + k2r

4 + k3r6)

yc = y(1 + k1r2 + k2r

4 + k3r6)

◮ Tangential distortion: Lenses not perfectly parallel to theimaging plane.

xc = x + (2p1xy + p2(r2 + 2x2))

yc = y + (p1(r2 + 2y2) + 2p2xy)

◮ Distortion coefficients: [k1k2p1p2k3]


Outline

Image Acquisition

Calibration

Introduction


Extrinsic Camera Parameters

There are six extrinsic camera parameters:

◮ Three are for the position of the center of projection,

◮ Three are for the orientation of the image plane coordinateframe.


Outline

Image Acquisition

Calibration

Introduction


Coordinate Transformations

◮ In general, 3D world coordinates of a point – Not specified inframe with origin at the center of projection and X3 axis liesalong the optical axis.

◮ A more convenient frame – more likely be specified,


Outline

Image Acquisition

Calibration

Introduction


Coordinate Transformations - 2

◮ Consider the transformation of coordinate system from thisother frame to the standard coordinate system

u = KTX

where T is a 4 × 4 homogeneous transformation matrix.

T =

∣∣∣∣R t

0T3 1

∣∣∣∣

1. The top 3 × 3 part - Rotation matrix R and encodes cameraorientation

2. The final column - Homogeneous vector containing t

corresponding camera translation from the world frame origin


Outline

Image Acquisition

Calibration

Introduction



◮ 6 degrees of freedom - Extrinsic camera parameters.

◮ the camera calibration matrix C : Combine 3 × 4 matrix K

and 4 × 4 matrix T → 3 × 4 as:

C =

∣∣∣∣∣∣

fr1 + o1r3 ft1 + o1t3fr2 + o2r3 ft2 + o2t3

r3 t3

∣∣∣∣∣∣

R =

∣∣∣∣∣∣

r1r2r3

∣∣∣∣∣∣t =

∣∣∣∣∣∣

t1t2t3

∣∣∣∣∣∣

◮ The vectors ri - Row vectors of the rotation matrix

◮ The matrix C, like the matrix K, – Rank three.


Outline

Image Acquisition

Calibration

Calibration

Least Squares Approach

Lagrangian Approach

Solving for Camera Parameters

Orthographic Projection

Camera Parameters

◮ Camera calibration◮ In order to deduce three-dimensional geometric information

from an image → The parameters that relate the position of apoint in a scene to its position in the image.

◮ Estimating the intrinsic and extrinsic parameters of a camera.


Outline

Image Acquisition

Calibration

Calibration


Lagrangian Approach



Calibration Procedure

◮ Calibration - Estimating the intrinsic and extrinsic parametersof the camera.

◮ Calibration Problem: Asume that we are given N pointsX j ∈ R3 as well as their corresponding x j , j = 1, · · · ,N.

◮ A two stage process:

1. Estimating the matrix C,2. Estimating the intrinsic and extrinsic parameters from C.

◮ In many cases, the second stage is not necessary.

◮ Solution: Linear and non-linear methods


Outline

Image Acquisition

Calibration

Calibration


Lagrangian Approach



Solving for Calibration Matrix

The approaches vary in the numerical methods used to solve thesetup problem.

◮ Least Squares Approach ⇒ Matrix inverse finding & matrixalgebra

◮ Lagrangian Formulation ⇒ Multi-dimensional optimization


Outline

Image Acquisition

Calibration

Calibration


Lagrangian Approach



General Formulation

Transforming the transformation equations into a linear formIn homogeneous coordinates, the relationship between the imagepoints with coordinates x j and the 3D reference point coordinatesX j

∣∣∣∣∣∣

s juj1

s ju2s j

∣∣∣∣∣∣=

∣∣∣∣∣∣

q11 q12 q13 q14q21 q22 q23 q24q31 q32 q33 q34

∣∣∣∣∣∣

∣∣∣∣∣∣∣∣∣

Xj1

Xj2

Xj3

1

∣∣∣∣∣∣∣∣∣→ s j

∣∣∣∣∣∣

uj1

uj2

1

∣∣∣∣∣∣=

∣∣∣∣∣∣

qT1 q14qT2 q24qT3 q34

∣∣∣∣∣∣X j

where qTk = [qk1qk2qk3].


Outline

Image Acquisition

Calibration

Calibration


Lagrangian Approach



General Formulation

Set q34 = 1.Substituting for s = X

j1q31 + X

j2q32 + X

j3q33 + 1,

(X

j1q31 + X

j2q32 + X

j3q33 + 1

)uj1 = X

j1q11 + X

j2q12 + X

j3q13 + q14

(X

j1q31 + X

j2q32 + X

j3q33 + 1

)uj2 = X

j1q21 + X

j2q22 + X

j3q23 + q24

Define x ∈ R2N as x =∑

e j ⊗ x j .Let q ∈ R11 be the concatenated vectorqT = [q11q12q13q14q21q22q23q24q31q32q33]

Letting D to be the large matrix consisting of known image andworld coordinates with dimensions 2N × 11d ∈ R2N as dT =

[u11u

12 · · · u

N1 u

N2

]


Outline

Image Acquisition

Calibration

Calibration


Lagrangian Approach



General Formulation

Dq = d

D =

∣∣∣∣∣∣∣∣∣∣∣

X 11 X 1

2 X 13 1 0 0 0 0 −u11X

11 −u11X

12 −u11X

13

0 0 0 0 X 11 X 1

2 X 13 1 −u12X

11 −u12X

12 −u12X

13

......

......

......

......

......

...XN1 XN

2 XN3 1 0 0 0 0 −uN1 X

N1 −uN1 X

N2 −uN1 X

N3

0 0 0 0 XN1 XN

2 XN3 1 −uN2 X

N1 −uN2 X

N2 −uN2 X

N3

∣∣∣∣∣∣∣∣∣∣∣


Outline

Image Acquisition

Calibration

Calibration


Lagrangian Approach



Least Squares Problem

‖Dq − d‖2 = (Dq − d)T (Dq − d)

Our objective:minq (Dq − d)T (Dq − d)

Differentiating wrt q and setting to 0 gives

DT (Dq − d) = 0 (1)

=⇒ DTDq = DTd (2)

=⇒ q =(DTD

)−1

DTd (3)

(DTD

)−1

DT is known as the pseudo-inverse of D.


Outline

Image Acquisition

Calibration

Calibration


Lagrangian Approach



Least Squares Problem (cont.)

◮ q can only be estimated if DTD is invertible.

◮ With D being a 2N × 11 matrix → DTD must be of full rank.

◮ At least 11 equations which means that N = 6.

◮ Furthermore, the rank of D ≥ 11.

◮ The reference points {Xi |1 = i = N,N = 6} must not lie in acertain configuration, which can be defined mathematically.

◮ If six or more points are chosen at random, and do not lie ona plane, then → This situation will not occur


Outline

Image Acquisition

Calibration

Calibration


Lagrangian Approach



Lagrangian Formulation

D ′q′ = 0

D – 2N × 12 matrix D = [D − d ]q′ = [q q34] – concatenated vector with unknown q34.Problem Formulation: Minimizing (D ′q′)T (D ′q′) subject to theconstraint that

q′Tq′ − 1 = 0

Let γ be a Lagrange multiplier. The Lagrangian to be minimized:

(D ′q′)T (D ′q′) + γ(q′Tq′ − 1)


Outline

Image Acquisition

Calibration

Calibration


Lagrangian Approach



Lagrangian Formulation (cont.)

Taking gradient wrt q′ ,

D ′TD ′q′ − γq′ = 0

q′Tq′ = 1

With some mathematical manipulation

q′TD ′TD ′q′ = γq′Tq′

= γ

Minimize (D ′q′)T (D ′q′) → Minimize γ.γ is an eigenvalue with corresponding q′.Conclude that q′ should be the eigenvector that has the smallesteigenvalue of D ′TD ′.


Outline

Image Acquisition

Calibration

Calibration


Lagrangian Approach



Lagrangian Formulation (cont.)

From matrix algebra, rank(D ′TD ′) + null(D ′TD ′) = m m = 12.In our case, N > m – hence the following three cases:

1. rank(D ′TD ′) = 12 =⇒ One solution to the system

2. rank(D ′TD ′) = 11 There is a unique solution up to a scalefactor.

3. rank(D ′TD ′) < 11 Infinite solutions

In real applications, rank(D ′) = 12 as noise affects the rank of thematrix.The smallest eigenvalue of D ′TD ′ 6= 0, but a small positivenumber.The amount of noise ≈ Ratio of smallest and largest eigenvaluesD ′TD ′.


Outline

Image Acquisition

Calibration

Calibration


Lagrangian Approach




C – Defined up to a unknown scale factor s, so that ‖q3‖ 6= 1First recall that

C =

∣∣∣∣∣∣

fr1 + o1r3 ft1 + o1t3fr2 + o2r3 ft2 + o2t3

r3 t3

∣∣∣∣∣∣= s

∣∣∣∣∣∣

qT1 q14qT2 q24qT3 q34

∣∣∣∣∣∣


Outline

Image Acquisition

Calibration

Calibration


Lagrangian Approach



Solving for Camera Parameters (cont.)

◮ r3 – A row of the rotation matrix R → ‖sq3‖ = ‖r3‖ = 1 →

s = ±1

‖q3‖2 solutions

◮ r3 = sq3 (2 solutions)

◮

t3 = sq34 = ±q34

‖q3‖2 solutions


Outline

Image Acquisition

Calibration

Calibration


Lagrangian Approach




s2qT1 q3 = (sqT1 )(sq3) = (fr1+o1r3)T r3 = o1r

T3 r3 = o1 =⇒ o1 = s2qT1 q3

Similarly

s2qT2 q3 = (sqT2 )(sq3) = (fr2+o2r3)T r3 = o2r

T3 r3 = o2 =⇒ o2 = s2qT2 q3


Outline

Image Acquisition

Calibration

Calibration


Lagrangian Approach




Consider s2q1 × q3 and first note that ‖r1 × r3‖ = 1

s2q1×q3 = (sq1)×(sq3) = (fr1+o1r3)×r3 = fr1×r3 =⇒ f = ±s2 ‖q1 × q3‖

Also noting that ‖r2 × r3‖ = 1

s2q2×q3 = (sq2)×(sq3) = (fr2+o1r3)×r3 = fr2×r3 =⇒ f = ±s2 ‖q2 × q3‖

These two values must be identical ideally. If not, reconsider theaccuracy of the data and the calibration computation procedure


Outline

Image Acquisition

Calibration

Calibration


Lagrangian Approach




r1 =s

f(q1 − o1q3)

r2 =s

f(q2 − o2q3)

t1 =1

f(sq14 − o1t3)

t2 =1

f(sq24 − o2t3)

Also to be noted:

(q1 × q3)T (q2 × q3) = ((fr1 + o1r3)× rT3 )((fr2 + o2r3)× r3

= (fr1 × r3)T (fr2 × r3)

= 0

where × – The exterior product or wedge product of vectors thatH.I. Bozma EE 576 - Camera & Image Formation

Outline

Image Acquisition

Calibration

Calibration


Lagrangian Approach




In regards to the found parameters,

◮ Due to the ± in the computations of s and f, four sets ofsolutions exist.

◮ Each correspond to whether the origin of the coordinates is infront of or behind the camera and the choice of the opticalaxis.

◮ Two solutions can be eliminated if a right-hand coordinatesystem is adapted.


Outline

Image Acquisition

Calibration

Calibration


Lagrangian Approach




Consider translation of f along the X3-axis of standard coordinateframe,

◮ the origin and the center of the image plane - coincident

◮ The focal point - Positioned at X3 = −f .

◮ No rotation involved in this transformation, hence →

C =

∣∣∣∣∣∣

f 0 0 00 f 0 00 0 1 f

∣∣∣∣∣∣


Outline

Image Acquisition

Calibration

Calibration


Lagrangian Approach




Consider translation of f along the X3-axis of standard coordinateframe,

◮ Assuming that the pixel width and height are both 1.

C =

∣∣∣∣∣∣

1 0 0 00 1 0 00 0 1

f1

∣∣∣∣∣∣Letting f → ∞ C =

∣∣∣∣∣∣

1 0 0 00 1 0 00 0 0 1

∣∣∣∣∣∣

◮ Orthographic projection parallel to X3 axis → x1 = X1 andx2 = X2


Outline

Image Acquisition

Calibration

Calibration


Lagrangian Approach



D. G. Lowe, “Object recognition from local scale-invariantfeatures,” in ICCV, 1999, pp. 1150–1157.

J. Sivic and A. Zisserman, “Efficient visual search of videoscast as text retrieval,” Pattern Analysis and Machine

Intelligence, IEEE Transactions on, vol. 31, no. 4, pp.591–606, April 2009.

A. Torralba, K. P. Murphy, W. T. Freeman, and M. A. Rubin,“Context-based vision system for place and objectrecognition,” Computer Vision, IEEE Int. Conf. on, vol. 1, p.273, 2003.

O. Erkent and H. I. Bozma, “Bubble Space and PlaceRepresentation in Topological Maps,” The Int. J. of Rob. Res.,vol. 32, no. 6, pp. 671 – 688, 2013.


ee 576 - camera & image...

Documents