understanding the image formation and the camera...

Understanding Understanding

the image formation the image formation

andand

the camera modelthe camera model

The geometric model

line

100,100,100

200,200,100

color

0,0,0

0,255,0

Extensão: DWG, DXF, METAFILE

Understanding the image formation (1)

The geometric model

Plan

Elevation northElevation east

Elevation west


The geometric aspect

Axonometric view Perspective view


The photometric (radiometric) aspect

Material and lightning effects

Factors in Image Formation

• Geometry concerned with the relationship between points in the three

dimensional world and their two dimensional images

• Radiometry concerned with the relationship between the amount of

light radiating from a surface and the amount of light incident at its

image

• Photometry concerned with ways of measuring the intensity of light

• Digitization – concerned with ways of converting continuous signals

(in both space and time) to digital representations

Positive converging lens

Thin Lens Equation

fSS

111

21

=+1

2

S

S

h

hM

o

i −==

oh

ih

The Pin Hole Camera

(Forsyth & Ponce)

v

y

P(x,y,x)

f zv

x

v

)d,py,px(P′

Projection plane

Projectioncenter

f

)d,py,px(P −−−′

f

x

z

x p=

fz

x

z

x.fxp ==

The equations of perspective

projection

The Perspective equations and

coordinate systems

Perspective are obtained considering :

• the world and camera coordinate systems are

coincident

• Y-axis aligned with y-axis of camera

• X-axis aligned with x-axis of camera

• Z-axis along central projection ray

In general case coordinate systems are not coincident.

Imaging geometry – coordinate

systems

Geometric transforms are

necessary to transform between

reference systems

• Why not always use the

camera reference frame as the

world reference frame?

– The camera might be moving

– We could have several

cameras at different locations.

Geometric Transform

x

z

y

∃z P

P’

ΘΘ

Θ−Θ

=

Z

Y

X

100

0zcoszsin

0zsinzcos

'Z

'Y

'X

Homogeneous Coordinates

Homogeneous coordinate systems allows to transform between reference

frames with a single matrix multiplication.

• In 3D space a point is written

=

w

wz

wy

wx

ph

Usually w=1 and the homogeneous coordinates are obtained appending

a 1 on the end of each set of coordinates

Cartesian Coordinates Homogeneous Coordinates

=

z

y

x

pc

Basic geometric transformations

=

1

z

y

x

M

W

Z

Y

X

A geometric transformation is represented

by the equation

T d d d

d

d

dx y z

x

y

z

( , , ) =

1 0 0

0 1 0

0 0 1

0 0 0 1

S s s s

s

s

sx y z

x

y

z

( , , ) =

0 0 0

0 0 0

0 0 0

0 0 0 1

Translation Scaling

Rotation transform

x

z

y

θθ

θ−θ

=θ

1000

0100

00cossen

00sencos

)(R z

−=

1000

0cossen0

0sencos0

0001

)(R xαα

ααα

ββ−

ββ

=β

1000

0cos0sen

0010

0sen0cos

)(R y

The rotation angle is

measured counterclockwise

Concatenation of transformations

x

y

xd

yd

x

y

xd

yd

x

y

xd

yd

x

y

xd

yd

z z z

T Tr xd yd Rz Tr xd yd= − −( ) ( ) ( , , )0 180 0

Transformation between

reference systems

)0,ay,ax(wP =

Zw

Yw

xa

ya

P

y

Yc

Xw

yc

xc

Xc

Zc

)0,cyay,cxax(cP −−=

wP)0,cy,cx(TcP −−=

−

−

=

−

−

1

0

Ya

Xa

1000

0100

Yc010

Xc001

1

0

YcYa

XcXa

Transformation between

reference systems

)0,ay,ax(wP =

)0),cxax(),cyay((cP −−−−=

wP)0,cy,cx(T)180(zRcP −−°=

Zw

Yw

xa

ya

P

y

Xw

yc

xc

Xc

Zc

Xc

−

−

°°

°−°

1000

0100

Yc010

Xc001

1000

0100

00180cos180sen

00180sen180cos

The Perspective transformation

matrix (1)

P(x,y,x)

fz

v

xv

yv

′P xp yp f( , , )

Projection plane

Projection

center

f

x

z

x p=

fx

x

z

x.fxp ==

=

f

fz

y

fz

x

pz

py

px

=

=

1

z

y

x

0f

100

0100

0010

0001

f

z

y

x

=

0f100

100

0010

0001

M0

Origin of camera coordinate systems at the center of projection

The perspective transformation

matrix (2)

P(x,y,x)

fz

v

xv

yv

′P xp yp d( , , )

Projection plane

Projection center

−

=

1f100

100

0010

0001

M0

f

px

fz

x=

−−

−=−=

fz1

x

f-z

f.xpx

Origin of camera coordinate systems at the image plane

The geometry of image formation

=

1

Z

Y

X

matrix

tiontransforma

systemscoordinate

cameratoworld

matrix

projection

C

C

C

C

w

w

w

4h

3h

2h

1h

4h2hc

p

4h1hc

p

C/CY

C/CX

=

=

Homogeneous

coordinates in world

reference system

Homogeneous

coordinates in camera

reference system

Cartesian coordinates

of the projected point

in camera reference

system

The camera model (1)

=

1

Z

Y

X

1000

trrr

trrr

trrr

0f

100

0100

0010

0001

C

C

C

C

w

w

w

z333231

y232221

x131211

4h

3h

2h

1h

=

1

Z

Y

X

ft

fr

fr

fr

trrr

trrr

trrr

C

C

C

C

w

w

w

z333231

z333231

y232221

x131211

4h

3h

2h

1h

4h2hc

p

4h1hc

p

C/CY

C/CX

=

=


4h2hc

p

4h1hc

p

C/CY

C/CX

=

=

≈

1

Z

Y

X

ft

fr

fr

fr

trrr

trrr

C

C

C

w

w

w

z333231

y232221

x131211

4h

2h

1h

Ch3 can be disregarded for image purposes

=

1

Z

Y

X

ft

fr

fr

fr

trrr

trrr

trrr

C

C

C

C

w

w

w

z333231

z333231

y232221

x131211

4h

3h

2h

1h

Matrix (3x4) Camera Model


=

1

Z

Y

X

ft

fr

fr

fr

trrr

trrr

C

C

C

w

w

w

z333231

y232221

x131211

4h

2h

1h

Extrinsic

camera

parameters

=

1

Z

Y

X

trrr

trrr

trrr

f100

010

001

C

C

C

w

w

w

z333231

y232221

x131211

4h

2h

1h

Intrinsic

camera

parameter

Zc

XcYc

Optical

axis

P

Xim

Yim

Image plane

Oc

Oi

Ox

Oy

Sx

Sy

The image reference system

PrincipalPoint (Ox,Oy)

OySy

YY

OxSx

XX

c

pim

p

c

pim

p

−=

+=


≈

1

Z

Y

X

trrr

trrr

trrr

f100

010

001

C

C

C

w

w

w

z333231

y232221

x131211

4h

2h

1h

Extrinsic

camera

parameters

Intrinsic

camera

parameters

≈

1

Z

Y

X

trrr

trrr

trrr

f100

010

001

100

0Sy

10

00Sx

1

000

Oy10

Ox01

C

C

C

w

w

w

z333231

y232221

x131211

4h

2h

1h

Metric coord.Metric

coord.

Pixel

coord.


≈

1

Z

Y

X

trrr

trrr

trrr

f100

010

001

100

0Sy

10

00Sx

1

000

Oy10

Ox01

C

C

C

w

w

w

z333231

y232221

x131211

4h

2h

1h

Metric

coord.

Extrinsic

camera

parameters

Intrinsic

camera

parameters

Pixel

coord.

≈

1

Z

Y

X

trrr

trrr

trrr

f100

OySy

10

Ox0Sx

1

C

C

C

w

w

w

z333231

y232221

x131211

4h

2h

1h

The skew angle of the image

coordinate axes (1)

Xc γ

YcP

Xa

Ya

P

The skew factor allows for a projective camera (as

opposed to a perspective camera) where the X and Y

axes may not be orthogonal, and/or the XY plane may

not be orthogonal to Z

The skew angle of the image

coordinate axes (2)

Xc

)tan(YXX

YY

c

p

c

p

a

p

c

p

a

p

γ−=

=

=

c

c

c

a

a

a

Z

Y

X

100

010

0tan1

Z

Y

X γ

γ

YcP

Xa

Ya

P


γ

f100

010

001

100

0Sy

10

00Sx

1

100

010

0)tan(1

100

Oy10

Ox01

Metric

coord.

Extrinsic

camera

parameters

Intrinsic

camera

parameters

Pixel

coord.

γ

=

1

Z

Y

X

trrr

trrr

trrr

f100

OySy

10

OxSy

)tan(Sx

1

C

C

C

w

w

w

z333231

y232221

x131211

4h

2h

1h

The camera model

Metric

coord.

Extrinsic

camera

parameters

Intrinsic

camera

parameters

Pixel

coord.

γ

=

1

Z

Y

X

trrr

trrr

trrr

f100

OySy

10

OxSx

)tan(Sx

1

C

C

C

w

w

w

z333231

y232221

x131211

4h

2h

1h

γ

=

1

Z

Y

X

trrr

trrr

trrr

100

OySy

f0

OxSx

)tan(Sx

f

f

1

C

C

C

w

w

w

z333231

y232221

x131211

4h

2h

1h

Chromatic Aberration

Spherical Aberration

Radial distortion

In several cases, optics introduces image distortions

that become evident at the periphery of the image

Modeling radial distortions

+ Pillow - Barrel

2

d

2

d

2

4

2

2

1d

4

2

2

1d

yxr

)rkrk1(yy

)rkrk1(xx

+=

++=

++=

tscoefficiendistortionKeK

valuetrueX

valuemeasuredX

21

d

⇒

⇒

⇒

Camera CalibrationCamera Calibration

and and

Pose estimationPose estimation

Camera Pose Estimation

• What is Camera Pose Estimation?

Camera pose estimation is the problem of determining the

position and orientation of an internally calibrated camera

from known 3D reference points and their images

• Pose estimation is important when we need to

Reconstruct a world model

Interact with the world (Robot, hand-eye coordination, etc.)

• Variety of Techniques: Basically the determination of

extrinsic camera parameters

Camera calibration

• What is Camera Calibration?

Primarily, finding the quantities internal to the camera that

affect the imaging process

• calibration is important when we need to

Reconstruct a world model

Interact with the world (Robot, hand-eye coordination,

etc.)

• Variety of Techniques: Basically the determination of

intrinsic camera parameters

Methodology

• The information available for camera pose estimation or

camera calibration is usually given in the form of point

correspondences between 3-D points, on the object or in

the scene, and their 2-D projections on the image plane of

the camera.

• Obtain equations that describe imaging formation based

on the camera model parameters.

The camera model

Metric

coord.

Extrinsic

camera

parameters

Intrinsic

camera

parameters

Pixel

coord.

γ

=

1

Z

Y

X

trrr

trrr

trrr

f100

OySy

10

OxSx

)tan(Sx

1

C

C

C

w

w

w

z333231

y232221

x131211

4h

2h

1h

γ

=

1

Z

Y

X

trrr

trrr

trrr

100

OySy

f0

OxSx

)tan(Sx

f

f

1

C

C

C

w

w

w

z333231

y232221

x131211

4h

2h

1h

The camera intrinsic parameters

γ

≅

1

Z

Y

X

trrr

trrr

trrr

100

OySy

f0

OxSx

)tan(Sx

f

C

C

C

w

w

w

z333231

y232221

x131211

4h

2h

1h

100

OySy

f0

Ox0Sx

fNeglecting the skew

angle the matrix of

intrinsic parameters

reduces to

equal up to

a scale factor

The camera intrinsic parameters

100

OySy

f0

Ox0Sx

f•Five intrinsic parameters

•parameters are not independent

Defining: Sxffx = Sx

Syfy

fx ==α,

100

Oyf0

Ox0f

y

x

Syffy =

•Four independent parameters

•fx and fy scaling in x and y directions.

•fx- focal length in horizontal pixels

•α – aspect ratio

e

The camera extrinsic parameters

z333231

y232221

x131211

trrr

trrr

trrr

A camera has only 6 extrinsic independent

parameters (3 translation and 3 rotation) describing

the location and orientation of the camera with

respect to the world coordinate frame.

•Twelve parameters

•parameters are not independent

Camera and Calibration Target

• Calibration target: 2 planes at right angle with checkerboard patterns (Tsai grid)

• Positions of pattern corners are known only with respect to a coordinate system of the target

Determining intrinsic and

extrinsic parameters (1)

z

w

33

w

32

w

31

x

w

13

w

12

w

11x

4h

1h

tZrYrXr

tZrYrXrfx

C

C

+++

+++==

z

w

33

w

32

w

31

y

w

23

w

22

w

21

y

4h

2h

tZrYrXr

tZrYrXrfy

C

C

+++

+++==

≅

1

Z

Y

X

trrr

trrr

trrr

100

0f0

00f

C

C

C

w

w

w

z333231

y232221

x131211

y

x

4h

2h

1h

Same denominator

Determining intrinsic and

extrinsic parameters (2)

)tZrYrXr(fy)tZrYrXr(fx x

w

i13

w

i12

w

i11xiy

w

i23

w

i22

w

i21yi +++=+++

0)tZrYrXr(fy)tZrYrXr(fx x

w

i13

w

i12

w

i11xiy

w

i23

w

i22

w

i21yi =+++−+++

Writing the last equation for N corresponding pairs leads

to a system of equations and the determination of the

intrinsic and extrinsic parameters.

See Trucco and Verri, Introductory Techniques for 3D

Computer Vision, Prentice Hall, 1998 for a detailed solution.

Estimation of the Projection

Matrix (1)

;

≅

1

Z

Y

X

trrr

trrr

trrr

100

Oyf0

Ox0f

C

C

C

w

w

w

z333231

y232221

x131211

y

x

4h

2h

1h

Intrinsic and extrinsic parameters can combined in a

single (3x4) projection matrix

=

1

Z

Y

X

aaaa

aaaa

aaaa

C

C

C

w

w

w

34333231

24232221

14131211

4h

2h

1h


Matrix (2)

=

1

Z

Y

X

aaaa

aaaa

aaaa

C

C

C

w

w

w

34333231

24232221

14131211

4h

2h

1h

34

w

33

w

32

w

314h

24

w

23

w

22

w

212h

14

w

13

w

12

w

111h

aZaYaXaC

aZaYaXaC

aZaYaXaC

+++=

+++=

+++=;

4h1h CxC = 4h2h CyC =

0ayayZayYayXaZaYaXa

0axaxZaxYaxXaZaYaXa

2434

w

33

w

32

w

31

w

23

w

22

w

21

1434

w

33

w

32

w

31

w

13

w

12

w

11

=+−−+−++

=+−−−−++


Matrix (3)

=

−−−−

−−−−

−−−−

−−−−

0

0

0

0

a

a

.

.

.

a

a

'

yZyYyXy1ZYX0000

..

..

..

xZxYxXx00001ZYX

yZyYyXy1ZYX0000

xZxYxXx00001ZYX

34

33

12

11

nnnnnnnnnn

2222222222

1111111111

1111111111

0yayZayYayXaaZaYaXa

0xaxZaxYaxXaaZaYaXa

34

w

33

w

32

w

3124

w

23

w

22

w

21

34

w

33

w

32

w

3114

w

13

w

12

w

11

=−−+−+++

=−−−−+++

12x1212x1


Matrix (3)

=

−−−

−−−

−−−

n

1

1

33

12

11

nnnnnnnnn

222

111111111

111111111

x

y

x

a

.

.

.

a

a

'

ZxYxXx00001ZYX

..

..

....1ZYX

ZyYyXy1ZYX0000

ZxYxXx00001ZYX

L

aij are obtained up to a unknown scale factor

11x11 11x1

1a34 =

Computing camera parameters

=

34333231

24232221

14131211

z333231

y232221

x131211

y

x

aaaa

aaaa

aaaa

trrr

trrr

trrr

100

Oyf0

Ox0f

++++

++++

=

z333231

zyyy33y23y32y22y31y21y

zxxx33x13x32x12x31x11x

trrr

tOtfrOrfrOrfrOrf

tOtfrOrfrOrfrOrf

A

34z

2131y21y

1131x11x

at

...

arOrf

arOrf

=

=+

=+

understanding the image formation and the camera...

Documents