camera calibration- geometry - ucsb · camera calibration certainly, not all 3x4 matrices are like...
TRANSCRIPT
Camera Calibration - Geometry
MotivationThink about the application of drawing the first-down line in football gameCalibration steps:
Contour mappingCamera calibration – position
During the game:Cameras: three, PTZ readings provided in real-timePeople: spotter, line-position technician, 1st & ten operator, trouble-shooterComputers: five
Gather PC: receive PTZ dataTally: keep track of on-air camera and choose 3D map to useF Ten: video display and overlay 3D mapMatte: pattern recognition (player/field classification)Render: receives data from all other, draws the line
Pan encoder
Tilt encoder
RemoteSensor
Camera CalibrationStatic:
Where is a camera placed with respect to the field?How is the camera aimed?
Dynamic:How is the camera aim’s changed?
In essence:Camera coordinates (i,j) must be able to correlate into world coordinates (x,y,z)
Coordinate systems
World
Camera
Other ApplicationsAutonomous navigationPhotogrammetry and remote sensingSoldering and weldingInspectionAlmost all CV algorithms (except those deal entirely with 2D images) perform camera calibration
Calibration & RegistrationCamera calibration
Intrinsic parameters (e.g., focal length, aspect ratio, lens distortion)Independent of camera placement
Pose registrationExtrinsic parametersIndependent of choice of cameras
Mathematics of Image Formation
A sequence of coordinate transforms + projection (lens transform)3D world coordinate3D camera coordinate2D ideal image coordinate2D real image coordinateWorld to camera: rigid body transform (R and T)Camera to ideal: ideal projectionIdeal to real: real CCD and lens
Coordinate systems
W
C
world
camera
ideal image
real image
2D examples
x
y
x’
y’
Trans+Rot
x
y
x’
y’
PTrans
x’
y’
x
y
Rot
3D Translation
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
zyx
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡+
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡=
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
z
y
x
TTT
zyx
zyx
'''
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
'''
zyx
X
Y
Z
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
z
y
x
TTT
3D: Euler Angle Rotation
X Y
Z
XY
Z
XY
Z
xyz
xyz
'''
cos sinsin cos
1
0 00 0
0 0 1 00 0 0 1 1
⎡
⎣
⎢⎢⎢⎢
⎤
⎦
⎥⎥⎥⎥
=
−⎡
⎣
⎢⎢⎢⎢
⎤
⎦
⎥⎥⎥⎥
⎡
⎣
⎢⎢⎢⎢
⎤
⎦
⎥⎥⎥⎥
θ θθ θ
xyz
xyz
'''
cos sinsin cos
1
1 0 0 00 00 00 0 0 1 1
⎡
⎣
⎢⎢⎢⎢
⎤
⎦
⎥⎥⎥⎥
=−
⎡
⎣
⎢⎢⎢⎢
⎤
⎦
⎥⎥⎥⎥
⎡
⎣
⎢⎢⎢⎢
⎤
⎦
⎥⎥⎥⎥
θ θθ θ
⎥⎥⎥⎥⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢⎢⎢⎢⎢
⎣
⎡
⎥⎥⎥⎥⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢⎢⎢⎢⎢
⎣
⎡
−=
⎥⎥⎥⎥⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢⎢⎢⎢⎢
⎣
⎡
11000
0sin0cos
0010
0cos0sin
1
'
'
'
z
y
x
z
y
x
θθ
θθ
GenerallyA translation Followed by a rotation around xFollowed by another translationFollowed by a rotation around yFollowed by another translationFollowed by a rotation around zEtc.
LL )))((( 321 TTTPRRR +++xyz
Sidebar: Representation of Coordinates
use (X,Y,Z)T for 3D and (x,y) T for 2D results in different representation for translation and rotation P’3x1=R3x3P3x1+T3x1
Homogeneous coordinates P’ 4x1=M4x4P4x1
)/,/,/(),,,(0),,,,(),,(
wwzwwywwxwwzwywxwwwzwywxzyx
→≠→
PTR
P ⎥⎦
⎤⎢⎣
⎡=
10'
3D Translation
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
1zyx
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
1'''
zyx
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
1z
y
x
TTT
X
Y
Z
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
=
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
11000100010001
1'''
zyx
TTT
zyx
z
y
x
3D: Euler Angle Rotation
X Y
Z
XY
Z
XY
Z
xyz
xyz
'''
cos sinsin cos
1
0 00 0
0 0 1 00 0 0 1 1
⎡
⎣
⎢⎢⎢⎢
⎤
⎦
⎥⎥⎥⎥
=
−⎡
⎣
⎢⎢⎢⎢
⎤
⎦
⎥⎥⎥⎥
⎡
⎣
⎢⎢⎢⎢
⎤
⎦
⎥⎥⎥⎥
θ θθ θ
xyz
xyz
'''
cos sinsin cos
1
1 0 0 00 00 00 0 0 1 1
⎡
⎣
⎢⎢⎢⎢
⎤
⎦
⎥⎥⎥⎥
=−
⎡
⎣
⎢⎢⎢⎢
⎤
⎦
⎥⎥⎥⎥
⎡
⎣
⎢⎢⎢⎢
⎤
⎦
⎥⎥⎥⎥
θ θθ θ
⎥⎥⎥⎥⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢⎢⎢⎢⎢
⎣
⎡
⎥⎥⎥⎥⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢⎢⎢⎢⎢
⎣
⎡
−=
⎥⎥⎥⎥⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢⎢⎢⎢⎢
⎣
⎡
11000
0sin0cos
0010
0cos0sin
1
'
'
'
z
y
x
z
y
x
θθ
θθ
GenerallyA translation Followed by a rotation around xFollowed by another translationFollowed by a rotation around yFollowed by another translationFollowed by a rotation around zEtc.A much elegant representation in terms of matrix operationEasy concatenation
PTRTRTR 123 xyzL
View Normalization – Hard WayMake world coordinates make sense to a cameraTransform world coordinates into camera coordinatesOne translation
Zero out camera origin
Three rotations –line up coordinate axesRotate about YRotate about XRotate about Z
TPRRR yxz
X
Y
Z
X
Y
Z
X
Y
Z
X
Y
Z
Y rotation
X rotation
Z rotation
v
wu
World to Camera – Easy Way
),,( zyx uuu
),,( zyx vvv ),,( zyx www
X
Y
Z
),,( zyx cccworldworldcameraworldcamera
camera PTR
P ⎥⎦
⎤⎢⎣
⎡= ←←
10
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
⋅−⋅−⋅−
=
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
−−−
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
=←
1000
1000100010001
1000000
crcrcr
M
zzyx
yzyx
xzyx
z
y
x
zyx
zyx
zyx
worldcamera
wwwvvvuuu
ccc
wwwvvvuuu
Viewing Transform in OpenGL
void gluLookAt (GLdouble eyex, eyey, eyez,GLdouble centerx, centery, centerz, GLdouble upx, upy, upz)
X
Y
Z
Y
Z
X
W
up
U
V
eye and center: local w(z) directionup and local w(z): local v(y) directionlocal v(y) and w(z) directions: local u(x) direction
Viewing Transform (cont.)
Viewing Normalization
w e ce c
u u p wu p w
v w u
= −−
= ××
= ×
| |
| |
uvw
u u uv v vw w w
xyz
x y z
x y z
x y z
1
000
0 0 0 1 1
⎡
⎣
⎢⎢⎢⎢
⎤
⎦
⎥⎥⎥⎥
=
⎡
⎣
⎢⎢⎢⎢
⎤
⎦
⎥⎥⎥⎥
⎡
⎣
⎢⎢⎢⎢
⎤
⎦
⎥⎥⎥⎥
Figuring out [u, v, w] in [x, y, z] systemApplying a rotation to transform [x, y, z] coordinates into [u, v, w] coordinates
Camera to (Ideal) ImageRepresented by a projection
focal length f, aspect ratio 1
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
=⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
←
11 camera
camera
camera
cameraidealideal
ideal
ZYX
yx
M⎪⎪⎩
⎪⎪⎨
⎧
=
=
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡=
camera
cameraideal
camera
camreaideal
cameraideal
ZYfy
ZXfx
ff
:0100000000
Pp
f
),( , cameracameracamera ZYX
),( idealideal yx
cameraX
cameraY
cameraZZcamera
Xcamera
f xideal
Intrinsic parameters5 intrinsic parameters account for
The focal length ( f )The principal point (C0)=(u0,v0)
Where the optical axis intersects the image planePixel aspect ratio (ku, kv)
Pixels aren’t necessarily squareAngle between the axes (θ)
Skewness in manufacturing
θ
Intrinsic parameters
realx
idealx
idealy xcamera
ycamera
z
1/kv
1/ku
Principal pointC0 = (u0, v0)
f
θ
Orealy
Sensor array – real image plane
Normalized (ideal) image plane
idealov
ou
idealidealrealreal
vkuk
p
pMp
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡=
= ←
1000
0
Ideal Image to Real Image: Square Grid
idealx
idealy
realx
realy
0u
0v
uk
vk
⎪⎪⎩
⎪⎪⎨
⎧
−=−=
−=−=
ocamera
cameravoidealvreal
ocamera
camreauoidealureal
vZYfkvxky
uZXfkuxkx
idealov
ouu
real vkukk
pp⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡=
100sin/0
cotθθ
idealx
idealy
realx
realy
0u
0v
θuk
vk
Ideal Image to Real Image:Non-square grid
Putting It All Together
34333231
24232221
34333231
14131211
34333231
24232221
14131211
qZqYqXqqZqYqXqy
qZqYqXqqZqYqXqx
qqqqqqqqqqqq
worldworldworld
worldworldworldreal
worldworldworld
worldworldworldreal
worldworldworldreal
worldworldcameracameraidealidealrealreal
++++++
=
++++++
=
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡==
=
←
←←←
PPM
PMMMp
UsageGoverning equation
Off-line processGiven known 3D coordinates (landmarks) and their 2D projections, calculate q’s
On-line processGiven arbitrary 3D coordinates (first down at 30 yards) and q’s, calculate 2D coordinates (where to draw the first-and-ten line)Given arbitrary 2D coordinates (images of a vehicle) and q’s, calculate 3D coordinates (where to aim the gun to fire)
worldworldworldcameracameraidealidealrealreal
qqqqqqqqqqqq
PPMMMp⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡== ←←←
34333231
24232221
14131211
Camera Calibration and Registration
First stepEstimate the combined transformation matrix
Second stepEstimate intrinsic camera parametersEstimate extrinsic camera parameters
SolutionUsing objects of known sizes and shapes (6 points at least)Each point provides two constraints (x,y)A checked board pattern placed at different depths
worldreal←M
Calibration software availableLots of it these days!E.g., Camera Calibration Toolkit for Matlab
From the Computational Vision Group at Caltech (Bouguet)http://www.vision.caltech.edu/bouguetj/calib_doc/Includes lots of links to calibration tools and research
0)()(,1||
1001000000
100100000000
1000
0
100100000000
1000
0
34
24
14
=×⋅×=
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡++++
=
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡=
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
⎥⎦
⎤⎢⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡=
32313
3
2
1
3
32
31
3
2
1
3
2
1
qqqqq
qqq
rrrrr
rrr
rrr
PTR
p
qqq
ttvtvtutu
ttt
vu
ttt
ff
vkuk
ff
vkuk
T
T
T
z
zoyvov
zoxuou
z
y
x
ov
ou
z
y
x
ov
ou
worldov
ou
real
αααα
αα
Putting It All Together
Camera CalibrationCertainly, not all 3x4 matrices are like above
3x4 matrices have 11 free parameters (with a scale factor that cannot be decided uniquely)matrix in the previous slide has 10 parameters (2 scale, 2 camera center, 3 translation, 3 rotation)additional constraints can be very useful
to solve for the matrix, andto compute the parameters
Theorem: 3x4 matrices can be put in the form of the previous slide if and only if the following two constraints are satisfied
0)()(,1|| =×⋅×= 32313 qqqqq
Finding the transform matrix
0AQ
PqPqPqPq
PqPqPqPq
P
qqq
qqq
Pqqq
p
323
2
313
1
3
2
1
3
2
1
3
2
1
=⇒
=−+−⇒=++
=−+−⇒=++
⎥⎦
⎤⎢⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=
0
0
11
342433
343
243
341433
343
143
3
34
24
14
34
24
14
34
24
14
vqqvyqq
uqquxqq
qqq
ZYX
qqq
wwywx
qqq
worldT
worldT
realworld
Tworld
T
worldT
worldT
realworld
Tworld
T
world
T
T
T
world
world
world
T
T
T
real
real
worldT
T
T
real
Each data point provide two equations, with at least 6 points we will have 12 equations for solving 11 numbers up to a scale factorLagrange multipliers can be used to incorporate other constraints
The usual constraint is q32=1
Afterward, both intrinsic and extrinsic parameters can be recovered
Finding the transform matrix (cont.)
Details
Solved by Langrage multiplier)||1(min
1|| subject tomin2
32
32
qAQ
qAQ
0AQ
−+
=
=
λ
Really Ugly Detailsq3 is the eigen vector with the smallest eign value of matrix DTEDE is a positive matrix with positive eigen valuesDTED represents a similarity transform and has the same eigen values
[ ]
331
331
31
333
3
23
23
23
2
3
23
2
))((
))((
)('
0)'(
0)'('
)|q|1('min
)|q|1(q
'min
)|q|1(min
qDqCCCCID
qqDCCCCDDD
DqCCCq
qDqDCqDqE
DqCCqCqE
DqCqE
qDCE
E
TTT
TTTT
TT
TT
TT
λ
λ
λ
λ
λ
λ
=−
=−
−=
=−+=∂∂
=+=∂∂
−++=
−+⎥⎦
⎤⎢⎣
⎡=
−+=
−
−
−
AQ
E
voy
uox
vT
oT
uT
o
ovovv
ououu
o
o
TTzz
T
T
T
z
zoyvov
zoxuou
qvqtquqt
vr
ur
vv
uu
v
u
rr
qtqt
qqq
trtvtvtutu
αα
α
α
αξα
αξα
ξ
ξ
αααα
/)(/)(
/)(
/)(
3424
3414
2
1
22
21
21
33
3434
34
24
14
3
−=−=−=
−=
−±=−=
−±=−=
=
=
±==
±==
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡++++
32
3T
1
2T
22T
2
T1
T1
3T
2
3T
1
33
3
2
1
32
31
qqqq
qqqq
qqq
rrrr
Finding the Camera Parameters
realx
There are Four Solutions
),,( zyx uuu
),,( zyx vvv ),,( zyx www
),,( zyx ccc
X
Y
Z
realx
realx
There are Four Solutions
All four solutions satisfy the given 3D and 2D coordinate constraints
),,( zyx uuu
),,( zyx vvv ),,( zyx www
),,( zyx ccc
X
Y
Z
X’
Y’ Z’
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡++++
=⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
WZYX
ttvtvtutu
wyx
z
zoyvov
zoxuou
3
32
31
rrrrr
αααα
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡++++
=⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
−−−
WZYX
ttvtvtutu
wyx
z
zoyvov
zoxuou
3
32
31
rrrrr
αααα
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
−−−−
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡++++
=⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
WZYX
ttvtvtutu
wyx
z
zoyvov
zoxuou
3
32
31
rrrrr
αααα
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
−−−−
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡++++
=⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
−−−
WZYX
ttvtvtutu
wyx
z
zoyvov
zoxuou
3
32
31
rrrrr
αααα
4 Solutions
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
−−+−+−+−+−
=⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡⇒
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡++++
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
−−
−=
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡⇒
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡++++
=⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡⇒
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡++++
=⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
−−−
WZYX
ttvtvtutu
wyx
WZYX
ttvtvtutu
wyx
WZYX
ttvtvtutu
wyx
WZYX
ttvtvtutu
wyx
z
zoyvov
zoxuou
z
zoyvov
zoxuou
z
zoyvov
zoxuou
z
zoyvov
zoxuou
3
32
31
3
32
31
3
32
31
3
32
31
rrrrr
rrrrr
rrrrr
rrrrr
)()()()(
11
1
αααα
αααα
αααα
αααα
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
−−+−+−+−+−
=⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡⇒
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
−−
−−
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡++++
=⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡⇒
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
−−−−
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡++++
=⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
WZYX
ttvtvtutu
wyx
WZYX
ttvtvtutu
wyx
WZYX
ttvtvtutu
wyx
z
zoyvov
zoxuou
z
zoyvov
zoxuou
z
zoyvov
zoxuou
3
32
31
3
32
31
3
32
31
rrrrr
rrrrr
rrrrr
)()()()(
11
11
αααα
αααα
αααα
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡++++
=⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡⇒
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
−−
−−
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡++++
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
−−
−=
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡⇒
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
−−
−−
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡++++
=⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
−−
−⇒
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
−−−−
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡++++
=⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
−−−
WZYX
ttvtvtutu
wyx
WZYX
ttvtvtutu
wyx
WZYX
ttvtvtutu
wyx
WZYX
ttvtvtutu
wyx
z
zoyvov
zoxuou
z
zoyvov
zoxuou
z
zoyvov
zoxuou
z
zoyvov
zoxuou
3
32
31
3
32
31
3
32
31
3
32
31
rrrrr
rrrrr
rrrrr
rrrrr
αααα
αααα
αααα
αααα
11
11
11
1
11
11
11
1
What does it mean?Positive
Negative
Positive
Negative
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
−−−+−−+−−+−−+−
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡+−−+−−+−−+−−
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
−−−+−−+−−+−−+−
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡++++
z
zoyvov
zoxuou
z
zoyvov
zoxuou
z
zoyvov
zoxuou
z
zoyvov
zoxuou
ttvtvtutu
ttvtvtutu
ttvtvtutu
ttvtvtutu
3
32
31
3
32
31
3
32
31
3
32
31
rrrrr
rrrrr
rrrrr
rrrrr
)()()()()()()()(
))(())(())(())((
)()()()()()()()(
αααα
αααα
αααα
αααα
Coordinate Reflection
Recall that r1, r2, and r3 are the directions of the camera’s x, y, and z axes in global systemT = -RC
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
−−−+−−+−−+−−+−
z
zoyvov
zoxuou
ttvtvtutu
3
32
31
rrrrr
)()()()()()()()(
αααα
World
camera
Reflected camera
Focal length reflection
The z-axis is not reflected But focal length is (similar effect)
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡+−−+−−+−−+−−
z
zoyvov
zoxuou
ttvtvtutu
3
32
31
rrrrr
))(())(())(())((
αααα
World
camera
Reflected camera
Focal Length + Coordinate Reflection
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
−−−+−−+−−+−−+−
z
zoyvov
zoxuou
ttvtvtutu
3
32
31
rrrrr
)()()()()()()()(
αααα
World
camera
Degeneration Conditions3D points should not
Lie on a single planeLie on a twisted space curve (very rare)Detail can be found in Faugeras
Other FormulationsWith a calibration pattern
Flexible placement of the pattern
With a calibration patternsUse vanishing points and vanishing lines
Calibrating against lens distortionFisheye and others
Calibrating pan-tilt-zoom camerasPan and tilt axis direction, placement, angle
Registration vs. calibration Many times vs. onceOn-line vs. off-line
Flexible Pattern Placementhttp://research.microsoft.com/~zhang/calib/
Step 1: Intrinsic Parametersximage=Hxplane=K[R1,R2,T]xplane
[ ][ ]
21
211
121
1
11
11
11
11
21
11
2131
21
11
21321
21
0)()(0)(
][
][][
hKKhhKKhhKKhhKhKhKhKhKhK
TRRhKhKhK
TRRKhhhTRRKH
−−−−−−
−−−−−−
−−−
====
⇒
=⇒
=⇒=
TTTTTT
TTT
One homography8 DOFs6 extrinsic parameters (3 rotation + 3 translation)2 constraints on intrinsic parameters3 planes in general configuration
Image of absolute conic
Step 2: Extrinsic Parameters
[ ]
21
11
213
31
21
2
11
1
21321
21
11
][][
hKhK
RRRhKT
hKR
hKR
TRRKhhhTRRKH
−−
−
−
−
==
×==
=
=⇒
=⇒=
λ
λ
λ
λ
Step 3: Intrinsic + Extrinsic Parameters
ximage=Hxplane=K[R1,R2,T]xplane
2model21
1 1)][( ij
n
i
m
jij xTRRKximage −∑∑
= =
Nonlinear optimization Using Lenvenberg-Marquardt in MinpackK from the previous step as initial guess
# of images # of points/images
Calibrating for radial distortionThe camera lens also introduces errors of several type (as we’ve already discussed):
Spherical aberrationComaChromatic aberrationVignettingAstigmatismMisfocusRadial distortion
Of these, radial distortion is the most significant in most systems, and it can be corrected for
Radial distortionVariation in the magnification for object points at different distances from the optical axisEffect increases with distance from the optical axisStraight lines become bent!Two main descriptions: barrel distortion and pincushion distortionCan be modeled and corrected for
Barrel PincushionCorrect
Correcting for radial distortion
Original Corrected
Modeling Lens DistortionRadial, Barrel, Pincushion, etc.
Modeled as bi-cubic (or bi-linear) with more parameters to compensate forVery hard to solve
[ ]⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢
⎣
⎡
⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢
⎣
⎡
=
1
12
3
444241
242221
141211
23
ideal
ideal
ideal
xxx
xxx
xxx
idealidealidealreal yyy
aaa
aaaaaa
xxxx
L
LLLL
L
L
Modeling radial distortionThe radial distortion can be modeled as a polynomial function (λ) of d2, where d is the distance between the image center and the image point
Called the radial alignment constraint
⎥⎦
⎤⎢⎣
⎡=⎥
⎦
⎤⎢⎣
⎡vu
dvu
d
d )(λ
e.g., λ(d) = 1 + κ1d2 + κ2d4 + κ3d6
q distortion coefficients (q <4)
Modeling radial distortion
Now do calibration by estimating the 11+q parameters
P0
TRK ⎥
⎦
⎤⎢⎣
⎡
⎥⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢⎢
⎣
⎡
=⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
1100
0)','(
10
00)','(
1
1Tvu
vu
vu
λ
λ
Since d is a function of u and v, we could also write λ as λ(u, v)
where (u’, v’) comes from K[R|T]P
Dynamic PTZ CamerasPan-tilt Tilt-pan
θ φ
Additional Steps
Move to the center of pan (tilt)Rotate (pan, tilt)Move back to the optical center
worldworldreal
worldworldcamerapanpantilttiltcameraidealidealrealreal
f
fpt
PM
PMTRTTRTMMp nn
),,(
)()()( 11
φθ
θφ
←
←−−
←←
=
=
Additional DOFs (Variables)Both pan and tilt are rotationsA rotation is characterized by
Center of rotationAxis of rotationRotation angle
Efficient and accurate procedures for all DOFs, Compared to existing techniques that
Calibrate only center of rotationUse expensive iterative optimization procedures
General Proceduresθ requested =φ requested =0, calibrate the slave camera
Moving θ requested to some known angle (but keeping φ requested fixed), calibrate the slave camera again
worldcameraworldcamera f ←← = MM ),0(
worldcamerapanrealizedpanrealizedworldcamera pf ←
−← = MTRTM n )(),( 1 θθ
worldworldcamerapanpantilttiltcameraidealidealrealreal ptf PMTRTTRTMMp nn ←
−−←←= )0()0()( 11
worldworldcamerapanpantilttiltcameraidealidealrealreal ptf PMTRTTRTMMp nn ←
−−←←= )()0()( 11 θ
Details
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
==
=
←←−
←−
←
1000
),(),(
)(),(
34333231
24232221
14131211
1
1
mmmmmmmmmmmm
ff
f
cameraworldrealizedworldcamerapanrealizednpan
worldcamerapanrealizednpanrealizedworldcamera
p
p
MMTRT
MTRTM
θθ
θθ
⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢
⎣
⎡
−⋅−⋅−⋅
=
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
−−−
⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢
⎣
⎡
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
=−
101000100010001
10000
1000100010001
),(33
22
11
3
2
1
1
zT
yT
xT
z
y
x
T
T
T
z
y
x
panrealizednpan TTT
TTT
TTT
fp rTr
rTrrTr
rrr
TRT θ
4 wherecos2
14,
14,
144
11
21221
23113
22332
∑=− ==
−
−=
−
−=
−
−=
iii
realized
ppp
mww
wwmmn
wwmmn
wwmmn
zyx
θ343
242
141
mT
mTmT
z
y
x
=+⋅−
=+⋅−=+⋅−
rT
rTrT
Basic ProceduresCalibrating the rotation angles
if the realized angles of rotation (θrealized and φrealized) are close to the requested ones (θrequested and φrequested), and
Calibrating the location Tp(f) and Tt(f) and the orientation np and nt
of the rotation axis
ProceduresOuter loop
Vary the zoom setting of the camera Determine Tp(f) and Tt(f) through interpolation
Inner loopExecute a wide range of pan (or tilt) movements with a fixed camera zoomDetermine θrealized =f(θrequested ) and φrealized=g(φrequested) through interpolation. Calibrate Tp(fo), Tt(fo), np and nt
Loop bodyBasic procedure illustrated above
Comparison with Stanford Procedures*
Different Calibration proceduresVirtual landmark generation vs. checkerboard
Criticism: large angular range can be achieved with
Small landmarks at close distanceDispersed landmarks at large distance
Accuracy and simplicity favors close-distance approach
Comparison with Stanford Procedures*
Different solution methodsOptimization vs. closed-form solutionCriticism: Nonlinear optimization without good initial guess is expensive and unreliable
211 })()()({minarg worldworldcamerapanpantilttiltcameraidealidealrealreal ptf PMTRTTRTMMp nn
nT,←
−−←←− θφ
* Calibrating pan-tilt camera in wide-area surveillance networks, Davis and Chen, ICCV, 2003
RegistrationCalibration is usually done only once (unless you change the zoom)Registration needs to be done
Once – stationary cameraContinuously – moving camera (pan-tilt, mobile robot)
AlgorithmsFast Small number of registration points
Alternativeαx=K[R|T]X
K is known (assume fixed focal length)
αK-1x=[R|T]XAn observed landmark provides two constraints R and TSix observed landmarks theoretically solve for R and T
Church’s AlgorithmChurch’s algorithm
Match face angles Iterative (fast and accurate with good initial guess at video frame rate)Three landmarks only
x
yz
αβ
γ
hαhβ
hγ
),,( hhh ZYX
),,( ZYX