© 2005 yusuf akgul gebze institute of technology department of computer engineering computer vision...
TRANSCRIPT
© 2005 Yusuf Akgul
Gebze Institute of TechnologyDepartment of Computer
Engineering
Computer Vision
Geometric Camera Calibration
© 2005 Yusuf Akgul
Geometric Camera Calibration• Geometric camera calibration is the estimation
of camera parameters (intrinsic, extrinsic, or both)
• Why do we need camera calibration?
• Most vision techniques need camera parameters but there are a few that don’t need the parameters.
© 2005 Yusuf Akgul
Camera Calibration• We need to link the
known world coordinates with the corresponding image coordinates.
• To be able to know the world coordinates, we need an object whose measurements are known, which are called calibration patterns.
© 2005 Yusuf Akgul
Camera Calibration
Is it enough to solve just the above matrix to do the camera calibration?
© 2005 Yusuf Akgul
Camera Calibration
We will see two ways of calibration
• First method: Direct camera calibration estimates the intrinsic and extrinsic camera parameters directly.
• Second method: The projection matrix is estimated first and then the camera parameters.
So which one should we prefer?
© 2005 Yusuf Akgul
Direct Camera Calibration
Extrinsic parameters to recover are
• R, 3x3 rotation matrix
• T, 3D translation vector.
© 2005 Yusuf Akgul
Direct Camera Calibration
© 2005 Yusuf Akgul
Direct Camera Calibration
If we ignore radial distortions, then we have
•Focal length
•Pixel size in x and y
•Image center x and y
5 dependent parameters
© 2005 Yusuf Akgul
Direct Camera calibrationIntrinsic camera parameters
© 2005 Yusuf Akgul
Direct Camera Calibration
© 2005 Yusuf Akgul
Direct Camera CalibrationTwo main steps
• Assume the image center is known, estimate all the remaining parameters. We will assume (ox, oy) is (0,0)
• Estimate the image center later.
© 2005 Yusuf Akgul
Direct Camera Calibration
Denominators are the same! So, for each
pair, we can write
© 2005 Yusuf Akgul
Direct Camera Calibration
• If we write N equations like this one, we would have a homogenous linear system of equations of the form
),,,,,,,(
),,,,,,,(
131211232221
87654321
xy TrrrTrrr
vvvvvvvv
© 2005 Yusuf Akgul
Direct Camera Calibration
How do we solve v?
© 2005 Yusuf Akgul
Direct Camera Calibration• Homogeneous System of N Linear Equations
– Given N corresponding pairs {(Xi, Yi,, Zi) <-> (xi, yi) }, i=1,2,…N
– 8 unknowns v = (v1,…,v8)T, 7 independent parameters
• The system has a nontrivial solution (up to a scale) – IF N >= 7 and N points are not coplanar => Rank
(A) = 7– Can be determined from the Singular Value
Decomposition (SVD) of A
© 2005 Yusuf Akgul
• Singular Value Decomposition:– Any mxn matrix can be written as the product of
three matrices TUDVA
Singular values i are fully determined by A D is diagonal: dij =0 if ij; dii = i (i=1,2,…,n)
1 2 … N 0
Both U and V are not unique Columns of each are mutual orthogonal vectors
nnnn
n
n
n
mmmm
m
m
mnmm
n
n
vvv
vvv
vvv
uuu
uuu
uuu
aaa
aaa
aaa
21
22212
121112
1
21
22221
11211
21
22221
11211
00
0
0
00
V1U1
© 2005 Yusuf Akgul
Singular Value Decomposition• 1. Singularity and Condition Number
– nxn A is nonsingular IFF all singular values are nonzero– Condition number : degree of singularity of A
• A is ill-conditioned if 1/C is comparable to the arithmetic precision of your machine; almost singular
• 2. Rank of a square matrix A– Rank (A) = number of nonzero singular values
• 3. Inverse of a square Matrix– If A is nonsingular– In general, the pseudo-inverse of A
• 4. Eigenvalues and Eigenvectors– Eigenvalues of both ATA and AAT are si2 (si > 0)– The columns of U are the eigenvectors of AAT (mxm)– The columns of V are the eigenvectors of ATA (nxn)
TUDVA
nC /1
TUVDA 10
iiT
iuuAA 2
iiT
ivAvA 2
© 2005 Yusuf Akgul
• Least Squares– Solve a system of m equations for n unknowns x(m >=
n)– A is a mxn matrix of the coefficients – b (0) is the m-D vector of the data – Solution:
– How to solve: compute the pseudo-inverse of ATA by SVD
• (ATA)+ is more likely to coincide with (ATA)-1 given m > n• Always a good idea to look at the condition number of ATA
bAx
bAAxA TT bAAAx TT )(
nxn matrix Pseudo-inverse
Singular Value Decomposition
© 2005 Yusuf Akgul
• Homogeneous System– m equations for n unknowns x(m >= n-1)– Rank (A) = n-1 (by looking at the SVD of A)– A non-trivial solution (up to a arbitrary scale) by SVD:– Simply proportional to the eigenvector corresponding
to the only zero eigenvalue of ATA (nxn matrix)
• Note:– All the other eigenvalues are positive because
Rank (A)=n-1
– In practice, the eigenvector (i.e. vn) corresponding to the minimum eigenvalue of ATA, i.e. n
2
0Ax
iiT
ivAvA 2
Singular Value Decomposition
© 2005 Yusuf Akgul
Singular Value Decomposition• Problem Statements
– Numerical estimate of a matrix A whose entries are not independent
– Errors introduced by noise alter the estimate to Â
• Enforcing Constraints by SVD– Take orthogonal matrix A as an example
– Find the closest matrix to Â, which satisfies the constraints exactly
• SVD of  • Observation: D = I (all the singular values are 1) if A is
orthogonal• Solution: changing the singular values to those expected
TUDVA ˆ
© 2005 Yusuf Akgul
• Homogeneous System of N Linear Equations – Given N corresponding pairs {(Xi, Yi,, Zi) <-> (xi, yi) }, i=1,2,…
N– 8 unknowns v = (v1,…,v8)T, 7 independent parameters
• The system has a nontrivial solution (up to a scale) – IF N >= 7 and N points are not coplanar => Rank (A) = 7– Can be determined from the SVD of A– Rows of VT: eigenvectors {ei} of ATA– Solution: the 8th row e8 corresponding to the only zero singular
value 8=0
• Practical Consideration– The errors in localizing image and world points may make the
rank of A to be maximum (8)– In this case select the eigenvector corresponding to the
smallest eigenvalue.
0Av
TUDVA
8ev c
Direct Camera Calibration
© 2005 Yusuf Akgul
• Equations for scale factor and aspect ratio
• How do we find the scale factor and alpha?• How do we find other parameters?
),,,,,,,( 131211232221 xy TrrrTrrr v v1 v2 v3 v4 v5 v6 v7 v8
Direct Camera CalibrationHow do we find the scale factor and alpha?
© 2005 Yusuf Akgul
• Equations for scale factor and aspect ratio
• Knowledge: R is an orthogonal matrix
• Second row (i=2):
• First row (i=1)
),,,,,,,( 131211232221 xy TrrrTrrr v
T
T
T
ij
rrr
rrr
rrr
r
3
2
1
333231
232221
131211
33R
R
R
R
ji if 0
ji if 1RRT
1222232221
rrr2
322
21|| vvv
1222131211
rrr 27
26
25|| vvv
||
v1 v2 v3 v4 v5 v6 v7 v8
Direct Camera CalibrationHow do we find the scale factor and alpha?
© 2005 Yusuf Akgul
Rotation R and Translation TRotation R and Translation T• Equations for first 2 rows of R and T gave us and ||
• First 2 rows of R and T can be found up to a common sign s (+ or -)
• The third row of the rotation matrix by vector product
• Remaining Questions :
– How to find the sign s?– Is R orthogonal?– How to find Tz and fx, fy?
),,,,,,,( || 131211232221 xy TrrrTrrrs v
yxTT sTsTss ,,, 21 RR
TTTTT ss 21213 RRRRR
T
T
T
ij
rrr
rrr
rrr
r
3
2
1
333231
232221
131211
33R
R
R
R
Direct Camera Calibration
© 2005 Yusuf Akgul
Find the sign sFind the sign s
• Facts:– fx > 0– Zc >0– x known– Xw,Yw,Zw known
• Solution Check the sign of Xc Should be opposite to
x
zwww
ywwwyy
zwww
xwwwxx
TZrYrXr
TZrYrXrf
Zc
Ycfy
TZrYrXr
TZrYrXrf
Zc
Xcfx
333231
232221
333231
131211
Zw
Xw
Yw
Y
X
Z
xyO
PwP
p
xim
yim
(xim,yim)
Direct Camera Calibration
© 2005 Yusuf Akgul
Rotation R : OrthogonalityRotation R : Orthogonality• Question:
– First 2 rows of R are calculated without using the mutual orthogonal constraint
• Solution: – Use SVD of estimate R
TTTTT ss 21213 RRRRR
T
T
T
ij
rrr
rrr
rrr
r
3
2
1
333231
232221
131211
33
R
R
R
R
?ˆˆ IRR T
TUDVR ˆ TUIVR
Replace the diagonal matrix D with the 3x3 identity matrix
Direct Camera Calibration
© 2005 Yusuf Akgul
Direct Camera Calibration
Find Tz, Fx and FyFind Tz, Fx and Fy• Solution
– Solve the system of N linear equations with two unknown
• Tx, fx
– Least Square method
– SVD method to find inverse
zwww
xwwwx TZrYrXr
TZrYrXrfx
333231
131211
bA
x
z
f
T
)()( 333231131211 wwwxxwwwz ZrYrXrxfTZrYrXrxT
ai2ai1 bi
bAAA TT
x
z
f
T 1)(ˆ
ˆ
© 2005 Yusuf Akgul
1. Measure N 3D coordinates (Xi, Yi,Zi)2. Locate their corresponding image
points (xi,yi) - Edge, Corner, Hough3. Build matrix A of a homogeneous
system Av = 0 4. Compute SVD of A , solution v5. Determine aspect ratio and scale ||6. Recover the first two rows of R and
the first two components of T up to a sign
7. Determine sign s of by checking the projection equation
8. Compute the 3rd row of R by vector product, and enforce orthogonality constraint by SVD
9. Solve Tz and fx using Least Square and SVD, then fy = fx /
Yw
Xw
Zw
Direct Camera Calibration Algorithm
© 2005 Yusuf Akgul
Estimating the Image CenterEstimating the Image Center• Vanishing points:
– Due to perspective projection, all parallel lines in 3D space appear to meet in a point on the image - the vanishing point, which is the common intersection of all the image lines
VP1
VP2
Direct Camera Calibration
© 2005 Yusuf Akgul
Direct Camera Calibration
Estimating the Image Center
Estimating the Image Center
• Orthocenter Theorem:
– Input: three mutually orthogonal sets of parallel lines in an image
– T: a triangle on the image plane defined by the three vanishing points
– Image center = orthocenter of triangle T
– Orthocenter of a triangle is the common intersection of the three altitudes VP1
VP2
VP3
h3
h1
h1
(ox,oy)
© 2005 Yusuf Akgul
VP1
VP2
VP3
h3
h1
h1
(ox,oy) ?
Direct Camera Calibration
Estimating the Image Center
Estimating the Image Center
Wrong aspect ratio would produce wrong image center.
© 2005 Yusuf Akgul
HomeworkRead the Projection Matrix
Based Camera Calibration In Your Book.