geometric camera calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf ·...
TRANSCRIPT
![Page 1: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/1.jpg)
Geometric Camera Calibration
Instructor: Jason Corso (jjcorso)!web.eecs.umich.edu/~jjcorso/t/598F14!
!
Materials on these slides have come from many sources in addition to myself; I am infinitely grateful to these, especially Greg Hager, Silvio Savarese, and Steve Seitz.!
EECS 598-08 Fall 2014!Foundations of Computer Vision!!
Readings: FP 1.3; SZ 6.3 (FL 4.6; extra notes) !Date: 9/17/14!!
![Page 2: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/2.jpg)
Plan
• Review Perspective Projection!• Geometric Camera Calibration!
– Indirect camera calibration!• Solve for projection matrix then the parameters!
– Direct camera calibration!– Multi-planes method!
• Example with the Matlab Toolbox!• Catadioptric Sensing!
– Different slide-deck. (See Chris Geyer’s CVPR 2003 Tutorial)!
• Other calibration methods not covered!– Vanishing points-based method (see SZ)!– Self-calibration!
2
![Page 3: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/3.jpg)
Projection equation
• The projection matrix models the cumulative effect of all parameters • Useful to decompose into a series of operations
ΠXx =
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=
1************
ZYX
ssysx
⎥⎥⎦
⎤
⎢⎢⎣
⎡
⎥⎥⎦
⎤
⎢⎢⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
−
−
=110100
00100001
100'0'0
31
1333
31
1333
x
xx
x
xxcy
cx
yfsxfs
000 TIRΠ
projection intrinsics rotation translation
identity matrix
Camera parameters A camera is described by several parameters
• Translation T of the optical center from the origin of world coords • Rotation R of the image plane • focal length , principle point (x’c, y’c), pixel size (sx, sy) • blue parameters are called “extrinsics,” red are “intrinsics”
• The definitions of these parameters are not completely standardized – especially intrinsics—varies from one book to another Source: S Seitz slides.!
3
![Page 4: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/4.jpg)
Oc
focal length:
Source: S Savarese slides.!
Projective Camera 4
![Page 5: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/5.jpg)
Projective Camera: The Normalized Image Plane 5
• The normalized image plane is parallel to the physical retina (e.g., ccd) but located at unit distance ( = 1) from the pinhole.!
Image Source: Forsyth and Ponce Book.!
![Page 6: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/6.jpg)
Projective Camera: The Normalized Image Plane
• Physical pixels in the retina (e.g. ccd) may not be square, so we have two additional scale parameters.!
• Units:!– is a distance expressed in meters!– A pixel will have dimensions where and are in!
• Can replace dependent pixel parameters !
6
![Page 7: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/7.jpg)
Source: S Savarese slides.!
Projective Camera 7
focal length: image center-point:
![Page 8: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/8.jpg)
Source: S Savarese slides.!
Projective Camera 8
focal length: image center-point: non-square pixels: skew angle:
![Page 9: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/9.jpg)
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=
10100000
zyx
vus
P' o
o
β
α
K has 5 degrees of freedom! Source: S Savarese slides.!
Projective Camera 9
focal length: image center-point: non-square pixels: skew angle:
![Page 10: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/10.jpg)
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡ −
=ʹ′
1zyx
01000v0
0ucot
P o
o
sinθβ
θαα
K has 5 degrees of freedom! Source: S Savarese slides.!
Projective Camera 10
focal length: image center-point: non-square pixels: skew angle:
![Page 11: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/11.jpg)
Ow
iw
kw
jw
Source: S Savarese slides.!
Projective Camera 11
focal length: image center-point: non-square pixels: skew angle: rotation, translation:
![Page 12: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/12.jpg)
Ow
iw
kw
jw
Internal parameters External parameters
Source: S Savarese slides.!
Projective Camera 12
focal length: image center-point: non-square pixels: skew angle: rotation, translation:
![Page 13: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/13.jpg)
Properties of Pinhole Perspective Projection
• Distant objects appear smaller!• Points project to points!• Lines project to lines!
• Angles are not preserved.!• Parallel lines meet!!
13
Source: S. Savarese slides.!
Vanishing Point!
![Page 14: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/14.jpg)
wPMP =ʹ′ [ ] wPTRK=Internal parameters
External parameters
Source: S Savarese slides.!
Projective Camera 14
![Page 15: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/15.jpg)
Projective Camera 15
Internal parameters External parameters
![Page 16: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/16.jpg)
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡ −
=
100v0
ucot
K o
o
sinθβ
θαα
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=T3
T2
T1
Rrrr
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=
z
y
x
ttt
T
43×
Projective Camera 16
![Page 17: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/17.jpg)
Estimate intrinsic and extrinsic parameters from 1 or multiple images
Change notation: P = Pw
p = P’ ⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡ −
=
100v0
ucot
K o
o
sinθβ
θαα
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=T3
T2
T1
Rrrr
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=
z
y
x
ttt
T
43×
Source: S Savarese slides.!
Goal of Calibration 17
![Page 18: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/18.jpg)
• P1… Pn with known positions in [Ow,iw,jw,kw] • p1, … pn known positions in the image Goal: compute intrinsic and extrinsic parameters
jC
Calibration rig
Source: S Savarese slides.!
The Calibration Problem 18
![Page 19: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/19.jpg)
• P1… Pn with known positions in [Ow,iw,jw,kw] • p1, … pn known positions in the image Goal: compute intrinsic and extrinsic parameters
jC
Calibration rig
image!
Source: S Savarese slides.!
The Calibration Problem 19
![Page 20: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/20.jpg)
jC
Calibration rig
How many correspondences do we need? • M has 11 unknown • We need 11 equations • 6 correspondences would do it
image!
Source: S Savarese slides.!
The Calibration Problem 20
![Page 21: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/21.jpg)
image!jC
Calibration rig
In practice, using more than 6 correspondences enables more robust results
Source: S Savarese slides.!
The Calibration Problem 21
![Page 22: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/22.jpg)
jC
ii PMP → ⎥⎦
⎤⎢⎣
⎡=→
i
ii v
up
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=
3
2
1
Mmmm
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
=
i3
i2
i3
i1
PPPP
mmmm
in pixels Source: S Savarese slides.!
The Calibration Problem 22
![Page 23: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/23.jpg)
i3
i1i P
Pumm
=
i2i3i P)P(v mm =→
i1i3i P)P(u mm =→
i3
i2i P
Pvmm
=
⎥⎦
⎤⎢⎣
⎡
i
i
vu
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
=
i3
i2
i3
i1
PPPP
mmmm
0P)P(v 23 =−→ iii mm
0P)P(u 13 =−→ iii mm
Source: S Savarese slides.!
The Calibration Problem 23
![Page 24: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/24.jpg)
…
…
0)( 12131 =− PPv mm
0)( 11131 =− PPu mm
0)( 23 =− iii PPv mm
0)( 13 =− iii PPu mm
0)( 23 =− nnn PPv mm
0)( 13 =− nnn PPu mm
Source: S Savarese slides.!
The Calibration Problem 24
![Page 25: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/25.jpg)
⎥⎦
⎤⎢⎣
⎡=⎥
⎦
⎤⎢⎣
⎡=
2221
1211
2221
1211
BBBB
BAAAA
A
What is AB ?
⎥⎦
⎤⎢⎣
⎡
++
++=
2222122121221121
2212121121121111
BABABABABABABABA
AB
Source: S Savarese slides.!
Block Matrix Multiplication 25
![Page 26: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/26.jpg)
2n x 12 12x1
1x4
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
=T3
T2
T1def
mmm
m
4x1
…
Homogenous linear system
known unknown
0)( 12131 =+− PPv mm
0)( 11131 =+− PPu mm
0)( 23 =+− nnn PPv mm
0)( 13 =+− nnn PPu mm
Source: S Savarese slides.!
The Calibration Problem 26
![Page 27: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/27.jpg)
P! m 0 =
Rectangular system (M>N)
• 0 is always a solution
Minimize |P m|2
under the constraint |m|2 =1
M=number of equations = 2n N=number of unknown = 11
• To find non-zero solution
Source: S Savarese slides.!
Homogeneous M x N Linear Systems 27
![Page 28: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/28.jpg)
• How do we solve this homogenous linear system?
• Using DLT (Direct Linear Transformation) algorithm via SVD decomposition!
Source: S Savarese slides.!
The Calibration Problem 28
![Page 29: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/29.jpg)
12
1
1
.−−
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
=Λ= SSSSA
Nλ
λ
λ
Eigenvectors of A are columns of S
[ ]NS vv1=
Eigendecomposition
Source: S Savarese slides.!
Eigenvalues and Eigenvectors 29
![Page 30: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/30.jpg)
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
=Σ
Nσ
σ
σ
.2
1
1−Σ= VUA
ii λσ = σ = singular value λ = eigenvalue of At A
U, V = orthogonal matrix
Source: S Savarese slides.!
Singular Value Decomposition 30
![Page 31: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/31.jpg)
Properties of SVD
• Recall the singular values of a matrix are related to its rank.!• Recall that Ax = 0 can have a nonzero x as solution only if A
is singular.!• Finally, note that the matrix V of the SVD is an orthogonal
basis for the domain of A; in particular the zero singular values are the basis vectors for the null space.!
• Putting all this together, we see that A must have rank 7 (in this particular case) and thus x must be a vector in this subspace. !
• Clearly, x is defined only up to scale.!
31
Source: G Hager slides.!
![Page 32: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/32.jpg)
ix ixʹ′
H
0Ai =hFunction of measurements
unknown
ii xHx =ʹ′Source: S Savarese slides.!
DLT algorithm (Direct Linear Transformation) 32
![Page 33: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/33.jpg)
1212T
121212n2 VDU ×××
Last column of V gives m
M iPM ip→
SVD decomposition of P
Why? See pag 593 of AZ
Source: S Savarese slides.!
The Calibration Problem 33
![Page 34: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/34.jpg)
Tnnnnnmnm VDUP ×××× =
Tnnnmmmnm VDUP ×××× =
Has n orthogonal columns
Orthogonal matrix
• This is one of the possible SVD decompositions • This is typically used for efficiency • The classic SVD is actually:
• Thanks to Pat O’ Keefe!!
orthogonal Orthogonal Source: S Savarese slides.!
Clarification about SVD 34
![Page 35: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/35.jpg)
Estimated values
Intrinsic
Source: S Savarese slides.!
Extracting Camera Parameters 35
![Page 36: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/36.jpg)
Source: S Savarese slides.!
Theorem (Faugeras, 1993) 36
![Page 37: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/37.jpg)
Estimated values
Intrinsic
Source: S Savarese slides.!
Extracting Camera Parameters 37
![Page 38: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/38.jpg)
Estimated values
Extrinsic
Source: S Savarese slides.!
Extracting Camera Parameters 38
![Page 39: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/39.jpg)
Degenerate cases
• Pi’s cannot lie on the same plane! • Points cannot lie on the intersection curve of two quadric surfaces
Source: S Savarese slides.!
39
![Page 40: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/40.jpg)
Taking lens distortions into account • Chromatic Aberration!• Spherical aberration !• Radial Distortion!
Source: S Savarese slides.!
40
![Page 41: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/41.jpg)
No distortion
Pin cushion
Barrel
Dealing with Radial Distortion As Well
– Caused by imperfect lenses!– Deviations are most noticeable for rays that pass through
the edge of the lens!
Source: S Savarese slides.!
41
![Page 42: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/42.jpg)
Issues with lenses: Radial Distortion
Pin cushion
Barrel (fisheye lens) Source: S Savarese slides.!
42
![Page 43: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/43.jpg)
Source: S Savarese slides.!
Radial Distortion 43
![Page 44: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/44.jpg)
Image magnification in(de)creases with distance from the optical center
ii
ii p
vu
PM
100
00
00
1
1
=⎥⎦
⎤⎢⎣
⎡→
⎥⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢⎢
⎣
⎡
λ
λd
v
vucvbuad 222 ++=
u
∑±==
3
1p
2ppdκ1λ
Polynomial function
Distortion coefficient
To model radial behavior
Source: S Savarese slides.!
Radial Distortion 44
![Page 45: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/45.jpg)
⎥⎦
⎤⎢⎣
⎡=
i
ii v
up
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=
3
2
1
Qqqq
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
=
i3
i2
i3
i1
PPPP
qqqq
ii
ii p
vu
PM
100
00
00
1
1
=⎥⎦
⎤⎢⎣
⎡→
⎥⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢⎢
⎣
⎡
λ
λ
Q
⎩⎨⎧
=
=
PPvPPu
2i3i
i1i3i
qqqq
Is this a linear system of equations?
No! why? Source: S Savarese slides.!
Radial Distortion 45
![Page 46: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/46.jpg)
)(PfX =
measurement parameter
f( ) is nonlinear
- Newton Method - Levenberg-Marquardt Algorithm
• Iterative, starts from initial solution • May be slow if initial solution far from real solution • Estimated solution may be function of the initial solution • Newton requires the computation of J, H • Levenberg-Marquardt doesn’t require the computation of H
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
=
i3
i2
i3
i1
PPPP
qqqq
⎥⎦
⎤⎢⎣
⎡
i
i
vu
Source: S Savarese slides.!
General Calibration Problem 46
![Page 47: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/47.jpg)
A possible algorithm 1. Solve linear part of the system to find approximated solution 2. Use this solution as initial condition for the full system 3. Solve full system using Newton or L.M.
)(PfX =
measurement parameter
f( ) is nonlinear
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
=
i3
i2
i3
i1
PPPP
qqqq
⎥⎦
⎤⎢⎣
⎡
i
i
vu
Source: S Savarese slides.!
General Calibration Problem 47
![Page 48: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/48.jpg)
Typical assumptions:
- zero-skew, square pixel - uo, vo = known center of the image - no distortion
Just estimate f and R, T
)(PfX =
measurement parameter
f( ) is nonlinear
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
=
i3
i2
i3
i1
PPPP
qqqq
⎥⎦
⎤⎢⎣
⎡
i
i
vu
Source: S Savarese slides.!
General Calibration Problem 48
![Page 49: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/49.jpg)
Can estimate m1 and m2 and ignore the radial distortion?!
⎥⎦
⎤⎢⎣
⎡=
i
ii v
up
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
=
i3
i2
i3
i1
PPPP
1
mmmm
λ
d
v
u Hint: slopevu
i
i =
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
=
i3
i2
i3
i1
PPPP
qqqq
Source: S Savarese slides.!
Radial Distortion 49
![Page 50: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/50.jpg)
Estimating m1 and m2…!
⎥⎦
⎤⎢⎣
⎡=
i
ii v
up
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
=
i3
i2
i3
i1
PPPP
1
mmmm
λ
0)()( 121111 =− PuPv mm0)()( 21 =− iiii PuPv mm
0)()( 21 =− nnnn PuPv mm…
0Q =n ⎥⎦
⎤⎢⎣
⎡=
2
1
mm
n
Tsai technique [87]
i
i
i
i
i
i
i
i
PP
PPPP
vu
2
1
3
2
3
1
)()()()(
mm
mmmm
==
Source: S Savarese slides.!
Radial Distortion 50
![Page 51: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/51.jpg)
Once that m1 and m2 are estimated…!
⎥⎦
⎤⎢⎣
⎡=
i
ii v
up
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
=
i3
i2
i3
i1
PPPP
1
mmmm
λ
3m is non linear function of 1m 2m λ
There are some degenerate configurations for which m1 and m2 cannot be computed
, ,
Source: S Savarese slides.!
Radial Distortion 51
![Page 52: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/52.jpg)
Direct Calibration: The Algorithm
1. Compute image center from orthocenter!2. Compute the Intrinsic matrix (6.8)!3. Compute solution with SVD!4. Compute gamma and alpha!5. Compute R (and normalize)!6. Compute fx and and Tz!
Source: G Hager slides.!
52
![Page 53: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/53.jpg)
Basic Equations
Source: G Hager slides.!
53
![Page 54: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/54.jpg)
Basic Equations
Source: G Hager slides.!
54
![Page 55: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/55.jpg)
Basic Equations
one of these for each point!
Source: G Hager slides.!
55
![Page 56: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/56.jpg)
Basic Equations
Source: G Hager slides.!
56
![Page 57: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/57.jpg)
Properties of SVD Again
• Recall the singular values of a matrix are related to its rank.!• Recall that Ax = 0 can have a nonzero x as solution only if A is singular.!• Finally, note that the matrix V of the SVD is an orthogonal basis for the domain of
A; in particular the zero singular values are the basis vectors for the null space.!
• Putting all this together, we see that A must have rank 7 (in this particular case) and thus x must be a vector in this subspace. !
• Clearly, x is defined only up to scale.!
Source: G Hager slides.!
57
![Page 58: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/58.jpg)
Basic Equations
We now know Rx and Ry up to a sign and gamma.!Rz = Rx x Ry!!We will probably use another SVD to orthogonalize!this system (R = U D V’; set D to I and multiply).!
Source: G Hager slides.!
58
![Page 59: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/59.jpg)
Last Details about Direct Calibration
• We still need to compute the correct sign.!– note that the denominator of the original equations must be
positive (points must be in front of the cameras)!– Thus, the numerator and the projection must disagree in sign.!– We know everything in numerator and we know the projection,
hence we can determine the sign.!
• We still need to compute Tz and fx!– we can formulate this as a least squares problem on those two
values using the first equation.!
Source: G Hager slides.!
59
![Page 60: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/60.jpg)
Self-Calibration
• Calculate the intrinsic parameters solely from point correspondences from multiple images.!
• Static scene and intrinsics are assumed.!• No expensive apparatus.!• Highly flexible but not well-established.!• Projective Geometry – image of the absolute conic.!
Source: G Hager slides.!
![Page 61: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/61.jpg)
Multi-Plane Calibration
• Hybrid method: Photogrammetric and Self-Calibration.!• Uses a planar pattern imaged multiple times (inexpensive).!• Used widely in practice and there are many
implementations.!• Based on a group of projective transformations called
homographies.!
• m be a 2d point [u v 1]’ and M be a 3d point [x y z 1]’.!
• Projection is !
Source: G Hager slides.!
![Page 62: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/62.jpg)
Planar Homographies
• First Fundamental Theorem of Projective Geometry:!– There exists a unique homography that performs a change of basis
between two projective spaces of the same dimension.!
– Projection Becomes !
– Notice that the homography is defined up to scale (s).!
Source: G Hager slides.!
![Page 63: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/63.jpg)
Computing the Intrinsics
• We know that!
• From one homography, how many constraints on the intrinsic parameters can we obtain? !– Extrinsics have 6 degrees of freedom.!– The homography has 8 degrees of freedom.!– Thus, we should be able to obtain 2 constraints per
homography.!
• Use the constraints on the rotation matrix columns…!
Source: G Hager slides.!
![Page 64: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/64.jpg)
Computing Intrinsics
• Rotation Matrix is orthonormal:!
• Write the homography in terms of its columns…!
Source: G Hager slides.!
![Page 65: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/65.jpg)
Computing Intrinsics
• Derive the two constraints:!
Source: G Hager slides.!
![Page 66: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/66.jpg)
Closed-Form Solution
• Notice is symmetric, 6 parameters can be written as a vector b.!• From the two constraints, we have!
• Stack up n of these for n images and build a 2n*6 system.!• Solve with SVD (yet again). !• Extrinsics “fall-out” of the result easily.!
Source: G Hager slides.!
![Page 67: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/67.jpg)
Non-linear Refinement
• Closed-form solution minimized algebraic distance.!• Since full-perspective is a non-linear model!
– Can include distortion parameters (radial, tangential)!– Minimize squared distance with a non-linear method.!
Source: G Hager slides.!
![Page 68: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/68.jpg)
Camera Calibration Toolbox for Matlab!J. Bouguet – [1998-2000]! !http://www.vision.caltech.edu/bouguetj/calib_doc/index.html#examples!
Source: S Savarese slides.!
Example Calibration Procedure 68
![Page 69: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/69.jpg)
Source: S Savarese slides.!
Example Calibration Procedure 69
![Page 70: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/70.jpg)
Source: S Savarese slides.!
Example Calibration Procedure 70
![Page 71: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/71.jpg)
Source: S Savarese slides.!
Example Calibration Procedure 71
![Page 72: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/72.jpg)
Source: S Savarese slides.!
Example Calibration Procedure 72
![Page 73: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/73.jpg)
Source: S Savarese slides.!
Example Calibration Procedure 73
![Page 74: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/74.jpg)
Source: S Savarese slides.!
Example Calibration Procedure 74
![Page 75: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/75.jpg)
Source: S Savarese slides.!
Example Calibration Procedure 75
![Page 76: Geometric Camera Calibrationweb.eecs.umich.edu/.../files/lecture_0917_calibration.pdf · 2016-09-07 · – Direct camera calibration! – Multi-planes method! • Example with the](https://reader033.vdocument.in/reader033/viewer/2022050517/5fa0f4816a0470388f190e6f/html5/thumbnails/76.jpg)
Next Lecture: Photometric and Radiometric Aspects
• Reading: FP 2, 3; SZ 2.2, 2.3!
76