cs5670 : computer vision - cornell university · 2019-03-28 · •midterm grades are out –please...
TRANSCRIPT
![Page 1: CS5670 : Computer Vision - Cornell University · 2019-03-28 · •Midterm grades are out –Please submit any regrade requests via Gradescope –Solutions will be available soon](https://reader034.vdocument.in/reader034/viewer/2022042601/5fa1950671e1753a9e26cd34/html5/thumbnails/1.jpg)
Two-view geometry
CS5670 : Computer VisionNoah Snavely
![Page 2: CS5670 : Computer Vision - Cornell University · 2019-03-28 · •Midterm grades are out –Please submit any regrade requests via Gradescope –Solutions will be available soon](https://reader034.vdocument.in/reader034/viewer/2022042601/5fa1950671e1753a9e26cd34/html5/thumbnails/2.jpg)
Reading
• Reading: Szeliski, Ch. 7.2
![Page 3: CS5670 : Computer Vision - Cornell University · 2019-03-28 · •Midterm grades are out –Please submit any regrade requests via Gradescope –Solutions will be available soon](https://reader034.vdocument.in/reader034/viewer/2022042601/5fa1950671e1753a9e26cd34/html5/thumbnails/3.jpg)
Fundamental matrix song
https://www.youtube.com/watch?v=DgGV3l82NTk
![Page 4: CS5670 : Computer Vision - Cornell University · 2019-03-28 · •Midterm grades are out –Please submit any regrade requests via Gradescope –Solutions will be available soon](https://reader034.vdocument.in/reader034/viewer/2022042601/5fa1950671e1753a9e26cd34/html5/thumbnails/4.jpg)
Annoucements
• Midterm grades are out
– Please submit any regrade requests via Gradescope
– Solutions will be available soon
• Project 3 code due tomorrow, 3/28, by 11:59pm
– Artifact due on 3/29
![Page 5: CS5670 : Computer Vision - Cornell University · 2019-03-28 · •Midterm grades are out –Please submit any regrade requests via Gradescope –Solutions will be available soon](https://reader034.vdocument.in/reader034/viewer/2022042601/5fa1950671e1753a9e26cd34/html5/thumbnails/5.jpg)
Back to stereo
• Where do epipolar lines come from?
![Page 6: CS5670 : Computer Vision - Cornell University · 2019-03-28 · •Midterm grades are out –Please submit any regrade requests via Gradescope –Solutions will be available soon](https://reader034.vdocument.in/reader034/viewer/2022042601/5fa1950671e1753a9e26cd34/html5/thumbnails/6.jpg)
Two-view geometry
• Where do epipolar lines come from?
epipolar plane
epipolar lineepipolar line
0
3D point lies somewhere along r
(projection of r)
Image 1 Image 2
![Page 7: CS5670 : Computer Vision - Cornell University · 2019-03-28 · •Midterm grades are out –Please submit any regrade requests via Gradescope –Solutions will be available soon](https://reader034.vdocument.in/reader034/viewer/2022042601/5fa1950671e1753a9e26cd34/html5/thumbnails/7.jpg)
Fundamental matrix
• This epipolar geometry of two views is described by a Very Special 3x3 matrix , called the fundamental matrix
• maps (homogeneous) points in image 1 to lines in image 2!
• The epipolar line (in image 2) of point p is:
• Epipolar constraint on corresponding points:
epipolar plane
epipolar lineepipolar line
0
(projection of ray)
Image 1 Image 2
![Page 8: CS5670 : Computer Vision - Cornell University · 2019-03-28 · •Midterm grades are out –Please submit any regrade requests via Gradescope –Solutions will be available soon](https://reader034.vdocument.in/reader034/viewer/2022042601/5fa1950671e1753a9e26cd34/html5/thumbnails/8.jpg)
Fundamental matrix
• Two Special points: e1 and e2 (the epipoles): projection of one camera into the other
epipolar plane
epipolar lineepipolar line
0
(projection of ray)
![Page 9: CS5670 : Computer Vision - Cornell University · 2019-03-28 · •Midterm grades are out –Please submit any regrade requests via Gradescope –Solutions will be available soon](https://reader034.vdocument.in/reader034/viewer/2022042601/5fa1950671e1753a9e26cd34/html5/thumbnails/9.jpg)
Fundamental matrix
• Two Special points: e1 and e2 (the epipoles): projection of one camera into the other
• All of the epipolar lines in an image pass through the epipole
0
![Page 10: CS5670 : Computer Vision - Cornell University · 2019-03-28 · •Midterm grades are out –Please submit any regrade requests via Gradescope –Solutions will be available soon](https://reader034.vdocument.in/reader034/viewer/2022042601/5fa1950671e1753a9e26cd34/html5/thumbnails/10.jpg)
Epipoles
![Page 11: CS5670 : Computer Vision - Cornell University · 2019-03-28 · •Midterm grades are out –Please submit any regrade requests via Gradescope –Solutions will be available soon](https://reader034.vdocument.in/reader034/viewer/2022042601/5fa1950671e1753a9e26cd34/html5/thumbnails/11.jpg)
Properties of the Fundamental Matrix
• is the epipolar line associated with
• is the epipolar line associated with
• and
• is rank 2
• How many parameters does F have?11
T
![Page 12: CS5670 : Computer Vision - Cornell University · 2019-03-28 · •Midterm grades are out –Please submit any regrade requests via Gradescope –Solutions will be available soon](https://reader034.vdocument.in/reader034/viewer/2022042601/5fa1950671e1753a9e26cd34/html5/thumbnails/12.jpg)
Fundamental matrix
• Why does F exist?
• Let’s derive it…
0
![Page 13: CS5670 : Computer Vision - Cornell University · 2019-03-28 · •Midterm grades are out –Please submit any regrade requests via Gradescope –Solutions will be available soon](https://reader034.vdocument.in/reader034/viewer/2022042601/5fa1950671e1753a9e26cd34/html5/thumbnails/13.jpg)
Fundamental matrix – calibrated case
0
: intrinsics of camera 1 : intrinsics of camera 2
: rotation of image 2 w.r.t. camera 1
: ray through p in camera 1’s (and world) coordinate system
: ray through q in camera 2’s coordinate system
![Page 14: CS5670 : Computer Vision - Cornell University · 2019-03-28 · •Midterm grades are out –Please submit any regrade requests via Gradescope –Solutions will be available soon](https://reader034.vdocument.in/reader034/viewer/2022042601/5fa1950671e1753a9e26cd34/html5/thumbnails/14.jpg)
Fundamental matrix – calibrated case
• , , and are coplanar
• epipolar plane can be represented as
0
![Page 15: CS5670 : Computer Vision - Cornell University · 2019-03-28 · •Midterm grades are out –Please submit any regrade requests via Gradescope –Solutions will be available soon](https://reader034.vdocument.in/reader034/viewer/2022042601/5fa1950671e1753a9e26cd34/html5/thumbnails/15.jpg)
Fundamental matrix – calibrated case
0
![Page 16: CS5670 : Computer Vision - Cornell University · 2019-03-28 · •Midterm grades are out –Please submit any regrade requests via Gradescope –Solutions will be available soon](https://reader034.vdocument.in/reader034/viewer/2022042601/5fa1950671e1753a9e26cd34/html5/thumbnails/16.jpg)
Fundamental matrix – calibrated case
• One more substitution:– Cross product with t (on left) can be represented as a 3x3 matrix
0
![Page 17: CS5670 : Computer Vision - Cornell University · 2019-03-28 · •Midterm grades are out –Please submit any regrade requests via Gradescope –Solutions will be available soon](https://reader034.vdocument.in/reader034/viewer/2022042601/5fa1950671e1753a9e26cd34/html5/thumbnails/17.jpg)
Fundamental matrix – calibrated case
0
![Page 18: CS5670 : Computer Vision - Cornell University · 2019-03-28 · •Midterm grades are out –Please submit any regrade requests via Gradescope –Solutions will be available soon](https://reader034.vdocument.in/reader034/viewer/2022042601/5fa1950671e1753a9e26cd34/html5/thumbnails/18.jpg)
Fundamental matrix – calibrated case
0
: ray through p in camera 1’s (and world) coordinate system
: ray through q in camera 2’s coordinate system
{
the Essential matrix
![Page 19: CS5670 : Computer Vision - Cornell University · 2019-03-28 · •Midterm grades are out –Please submit any regrade requests via Gradescope –Solutions will be available soon](https://reader034.vdocument.in/reader034/viewer/2022042601/5fa1950671e1753a9e26cd34/html5/thumbnails/19.jpg)
Cross-product as linear operator
Useful fact: Cross product with a vector t can be represented as multiplication with a (skew-symmetric) 3x3 matrix
![Page 20: CS5670 : Computer Vision - Cornell University · 2019-03-28 · •Midterm grades are out –Please submit any regrade requests via Gradescope –Solutions will be available soon](https://reader034.vdocument.in/reader034/viewer/2022042601/5fa1950671e1753a9e26cd34/html5/thumbnails/20.jpg)
Fundamental matrix – uncalibrated case
0
the Fundamental matrix
: intrinsics of camera 1 : intrinsics of camera 2
: rotation of image 2 w.r.t. camera 1
![Page 21: CS5670 : Computer Vision - Cornell University · 2019-03-28 · •Midterm grades are out –Please submit any regrade requests via Gradescope –Solutions will be available soon](https://reader034.vdocument.in/reader034/viewer/2022042601/5fa1950671e1753a9e26cd34/html5/thumbnails/21.jpg)
Rectified case
![Page 22: CS5670 : Computer Vision - Cornell University · 2019-03-28 · •Midterm grades are out –Please submit any regrade requests via Gradescope –Solutions will be available soon](https://reader034.vdocument.in/reader034/viewer/2022042601/5fa1950671e1753a9e26cd34/html5/thumbnails/22.jpg)
Stereo image rectification
• reproject image planes onto a common• plane parallel to the line between optical centers• pixel motion is horizontal after this transformation• two homographies (3x3 transform), one for each input
image reprojection➢ C. Loop and Z. Zhang. Computing Rectifying Homographies for Stereo
Vision. IEEE Conf. Computer Vision and Pattern Recognition, 1999.
![Page 23: CS5670 : Computer Vision - Cornell University · 2019-03-28 · •Midterm grades are out –Please submit any regrade requests via Gradescope –Solutions will be available soon](https://reader034.vdocument.in/reader034/viewer/2022042601/5fa1950671e1753a9e26cd34/html5/thumbnails/23.jpg)
Original stereo pair
After rectification
![Page 24: CS5670 : Computer Vision - Cornell University · 2019-03-28 · •Midterm grades are out –Please submit any regrade requests via Gradescope –Solutions will be available soon](https://reader034.vdocument.in/reader034/viewer/2022042601/5fa1950671e1753a9e26cd34/html5/thumbnails/24.jpg)
Relationship with homography?
Images taken from the same center of projection? Use a homography!
![Page 25: CS5670 : Computer Vision - Cornell University · 2019-03-28 · •Midterm grades are out –Please submit any regrade requests via Gradescope –Solutions will be available soon](https://reader034.vdocument.in/reader034/viewer/2022042601/5fa1950671e1753a9e26cd34/html5/thumbnails/25.jpg)
Questions?
![Page 26: CS5670 : Computer Vision - Cornell University · 2019-03-28 · •Midterm grades are out –Please submit any regrade requests via Gradescope –Solutions will be available soon](https://reader034.vdocument.in/reader034/viewer/2022042601/5fa1950671e1753a9e26cd34/html5/thumbnails/26.jpg)
Estimating F
• If we don’t know K1, K2, R, or t, can we estimate F for two images?
• Yes, given enough correspondences
![Page 27: CS5670 : Computer Vision - Cornell University · 2019-03-28 · •Midterm grades are out –Please submit any regrade requests via Gradescope –Solutions will be available soon](https://reader034.vdocument.in/reader034/viewer/2022042601/5fa1950671e1753a9e26cd34/html5/thumbnails/27.jpg)
Estimating F – 8-point algorithm
• The fundamental matrix F is defined by
0=Fxx'
for any pair of matches x and x’ in two images.
• Let x=(u,v,1)T and x’=(u’,v’,1)T,
=
333231
232221
131211
fff
fff
fff
F
each match gives a linear equation
0'''''' 333231232221131211 =++++++++ fvfuffvfvvfuvfufvufuu
![Page 28: CS5670 : Computer Vision - Cornell University · 2019-03-28 · •Midterm grades are out –Please submit any regrade requests via Gradescope –Solutions will be available soon](https://reader034.vdocument.in/reader034/viewer/2022042601/5fa1950671e1753a9e26cd34/html5/thumbnails/28.jpg)
8-point algorithm
0
1´´´´´´
1´´´´´´
1´´´´´´
33
32
31
23
22
21
13
12
11
222222222222
111111111111
=
f
f
f
f
f
f
f
f
f
vuvvvvuuuvuu
vuvvvvuuuvuu
vuvvvvuuuvuu
nnnnnnnnnnnn
• Like with homographies, instead of solving , we seek f to minimize , least eigenvector of .
0=AfAf
AA
![Page 29: CS5670 : Computer Vision - Cornell University · 2019-03-28 · •Midterm grades are out –Please submit any regrade requests via Gradescope –Solutions will be available soon](https://reader034.vdocument.in/reader034/viewer/2022042601/5fa1950671e1753a9e26cd34/html5/thumbnails/29.jpg)
8-point algorithm – Problem?
• F should have rank 2
• To enforce that F is of rank 2, F is replaced by F’ that minimizes subject to the rank constraint. 'FF −
• This is achieved by SVD. Let , where
, let
then is the solution.
= VUF Σ
=
3
2
1
00
00
00
Σ
=
000
00
00
Σ' 2
1
= VUF Σ''
![Page 30: CS5670 : Computer Vision - Cornell University · 2019-03-28 · •Midterm grades are out –Please submit any regrade requests via Gradescope –Solutions will be available soon](https://reader034.vdocument.in/reader034/viewer/2022042601/5fa1950671e1753a9e26cd34/html5/thumbnails/30.jpg)
8-point algorithm
% Build the constraint matrix
A = [x2(1,:)'.*x1(1,:)' x2(1,:)'.*x1(2,:)' x2(1,:)' ...
x2(2,:)'.*x1(1,:)' x2(2,:)'.*x1(2,:)' x2(2,:)' ...
x1(1,:)' x1(2,:)' ones(npts,1) ];
[U,D,V] = svd(A);
% Extract fundamental matrix from the column of V
% corresponding to the smallest singular value.
F = reshape(V(:,9),3,3)';
% Enforce rank2 constraint
[U,D,V] = svd(F);
F = U*diag([D(1,1) D(2,2) 0])*V';
![Page 31: CS5670 : Computer Vision - Cornell University · 2019-03-28 · •Midterm grades are out –Please submit any regrade requests via Gradescope –Solutions will be available soon](https://reader034.vdocument.in/reader034/viewer/2022042601/5fa1950671e1753a9e26cd34/html5/thumbnails/31.jpg)
8-point algorithm
• Pros: it is linear, easy to implement and fast
• Cons: susceptible to noise
![Page 32: CS5670 : Computer Vision - Cornell University · 2019-03-28 · •Midterm grades are out –Please submit any regrade requests via Gradescope –Solutions will be available soon](https://reader034.vdocument.in/reader034/viewer/2022042601/5fa1950671e1753a9e26cd34/html5/thumbnails/32.jpg)
0
1´´´´´´
1´´´´´´
1´´´´´´
33
32
31
23
22
21
13
12
11
222222222222
111111111111
=
f
f
f
f
f
f
f
f
f
vuvvvvuuuvuu
vuvvvvuuuvuu
vuvvvvuuuvuu
nnnnnnnnnnnn
Problem with 8-point algorithm
~10000 ~10000 ~10000 ~10000~100 ~100 1~100 ~100
!
Orders of magnitude difference
between column of data matrix
→ least-squares yields poor results
![Page 33: CS5670 : Computer Vision - Cornell University · 2019-03-28 · •Midterm grades are out –Please submit any regrade requests via Gradescope –Solutions will be available soon](https://reader034.vdocument.in/reader034/viewer/2022042601/5fa1950671e1753a9e26cd34/html5/thumbnails/33.jpg)
Normalized 8-point algorithm
(0,0)
(700,500)
(700,0)
(0,500)
(1,-1)
(0,0)
(1,1)(-1,1)
(-1,-1)
−
−
1
1500
2
10700
2
normalized least squares yields good results
Transform image to ~[-1,1]x[-1,1]
![Page 34: CS5670 : Computer Vision - Cornell University · 2019-03-28 · •Midterm grades are out –Please submit any regrade requests via Gradescope –Solutions will be available soon](https://reader034.vdocument.in/reader034/viewer/2022042601/5fa1950671e1753a9e26cd34/html5/thumbnails/34.jpg)
Normalized 8-point algorithm
1. Transform input by ,
2. Call 8-point on to obtain
3.
ii Txx =ˆ'
i
'
i Txx =ˆ
'
ii xx ˆ,ˆ
TFTF ˆΤ'=
F̂
0=Fxx'
0ˆ'ˆ 1 =−−xFTTx'
F̂
![Page 35: CS5670 : Computer Vision - Cornell University · 2019-03-28 · •Midterm grades are out –Please submit any regrade requests via Gradescope –Solutions will be available soon](https://reader034.vdocument.in/reader034/viewer/2022042601/5fa1950671e1753a9e26cd34/html5/thumbnails/35.jpg)
Normalized 8-point algorithm
A = [x2(1,:)'.*x1(1,:)' x2(1,:)'.*x1(2,:)' x2(1,:)' ...
x2(2,:)'.*x1(1,:)' x2(2,:)'.*x1(2,:)' x2(2,:)' ...
x1(1,:)' x1(2,:)' ones(npts,1) ];
[U,D,V] = svd(A);
F = reshape(V(:,9),3,3)';
[U,D,V] = svd(F);
F = U*diag([D(1,1) D(2,2) 0])*V';
% Denormalise
F = T2'*F*T1;
[x1, T1] = normalise2dpts(x1);
[x2, T2] = normalise2dpts(x2);
![Page 36: CS5670 : Computer Vision - Cornell University · 2019-03-28 · •Midterm grades are out –Please submit any regrade requests via Gradescope –Solutions will be available soon](https://reader034.vdocument.in/reader034/viewer/2022042601/5fa1950671e1753a9e26cd34/html5/thumbnails/36.jpg)
Results (ground truth)
![Page 37: CS5670 : Computer Vision - Cornell University · 2019-03-28 · •Midterm grades are out –Please submit any regrade requests via Gradescope –Solutions will be available soon](https://reader034.vdocument.in/reader034/viewer/2022042601/5fa1950671e1753a9e26cd34/html5/thumbnails/37.jpg)
Results (8-point algorithm)
![Page 38: CS5670 : Computer Vision - Cornell University · 2019-03-28 · •Midterm grades are out –Please submit any regrade requests via Gradescope –Solutions will be available soon](https://reader034.vdocument.in/reader034/viewer/2022042601/5fa1950671e1753a9e26cd34/html5/thumbnails/38.jpg)
Results (normalized 8-point algorithm)
![Page 39: CS5670 : Computer Vision - Cornell University · 2019-03-28 · •Midterm grades are out –Please submit any regrade requests via Gradescope –Solutions will be available soon](https://reader034.vdocument.in/reader034/viewer/2022042601/5fa1950671e1753a9e26cd34/html5/thumbnails/39.jpg)
What about more than two views?
• The geometry of three views is described by a 3 x 3 x 3 tensor called the trifocal tensor
• The geometry of four views is described by a 3 x 3 x 3 x 3 tensor called the quadrifocaltensor
• After this it starts to get complicated…
![Page 40: CS5670 : Computer Vision - Cornell University · 2019-03-28 · •Midterm grades are out –Please submit any regrade requests via Gradescope –Solutions will be available soon](https://reader034.vdocument.in/reader034/viewer/2022042601/5fa1950671e1753a9e26cd34/html5/thumbnails/40.jpg)
Large-scale structure from motion
Dubrovnik, Croatia. 4,619 images (out of an initial 57,845).Total reconstruction time: 23 hoursNumber of cores: 352