Download - Assignment 2
![Page 1: Assignment 2](https://reader036.vdocument.in/reader036/viewer/2022070415/56814f01550346895dbc9438/html5/thumbnails/1.jpg)
Assignment 2Compute F automatically from image
pair
(putative matches, 8-point, 7-point, iterative, RANSAC, guided matching)
(due by Wednesday 19/03/03)
![Page 2: Assignment 2](https://reader036.vdocument.in/reader036/viewer/2022070415/56814f01550346895dbc9438/html5/thumbnails/2.jpg)
Rectification and structure computation
class 15
Multiple View GeometryComp 290-089Marc Pollefeys
![Page 3: Assignment 2](https://reader036.vdocument.in/reader036/viewer/2022070415/56814f01550346895dbc9438/html5/thumbnails/3.jpg)
Multiple View Geometry course schedule(subject to change)
Jan. 7, 9 Intro & motivation Projective 2D Geometry
Jan. 14, 16
(no class) Projective 2D Geometry
Jan. 21, 23
Projective 3D Geometry (no class)
Jan. 28, 30
Parameter Estimation Parameter Estimation
Feb. 4, 6 Algorithm Evaluation Camera Models
Feb. 11, 13
Camera Calibration Single View Geometry
Feb. 18, 20
Epipolar Geometry 3D reconstruction
Feb. 25, 27
Fund. Matrix Comp. Fund. Matrix Comp.
Mar. 4, 6 Structure Comp. Planes & Homographies
Mar. 18, 20
Trifocal Tensor Three View Reconstruction
Mar. 25, 27
Multiple View Geometry
MultipleView Reconstruction
Apr. 1, 3 Bundle adjustment Papers
Apr. 8, 10
Auto-Calibration Papers
Apr. 15, 17
Dynamic SfM Papers
Apr. 22, 24
Cheirality Project Demos
![Page 4: Assignment 2](https://reader036.vdocument.in/reader036/viewer/2022070415/56814f01550346895dbc9438/html5/thumbnails/4.jpg)
Two-view geometry
Epipolar geometry
3D reconstruction
F-matrix comp.
Structure comp.
![Page 5: Assignment 2](https://reader036.vdocument.in/reader036/viewer/2022070415/56814f01550346895dbc9438/html5/thumbnails/5.jpg)
Automatic computation of F
(i) Interest points(ii) Putative correspondences(iii) RANSAC (iv) Non-linear re-estimation of F(v) Guided matching(repeat (iv) and (v) until stable)
![Page 6: Assignment 2](https://reader036.vdocument.in/reader036/viewer/2022070415/56814f01550346895dbc9438/html5/thumbnails/6.jpg)
Select strongest features (e.g. 1000/image)
Feature points
![Page 7: Assignment 2](https://reader036.vdocument.in/reader036/viewer/2022070415/56814f01550346895dbc9438/html5/thumbnails/7.jpg)
Evaluate ZNCC,SSD,SAD for all features with similar coordinates
Keep mutual best matchesKeep mutual best matches
Still many wrong matches!Still many wrong matches!
10101010 ,,´´, e.g. hhww yyxxyx
?
Feature matching
![Page 8: Assignment 2](https://reader036.vdocument.in/reader036/viewer/2022070415/56814f01550346895dbc9438/html5/thumbnails/8.jpg)
Step 1. Extract featuresStep 2. Compute a set of potential matchesStep 3. do
Step 3.1 select minimal sample (i.e. 7 matches)
Step 3.2 compute solution(s) for F
Step 3.3 determine inliers
until (#inliers,#samples)<95%
samples#7)1(1
matches#inliers#
#inliers 90%
80%
70% 60%
50%
#samples
5 13 35 106 382
Step 4. Compute F based on all inliersStep 5. Look for additional matchesStep 6. Refine F based on all correct matches
(generate hypothesis)
(verify hypothesis)
RANSAC
![Page 9: Assignment 2](https://reader036.vdocument.in/reader036/viewer/2022070415/56814f01550346895dbc9438/html5/thumbnails/9.jpg)
restrict search range to neighborhood of epipolar line (1.5 pixels)
relax disparity restriction (along epipolar line)
Finding more matches: guided matching
![Page 10: Assignment 2](https://reader036.vdocument.in/reader036/viewer/2022070415/56814f01550346895dbc9438/html5/thumbnails/10.jpg)
geometric relations between two views is fully
described by recovered 3x3 matrix F
two-view geometry
![Page 11: Assignment 2](https://reader036.vdocument.in/reader036/viewer/2022070415/56814f01550346895dbc9438/html5/thumbnails/11.jpg)
Image pair rectification
simplify stereo matching by warping the images
Apply projective transformation so that epipolar linescorrespond to horizontal scanlines
e
e
map epipole e to (1,0,0)
try to minimize image distortion
problem when epipole in (or close to) the image
He001
![Page 12: Assignment 2](https://reader036.vdocument.in/reader036/viewer/2022070415/56814f01550346895dbc9438/html5/thumbnails/12.jpg)
Planar rectification
Bring two views Bring two views to standard stereo setupto standard stereo setup
(moves epipole to )(not possible when in/close to image)
~ image size
(calibrated)(calibrated)
Distortion minimization(uncalibrated)
(standard approach)
![Page 13: Assignment 2](https://reader036.vdocument.in/reader036/viewer/2022070415/56814f01550346895dbc9438/html5/thumbnails/13.jpg)
![Page 14: Assignment 2](https://reader036.vdocument.in/reader036/viewer/2022070415/56814f01550346895dbc9438/html5/thumbnails/14.jpg)
![Page 15: Assignment 2](https://reader036.vdocument.in/reader036/viewer/2022070415/56814f01550346895dbc9438/html5/thumbnails/15.jpg)
Polar re-parameterization around epipoles
Requires only (oriented) epipolar geometry
Preserve length of epipolar linesChoose so that no pixels are
compressed
original image rectified image
Polar rectification(Pollefeys et al. ICCV’99)
Works for all relative motionsGuarantees minimal image size
![Page 16: Assignment 2](https://reader036.vdocument.in/reader036/viewer/2022070415/56814f01550346895dbc9438/html5/thumbnails/16.jpg)
polar rectification: example
![Page 17: Assignment 2](https://reader036.vdocument.in/reader036/viewer/2022070415/56814f01550346895dbc9438/html5/thumbnails/17.jpg)
polar rectification: example
![Page 18: Assignment 2](https://reader036.vdocument.in/reader036/viewer/2022070415/56814f01550346895dbc9438/html5/thumbnails/18.jpg)
Example: Béguinage of Leuven
Does not work with standard Homography-based approaches
![Page 19: Assignment 2](https://reader036.vdocument.in/reader036/viewer/2022070415/56814f01550346895dbc9438/html5/thumbnails/19.jpg)
Example: Béguinage of Leuven
![Page 20: Assignment 2](https://reader036.vdocument.in/reader036/viewer/2022070415/56814f01550346895dbc9438/html5/thumbnails/20.jpg)
Stereo matching
• attempt to match every pixel• use additional constraints
![Page 21: Assignment 2](https://reader036.vdocument.in/reader036/viewer/2022070415/56814f01550346895dbc9438/html5/thumbnails/21.jpg)
Exploiting motion and scene constraints
• Ordering constraint• Uniqueness constraint• Disparity limit• Disparity continuity constraint
• Epipolar constraint Epipolar constraint (through rectification)
![Page 22: Assignment 2](https://reader036.vdocument.in/reader036/viewer/2022070415/56814f01550346895dbc9438/html5/thumbnails/22.jpg)
Ordering constraint
11 22 33 4,54,5 66 11 2,32,3 44 55 66
2211 33 4,54,5 6611
2,32,3
44
55
66
surface slicesurface slice surface as a pathsurface as a path
occlusion right
occlusion left
![Page 23: Assignment 2](https://reader036.vdocument.in/reader036/viewer/2022070415/56814f01550346895dbc9438/html5/thumbnails/23.jpg)
Uniqueness constraint
• In an image pair each pixel has at most one corresponding pixel• In general one corresponding pixel• In case of occlusion there is none
![Page 24: Assignment 2](https://reader036.vdocument.in/reader036/viewer/2022070415/56814f01550346895dbc9438/html5/thumbnails/24.jpg)
Disparity constraint
surface slicesurface slice surface as a pathsurface as a path
bounding box
dispa
rity b
and
use reconsructed features to determine bounding box
constantdisparitysurfaces
![Page 25: Assignment 2](https://reader036.vdocument.in/reader036/viewer/2022070415/56814f01550346895dbc9438/html5/thumbnails/25.jpg)
Disparity continuity constraint
• Assume piecewise continuous surface
piecewise continuous disparity• In general disparity changes
continuously• discontinuities at occluding
boundaries
![Page 26: Assignment 2](https://reader036.vdocument.in/reader036/viewer/2022070415/56814f01550346895dbc9438/html5/thumbnails/26.jpg)
Stereo matching
Optimal path(dynamic programming )
Similarity measure(SSD or NCC)
Constraints• epipolar
• ordering
• uniqueness
• disparity limit
• disparity gradient limit
Trade-off
• Matching cost (data)
• Discontinuities (prior)
(Cox et al. CVGIP’96; Koch’96; Falkenhagen´97; Van Meerbergen,Vergauwen,Pollefeys,VanGool IJCV‘02)
![Page 27: Assignment 2](https://reader036.vdocument.in/reader036/viewer/2022070415/56814f01550346895dbc9438/html5/thumbnails/27.jpg)
Hierarchical stereo matching
Dow
nsam
plin
g
(Gau
ssia
n p
yra
mid
)
Dis
pari
ty p
rop
ag
ati
on
Allows faster computation
Deals with large disparity ranges
(Falkenhagen´97;Van Meerbergen,Vergauwen,Pollefeys,VanGool IJCV‘02)
![Page 28: Assignment 2](https://reader036.vdocument.in/reader036/viewer/2022070415/56814f01550346895dbc9438/html5/thumbnails/28.jpg)
Disparity map
image I(x,y) image I´(x´,y´)Disparity map D(x,y)
(x´,y´)=(x+D(x,y),y)
![Page 29: Assignment 2](https://reader036.vdocument.in/reader036/viewer/2022070415/56814f01550346895dbc9438/html5/thumbnails/29.jpg)
Example: reconstruct image from neighboring
images
![Page 30: Assignment 2](https://reader036.vdocument.in/reader036/viewer/2022070415/56814f01550346895dbc9438/html5/thumbnails/30.jpg)
Multi-view depth fusion
• Compute depth for every pixel of reference image• Triangulation• Use multiple views• Up- and down
sequence• Use Kalman filter
(Koch, Pollefeys and Van Gool. ECCV‘98)
Allows to compute robust texture
![Page 31: Assignment 2](https://reader036.vdocument.in/reader036/viewer/2022070415/56814f01550346895dbc9438/html5/thumbnails/31.jpg)
Point reconstruction
PXx XP'x'
![Page 32: Assignment 2](https://reader036.vdocument.in/reader036/viewer/2022070415/56814f01550346895dbc9438/html5/thumbnails/32.jpg)
linear triangulation
XP'x'PXx
0XP'x
0XpXp0XpXp0XpXp
1T2T
2T3T
1T3T
yxyx
2T3T
1T3T
2T3T
1T3T
p'p''p'p''pppp
A
yxyx
0AX
homogeneous
1X
)1,,,( ZYX
inhomogeneous
invariance?
e)(HX)(AH-1
algebraic error yes, constraint no (except for affine)
![Page 33: Assignment 2](https://reader036.vdocument.in/reader036/viewer/2022070415/56814f01550346895dbc9438/html5/thumbnails/33.jpg)
geometric error
0x̂F'x̂ subject to )'x̂,(x')x̂(x, T22 dd
X̂P''x̂ and X̂Px̂ subject toly equivalentor
possibility to compute using LM (for 2 or more points)
or directly (for 2 points)
![Page 34: Assignment 2](https://reader036.vdocument.in/reader036/viewer/2022070415/56814f01550346895dbc9438/html5/thumbnails/34.jpg)
Geometric error
Reconstruct matches in projective frame by minimizing the reprojection error
(see Hartley&Sturm,CVIU´97)Non-iterative optimal solution
![Page 35: Assignment 2](https://reader036.vdocument.in/reader036/viewer/2022070415/56814f01550346895dbc9438/html5/thumbnails/35.jpg)
Optimal 3D point in epipolar plane
Given an epipolar plane, find best 3D point for (x1,x2)
x1
x2
l1 l2
l1x1
x2l2
x1´
x2´
Select closest points (x1´,x2´) on epipolar lines
Obtain 3D point through exact triangulationGuarantees minimal reprojection error (given this epipolar plane)
![Page 36: Assignment 2](https://reader036.vdocument.in/reader036/viewer/2022070415/56814f01550346895dbc9438/html5/thumbnails/36.jpg)
Optimal epipolar plane
• Reconstruct matches in projective frame by minimizing the reprojection error
• Non-iterative methodDetermine the epipolar plane for reconstruction
Reconstruct optimal point from selected epipolar plane
222
211 XP,xXP,x dd
(Hartley and Sturm, CVIU´97)
222
211 αl,xαl,x DD
(polynomial of degree 6check all minima, incl ∞)
m1
m2
l1 l2
3DOF
1DOF
![Page 37: Assignment 2](https://reader036.vdocument.in/reader036/viewer/2022070415/56814f01550346895dbc9438/html5/thumbnails/37.jpg)
Reconstruction uncertainty
consider angle between rays
![Page 38: Assignment 2](https://reader036.vdocument.in/reader036/viewer/2022070415/56814f01550346895dbc9438/html5/thumbnails/38.jpg)
Line reconstruction
P'l'Pl
T
T
L
doesn‘t work for epipolar plane
![Page 39: Assignment 2](https://reader036.vdocument.in/reader036/viewer/2022070415/56814f01550346895dbc9438/html5/thumbnails/39.jpg)
Next class: Scene and plane homographies