![Page 1: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/1.jpg)
![Page 2: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/2.jpg)
Chaplin, Modern Times, 1936
![Page 3: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/3.jpg)
[A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection set]
![Page 4: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/4.jpg)
Two cameras, simultaneous
views
Single moving camera and
static scene
Computer vision as world measurement
![Page 5: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/5.jpg)
Multiple view geometry
Hartley and Zisserman
Lowe
Camera calibration
Epipolar geometry
Dense depth
map estimation
Owiw
kw
jwR,t
X
x XtRKx =
![Page 6: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/6.jpg)
Multi-view geometry problems
• Camera ‘Motion’: Given a set of corresponding 2D/3D
points in two or more images, compute the camera
parameters.
Camera 1Camera 2 Camera 3
R1,t1 R2,t2R3,t3? ? ? Slide credit:
Noah Snavely
![Page 7: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/7.jpg)
Multi-view geometry problems
• Stereo correspondence: Given known camera
parameters and a point in one of the images, where could
its corresponding points be in the other images?
Camera 3
R3,t3
Camera 1Camera 2
R1,t1 R2,t2Slide credit:
Noah Snavely
![Page 8: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/8.jpg)
Multi-view geometry problems
• Structure from Motion: Given projections of the same 3D
point in two or more images, compute the 3D coordinates
of that point
Camera 3
R3,t3 Slide credit:
Noah Snavely
?
Camera 1Camera 2
R1,t1 R2,t2? ? ?
![Page 9: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/9.jpg)
Multi-view geometry problems
• Optical flow: Given two images, find the location of a world
point in a second close-by image with no camera info.
Camera 1
Camera 2
![Page 10: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/10.jpg)
Essential Matrix
(Longuet-Higgins, 1981)
Essential matrix
0)]ˆ([ˆ = xRtx RtExExT
== with0ˆˆ
X
x x’
E is a 3x3 matrix which relates
corresponding pairs of normalized
homogeneous image points across pairs of
images – for K calibrated cameras.
Estimates relative position/orientation. Note: [t]× is matrix representation of cross product
ො𝑥′ො𝑥
![Page 11: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/11.jpg)
Fundamental matrix for uncalibrated cases
1with0
−− == KEKFxFxTT
• F x’ = 0 is the epipolar line l associated with x’
• FTx = 0 is the epipolar line l’ associated with x
• F is singular (rank two): det(F)=0
• F e’ = 0 and FTe = 0 (nullspaces of F = e’; nullspace of FT = e’)
• F has seven degrees of freedom: 9 entries but defined up to scale, det(F)=0
X
x x’
Intrinsics K
Intrinsics K’
![Page 12: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/12.jpg)
Fundamental matrix
Let x be a point in left image, x’ in right image
Epipolar relation
• x maps to epipolar line l’
• x’ maps to epipolar line l
Epipolar mapping described by a 3x3 matrix F:
It follows that:
l’l
x x’
𝑙′ = 𝐹𝑥𝑙 = 𝐹𝑇𝑥′
𝑥′𝐹𝑥 = 0
![Page 13: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/13.jpg)
Fundamental matrix
This matrix F is called
• the “Essential Matrix”
– when image intrinsic parameters are known
• the “Fundamental Matrix”
– more generally (uncalibrated case)
Can solve for F from point correspondences
• Each (x, x’) pair gives one linear equation in entries of F
• F has 9 entries, but really only 7 degrees of freedom.
• With 8 points it is simple to solve for F, but it is also possible
with 7. See Marc Pollefey’s notes for a nice tutorial
𝑥′𝐹𝑥 = 0
![Page 14: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/14.jpg)
VLFeat’s 800 most confident matches among 10,000+ local features.
![Page 15: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/15.jpg)
Algorithm:
1. Sample (randomly) the number of points required to fit the model (s=2)
2. Solve for model parameters using samples
3. Score by the fraction of inliers within a preset threshold of the model
Repeat 1-3 until the best model is found with high confidence
RANSAC
14=InliersN
![Page 16: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/16.jpg)
Epipolar lines
![Page 17: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/17.jpg)
Keep only the matches at are “inliers” with respect to the “best” fundamental matrix
![Page 18: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/18.jpg)
Stereo image rectification
![Page 19: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/19.jpg)
Stereo image rectification
• Reproject image planes onto a common plane parallel to the line between camera centers
• Pixel motion is horizontal after this transformation
• Two homographies (3x3 transform), one for each input image reprojection
➢ C. Loop and Z. Zhang. Computing Rectifying Homographies for Stereo Vision. IEEE Conf. Computer Vision and Pattern Recognition, 1999.
![Page 20: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/20.jpg)
Rectification example
![Page 21: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/21.jpg)
A photon’s life choices
• Absorption
• Diffusion
• Reflection
• Transparency
• Refraction
• Fluorescence
• Subsurface scattering
• Phosphorescence
• Interreflection
λ
light source
?
James Hays
![Page 22: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/22.jpg)
A photon’s life choices
• Absorption
• Diffusion
• Reflection
• Transparency
• Refraction
• Fluorescence
• Subsurface scattering
• Phosphorescence
• Interreflection
λ
light source
James Hays
![Page 23: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/23.jpg)
A photon’s life choices
• Absorption
• Diffuse Reflection
• Reflection
• Transparency
• Refraction
• Fluorescence
• Subsurface scattering
• Phosphorescence
• Interreflection
λ
light source
James Hays
Perfect diffuse
= Lambertian
= Equal in all directions
![Page 24: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/24.jpg)
A photon’s life choices
• Absorption
• Diffusion
• Specular Reflection
• Transparency
• Refraction
• Fluorescence
• Subsurface scattering
• Phosphorescence
• Interreflection
λ
light source
James Hays
Perfect specular
= mirror reflection
= only one direction
![Page 25: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/25.jpg)
A photon’s life choices
• Absorption
• Diffusion
• Specular (Glossy) Reflection
• Transparency
• Refraction
• Fluorescence
• Subsurface scattering
• Phosphorescence
• Interreflection
λ
light source
James Hays
Glossy reflection
= ‘specular lobe’
= varying across directions
![Page 26: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/26.jpg)
A photon’s life choices
• Absorption
• Diffusion
• Reflection
• Transparency
• Refraction
• Fluorescence
• Subsurface scattering
• Phosphorescence
• Interreflection
λ
light source
James Hays
![Page 27: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/27.jpg)
A photon’s life choices
• Absorption
• Diffusion
• Reflection
• Transparency
• Refraction
• Fluorescence
• Subsurface scattering
• Phosphorescence
• Interreflection
λ
light source
James Hays
![Page 28: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/28.jpg)
A photon’s life choices
• Absorption
• Diffusion
• Reflection
• Transparency
• Refraction
• Fluorescence
• Subsurface scattering
• Phosphorescence
• Interreflection
λ1
light source
λ2
James Hays
![Page 29: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/29.jpg)
A photon’s life choices
• Absorption
• Diffusion
• Reflection
• Transparency
• Refraction
• Fluorescence
• Subsurface scattering
• Phosphorescence
• Interreflection
λ
light source
James Hays
![Page 30: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/30.jpg)
A photon’s life choices
• Absorption
• Diffusion
• Reflection
• Transparency
• Refraction
• Fluorescence
• Subsurface scattering
• Phosphorescence
• Interreflection
t=1
light source
t=n
James Hays
![Page 31: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/31.jpg)
A photon’s life choices
• Absorption
• Diffusion
• Reflection
• Transparency
• Refraction
• Fluorescence
• Subsurface scattering
• Phosphorescence
• Interreflection
λ
light source
(Specular Interreflection)
James Hays
![Page 32: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/32.jpg)
Lambertian Reflectance
In computer vision, surfaces are often assumed to be ideal diffuse reflectors with no dependence on viewing direction.
This is obviously nonsense, but a useful model!
James Hays
![Page 33: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/33.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Correspondence problem
Multiple match
hypotheses
satisfy epipolar
constraint, but
which is correct?
Figure from Gee & Cipolla 1999
![Page 34: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/34.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Dense correspondence search
For each epipolar line:
For each pixel / window in the left image:
• Compare with every pixel / window on same
epipolar line in right image• Pick position with minimum match cost (e.g., SSD,
normalized correlation)
Adapted from Li Zhang
![Page 35: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/35.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Think-Pair-Share
How can we solve this problem?
For which ‘real-world’ phenomena will this work?
For which will it not?
![Page 36: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/36.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Correspondence problem
Source: Andrew Zisserman
Intensity
profiles
![Page 37: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/37.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Correspondence problem
Neighborhoods of corresponding points are
similar in intensity patterns.
Source: Andrew Zisserman
![Page 38: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/38.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Correlation-based window matching
![Page 39: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/39.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Correlation-based window matching
![Page 40: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/40.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Correlation-based window matching
![Page 41: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/41.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Correlation-based window matching
![Page 42: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/42.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Correlation-based window matching
???
Textureless regions are non-distinct; high ambiguity for matches.
![Page 43: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/43.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
W = 3 W = 20
Figures from Li Zhang
Want window large enough to have sufficient intensity
variation, yet small enough to contain only pixels with
about the same disparity.
Effect of window size
![Page 44: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/44.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Problem: Occlusion
• Uniqueness says “up to match” per pixel
• When is there no match?
Occluded pixels
![Page 45: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/45.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Disparity gradient constraint
• Assume piecewise continuous surface, so want disparity
estimates to be locally smooth
Figure from Gee &
Cipolla 1999
![Page 46: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/46.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Ordering constraint
• Points on same surface (opaque object) will be in same
order in both views
Figure from Gee &
Cipolla 1999
![Page 47: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/47.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Ordering constraint
Figures from Forsyth & Ponce
• Won’t always hold, e.g. consider transparent object, or
an occluding surface
![Page 48: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/48.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Stereo – Tsukuba test scene (now old)
![Page 49: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/49.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Results with window search
Window-based matching
(best window size)
‘Ground truth’
![Page 50: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/50.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Better solutions
• Beyond individual correspondences to estimate
disparities:
• Optimize correspondence assignments jointly
• Scanline at a time (DP)
• Full 2D grid (graph cuts)
![Page 51: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/51.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Scanline stereo
• Try to coherently match pixels on the entire scanline
• Different scanlines are still optimized independently
Left image Right image
inte
nsit
y
![Page 52: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/52.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
“Shortest paths” for scan-line stereoLeft image
Right image
Can be implemented with dynamic programming
Ohta & Kanade ’85, Cox et al. ’96, Intille & Bobick, ‘01
leftS
rightS
q
p
t
s
I
I
Slide credit: Y. Boykov
Right
occlusion
Left
occlusion
Left
occ
lusi
on
Rightocclusion
one-to-one
![Page 53: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/53.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Coherent stereo on 2D grid• Scanline stereo generates streaking artifacts
• Can’t use dynamic programming to find spatially
coherent disparities/ correspondences on a 2D grid
![Page 54: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/54.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Stereo as energy minimization
• What defines a good stereo correspondence?
1. Match quality
• Want each pixel to find a good match in the other image
2. Smoothness
• If two pixels are adjacent, they should (usually) move about
the same amount
![Page 55: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/55.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Stereo matching as energy minimization
I1I2 D
Energy functions of this form can be minimized using graph cuts.
Y. Boykov, O. Veksler, and R. Zabih, Fast Approximate Energy Minimization via Graph Cuts, PAMI 2001
W1(i) W2(i+D(i)) D(i)
)(),,( smooth21data DEDIIEE +=
( ) −=ji
jDiDE,neighbors
smooth )()(( )2
21data ))(()( +−=i
iDiWiWE
Source: Steve Seitz
![Page 56: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/56.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Better results…
Graph cut methodBoykov et al., Fast Approximate Energy Minimization via Graph Cuts,
International Conference on Computer Vision, September 1999.
Ground truth
For the latest and greatest: http://www.middlebury.edu/stereo/
![Page 57: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/57.jpg)
Motion and Optic FlowCS 4495 Computer Vision – A. Bobick
Challenges
• Low-contrast ‘textureless’ image regions
• Occlusions
• Violations of brightness constancy
• Specular reflections
• Really large baselines
• Foreshortening and appearance change
• Camera calibration errors
![Page 58: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/58.jpg)
SIFT + Fundamental Matrix + RANSAC + Sparse correspondence
![Page 59: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/59.jpg)
SIFT + Fundamental Matrix + RANSAC + dense correspondence
Building Rome in a Day
By Sameer Agarwal, Yasutaka Furukawa, Noah Snavely, Ian Simon, Brian Curless, Steven M. Seitz, Richard Szeliski
Communications of the ACM, Vol. 54 No. 10, Pages 105-112 2009
![Page 60: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/60.jpg)
Building Rome in a Day
By Sameer Agarwal, Yasutaka Furukawa, Noah Snavely, Ian Simon, Brian Curless, Steven M. Seitz, Richard Szeliski
Communications of the ACM, Vol. 54 No. 10, Pages 105-112
SIFT + Fundamental Matrix + RANSAC + dense correspondence
![Page 61: Chaplin, Modern Times, 1936 · 2020. 4. 6. · Chaplin, Modern Times, 1936 [A Bucket of Water and a Glass Matte: Special Effects in Modern Times; bonus feature on The Criterion Collection](https://reader033.vdocument.in/reader033/viewer/2022051903/5ff4dc8fa1e20c3f007e6d1d/html5/thumbnails/61.jpg)
SIFT + Fundamental Matrix + RANSAC + dense correspondence