Download - Computer Vision: Stereo - SILP LAB
![Page 1: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/1.jpg)
Multi-view stereo
Many slides adapted from S. Seitz
![Page 2: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/2.jpg)
The third eye can be used for verification
Beyond two-view stereo
![Page 3: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/3.jpg)
The third eye can be used for verification
Beyond two-view stereo
![Page 4: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/4.jpg)
The third eye can be used for verification
Beyond two-view stereo
![Page 5: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/5.jpg)
• Pick a reference image, and slide the corresponding
window along the corresponding epipolar lines of all
other images, using inverse depth relative to the first
image as the search parameter
M. Okutomi and T. Kanade, “A Multiple-Baseline Stereo System,” IEEE Trans. on
Pattern Analysis and Machine Intelligence, 15(4):353-363 (1993).
Multiple-baseline stereo
![Page 6: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/6.jpg)
Multiple-baseline stereo
• For larger baselines, must search larger
area in second image
1/z
width of
a pixel
width of
a pixel
1/z
pixel matching score
![Page 7: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/7.jpg)
Multiple-baseline stereo
Use the sum of
correlation scores to
rank matches
![Page 8: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/8.jpg)
Multiple-baseline stereo
Use the sum of
correlation scores to
rank matches
![Page 9: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/9.jpg)
I1 I2 I10
Multiple-baseline stereo results
M. Okutomi and T. Kanade, “A Multiple-Baseline Stereo System,” IEEE Trans. on
Pattern Analysis and Machine Intelligence, 15(4):353-363 (1993).
![Page 10: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/10.jpg)
Summary: Multiple-baseline stereo
• Basic Approach
• Choose a reference view
• Plot SSD vs. inverse depth instead of disparity
• Replace two-view SSD with sum of SSD over all baselines
• Limitations
• Must choose a reference view
• Occlusions become an issue for large baseline
• Possible solution: use a virtual view
![Page 11: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/11.jpg)
Plane Sweep Stereo
• Choose a virtual view
• Sweep family of planes at different depths with
respect to the virtual camera
virtual camera
input image
R. Collins. A space-sweep approach to true multi-image matching. CVPR 1996.
input image
![Page 12: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/12.jpg)
Plane Sweep Stereo
• Choose a virtual view
• Sweep family of planes at different depths with
respect to the virtual camera
virtual camera
input image
R. Collins. A space-sweep approach to true multi-image matching. CVPR 1996.
input image
![Page 13: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/13.jpg)
Plane Sweep Stereo
• Choose a virtual view
• Sweep family of planes at different depths with
respect to the virtual camera
each plane defines an image composite homography
virtual camera
composite
input image
R. Collins. A space-sweep approach to true multi-image matching. CVPR 1996.
input image
![Page 14: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/14.jpg)
Plane Sweep Stereo
• Choose a virtual view
• Sweep family of planes at different depths with
respect to the virtual camera
each plane defines an image composite homography
virtual camera
composite
input image
R. Collins. A space-sweep approach to true multi-image matching. CVPR 1996.
input image
![Page 15: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/15.jpg)
Plane Sweep Stereo
• Choose a virtual view
• Sweep family of planes at different depths with
respect to the virtual camera
each plane defines an image composite homography
virtual camera
composite
input image
R. Collins. A space-sweep approach to true multi-image matching. CVPR 1996.
input image
![Page 16: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/16.jpg)
Plane Sweep Stereo
• For each depth plane
• For each pixel in the composite image, compute the variance
• For each pixel, select the depth that gives the lowest variance
![Page 17: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/17.jpg)
Plane Sweep Stereo
• For each depth plane
• For each pixel in the composite image, compute the variance
• For each pixel, select the depth that gives the lowest variance
![Page 18: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/18.jpg)
Plane Sweep Stereo
• For each depth plane
• For each pixel in the composite image, compute the variance
• For each pixel, select the depth that gives the lowest variance
![Page 19: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/19.jpg)
Plane Sweep Stereo
• For each depth plane
• For each pixel in the composite image, compute the variance
• For each pixel, select the depth that gives the lowest variance
![Page 20: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/20.jpg)
Plane Sweep Stereo
• For each depth plane
• For each pixel in the composite image, compute the variance
• For each pixel, select the depth that gives the lowest variance
![Page 21: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/21.jpg)
Plane Sweep Stereo
• For each depth plane
• For each pixel in the composite image, compute the variance
• For each pixel, select the depth that gives the lowest variance
![Page 22: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/22.jpg)
Plane Sweep Stereo
• For each depth plane
• For each pixel in the composite image, compute the variance
• For each pixel, select the depth that gives the lowest variance
Can be accelerated using graphics hardware
R. Yang and M. Pollefeys. Multi-Resolution Real-Time Stereo on Commodity Graphics
Hardware, CVPR 2003
![Page 23: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/23.jpg)
Volumetric stereo
• In plane sweep stereo, the sampling of the scene
still depends on the reference view
• We can use a voxel volume to get a view-
independent representation
![Page 24: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/24.jpg)
Volumetric stereo
Scene Volume
V
Input Images
(Calibrated)
Goal: Determine occupancy, “color” of points in V
![Page 25: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/25.jpg)
Discrete formulation: Voxel Coloring
Discretized
Scene Volume
Input Images
(Calibrated)
Goal: Assign RGB values to voxels in Vphoto-consistent with images
![Page 26: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/26.jpg)
Photo-consistency
All ScenesPhoto-Consistent
Scenes
TrueScene
• A photo-consistent scene is a scene that exactly
reproduces your input images from the same camera
viewpoints
• You can’t use your input cameras and images to tell
the difference between a photo-consistent scene and
the true scene
![Page 27: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/27.jpg)
Space Carving
Space Carving Algorithm
Image 1 Image N
…...
K. N. Kutulakos and S. M. Seitz, A Theory of Shape by Space Carving, ICCV 1999
![Page 28: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/28.jpg)
Space Carving
Space Carving Algorithm
Image 1 Image N
…...
• Initialize to a volume V containing the true scene
K. N. Kutulakos and S. M. Seitz, A Theory of Shape by Space Carving, ICCV 1999
![Page 29: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/29.jpg)
Space Carving
Space Carving Algorithm
Image 1 Image N
…...
• Initialize to a volume V containing the true scene
• Choose a voxel on the current surface
K. N. Kutulakos and S. M. Seitz, A Theory of Shape by Space Carving, ICCV 1999
![Page 30: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/30.jpg)
Space Carving
Space Carving Algorithm
Image 1 Image N
…...
• Initialize to a volume V containing the true scene
• Choose a voxel on the current surface
• Project to visible input images
K. N. Kutulakos and S. M. Seitz, A Theory of Shape by Space Carving, ICCV 1999
![Page 31: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/31.jpg)
Space Carving
Space Carving Algorithm
Image 1 Image N
…...
• Initialize to a volume V containing the true scene
• Choose a voxel on the current surface
• Carve if not photo-consistent
• Project to visible input images
K. N. Kutulakos and S. M. Seitz, A Theory of Shape by Space Carving, ICCV 1999
![Page 32: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/32.jpg)
Space Carving
Space Carving Algorithm
Image 1 Image N
…...
• Initialize to a volume V containing the true scene
• Repeat until convergence
• Choose a voxel on the current surface
• Carve if not photo-consistent
• Project to visible input images
K. N. Kutulakos and S. M. Seitz, A Theory of Shape by Space Carving, ICCV 1999
![Page 33: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/33.jpg)
Which shape do you get?
True Scene
V
Source: S. Seitz
![Page 34: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/34.jpg)
Which shape do you get?
True Scene
V
Photo Hull
V
Source: S. Seitz
![Page 35: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/35.jpg)
Which shape do you get?
The Photo Hull is the UNION of all photo-consistent scenes in V
• It is a photo-consistent scene reconstruction
• Tightest possible bound on the true scene
True Scene
V
Photo Hull
V
Source: S. Seitz
![Page 36: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/36.jpg)
Space Carving Results: African Violet
Input Image (1 of 45) Reconstruction
ReconstructionReconstruction Source: S. Seitz
![Page 37: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/37.jpg)
Space Carving Results: Hand
Input Image(1 of 100)
Views of Reconstruction
![Page 38: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/38.jpg)
Reconstruction from Silhouettes
Binary Images
• The case of binary images: a voxel is photo-
consistent if it lies inside the object’s silhouette in all
views
![Page 39: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/39.jpg)
Reconstruction from Silhouettes
Binary Images
Finding the silhouette-consistent shape (visual hull):
• Backproject each silhouette
• Intersect backprojected volumes
• The case of binary images: a voxel is photo-
consistent if it lies inside the object’s silhouette in all
views
![Page 40: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/40.jpg)
Reconstruction from Silhouettes
Binary Images
Finding the silhouette-consistent shape (visual hull):
• Backproject each silhouette
• Intersect backprojected volumes
• The case of binary images: a voxel is photo-
consistent if it lies inside the object’s silhouette in all
views
![Page 41: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/41.jpg)
Reconstruction from Silhouettes
Binary Images
Finding the silhouette-consistent shape (visual hull):
• Backproject each silhouette
• Intersect backprojected volumes
• The case of binary images: a voxel is photo-
consistent if it lies inside the object’s silhouette in all
views
![Page 42: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/42.jpg)
Reconstruction from Silhouettes
Binary Images
Finding the silhouette-consistent shape (visual hull):
• Backproject each silhouette
• Intersect backprojected volumes
• The case of binary images: a voxel is photo-
consistent if it lies inside the object’s silhouette in all
views
![Page 43: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/43.jpg)
Volume intersection
![Page 44: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/44.jpg)
Volume intersection
![Page 45: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/45.jpg)
Volume intersection
Reconstruction Contains the True Scene
• But is generally not the same
![Page 46: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/46.jpg)
Voxel algorithm for volume intersection
Color voxel black if on silhouette in every image
![Page 47: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/47.jpg)
Voxel algorithm for volume intersection
Color voxel black if on silhouette in every image
![Page 48: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/48.jpg)
Properties of Volume Intersection
• Pros
• Easy to implement, fast
• Cons
• No concavities
![Page 49: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/49.jpg)
Properties of Volume Intersection
• Pros
• Easy to implement, fast
• Cons
• No concavities
![Page 50: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/50.jpg)
Properties of Volume Intersection
• Pros
• Easy to implement, fast
• Cons
• No concavities
![Page 51: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/51.jpg)
Properties of Volume Intersection
• Pros
• Easy to implement, fast
• Cons
• No concavities
• Reconstruction is not photo-consistent if texture information
is available
• Requires silhouette extraction
![Page 52: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/52.jpg)
Polyhedral volume intersection
B. Baumgart, Geometric Modeling for Computer Vision, Stanford Artificial Intelligence
Laboratory, Memo no. AIM-249, Stanford University, October 1974.
![Page 53: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/53.jpg)
Polyhedral volume intersection: Pros and cons
• Pros
• No voxelization artifacts
• Cons
• Depends on discretization of outlines
• Numerical problems when polygons to be intersected are
almost coplanar
• Does not take advantage of epipolar geometry
![Page 54: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/54.jpg)
Image-based visual hulls
Wojciech Matusik, Christopher Buehler, Ramesh Raskar, Steven Gortler, and Leonard
McMillan. Image-based Visual Hulls. In SIGGRAPH 2000
Virtual view
• Pick a pixel in the virtual view
• Project corresponding visual
ray into every other view to get
a set of epipolar lines
• Find intervals where epipolar
lines overlap with silhouettes
• Lift intervals back onto the 3D
ray and find their intersection
![Page 55: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/55.jpg)
Image-based visual hulls
Wojciech Matusik, Christopher Buehler, Ramesh Raskar, Steven Gortler, and Leonard
McMillan. Image-based Visual Hulls. In SIGGRAPH 2000
![Page 56: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/56.jpg)
Image-based visual hulls: Pros and cons
• Pros
• Can work in real time
• Takes advantage of epipolar geometry
• Cons
• Need to recompute the visual hull every time the virtual view
is changed
![Page 57: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/57.jpg)
Understanding the shape of visual hulls
• In principle, we can use epipolar geometry to compute
an exact representation of the visual hull in 3D
S. Lazebnik, Y. Furukawa, and J. Ponce, “Projective Visual Hulls”, IJCV 2007.
![Page 58: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/58.jpg)
Review: Multi-view stereo
• Multiple-baseline stereo
• Pick one input view as reference
• Inverse depth instead of disparity
• Plane sweep stereo
• Virtual view
• Volumetric stereo
• Photo-consistency
• Space carving
• Shape from silhouettes
• Visual hull: intersection of visual cones
![Page 59: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/59.jpg)
Reconstruction from Silhouettes
Binary Images
• The case of binary images: a voxel is photo-
consistent if it lies inside the object’s silhouette in all
views
![Page 60: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/60.jpg)
Reconstruction from Silhouettes
Binary Images
Finding the silhouette-consistent shape (visual hull):
• Backproject each silhouette
• Intersect backprojected volumes
• The case of binary images: a voxel is photo-
consistent if it lies inside the object’s silhouette in all
views
![Page 61: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/61.jpg)
Reconstruction from Silhouettes
Binary Images
Finding the silhouette-consistent shape (visual hull):
• Backproject each silhouette
• Intersect backprojected volumes
• The case of binary images: a voxel is photo-
consistent if it lies inside the object’s silhouette in all
views
![Page 62: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/62.jpg)
Reconstruction from Silhouettes
Binary Images
Finding the silhouette-consistent shape (visual hull):
• Backproject each silhouette
• Intersect backprojected volumes
• The case of binary images: a voxel is photo-
consistent if it lies inside the object’s silhouette in all
views
![Page 63: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/63.jpg)
Reconstruction from Silhouettes
Binary Images
Finding the silhouette-consistent shape (visual hull):
• Backproject each silhouette
• Intersect backprojected volumes
• The case of binary images: a voxel is photo-
consistent if it lies inside the object’s silhouette in all
views
![Page 64: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/64.jpg)
Photo-consistency vs. silhouette-consistency
True Scene
![Page 65: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/65.jpg)
Photo-consistency vs. silhouette-consistency
True Scene Photo Hull
![Page 66: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/66.jpg)
Photo-consistency vs. silhouette-consistency
True Scene Photo Hull Visual Hull
![Page 67: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/67.jpg)
Understanding the shape of visual hulls
• What part of the visual hull belongs to the surface?
![Page 68: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/68.jpg)
Understanding the shape of visual hulls
• The visual hull does not correspond to the true
surface because the epipolar constraint is not valid
for silhouette points
O O’
X
x x’
X’
Point on
visual hull
![Page 69: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/69.jpg)
Understanding the shape of visual hulls
• The visual hull does not correspond to the true
surface because the epipolar constraint is not valid
for silhouette points
X
x x’
X’
Point on
visual hullO
O’
![Page 70: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/70.jpg)
Understanding the shape of visual hulls
• The visual hull does not correspond to the true
surface because the epipolar constraint is not valid
for silhouette points
• Exception: frontier points
O O’
X
x x’
Epipolar
tangency
![Page 71: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/71.jpg)
Carved visual hulls
• The visual hull is a good starting point for optimizing
photo-consistency
• Easy to compute
• Tight outer boundary of the object
• Parts of the visual hull (rims) already lie on the surface and are
already photo-consistent
Yasutaka Furukawa and Jean Ponce, Carved Visual Hulls for Image-Based
Modeling, ECCV 2006.
![Page 72: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/72.jpg)
Carved visual hulls
1. Compute visual hull
2. Use dynamic programming to find rims and
constrain them to be fixed
3. Carve the visual hull to optimize photo-consistency
Yasutaka Furukawa and Jean Ponce, Carved Visual Hulls for Image-Based
Modeling, ECCV 2006.
![Page 73: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/73.jpg)
Carved visual hulls
Yasutaka Furukawa and Jean Ponce, Carved Visual Hulls for Image-Based
Modeling, ECCV 2006.
![Page 74: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/74.jpg)
Carved visual hulls: Pros and cons
• Pros
• Visual hull gives a reasonable initial mesh that can be
iteratively deformed
• Cons
• Need silhouette extraction
• Have to compute a lot of points that don’t lie on the object
• Finding rims is difficult
• The carving step can get caught in local minima
• Possible solution: use sparse feature
correspondences as initialization
![Page 75: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/75.jpg)
Feature-based stereo matching
T. Tuytelaars and L. Van Gool, "Matching Widely Separated Views based on Affine
Invariant Regions" Int. Journal on Computer Vision, 59(1), pp. 61-85, 2004.
![Page 76: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/76.jpg)
Feature-based stereo matching
• Pros
• Robust to clutter and occlusion
• Only find matches at reliable points
• Can use invariant local features to deal with foreshortening,
scale change, wide baselines
• Cons
• You only get a sparse cloud of points (or oriented patches),
not a dense depth map or a complete surface
![Page 77: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/77.jpg)
From feature matching to dense stereo
1. Extract features
2. Get a sparse set of initial matches
3. Iteratively expand matches to nearby locations
4. Use visibility constraints to filter out false matches
5. Perform surface reconstruction
Yasutaka Furukawa and Jean Ponce, Accurate, Dense, and Robust Multi-View
Stereopsis, CVPR 2007.
![Page 78: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/78.jpg)
From feature matching to dense stereo
Yasutaka Furukawa and Jean Ponce, Accurate, Dense, and Robust Multi-View
Stereopsis, CVPR 2007.
http://www-cvr.ai.uiuc.edu/~yfurukaw/
![Page 79: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/79.jpg)
Stereo from community photo collections
M. Goesele, N. Snavely, B. Curless, H. Hoppe, S. Seitz, Multi-View Stereo for
Community Photo Collections, ICCV 2007
http://grail.cs.washington.edu/projects/mvscpc/
![Page 80: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/80.jpg)
Stereo from community photo collections
M. Goesele, N. Snavely, B. Curless, H. Hoppe, S. Seitz, Multi-View Stereo for
Community Photo Collections, ICCV 2007
stereo laser scan
Comparison: 90% of points
within 0.128 m of laser scan
(building height 51m)
![Page 81: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/81.jpg)
Stereo from community photo collections
• Up to now, we’ve always assumed that camera
calibration is known
• For photos taken from the Internet, we need structure
from motion techniques to reconstruct both camera
positions and 3D points
![Page 82: Computer Vision: Stereo - SILP LAB](https://reader030.vdocument.in/reader030/viewer/2022020620/61e3b72efed0b96e2475bf67/html5/thumbnails/82.jpg)
Multi-view stereo: Summary
• Multiple-baseline stereo
• Pick one input view as reference
• Inverse depth instead of disparity
• Plane sweep stereo
• Virtual view
• Volumetric stereo
• Photo-consistency
• Space carving
• Shape from silhouettes
• Visual hull: intersection of visual cones
• Volumetric, polyhedral, image-based
• Carved visual hulls
• Feature-based stereo
• From sparse to dense correspondences