announcements
DESCRIPTION
Announcements. Project Update Extension: due Friday, April 20 Create web page with description, results Present your project in class (~10min/each) on Friday, April 27. Stereo Reconstruction Pipeline. Steps Calibrate cameras Rectify images Compute disparity Estimate depth. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/1.jpg)
Announcements
Project Update• Extension: due Friday, April 20
• Create web page with description, results
• Present your project in class (~10min/each) on Friday, April 27
![Page 2: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/2.jpg)
Stereo Reconstruction Pipeline
Steps• Calibrate cameras
• Rectify images
• Compute disparity
• Estimate depth
![Page 3: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/3.jpg)
Image Rectification
![Page 4: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/4.jpg)
Image Rectification
Image Reprojection• reproject image planes onto common
plane parallel to line between optical centers• a homography (3x3 transform)
applied to both input images• C. Loop and Z. Zhang. Computing Rectifying Homographies
for Stereo Vision. IEEE Conf. Computer Vision and Pattern Recognition, 1999. Show VM video
![Page 5: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/5.jpg)
C C’
Depth from Disparity
f
u u’
baseline
z
X
f
input image (1 of 2) [Szeliski & Kang ‘95]
depth map 3D rendering
![Page 6: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/6.jpg)
Disparity-Based Rendering
Render new views from raw disparity• S. M. Seitz and C. R. Dyer, View Morphing, Proc. SIGGRAPH
96, 1996, pp. 21-30.• L. McMillan and G. Bishop. Plenoptic
Modeling: An Image-Based Rendering System, Proc. of SIGGRAPH 95, 1995, pp. 39-46.
![Page 7: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/7.jpg)
Choosing the Baseline
What’s the optimal baseline?• Too small: large depth error
• Too large: difficult search problem
Large BaselineLarge Baseline Small BaselineSmall Baseline
![Page 8: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/8.jpg)
The Effect of Baseline on Depth Estimation
![Page 9: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/9.jpg)
![Page 10: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/10.jpg)
Multibaseline Stereo
Basic Approach• Choose a reference view
• Use your favorite stereo algorithm BUT> replace two-view SSD with SSD over all baselines
Limitations• Must choose a reference view (bad)
• Visibility!
CMU’s 3D Room Video
![Page 11: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/11.jpg)
Epipolar-Plane Images [Bolles 87]
http://www.graphics.lcs.mit.edu/~aisaksen/projects/drlf/epi/
Lesson: Lesson: Beware of Beware of occlusionsocclusions
![Page 12: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/12.jpg)
The Global Visibility Problem
Inverse Visibilityknown images
Unknown SceneUnknown Scene
Which points are visible in which images?
Known SceneKnown Scene
Forward Visibilityknown scene
![Page 13: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/13.jpg)
Volumetric Stereo
Scene VolumeScene Volume
VV
Input ImagesInput Images
(Calibrated)(Calibrated)
Goal: Goal: Determine transparency, radiance of points in VDetermine transparency, radiance of points in V
![Page 14: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/14.jpg)
Discrete Formulation: Voxel Coloring
Discretized Discretized
Scene VolumeScene Volume
Input ImagesInput Images
(Calibrated)(Calibrated)
Goal: Assign RGBA values to voxels in Vphoto-consistent with images
![Page 15: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/15.jpg)
Complexity and Computability
Discretized Discretized
Scene VolumeScene Volume
N voxelsN voxels
C colorsC colors
33
All Scenes (CN3)Photo-Consistent
Scenes
TrueScene
![Page 16: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/16.jpg)
Issues
Theoretical Questions• Identify class of all photo-consistent scenes
Practical Questions• How do we compute photo-consistent models?
![Page 17: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/17.jpg)
1. C=2 (silhouettes)• Volume intersection [Martin 81, Szeliski 93]
2. C unconstrained, viewpoint constraints• Voxel coloring algorithm [Seitz & Dyer 97]
3. General Case• Space carving [Kutulakos & Seitz 98]
Voxel Coloring Solutions
![Page 18: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/18.jpg)
Reconstruction from Silhouettes (C = 2)
Binary ImagesBinary Images
Approach: • Backproject each silhouette
• Intersect backprojected volumes
![Page 19: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/19.jpg)
Volume Intersection
Reconstruction Contains the True Scene• But is generally not the same
• In the limit (all views) get visual hull > Complement of all lines that don’t intersect S
![Page 20: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/20.jpg)
Voxel Algorithm for Volume Intersection
Color voxel black if on silhouette in every image• O(MN3), for M images, N3 voxels
• Don’t have to search 2N3 possible scenes!
![Page 21: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/21.jpg)
Properties of Volume Intersection
Pros• Easy to implement, fast
• Accelerated via octrees [Szeliski 1993]
Cons• No concavities
• Reconstruction is not photo-consistent
• Requires identification of silhouettes
![Page 22: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/22.jpg)
1. C=2 (silhouettes)• Volume intersection [Martin 81, Szeliski 93]
2. C unconstrained, viewpoint constraints• Voxel coloring algorithm [Seitz & Dyer 97]
3. General Case• Space carving [Kutulakos & Seitz 98]
Voxel Coloring Solutions
![Page 23: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/23.jpg)
1. Choose voxel1. Choose voxel2. Project and correlate2. Project and correlate
3.3. Color if consistentColor if consistent(standard deviation of pixel
colors below threshold)
Voxel Coloring Approach
Visibility Problem: Visibility Problem: in which images is each voxel visible?in which images is each voxel visible?
![Page 24: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/24.jpg)
LayersLayers
Depth Ordering: visit occluders first!
SceneScene
TraversalTraversal
Condition: Condition: depth order is the depth order is the same for all input viewssame for all input views
![Page 25: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/25.jpg)
For example: f = distance from separating plane
f
What is A View-Independent Depth Order?
A function f over a scene S and a camera volume C
CS
p
q v
Such that for all p and q in S, v in C
p occludes q from v only if f(p) < f(q)
![Page 26: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/26.jpg)
Panoramic Depth Ordering
• Cameras oriented in many different directions
• Planar depth ordering does not apply
![Page 27: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/27.jpg)
Panoramic Depth Ordering
Layers radiate outwards from camerasLayers radiate outwards from cameras
![Page 28: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/28.jpg)
Panoramic Layering
Layers radiate outwards from camerasLayers radiate outwards from cameras
![Page 29: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/29.jpg)
Panoramic Layering
Layers radiate outwards from camerasLayers radiate outwards from cameras
![Page 30: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/30.jpg)
Compatible Camera Configurations
Depth-Order Constraint• Scene outside convex hull of camera centers
Outward-Lookingcameras inside scene
Inward-Lookingcameras above scene
![Page 31: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/31.jpg)
Calibrated Image Acquisition
Calibrated Turntable360° rotation (21 images)
Selected Dinosaur ImagesSelected Dinosaur Images
Selected Flower ImagesSelected Flower Images
![Page 32: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/32.jpg)
Voxel Coloring Results (Video)
Dinosaur ReconstructionDinosaur Reconstruction72 K voxels colored72 K voxels colored7.6 M voxels tested7.6 M voxels tested7 min. to compute 7 min. to compute on a 250MHz SGIon a 250MHz SGI
Flower ReconstructionFlower Reconstruction70 K voxels colored70 K voxels colored7.6 M voxels tested7.6 M voxels tested7 min. to compute 7 min. to compute on a 250MHz SGIon a 250MHz SGI
![Page 33: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/33.jpg)
Limitations of Depth Ordering
A view-independent depth order may not exist
p q
Need more powerful general-case algorithms• Unconstrained camera positions
• Unconstrained scene geometry/topology
![Page 34: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/34.jpg)
1. C=2 (silhouettes)• Volume intersection [Martin 81, Szeliski 93]
2. C unconstrained, viewpoint constraints• Voxel coloring algorithm [Seitz & Dyer 97]
3. General Case• Space carving [Kutulakos & Seitz 98]
Voxel Coloring Solutions
![Page 35: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/35.jpg)
Space Carving Algorithm
Space Carving Algorithm
Image 1 Image N
…...
• Initialize to a volume V containing the true scene
• Repeat until convergence
• Choose a voxel on the current surface
• Carve if not photo-consistent
• Project to visible input images
![Page 36: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/36.jpg)
Convergence
Consistency Property• The resulting shape is photo-consistent
> all inconsistent points are removed
Convergence Property• Carving converges to a non-empty shape
> a point on the true scene is never removed
p
![Page 37: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/37.jpg)
What is Computable?
The Photo Hull is the UNION of all photo-consistent scenes in V• It is a photo-consistent scene reconstruction
• Tightest possible bound on the true scene
True SceneTrue Scene
VV
Photo HullPhoto Hull
VV
![Page 38: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/38.jpg)
![Page 39: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/39.jpg)
Space Carving Algorithm
The Basic Algorithm is Unwieldy• Complex update procedure
Alternative: Multi-Pass Plane Sweep• Efficient, can use texture-mapping hardware
• Converges quickly in practice
• Easy to implement
Results Algorithm
![Page 40: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/40.jpg)
Multi-Pass Plane Sweep• Sweep plane in each of 6 principle directions• Consider cameras on only one side of plane• Repeat until convergence
True Scene Reconstruction
![Page 41: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/41.jpg)
Multi-Pass Plane Sweep• Sweep plane in each of 6 principle directions• Consider cameras on only one side of plane• Repeat until convergence
![Page 42: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/42.jpg)
Multi-Pass Plane Sweep• Sweep plane in each of 6 principle directions• Consider cameras on only one side of plane• Repeat until convergence
![Page 43: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/43.jpg)
Multi-Pass Plane Sweep• Sweep plane in each of 6 principle directions• Consider cameras on only one side of plane• Repeat until convergence
![Page 44: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/44.jpg)
Multi-Pass Plane Sweep• Sweep plane in each of 6 principle directions• Consider cameras on only one side of plane• Repeat until convergence
![Page 45: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/45.jpg)
Multi-Pass Plane Sweep• Sweep plane in each of 6 principle directions• Consider cameras on only one side of plane• Repeat until convergence
![Page 46: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/46.jpg)
Space Carving Results: African Violet
Input Image (1 of 45) Reconstruction
ReconstructionReconstruction
![Page 47: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/47.jpg)
Space Carving Results: Hand
Input Image(1 of 100)
Views of Reconstruction
![Page 48: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/48.jpg)
House Walkthrough
24 rendered input views from inside and outside
![Page 49: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/49.jpg)
Space Carving Results: House
Input Image Input Image (true scene)(true scene)
ReconstructionReconstruction370,000 voxels370,000 voxels
![Page 50: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/50.jpg)
Space Carving Results: House
Input Image Input Image (true scene)(true scene)
ReconstructionReconstruction370,000 voxels370,000 voxels
![Page 51: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/51.jpg)
New View (true scene)New View (true scene)
Space Carving Results: House
ReconstructionReconstruction
New ViewNew View(true scene)(true scene)
ReconstructionReconstruction ReconstructionReconstruction(with new input view)(with new input view)
![Page 52: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/52.jpg)
Other Approaches
Level-Set Methods [Faugeras & Keriven 1998]
• Evolve implicit function by solving PDE’s
Probabilistic Voxel Reconstruction [DeBonet & Viola 1999]
• Solve for voxel uncertainty (also transparency)
Transparency and Matting [Szeliski & Golland 1998]
• Compute voxels with alpha-channel
Max Flow/Min Cut [Roy & Cox 1998]
• Graph theoretic formulation
Mesh-Based Stereo [Fua & Leclerc 1995], [Zhang & Seitz 2001]
• Mesh-based but similar consistency formulation
Virtualized Reality [Narayan, Rander, Kanade 1998]
• Perform stereo 3 images at a time, merge results
![Page 53: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/53.jpg)
Level Set Stereo [Faugeras & Keriven 1998]
Pose Stereo as Energy Minimization• First idea: find best surface S(u,v) to match images
• This is a variational minimization problem> solved by deforming surface infinitesimally
> deformation given by Euler-Lagrange equations
Problem—how to handle case where object is not a single surface?
• Can use level-set formulation
> represent the object as a function f(x,y,z) whose zero-set is the object’s surface
> evolve f instead of S
![Page 54: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/54.jpg)
Volume Intersection• Martin & Aggarwal, “Volumetric description of objects from multiple views”, Trans. Pattern
Analysis and Machine Intelligence, 5(2), 1991, pp. 150-158.
• Szeliski, “Rapid Octree Construction from Image Sequences”, Computer Vision, Graphics, and Image Processing: Image Understanding, 58(1), 1993, pp. 23-32.
Voxel Coloring and Space Carving• Seitz & Dyer, “Photorealistic Scene Reconstruction by Voxel Coloring”, Proc. Computer
Vision and Pattern Recognition (CVPR), 1997, pp. 1067-1073.
• Seitz & Kutulakos, “Plenoptic Image Editing”, Proc. Int. Conf. on Computer Vision (ICCV), 1998, pp. 17-24.
• Kutulakos & Seitz, “A Theory of Shape by Space Carving”, Proc. ICCV, 1998, pp. 307-314.
Bibliography
![Page 55: Announcements](https://reader036.vdocument.in/reader036/viewer/2022062517/56812dd9550346895d9325d1/html5/thumbnails/55.jpg)
Related References• Bolles, Baker, and Marimont, “Epipolar-Plane Image Analysis: An Approach to Determining
Structure from Motion”, International Journal of Computer Vision, vol 1, no 1, 1987, pp. 7-55.
• DeBonet & Viola, “Poxels: Probabilistic Voxelized Volume Reconstruction”, Proc. Int. Conf. on Computer Vision (ICCV) 1999.
• Faugeras & Keriven, “Variational principles, surface evolution, PDE's, level set methods and the stereo problem", IEEE Trans. on Image Processing, 7(3), 1998, pp. 336-344.
• Szeliski & Golland, “Stereo Matching with Transparency and Matting”, Proc. Int. Conf. on Computer Vision (ICCV), 1998, 517-524.
• Roy & Cox, “A Maximum-Flow Formulation of the N-camera Stereo Correspondence Problem”, Proc. ICCV, 1998, pp. 492-499.
• Fua & Leclerc, “Object-centered surface reconstruction: Combining multi-image stereo and shading", International Journal of Computer Vision, 16, 1995, pp. 35-56.
• Narayanan, Rander, & Kanade, “Constructing Virtual Worlds Using Dense Stereo”, Proc. ICCV, 1998, pp. 3-10.
Bibliography