from images to voxelsseitz/course/sigg00/slides/seitz-voxel.pdf · input: calibrated images from...
TRANSCRIPT
1
From Images to VoxelsFrom Images to Voxels
Steve SeitzSteve SeitzCarnegie Mellon UniversityCarnegie Mellon UniversityUniversity of WashingtonUniversity of Washingtonhttp://www.http://www.cscs..cmucmu..eduedu/~/~seitzseitz
SIGGRAPH 2000 Course onSIGGRAPH 2000 Course on3D Photography3D Photography
3D Reconstruction from Calibrated Images3D Reconstruction from Calibrated Images
6FHQH�9ROXPH6FHQH�9ROXPH
99
,QSXW�,PDJHV,QSXW�,PDJHV
�&DOLEUDWHG��&DOLEUDWHG�
Goal: Goal: Determine transparency, radiance of points in VDetermine transparency, radiance of points in V
2
Discrete Formulation: Voxel ColoringDiscrete Formulation: Voxel Coloring
'LVFUHWL]HG�'LVFUHWL]HG�
6FHQH�9ROXPH6FHQH�9ROXPH
,QSXW�,PDJHV,QSXW�,PDJHV
�&DOLEUDWHG��&DOLEUDWHG�
Goal: Goal: Assign RGBA values to voxels in VAssign RGBA values to voxels in Vphotophoto--consistentconsistent with imageswith images
Complexity and ComputabilityComplexity and Computability
'LVFUHWL]HG�'LVFUHWL]HG�
6FHQH�9ROXPH6FHQH�9ROXPH
N voxelsN voxelsC colorsC colors
33
$OO�6FHQHV��CN3�3KRWR�&RQVLVWHQW
6FHQHV
7UXH6FHQH
3
IssuesIssues
Theoretical QuestionsTheoretical Questions•• Identify class of Identify class of allall photophoto--consistent scenesconsistent scenes
Practical QuestionsPractical Questions•• How do we compute photoHow do we compute photo--consistent models?consistent models?
1. C=2 (silhouettes)1. C=2 (silhouettes)•• Volume intersection [Martin 81, Volume intersection [Martin 81, SzeliskiSzeliski 93]93]
2. C unconstrained, viewpoint constraints2. C unconstrained, viewpoint constraints•• Voxel coloring algorithm [Seitz & Dyer 97]Voxel coloring algorithm [Seitz & Dyer 97]
3. General Case3. General Case•• Space carving [Kutulakos & Seitz 98]Space carving [Kutulakos & Seitz 98]
Voxel Coloring SolutionsVoxel Coloring Solutions
4
Reconstruction from Silhouettes (C = 2)Reconstruction from Silhouettes (C = 2)
%LQDU\%LQDU\ ,PDJHV,PDJHV
Approach: Approach: •• BackprojectBackproject each silhouetteeach silhouette
•• Intersect backprojected volumesIntersect backprojected volumes
Volume IntersectionVolume Intersection
Reconstruction Contains the True SceneReconstruction Contains the True Scene•• But is generally not the sameBut is generally not the same
•• No concavitiesNo concavities
•• In the limit get In the limit get visual hullvisual hull
5
Voxel Algorithm for Volume IntersectionVoxel Algorithm for Volume Intersection
Color voxel black if on silhouette in every imageColor voxel black if on silhouette in every image•• O(MNO(MN33), for M images, N), for M images, N33 voxelsvoxels
•• Don’t have to search 2Don’t have to search 2 NN33 possible scenes!possible scenes!
Properties of Volume IntersectionProperties of Volume Intersection
ProsPros•• Easy to implement, fastEasy to implement, fast
•• Accelerated viaAccelerated via octreesoctrees [[SzeliskiSzeliski 1993]1993]
ConsCons•• No concavitiesNo concavities
•• Reconstruction is not photoReconstruction is not photo--consistentconsistent
•• Requires identification of silhouettesRequires identification of silhouettes
6
1. C=2 (silhouettes)1. C=2 (silhouettes)•• Volume intersection [Martin 81, Volume intersection [Martin 81, SzeliskiSzeliski 93]93]
2. C unconstrained, viewpoint constraints2. C unconstrained, viewpoint constraints•• Voxel coloring algorithm [Seitz & Dyer 97]Voxel coloring algorithm [Seitz & Dyer 97]
3. General Case3. General Case•• Space carving [Kutulakos & Seitz 98]Space carving [Kutulakos & Seitz 98]
Voxel Coloring SolutionsVoxel Coloring Solutions
����&KRRVH�YR[HO����&KRRVH�YR[HO
����3URMHFW�DQG�FRUUHODWH����3URMHFW�DQG�FRUUHODWH
����&RORU�LI�FRQVLVWHQW����&RORU�LI�FRQVLVWHQW
Voxel Coloring ApproachVoxel Coloring Approach
Visibility Problem: Visibility Problem: in which images is each voxel visible?in which images is each voxel visible?
7
The Global Visibility ProblemThe Global Visibility Problem
Inverse VisibilityInverse Visibilityknown imagesknown images
8QNQRZQ�6FHQH8QNQRZQ�6FHQH
Which Points are Visible in Which Images?Which Points are Visible in Which Images?
.QRZQ�6FHQH.QRZQ�6FHQH
Forward VisibilityForward Visibilityknown sceneknown scene
/D\HUV/D\HUV
Depth Ordering: Visit Depth Ordering: Visit OccludersOccluders First!First!
6FHQH6FHQH
7UDYHUVDO7UDYHUVDO
Condition: Condition: depth order is depth order is viewview--independentindependent
8
Panoramic Depth OrderingPanoramic Depth Ordering
•• Cameras oriented in many different directionsCameras oriented in many different directions
•• Planar depth ordering does not applyPlanar depth ordering does not apply
Panoramic Depth OrderingPanoramic Depth Ordering
Layers radiate outwards from camerasLayers radiate outwards from cameras
9
Panoramic LayeringPanoramic Layering
Layers radiate outwards from camerasLayers radiate outwards from cameras
Panoramic LayeringPanoramic Layering
Layers radiate outwards from camerasLayers radiate outwards from cameras
10
Compatible Camera ConfigurationsCompatible Camera Configurations
DepthDepth--Order ConstraintOrder Constraint•• Scene outside convex hull of camera centersScene outside convex hull of camera centers
OutwardOutward--LookingLookingcameras inside scenecameras inside scene
InwardInward--LookingLookingcameras above scenecameras above scene
Calibrated Image AcquisitionCalibrated Image Acquisition
Calibrated TurntableCalibrated Turntable360° rotation (21 images)360° rotation (21 images)
Selected Dinosaur ImagesSelected Dinosaur Images
Selected Flower ImagesSelected Flower Images
11
Voxel Coloring Results (Video)Voxel Coloring Results (Video)
'LQRVDXU�5HFRQVWUXFWLRQ'LQRVDXU�5HFRQVWUXFWLRQ���.�����.��YR[HOV�FRORUHGYR[HOV�FRORUHG����0�����0�YR[HOV�WHVWHGYR[HOV�WHVWHG��PLQ����PLQ��WR�FRPSXWH�WR�FRPSXWH�RQ�D����0+]�6*,RQ�D����0+]�6*,
)ORZHU�5HFRQVWUXFWLRQ)ORZHU�5HFRQVWUXFWLRQ���.�����.��YR[HOV�FRORUHGYR[HOV�FRORUHG����0�����0�YR[HOV�WHVWHGYR[HOV�WHVWHG��PLQ����PLQ��WR�FRPSXWH�WR�FRPSXWH�RQ�D����0+]�6*,RQ�D����0+]�6*,
Limitations of Depth OrderingLimitations of Depth Ordering
A ViewA View--Independent Depth Order May Not ExistIndependent Depth Order May Not Exist
S T
Need More Powerful GeneralNeed More Powerful General--Case AlgorithmsCase Algorithms•• Unconstrained camera positionsUnconstrained camera positions
•• Unconstrained scene geometry/topologyUnconstrained scene geometry/topology
12
A More Difficult Problem: WalkthroughA More Difficult Problem: Walkthrough
Input: calibrated images from arbitrary positionsInput: calibrated images from arbitrary positionsOutput: 3D model photoOutput: 3D model photo--consistent with all imagesconsistent with all images
WUHH
ZLQGRZ
1. C=2 (silhouettes)1. C=2 (silhouettes)•• Volume intersection [Martin 81, Volume intersection [Martin 81, SzeliskiSzeliski 93]93]
2. C unconstrained, viewpoint constraints2. C unconstrained, viewpoint constraints•• Voxel coloring algorithm [Seitz & Dyer 97]Voxel coloring algorithm [Seitz & Dyer 97]
3. General Case3. General Case•• Space carving [Kutulakos & Seitz 98]Space carving [Kutulakos & Seitz 98]
Voxel Coloring SolutionsVoxel Coloring Solutions
13
Space Carving AlgorithmSpace Carving Algorithm
Space Carving AlgorithmSpace Carving Algorithm
Image 1 Image N
…...
•• Initialize to a volume V containing the true sceneInitialize to a volume V containing the true scene
•• Repeat until convergenceRepeat until convergence
•• Choose a voxel on the current surfaceChoose a voxel on the current surface
•• Carve if not photoCarve if not photo--consistentconsistent•• Project to visible input imagesProject to visible input images
ConvergenceConvergence
Consistency PropertyConsistency Property•• The resulting shape is photoThe resulting shape is photo--consistentconsistent
>> all inconsistent voxels are removedall inconsistent voxels are removed
Convergence PropertyConvergence Property•• Carving converges to a nonCarving converges to a non--empty shapeempty shape
>> a point on the true scene is a point on the true scene is nevernever removedremoved
V’
V
p
14
Output of Space Carving: Photo HullOutput of Space Carving: Photo Hull
The The Photo HullPhoto Hull is the UNION of all photois the UNION of all photo--consistent scenes in Vconsistent scenes in V
•• Tightest possible bound on the true sceneTightest possible bound on the true scene
True SceneTrue Scene
VV
Photo HullPhoto Hull
VV
Space Carving AlgorithmSpace Carving Algorithm
The Basic Algorithm is UnwieldyThe Basic Algorithm is Unwieldy•• Complex update procedureComplex update procedure
Alternative: MultiAlternative: Multi--Pass Plane SweepPass Plane Sweep•• Efficient, can use textureEfficient, can use texture--mapping hardwaremapping hardware
•• Converges quickly in practiceConverges quickly in practice
•• Easy to implementEasy to implement
Results Related Methods
15
MultiMulti--Pass Plane SweepPass Plane Sweep
•• Sweep plane in each of 6 principle directionsSweep plane in each of 6 principle directions
•• Consider cameras on only one side of planeConsider cameras on only one side of plane
•• Repeat until convergenceRepeat until convergence
True Scene Reconstruction
MultiMulti--Pass Plane SweepPass Plane Sweep
•• Sweep plane in each of 6 principle directionsSweep plane in each of 6 principle directions
•• Consider cameras on only one side of planeConsider cameras on only one side of plane
•• Repeat until convergenceRepeat until convergence
16
MultiMulti--Pass Plane SweepPass Plane Sweep
•• Sweep plane in each of 6 principle directionsSweep plane in each of 6 principle directions
•• Consider cameras on only one side of planeConsider cameras on only one side of plane
•• Repeat until convergenceRepeat until convergence
MultiMulti--Pass Plane SweepPass Plane Sweep
•• Sweep plane in each of 6 principle directionsSweep plane in each of 6 principle directions
•• Consider cameras on only one side of planeConsider cameras on only one side of plane
•• Repeat until convergenceRepeat until convergence
17
MultiMulti--Pass Plane SweepPass Plane Sweep
•• Sweep plane in each of 6 principle directionsSweep plane in each of 6 principle directions
•• Consider cameras on only one side of planeConsider cameras on only one side of plane
•• Repeat until convergenceRepeat until convergence
MultiMulti--Pass Plane SweepPass Plane Sweep
•• Sweep plane in each of 6 principle directionsSweep plane in each of 6 principle directions
•• Consider cameras on only one side of planeConsider cameras on only one side of plane
•• Repeat until convergenceRepeat until convergence
18
MultiMulti--Pass Plane SweepPass Plane Sweep
•• Sweep plane in each of 6 principle directionsSweep plane in each of 6 principle directions
•• Consider cameras on only one side of planeConsider cameras on only one side of plane
•• Repeat until convergenceRepeat until convergence
Space Carving Results: African VioletSpace Carving Results: African Violet
,QSXW�,PDJH����RI�����,QSXW�,PDJH����RI����� 5HFRQVWUXFWLRQ5HFRQVWUXFWLRQ
5HFRQVWUXFWLRQ5HFRQVWUXFWLRQ5HFRQVWUXFWLRQ5HFRQVWUXFWLRQ
19
Space Carving Results: HandSpace Carving Results: Hand
,QSXW�,PDJH,QSXW�,PDJH���RI���������RI������
9LHZV�RI�5HFRQVWUXFWLRQ9LHZV�RI�5HFRQVWUXFWLRQ
House WalkthroughHouse Walkthrough
24 rendered input views from inside 24 rendered input views from inside andand outsideoutside
20
Space Carving Results: HouseSpace Carving Results: House
,QSXW�,PDJH�,QSXW�,PDJH��WUXH�VFHQH��WUXH�VFHQH�
5HFRQVWUXFWLRQ5HFRQVWUXFWLRQ��������YR[HOV��������YR[HOV
Space Carving Results: HouseSpace Carving Results: House
,QSXW�,PDJH�,QSXW�,PDJH��WUXH�VFHQH��WUXH�VFHQH�
5HFRQVWUXFWLRQ5HFRQVWUXFWLRQ��������YR[HOV��������YR[HOV
21
1HZ�9LHZ��WUXH�VFHQH�1HZ�9LHZ��WUXH�VFHQH�
Space Carving Results: HouseSpace Carving Results: House
5HFRQVWUXFWLRQ5HFRQVWUXFWLRQ
1HZ�9LHZ1HZ�9LHZ�WUXH�VFHQH��WUXH�VFHQH�
5HFRQVWUXFWLRQ5HFRQVWUXFWLRQ 5HFRQVWUXFWLRQ5HFRQVWUXFWLRQ�ZLWK�QHZ�LQSXW�YLHZ��ZLWK�QHZ�LQSXW�YLHZ�
Other ApproachesOther Approaches
LevelLevel--Set Methods Set Methods [Faugeras & [Faugeras & KerivenKeriven 1998]1998]•• Evolve implicit function by solving Evolve implicit function by solving PDE’sPDE’s
Transparency and Matting Transparency and Matting [[Szeliski Szeliski & & GollandGolland 1998]1998]•• Compute voxels with alphaCompute voxels with alpha--channelchannel
Max Flow/Min Cut Max Flow/Min Cut [Roy & Cox 1998][Roy & Cox 1998]•• Graph theoretic formulationGraph theoretic formulation
MeshMesh--Based Stereo Based Stereo [Fua & [Fua & LeclercLeclerc 95]95]•• MeshMesh--based but similar consistency formulationbased but similar consistency formulation
Virtualized Reality Virtualized Reality [[NarayanNarayan, , RanderRander, , Kanade Kanade 1998]1998]•• Perform stereo 3 images at a time, merge resultsPerform stereo 3 images at a time, merge results
22
Advantages of VoxelsAdvantages of Voxels•• NonNon--parametricparametric
>> can model arbitrary geometrycan model arbitrary geometry
>> can model arbitrary topologycan model arbitrary topology
•• Good reconstruction algorithmsGood reconstruction algorithms
•• Good rendering algorithms (Good rendering algorithms (splattingsplatting, LDI), LDI)
DisadvantagesDisadvantages•• Expensive to process hiExpensive to process hi--resres voxel gridsvoxel grids
•• Large number of parametersLarge number of parameters>> Simple scenes (e.g., planes) require lots of voxelsSimple scenes (e.g., planes) require lots of voxels
ConclusionsConclusions
Volume IntersectionVolume Intersection•• Martin & Martin & AggarwalAggarwal, “Volumetric description of objects from multiple views”, , “Volumetric description of objects from multiple views”,
Trans. Pattern Analysis and Machine Intelligence, 5(2), 1991, pTrans. Pattern Analysis and Machine Intelligence, 5(2), 1991, p p. 150p. 150--158.158.
•• SzeliskiSzeliski, “Rapid, “Rapid OctreeOctree Construction from Image Sequences”, Computer Vision, Construction from Image Sequences”, Computer Vision, Graphics, and Image Processing: Image Understanding, 58(1), 1993Graphics, and Image Processing: Image Understanding, 58(1), 1993 , pp. 23, pp. 23--32.32.
Voxel Coloring and Space CarvingVoxel Coloring and Space Carving•• Seitz & Dyer, “Photorealistic Scene Reconstruction by Voxel ColoSeitz & Dyer, “Photorealistic Scene Reconstruction by Voxel Colo ring”, Proc. ring”, Proc.
Computer Vision and Pattern Recognition (CVPR), 1997, pp. 1067Computer Vision and Pattern Recognition (CVPR), 1997, pp. 1067 --1073.1073.
•• Seitz & Kutulakos, “Plenoptic Image Editing”, Proc. Int. Conf. Seitz & Kutulakos, “Plenoptic Image Editing”, Proc. Int. Conf. on Computer on Computer Vision (ICCV), 1998, pp. 17Vision (ICCV), 1998, pp. 17 --24.24.
•• Kutulakos & Seitz, “A Theory of Shape by Space Carv ing”, Proc. Kutulakos & Seitz, “A Theory of Shape by Space Carv ing”, Proc. ICCV, 1998, pp. ICCV, 1998, pp. 307307--314.314.
BibliographyBibliography
23
Related ReferencesRelated References•• Faugeras & Faugeras & KerivenKeriven, “, “ VariationalVariational principles, surface evolution, principles, surface evolution, PDE'sPDE's , level set , level set
methods and the stereo problem", IEEE Trans. on Image Processingmethods and the stereo problem", IEEE Trans. on Image Processing , 7(3), 1998, , 7(3), 1998, pp. 336pp. 336--344.344.
•• Szeliski Szeliski & & GollandGolland, “Stereo Matching with Transparency and Matting”, Proc. Int. , “Stereo Matching with Transparency and Matting”, Proc. Int. Conf. on Computer Vision (ICCV), 1998, 517Conf. on Computer Vision (ICCV), 1998, 517 --524.524.
•• Roy & Cox, “A MaximumRoy & Cox, “A Maximum --Flow Formulation of the NFlow Formulation of the N --camera Stereo camera Stereo Correspondence Problem”, Proc. ICCV, 1998, pp. 492Correspondence Problem”, Proc. ICCV, 1998, pp. 492 --499.499.
•• Fua & Fua & LeclercLeclerc, “Object, “Object --centered surface reconstruction: Combining multicentered surface reconstruction: Combining multi --image image stereo and shading", Int. Journal of Computer Vision, 16, 1995, stereo and shading", Int. Journal of Computer Vision, 16, 1995, pp. 35pp. 35--56.56.
•• Narayanan, Narayanan, RanderRander, & , & KanadeKanade, “Constructing Virtual Worlds Using Dense , “Constructing Virtual Worlds Using Dense Stereo”, Proc. ICCV, 1998, pp. 3Stereo”, Proc. ICCV, 1998, pp. 3 --10.10.
BibliographyBibliography