from images to voxelsseitz/course/sigg00/slides/seitz-voxel.pdf · input: calibrated images from...

1

From Images to VoxelsFrom Images to Voxels

Steve SeitzSteve SeitzCarnegie Mellon UniversityCarnegie Mellon UniversityUniversity of WashingtonUniversity of Washingtonhttp://www.http://www.cscs..cmucmu..eduedu/~/~seitzseitz

SIGGRAPH 2000 Course onSIGGRAPH 2000 Course on3D Photography3D Photography

3D Reconstruction from Calibrated Images3D Reconstruction from Calibrated Images

6FHQH�9ROXPH6FHQH�9ROXPH

99

,QSXW�,PDJHV,QSXW�,PDJHV

�&DOLEUDWHG��&DOLEUDWHG�

Goal: Goal: Determine transparency, radiance of points in VDetermine transparency, radiance of points in V

2

Discrete Formulation: Voxel ColoringDiscrete Formulation: Voxel Coloring

'LVFUHWL]HG�'LVFUHWL]HG�

6FHQH�9ROXPH6FHQH�9ROXPH

,QSXW�,PDJHV,QSXW�,PDJHV

�&DOLEUDWHG��&DOLEUDWHG�

Goal: Goal: Assign RGBA values to voxels in VAssign RGBA values to voxels in Vphotophoto--consistentconsistent with imageswith images

Complexity and ComputabilityComplexity and Computability

'LVFUHWL]HG�'LVFUHWL]HG�

6FHQH�9ROXPH6FHQH�9ROXPH

N voxelsN voxelsC colorsC colors

33

$OO�6FHQHV��CN3�3KRWR�&RQVLVWHQW

6FHQHV

7UXH6FHQH

3

IssuesIssues

Theoretical QuestionsTheoretical Questions•• Identify class of Identify class of allall photophoto--consistent scenesconsistent scenes

Practical QuestionsPractical Questions•• How do we compute photoHow do we compute photo--consistent models?consistent models?

1. C=2 (silhouettes)1. C=2 (silhouettes)•• Volume intersection [Martin 81, Volume intersection [Martin 81, SzeliskiSzeliski 93]93]

2. C unconstrained, viewpoint constraints2. C unconstrained, viewpoint constraints•• Voxel coloring algorithm [Seitz & Dyer 97]Voxel coloring algorithm [Seitz & Dyer 97]

3. General Case3. General Case•• Space carving [Kutulakos & Seitz 98]Space carving [Kutulakos & Seitz 98]

Voxel Coloring SolutionsVoxel Coloring Solutions

4

Reconstruction from Silhouettes (C = 2)Reconstruction from Silhouettes (C = 2)

%LQDU\%LQDU\ ,PDJHV,PDJHV

Approach: Approach: •• BackprojectBackproject each silhouetteeach silhouette

•• Intersect backprojected volumesIntersect backprojected volumes

Volume IntersectionVolume Intersection

Reconstruction Contains the True SceneReconstruction Contains the True Scene•• But is generally not the sameBut is generally not the same

•• No concavitiesNo concavities

•• In the limit get In the limit get visual hullvisual hull

5

Voxel Algorithm for Volume IntersectionVoxel Algorithm for Volume Intersection

Color voxel black if on silhouette in every imageColor voxel black if on silhouette in every image•• O(MNO(MN33), for M images, N), for M images, N33 voxelsvoxels

•• Don’t have to search 2Don’t have to search 2 NN33 possible scenes!possible scenes!

Properties of Volume IntersectionProperties of Volume Intersection

ProsPros•• Easy to implement, fastEasy to implement, fast

•• Accelerated viaAccelerated via octreesoctrees [[SzeliskiSzeliski 1993]1993]

ConsCons•• No concavitiesNo concavities

•• Reconstruction is not photoReconstruction is not photo--consistentconsistent

•• Requires identification of silhouettesRequires identification of silhouettes

6

1. C=2 (silhouettes)1. C=2 (silhouettes)•• Volume intersection [Martin 81, Volume intersection [Martin 81, SzeliskiSzeliski 93]93]

2. C unconstrained, viewpoint constraints2. C unconstrained, viewpoint constraints•• Voxel coloring algorithm [Seitz & Dyer 97]Voxel coloring algorithm [Seitz & Dyer 97]

3. General Case3. General Case•• Space carving [Kutulakos & Seitz 98]Space carving [Kutulakos & Seitz 98]

Voxel Coloring SolutionsVoxel Coloring Solutions

����&KRRVH�YR[HO����&KRRVH�YR[HO

����3URMHFW�DQG�FRUUHODWH����3URMHFW�DQG�FRUUHODWH

����&RORU�LI�FRQVLVWHQW����&RORU�LI�FRQVLVWHQW

Voxel Coloring ApproachVoxel Coloring Approach

Visibility Problem: Visibility Problem: in which images is each voxel visible?in which images is each voxel visible?

7

The Global Visibility ProblemThe Global Visibility Problem

Inverse VisibilityInverse Visibilityknown imagesknown images

8QNQRZQ�6FHQH8QNQRZQ�6FHQH

Which Points are Visible in Which Images?Which Points are Visible in Which Images?

.QRZQ�6FHQH.QRZQ�6FHQH

Forward VisibilityForward Visibilityknown sceneknown scene

/D\HUV/D\HUV

Depth Ordering: Visit Depth Ordering: Visit OccludersOccluders First!First!

6FHQH6FHQH

7UDYHUVDO7UDYHUVDO

Condition: Condition: depth order is depth order is viewview--independentindependent

8

Panoramic Depth OrderingPanoramic Depth Ordering

•• Cameras oriented in many different directionsCameras oriented in many different directions

•• Planar depth ordering does not applyPlanar depth ordering does not apply

Panoramic Depth OrderingPanoramic Depth Ordering

Layers radiate outwards from camerasLayers radiate outwards from cameras

9

Panoramic LayeringPanoramic Layering

Layers radiate outwards from camerasLayers radiate outwards from cameras

Panoramic LayeringPanoramic Layering

Layers radiate outwards from camerasLayers radiate outwards from cameras

10

Compatible Camera ConfigurationsCompatible Camera Configurations

DepthDepth--Order ConstraintOrder Constraint•• Scene outside convex hull of camera centersScene outside convex hull of camera centers

OutwardOutward--LookingLookingcameras inside scenecameras inside scene

InwardInward--LookingLookingcameras above scenecameras above scene

Calibrated Image AcquisitionCalibrated Image Acquisition

Calibrated TurntableCalibrated Turntable360° rotation (21 images)360° rotation (21 images)

Selected Dinosaur ImagesSelected Dinosaur Images

Selected Flower ImagesSelected Flower Images

11

Voxel Coloring Results (Video)Voxel Coloring Results (Video)

'LQRVDXU�5HFRQVWUXFWLRQ'LQRVDXU�5HFRQVWUXFWLRQ���.�����.��YR[HOV�FRORUHGYR[HOV�FRORUHG����0�����0�YR[HOV�WHVWHGYR[HOV�WHVWHG��PLQ����PLQ��WR�FRPSXWH�WR�FRPSXWH�RQ�D����0+]�6*,RQ�D����0+]�6*,

)ORZHU�5HFRQVWUXFWLRQ)ORZHU�5HFRQVWUXFWLRQ���.�����.��YR[HOV�FRORUHGYR[HOV�FRORUHG����0�����0�YR[HOV�WHVWHGYR[HOV�WHVWHG��PLQ����PLQ��WR�FRPSXWH�WR�FRPSXWH�RQ�D����0+]�6*,RQ�D����0+]�6*,

Limitations of Depth OrderingLimitations of Depth Ordering

A ViewA View--Independent Depth Order May Not ExistIndependent Depth Order May Not Exist

S T

Need More Powerful GeneralNeed More Powerful General--Case AlgorithmsCase Algorithms•• Unconstrained camera positionsUnconstrained camera positions

•• Unconstrained scene geometry/topologyUnconstrained scene geometry/topology

12

A More Difficult Problem: WalkthroughA More Difficult Problem: Walkthrough

Input: calibrated images from arbitrary positionsInput: calibrated images from arbitrary positionsOutput: 3D model photoOutput: 3D model photo--consistent with all imagesconsistent with all images

WUHH

ZLQGRZ

1. C=2 (silhouettes)1. C=2 (silhouettes)•• Volume intersection [Martin 81, Volume intersection [Martin 81, SzeliskiSzeliski 93]93]

2. C unconstrained, viewpoint constraints2. C unconstrained, viewpoint constraints•• Voxel coloring algorithm [Seitz & Dyer 97]Voxel coloring algorithm [Seitz & Dyer 97]

3. General Case3. General Case•• Space carving [Kutulakos & Seitz 98]Space carving [Kutulakos & Seitz 98]

Voxel Coloring SolutionsVoxel Coloring Solutions

13

Space Carving AlgorithmSpace Carving Algorithm

Space Carving AlgorithmSpace Carving Algorithm

Image 1 Image N

…...

•• Initialize to a volume V containing the true sceneInitialize to a volume V containing the true scene

•• Repeat until convergenceRepeat until convergence

•• Choose a voxel on the current surfaceChoose a voxel on the current surface

•• Carve if not photoCarve if not photo--consistentconsistent•• Project to visible input imagesProject to visible input images

ConvergenceConvergence

Consistency PropertyConsistency Property•• The resulting shape is photoThe resulting shape is photo--consistentconsistent

>> all inconsistent voxels are removedall inconsistent voxels are removed

Convergence PropertyConvergence Property•• Carving converges to a nonCarving converges to a non--empty shapeempty shape

>> a point on the true scene is a point on the true scene is nevernever removedremoved

V’

V

p

14

Output of Space Carving: Photo HullOutput of Space Carving: Photo Hull

The The Photo HullPhoto Hull is the UNION of all photois the UNION of all photo--consistent scenes in Vconsistent scenes in V

•• Tightest possible bound on the true sceneTightest possible bound on the true scene

True SceneTrue Scene

VV

Photo HullPhoto Hull

VV

Space Carving AlgorithmSpace Carving Algorithm

The Basic Algorithm is UnwieldyThe Basic Algorithm is Unwieldy•• Complex update procedureComplex update procedure

Alternative: MultiAlternative: Multi--Pass Plane SweepPass Plane Sweep•• Efficient, can use textureEfficient, can use texture--mapping hardwaremapping hardware

•• Converges quickly in practiceConverges quickly in practice

•• Easy to implementEasy to implement

Results Related Methods

15

MultiMulti--Pass Plane SweepPass Plane Sweep

•• Sweep plane in each of 6 principle directionsSweep plane in each of 6 principle directions

•• Consider cameras on only one side of planeConsider cameras on only one side of plane

•• Repeat until convergenceRepeat until convergence

True Scene Reconstruction

MultiMulti--Pass Plane SweepPass Plane Sweep

•• Sweep plane in each of 6 principle directionsSweep plane in each of 6 principle directions

•• Consider cameras on only one side of planeConsider cameras on only one side of plane

•• Repeat until convergenceRepeat until convergence

16

MultiMulti--Pass Plane SweepPass Plane Sweep

•• Sweep plane in each of 6 principle directionsSweep plane in each of 6 principle directions

•• Consider cameras on only one side of planeConsider cameras on only one side of plane

•• Repeat until convergenceRepeat until convergence

MultiMulti--Pass Plane SweepPass Plane Sweep

•• Sweep plane in each of 6 principle directionsSweep plane in each of 6 principle directions

•• Consider cameras on only one side of planeConsider cameras on only one side of plane

•• Repeat until convergenceRepeat until convergence

17

MultiMulti--Pass Plane SweepPass Plane Sweep

•• Sweep plane in each of 6 principle directionsSweep plane in each of 6 principle directions

•• Consider cameras on only one side of planeConsider cameras on only one side of plane

•• Repeat until convergenceRepeat until convergence

MultiMulti--Pass Plane SweepPass Plane Sweep

•• Sweep plane in each of 6 principle directionsSweep plane in each of 6 principle directions

•• Consider cameras on only one side of planeConsider cameras on only one side of plane

•• Repeat until convergenceRepeat until convergence

18

MultiMulti--Pass Plane SweepPass Plane Sweep

•• Sweep plane in each of 6 principle directionsSweep plane in each of 6 principle directions

•• Consider cameras on only one side of planeConsider cameras on only one side of plane

•• Repeat until convergenceRepeat until convergence

Space Carving Results: African VioletSpace Carving Results: African Violet

,QSXW�,PDJH����RI�����,QSXW�,PDJH����RI����� 5HFRQVWUXFWLRQ5HFRQVWUXFWLRQ

5HFRQVWUXFWLRQ5HFRQVWUXFWLRQ5HFRQVWUXFWLRQ5HFRQVWUXFWLRQ

19

Space Carving Results: HandSpace Carving Results: Hand

,QSXW�,PDJH,QSXW�,PDJH���RI���������RI������

9LHZV�RI�5HFRQVWUXFWLRQ9LHZV�RI�5HFRQVWUXFWLRQ

House WalkthroughHouse Walkthrough

24 rendered input views from inside 24 rendered input views from inside andand outsideoutside

20

Space Carving Results: HouseSpace Carving Results: House

,QSXW�,PDJH�,QSXW�,PDJH��WUXH�VFHQH��WUXH�VFHQH�

5HFRQVWUXFWLRQ5HFRQVWUXFWLRQ��������YR[HOV��������YR[HOV

Space Carving Results: HouseSpace Carving Results: House

,QSXW�,PDJH�,QSXW�,PDJH��WUXH�VFHQH��WUXH�VFHQH�

5HFRQVWUXFWLRQ5HFRQVWUXFWLRQ��������YR[HOV��������YR[HOV

21

1HZ�9LHZ��WUXH�VFHQH�1HZ�9LHZ��WUXH�VFHQH�

Space Carving Results: HouseSpace Carving Results: House

5HFRQVWUXFWLRQ5HFRQVWUXFWLRQ

1HZ�9LHZ1HZ�9LHZ�WUXH�VFHQH��WUXH�VFHQH�

5HFRQVWUXFWLRQ5HFRQVWUXFWLRQ 5HFRQVWUXFWLRQ5HFRQVWUXFWLRQ�ZLWK�QHZ�LQSXW�YLHZ��ZLWK�QHZ�LQSXW�YLHZ�

Other ApproachesOther Approaches

LevelLevel--Set Methods Set Methods [Faugeras & [Faugeras & KerivenKeriven 1998]1998]•• Evolve implicit function by solving Evolve implicit function by solving PDE’sPDE’s

Transparency and Matting Transparency and Matting [[Szeliski Szeliski & & GollandGolland 1998]1998]•• Compute voxels with alphaCompute voxels with alpha--channelchannel

Max Flow/Min Cut Max Flow/Min Cut [Roy & Cox 1998][Roy & Cox 1998]•• Graph theoretic formulationGraph theoretic formulation

MeshMesh--Based Stereo Based Stereo [Fua & [Fua & LeclercLeclerc 95]95]•• MeshMesh--based but similar consistency formulationbased but similar consistency formulation

Virtualized Reality Virtualized Reality [[NarayanNarayan, , RanderRander, , Kanade Kanade 1998]1998]•• Perform stereo 3 images at a time, merge resultsPerform stereo 3 images at a time, merge results

22

Advantages of VoxelsAdvantages of Voxels•• NonNon--parametricparametric

>> can model arbitrary geometrycan model arbitrary geometry

>> can model arbitrary topologycan model arbitrary topology

•• Good reconstruction algorithmsGood reconstruction algorithms

•• Good rendering algorithms (Good rendering algorithms (splattingsplatting, LDI), LDI)

DisadvantagesDisadvantages•• Expensive to process hiExpensive to process hi--resres voxel gridsvoxel grids

•• Large number of parametersLarge number of parameters>> Simple scenes (e.g., planes) require lots of voxelsSimple scenes (e.g., planes) require lots of voxels

ConclusionsConclusions

Volume IntersectionVolume Intersection•• Martin & Martin & AggarwalAggarwal, “Volumetric description of objects from multiple views”, , “Volumetric description of objects from multiple views”,

Trans. Pattern Analysis and Machine Intelligence, 5(2), 1991, pTrans. Pattern Analysis and Machine Intelligence, 5(2), 1991, p p. 150p. 150--158.158.

•• SzeliskiSzeliski, “Rapid, “Rapid OctreeOctree Construction from Image Sequences”, Computer Vision, Construction from Image Sequences”, Computer Vision, Graphics, and Image Processing: Image Understanding, 58(1), 1993Graphics, and Image Processing: Image Understanding, 58(1), 1993 , pp. 23, pp. 23--32.32.

Voxel Coloring and Space CarvingVoxel Coloring and Space Carving•• Seitz & Dyer, “Photorealistic Scene Reconstruction by Voxel ColoSeitz & Dyer, “Photorealistic Scene Reconstruction by Voxel Colo ring”, Proc. ring”, Proc.

Computer Vision and Pattern Recognition (CVPR), 1997, pp. 1067Computer Vision and Pattern Recognition (CVPR), 1997, pp. 1067 --1073.1073.

•• Seitz & Kutulakos, “Plenoptic Image Editing”, Proc. Int. Conf. Seitz & Kutulakos, “Plenoptic Image Editing”, Proc. Int. Conf. on Computer on Computer Vision (ICCV), 1998, pp. 17Vision (ICCV), 1998, pp. 17 --24.24.

•• Kutulakos & Seitz, “A Theory of Shape by Space Carv ing”, Proc. Kutulakos & Seitz, “A Theory of Shape by Space Carv ing”, Proc. ICCV, 1998, pp. ICCV, 1998, pp. 307307--314.314.

BibliographyBibliography

23

Related ReferencesRelated References•• Faugeras & Faugeras & KerivenKeriven, “, “ VariationalVariational principles, surface evolution, principles, surface evolution, PDE'sPDE's , level set , level set

methods and the stereo problem", IEEE Trans. on Image Processingmethods and the stereo problem", IEEE Trans. on Image Processing , 7(3), 1998, , 7(3), 1998, pp. 336pp. 336--344.344.

•• Szeliski Szeliski & & GollandGolland, “Stereo Matching with Transparency and Matting”, Proc. Int. , “Stereo Matching with Transparency and Matting”, Proc. Int. Conf. on Computer Vision (ICCV), 1998, 517Conf. on Computer Vision (ICCV), 1998, 517 --524.524.

•• Roy & Cox, “A MaximumRoy & Cox, “A Maximum --Flow Formulation of the NFlow Formulation of the N --camera Stereo camera Stereo Correspondence Problem”, Proc. ICCV, 1998, pp. 492Correspondence Problem”, Proc. ICCV, 1998, pp. 492 --499.499.

•• Fua & Fua & LeclercLeclerc, “Object, “Object --centered surface reconstruction: Combining multicentered surface reconstruction: Combining multi --image image stereo and shading", Int. Journal of Computer Vision, 16, 1995, stereo and shading", Int. Journal of Computer Vision, 16, 1995, pp. 35pp. 35--56.56.

•• Narayanan, Narayanan, RanderRander, & , & KanadeKanade, “Constructing Virtual Worlds Using Dense , “Constructing Virtual Worlds Using Dense Stereo”, Proc. ICCV, 1998, pp. 3Stereo”, Proc. ICCV, 1998, pp. 3 --10.10.

BibliographyBibliography