![Page 1: CS664 Lecture #15: Multi-camera stereo - Cornell University...Multicamera stereo Obvious generalization of stereo: – More than two cameras We saw this a little in “voxel occupancy”,](https://reader033.vdocument.in/reader033/viewer/2022051605/600ff407721c8727436bf117/html5/thumbnails/1.jpg)
CS664 Lecture #15: Multi-camera stereo
Some material taken from:
Steve Seitz, University of Washingtonhttp://www.cs.washington.edu/homes/seitz/
![Page 2: CS664 Lecture #15: Multi-camera stereo - Cornell University...Multicamera stereo Obvious generalization of stereo: – More than two cameras We saw this a little in “voxel occupancy”,](https://reader033.vdocument.in/reader033/viewer/2022051605/600ff407721c8727436bf117/html5/thumbnails/2.jpg)
2
Announcements
Pick a vision paper to write a report on by the next week (10/25)– Sources: CVPR, ICCV, ECCV, PAMI, IJCV– Email your choice to rdz@cs
1-page report should discuss the paper, list its assumptions, give some advantages and disadvantages– Report due on 11/15
Next quiz: 10/25 (Tuesday)– Coverage through this lecture
![Page 3: CS664 Lecture #15: Multi-camera stereo - Cornell University...Multicamera stereo Obvious generalization of stereo: – More than two cameras We saw this a little in “voxel occupancy”,](https://reader033.vdocument.in/reader033/viewer/2022051605/600ff407721c8727436bf117/html5/thumbnails/3.jpg)
3
Graph cuts + EM for sloped surfacesDisparity 5 region “A”
Disparity 5 region “B”
Region “A”plane label
Graph cut solution(integer labels)
Graph cut solution(plane labels)
![Page 4: CS664 Lecture #15: Multi-camera stereo - Cornell University...Multicamera stereo Obvious generalization of stereo: – More than two cameras We saw this a little in “voxel occupancy”,](https://reader033.vdocument.in/reader033/viewer/2022051605/600ff407721c8727436bf117/html5/thumbnails/4.jpg)
4
Multicamera stereo
Obvious generalization of stereo:– More than two cameras
We saw this a little in “voxel occupancy”, which we solved with a binary graph cutThere is a lot of work on this topic, and it naturally involves other important ideasWe’ll look at some simple, elegant methods not based on energy minimization– And see how energy minimization can be used to
improve them!
![Page 5: CS664 Lecture #15: Multi-camera stereo - Cornell University...Multicamera stereo Obvious generalization of stereo: – More than two cameras We saw this a little in “voxel occupancy”,](https://reader033.vdocument.in/reader033/viewer/2022051605/600ff407721c8727436bf117/html5/thumbnails/5.jpg)
5
Multiview stereo
![Page 6: CS664 Lecture #15: Multi-camera stereo - Cornell University...Multicamera stereo Obvious generalization of stereo: – More than two cameras We saw this a little in “voxel occupancy”,](https://reader033.vdocument.in/reader033/viewer/2022051605/600ff407721c8727436bf117/html5/thumbnails/6.jpg)
6
Volumetric stereo
Scene VolumeScene VolumeVV
Input ImagesInput Images(Calibrated)(Calibrated)
Goal: Goal: Determine occupancy, Determine occupancy, ““colorcolor”” of points in Vof points in V
![Page 7: CS664 Lecture #15: Multi-camera stereo - Cornell University...Multicamera stereo Obvious generalization of stereo: – More than two cameras We saw this a little in “voxel occupancy”,](https://reader033.vdocument.in/reader033/viewer/2022051605/600ff407721c8727436bf117/html5/thumbnails/7.jpg)
7
Discrete formulation: Voxel coloring
Discretized Discretized Scene VolumeScene Volume
Input ImagesInput Images(Calibrated)(Calibrated)
Goal: Assign RGBA values to voxels in Vphoto-consistent with images
![Page 8: CS664 Lecture #15: Multi-camera stereo - Cornell University...Multicamera stereo Obvious generalization of stereo: – More than two cameras We saw this a little in “voxel occupancy”,](https://reader033.vdocument.in/reader033/viewer/2022051605/600ff407721c8727436bf117/html5/thumbnails/8.jpg)
8
Complexity and computability
Discretized Discretized Scene VolumeScene Volume
N voxelsN voxelsC colorsC colors
33
All Scenes (CN3)Photo-Consistent
Scenes
TrueScene
![Page 9: CS664 Lecture #15: Multi-camera stereo - Cornell University...Multicamera stereo Obvious generalization of stereo: – More than two cameras We saw this a little in “voxel occupancy”,](https://reader033.vdocument.in/reader033/viewer/2022051605/600ff407721c8727436bf117/html5/thumbnails/9.jpg)
9
Voxel coloring (Seitz)
1. Choose voxel1. Choose voxel2. Project and correlate2. Project and correlate3.3. Color if consistentColor if consistent
(standard deviation of pixel colors below threshold)
Visibility Problem: Visibility Problem: in which images is each voxel visible?in which images is each voxel visible?
![Page 10: CS664 Lecture #15: Multi-camera stereo - Cornell University...Multicamera stereo Obvious generalization of stereo: – More than two cameras We saw this a little in “voxel occupancy”,](https://reader033.vdocument.in/reader033/viewer/2022051605/600ff407721c8727436bf117/html5/thumbnails/10.jpg)
10
Depth ordering: occluders first!
LayersLayers
SceneSceneTraversalTraversal
Condition: Condition: depth order is the depth order is the same for all input viewssame for all input views
![Page 11: CS664 Lecture #15: Multi-camera stereo - Cornell University...Multicamera stereo Obvious generalization of stereo: – More than two cameras We saw this a little in “voxel occupancy”,](https://reader033.vdocument.in/reader033/viewer/2022051605/600ff407721c8727436bf117/html5/thumbnails/11.jpg)
11
Panoramic Depth Ordering
– Cameras oriented in many different directions– Planar depth ordering does not apply
![Page 12: CS664 Lecture #15: Multi-camera stereo - Cornell University...Multicamera stereo Obvious generalization of stereo: – More than two cameras We saw this a little in “voxel occupancy”,](https://reader033.vdocument.in/reader033/viewer/2022051605/600ff407721c8727436bf117/html5/thumbnails/12.jpg)
12
Layers radiate outwards from camerasLayers radiate outwards from cameras
![Page 13: CS664 Lecture #15: Multi-camera stereo - Cornell University...Multicamera stereo Obvious generalization of stereo: – More than two cameras We saw this a little in “voxel occupancy”,](https://reader033.vdocument.in/reader033/viewer/2022051605/600ff407721c8727436bf117/html5/thumbnails/13.jpg)
13
Layers radiate outwards from camerasLayers radiate outwards from cameras
![Page 14: CS664 Lecture #15: Multi-camera stereo - Cornell University...Multicamera stereo Obvious generalization of stereo: – More than two cameras We saw this a little in “voxel occupancy”,](https://reader033.vdocument.in/reader033/viewer/2022051605/600ff407721c8727436bf117/html5/thumbnails/14.jpg)
14
Layers radiate outwards from camerasLayers radiate outwards from cameras
![Page 15: CS664 Lecture #15: Multi-camera stereo - Cornell University...Multicamera stereo Obvious generalization of stereo: – More than two cameras We saw this a little in “voxel occupancy”,](https://reader033.vdocument.in/reader033/viewer/2022051605/600ff407721c8727436bf117/html5/thumbnails/15.jpg)
15
Compatible Camera Configurations
Outward-Lookingcameras inside scene
Inward-Lookingcameras above scene
![Page 16: CS664 Lecture #15: Multi-camera stereo - Cornell University...Multicamera stereo Obvious generalization of stereo: – More than two cameras We saw this a little in “voxel occupancy”,](https://reader033.vdocument.in/reader033/viewer/2022051605/600ff407721c8727436bf117/html5/thumbnails/16.jpg)
16
Voxel Coloring Results
Dinosaur ReconstructionDinosaur Reconstruction72 K voxels colored72 K voxels colored7.6 M voxels tested7.6 M voxels tested7 min. to compute 7 min. to compute on a 250MHz SGIon a 250MHz SGI
Flower ReconstructionFlower Reconstruction70 K voxels colored70 K voxels colored7.6 M voxels tested7.6 M voxels tested7 min. to compute 7 min. to compute on a 250MHz SGIon a 250MHz SGI
![Page 17: CS664 Lecture #15: Multi-camera stereo - Cornell University...Multicamera stereo Obvious generalization of stereo: – More than two cameras We saw this a little in “voxel occupancy”,](https://reader033.vdocument.in/reader033/viewer/2022051605/600ff407721c8727436bf117/html5/thumbnails/17.jpg)
17
Limitations of Depth Ordering
A view-independent depth order may not exist
p q
Need more powerful general-case algorithms– Unconstrained camera positions– Unconstrained scene geometry/topology
![Page 18: CS664 Lecture #15: Multi-camera stereo - Cornell University...Multicamera stereo Obvious generalization of stereo: – More than two cameras We saw this a little in “voxel occupancy”,](https://reader033.vdocument.in/reader033/viewer/2022051605/600ff407721c8727436bf117/html5/thumbnails/18.jpg)
18
Space Carving Algorithm
Image 1 Image N
…...
– Initialize to a volume V containing the true scene
– Repeat until convergence
– Choose a voxel on the current surface
– Carve if not photo-consistent– Project to visible input images
![Page 19: CS664 Lecture #15: Multi-camera stereo - Cornell University...Multicamera stereo Obvious generalization of stereo: – More than two cameras We saw this a little in “voxel occupancy”,](https://reader033.vdocument.in/reader033/viewer/2022051605/600ff407721c8727436bf117/html5/thumbnails/19.jpg)
19
Convergence
Consistency Property– The resulting shape is photo-consistent
• all inconsistent points are removed
Convergence Property– Carving converges to a non-empty shape
• a point on the true scene is never removed
p
![Page 20: CS664 Lecture #15: Multi-camera stereo - Cornell University...Multicamera stereo Obvious generalization of stereo: – More than two cameras We saw this a little in “voxel occupancy”,](https://reader033.vdocument.in/reader033/viewer/2022051605/600ff407721c8727436bf117/html5/thumbnails/20.jpg)
20
Which shape do you get?
The Photo Hull is the UNION of all photo-consistent scenes in V– It is a photo-consistent scene reconstruction– Tightest possible bound on the true scene
True SceneTrue Scene
VV
Photo HullPhoto Hull
VV
![Page 21: CS664 Lecture #15: Multi-camera stereo - Cornell University...Multicamera stereo Obvious generalization of stereo: – More than two cameras We saw this a little in “voxel occupancy”,](https://reader033.vdocument.in/reader033/viewer/2022051605/600ff407721c8727436bf117/html5/thumbnails/21.jpg)
Space Carving Results: Violet
Input Image (1 of 45) Reconstruction
ReconstructionReconstruction
![Page 22: CS664 Lecture #15: Multi-camera stereo - Cornell University...Multicamera stereo Obvious generalization of stereo: – More than two cameras We saw this a little in “voxel occupancy”,](https://reader033.vdocument.in/reader033/viewer/2022051605/600ff407721c8727436bf117/html5/thumbnails/22.jpg)
Space Carving Results: Hand
Input Image(1 of 100)
Views of Reconstruction
![Page 23: CS664 Lecture #15: Multi-camera stereo - Cornell University...Multicamera stereo Obvious generalization of stereo: – More than two cameras We saw this a little in “voxel occupancy”,](https://reader033.vdocument.in/reader033/viewer/2022051605/600ff407721c8727436bf117/html5/thumbnails/23.jpg)
23
Why use energy minimization?
Very similar to voxel occupancy example– Early hard decisions can be wrong
• Difficult to recover from them
No use of spatial smoothness!– Almost nothing in vision actually works unless
it uses spatial smoothness– No one really understands why voxel coloring
and space carving are exceptions– Very active area of research
![Page 24: CS664 Lecture #15: Multi-camera stereo - Cornell University...Multicamera stereo Obvious generalization of stereo: – More than two cameras We saw this a little in “voxel occupancy”,](https://reader033.vdocument.in/reader033/viewer/2022051605/600ff407721c8727436bf117/html5/thumbnails/24.jpg)
24
Multi-Camera Scene Reconstruction via Graph Cuts(Kolmogorov & Zabih, ECCV ’02)
![Page 25: CS664 Lecture #15: Multi-camera stereo - Cornell University...Multicamera stereo Obvious generalization of stereo: – More than two cameras We saw this a little in “voxel occupancy”,](https://reader033.vdocument.in/reader033/viewer/2022051605/600ff407721c8727436bf117/html5/thumbnails/25.jpg)
25
Comparison with stereo
Much harder problem than stereoIn stereo, most scene elements are visible in both cameras– It is common to ignore occlusions
Here, almost no scene elements are visible in all cameras– Visibility reasoning is vital
![Page 26: CS664 Lecture #15: Multi-camera stereo - Cornell University...Multicamera stereo Obvious generalization of stereo: – More than two cameras We saw this a little in “voxel occupancy”,](https://reader033.vdocument.in/reader033/viewer/2022051605/600ff407721c8727436bf117/html5/thumbnails/26.jpg)
26
Key issues
Visibility reasoningIncorporating spatial smoothnessComputational tractability– Only certain energy functions can be
minimized using graph cuts!
Handle a large class of camera configurationsTreat input images symmetrically
![Page 27: CS664 Lecture #15: Multi-camera stereo - Cornell University...Multicamera stereo Obvious generalization of stereo: – More than two cameras We saw this a little in “voxel occupancy”,](https://reader033.vdocument.in/reader033/viewer/2022051605/600ff407721c8727436bf117/html5/thumbnails/27.jpg)
27
Approach
Problem formulation– Discrete labels, not voxels– Carefully constructed energy function
Minimizing the energy via graph cuts– Local minimum in a strong sense– Use the regularity construction
Experimental results– Strong preliminary results
![Page 28: CS664 Lecture #15: Multi-camera stereo - Cornell University...Multicamera stereo Obvious generalization of stereo: – More than two cameras We saw this a little in “voxel occupancy”,](https://reader033.vdocument.in/reader033/viewer/2022051605/600ff407721c8727436bf117/html5/thumbnails/28.jpg)
28
Problem formulation
Discrete set of labels corresponding to different depths– For example, from a single camera
Camera pixel plus label = 3D pointGoal: find the best configuration– Labeling for each pixel in each camera– Minimize an energy function over configurations– Finding the exact minimum is NP-hard
![Page 29: CS664 Lecture #15: Multi-camera stereo - Cornell University...Multicamera stereo Obvious generalization of stereo: – More than two cameras We saw this a little in “voxel occupancy”,](https://reader033.vdocument.in/reader033/viewer/2022051605/600ff407721c8727436bf117/html5/thumbnails/29.jpg)
29
C1 p
r
q
C2
l = 2 l = 3
Depth labels
{ }ionconfigurat
3,,2,,3, qrp
Sample configuration
![Page 30: CS664 Lecture #15: Multi-camera stereo - Cornell University...Multicamera stereo Obvious generalization of stereo: – More than two cameras We saw this a little in “voxel occupancy”,](https://reader033.vdocument.in/reader033/viewer/2022051605/600ff407721c8727436bf117/html5/thumbnails/30.jpg)
30
Energy function has 3 terms: smoothness, data, visibility
Neighborhood systems involve 3D pointsSmoothness: spatial coherence (within camera)Data: photoconsistency (between cameras)– Two pixels looking at the same scene point should see
similar intensities
Visibility: prohibit certain configurations (between cameras)– A pixel in one camera can have its view blocked by a
scene element visible from another camera
![Page 31: CS664 Lecture #15: Multi-camera stereo - Cornell University...Multicamera stereo Obvious generalization of stereo: – More than two cameras We saw this a little in “voxel occupancy”,](https://reader033.vdocument.in/reader033/viewer/2022051605/600ff407721c8727436bf117/html5/thumbnails/31.jpg)
31
Smoothness neighborhood
r
C1p
l = 2 l = 3
Depth labels
Smoothness neighbors{ }1,,2, rp
![Page 32: CS664 Lecture #15: Multi-camera stereo - Cornell University...Multicamera stereo Obvious generalization of stereo: – More than two cameras We saw this a little in “voxel occupancy”,](https://reader033.vdocument.in/reader033/viewer/2022051605/600ff407721c8727436bf117/html5/thumbnails/32.jpg)
32
Smoothness term
Smoothness neighborhood involves pairs of 3D points from the same camera– We’ll assume it only depends on a pair of labels for
neighboring pixels• Usual 4- or 8-connected system among pixels
Smoothness penalty for configuration f is
– V must be a metric, i.e. robustified L1 (regularity)
( ){ }∑∈ smoothNqp
qp ffV ,
,
![Page 33: CS664 Lecture #15: Multi-camera stereo - Cornell University...Multicamera stereo Obvious generalization of stereo: – More than two cameras We saw this a little in “voxel occupancy”,](https://reader033.vdocument.in/reader033/viewer/2022051605/600ff407721c8727436bf117/html5/thumbnails/33.jpg)
33
Photoconsistency constraint
C2
q
C1 p
l = 2 l = 3
Depth labels
If this 3D point is visible in both cameras, pixels p and q should have similar intensities
![Page 34: CS664 Lecture #15: Multi-camera stereo - Cornell University...Multicamera stereo Obvious generalization of stereo: – More than two cameras We saw this a little in “voxel occupancy”,](https://reader033.vdocument.in/reader033/viewer/2022051605/600ff407721c8727436bf117/html5/thumbnails/34.jpg)
34
Photoconsistency neighborhood
C1 p
C2
ql = 2 l = 3
Depth labels
Photoconsistency neighbors{ }2,,2, qp
![Page 35: CS664 Lecture #15: Multi-camera stereo - Cornell University...Multicamera stereo Obvious generalization of stereo: – More than two cameras We saw this a little in “voxel occupancy”,](https://reader033.vdocument.in/reader033/viewer/2022051605/600ff407721c8727436bf117/html5/thumbnails/35.jpg)
35
Data (photoconsistency) termPhotoconsistency neighborhood Nphoto– Arbitrary set of pairs of 3D points (same depth)– Implementation: {hp,fpi,h q,fqi}∈ Nphoto if the projection
of hp,fpi on C2 is nearest to qOur data penalty for configuration f is
– Negative for technical reasons (regularity)
![Page 36: CS664 Lecture #15: Multi-camera stereo - Cornell University...Multicamera stereo Obvious generalization of stereo: – More than two cameras We saw this a little in “voxel occupancy”,](https://reader033.vdocument.in/reader033/viewer/2022051605/600ff407721c8727436bf117/html5/thumbnails/36.jpg)
36
Visibility constraint
C1 p
C2
ql = 2 l = 3
Depth labels
is an impossible configuration
{ }3,,2, qp
![Page 37: CS664 Lecture #15: Multi-camera stereo - Cornell University...Multicamera stereo Obvious generalization of stereo: – More than two cameras We saw this a little in “voxel occupancy”,](https://reader033.vdocument.in/reader033/viewer/2022051605/600ff407721c8727436bf117/html5/thumbnails/37.jpg)
37
Visibility neighborhood
C1 p
C2
ql = 2 l = 3
Depth labels
Visibility neighbors{ }3,,2, qp
![Page 38: CS664 Lecture #15: Multi-camera stereo - Cornell University...Multicamera stereo Obvious generalization of stereo: – More than two cameras We saw this a little in “voxel occupancy”,](https://reader033.vdocument.in/reader033/viewer/2022051605/600ff407721c8727436bf117/html5/thumbnails/38.jpg)
38
Visibility termVisibility neighborhood Nvis is all pairs of 3D points that violate the visibility constraint– Arbitrary set of pairs of points at different depths
• Needed for regularity– The pair of points come from different cameras– Current implementation: based on the photoconsistency
neighborhood
A configuration containing any pair of 3D points in the visibility neighborhood has infinite cost
![Page 39: CS664 Lecture #15: Multi-camera stereo - Cornell University...Multicamera stereo Obvious generalization of stereo: – More than two cameras We saw this a little in “voxel occupancy”,](https://reader033.vdocument.in/reader033/viewer/2022051605/600ff407721c8727436bf117/html5/thumbnails/39.jpg)
39
Tsukuba images
Our results, 4 interactions
![Page 40: CS664 Lecture #15: Multi-camera stereo - Cornell University...Multicamera stereo Obvious generalization of stereo: – More than two cameras We saw this a little in “voxel occupancy”,](https://reader033.vdocument.in/reader033/viewer/2022051605/600ff407721c8727436bf117/html5/thumbnails/40.jpg)
40
Comparison
Our results, 10 interactionsBest results [SS ’02]