The University of Ontario From photohulls to photoflux optimization Yuri Boykov University of Western Ontario Victor Lempitsky Moscow State University The University of Ontario Overview n Current methods for multi-view reconstruction Photohulls and space carving Deformable models n Flux for image segmentation n Photoflux for stereo n Proof of concept The University of Ontario Kutulakos and Seitz, 2000 Photohull S1S1 S2S2 binary, monotonic photoconsistency local decision to carve inconsistent points is OK region growing X it is guaranteed that The University of Ontario A slide from Kutulakos and Seitz (IJCV 2000) Which shape do you get? The Photo Hull is the UNION of all photo-consistent scenes in V It is a photo-consistent scene reconstruction Tightest possible bound on the true scene V V The University of Ontario V Which shape do you get? The Photo Hull is the UNION of all photo-consistent scenes in V It is a photo-consistent scene reconstruction Tightest possible bound on the true scene ? V The University of Ontario Photohull may not be well defined in some cases Rectangle of any size is a photoconsistent surface for this camera configuration object of arbitrary shape and surface texture The University of Ontario Space carving may underestimate the tightest bound on the true scene a larger photoconsistent surface photohull [K&S00] Why? The University of Ontario current voxels Photohull surface points Space Carving voxels inconsistent voxel current surface consistent surface The University of Ontario Non-monotone photoconsistency S1S1 S2S2 non-monotonic photoconsistency NOT safe to carve inconsistent points X it is possible that The University of Ontario Non-binary photoconsistency S1S1 S2S2 non-binary photoconsistency local answer no longer available To carve or not to carve? X The University of Ontario So, what exactly is photohull ? n Largest continuous photoconsistent surface? Defined only for binary monotone photoconsistency In some cases it may not exist OR n The output of voxel carving algorithm? Which version? approach to voxel vs. surface consistency threshold for making 0-1 photoconsistency decisions order of voxel carving for non-monotone photoconsistency The University of Ontario Minimal surfaces for multi-view reconstruction X Faugeras and Keriven, 98 Pons et al., 05 Vogiatzis et al., 05 non-binary photoconsistency S1S1 continuous space carving ? The University of Ontario X S1S X NO surface can both contract or expand even for monotonic photoconsistency Is the surface guaranteed to contract? S2S2 The University of Ontario Bottom line n Photohull surface based concept voxel based algorithms may not exist binary photoconsistency thresholding, may leak spatial monotonicity greedy carving can get fine details but noisy n Deformable models different surface based concept -surface based algorithms -solution exists for solid PDEs real-valued photoconsistency - energy-based regularization no spatial monotonisity -not greedy, may backtrack/expand no noise but may oversmooth The University of Ontario Multi-view Reconstruction versus Image Segmentation multi-view reconstruction (volumetric approach) image segmentation Greedy methods Regularization Flux-based methods Thresholding Region growing Voxel coloring Seitz and Dyer, 1997 Space carving Kutulakos and Seitz, 2002 Snakes Kass et al., 1988 Level-sets Malladi et al., 1994 Graph cuts Boykov and Jolly, 2001 Mash-based Esteban and Schmitt, 2004 Level-sets Faugeras and Keriven, 1998 Graph cuts Vogiatzis et al., 2005 Level-sets Vasilevsky and Sidiqqi, 2002 Kimmel et al., 2003 Graph cuts Kolmogorov and Boykov, 2005 This work The University of Ontario Flux n vector field: some vector defined at each point p stream of water with a given speed at each location C1 n flux: amount of water passing through a given contour flux(C1) > flux(C2) C2 n n changes sign with orientation The University of Ontario Why flux? n Regularization: shrinks n Flux: intelligent ballooning flux and length are geometric properties of boundary with opposite effect Gauss-Ostrogradsky (divergence) theorem The University of Ontario What flux is good for? Segmentation of thin objects [Vasilevsky,Siddiqi02] boundary with large flux Flux of image gradients: The University of Ontario boundary with large flux What flux is good for? Segmentation of thin objects [Vasilevsky,Siddiqi02] Flux of image gradients: Laplacian of image intensities _ _ _ _ _ or region optimizing Laplacian _ _ + + + The University of Ontario What flux is good for? Data-driven ballooning Vasilevsky and Sidiqqi, 2002 Kimmel and Bruckstein, 2003 (in the context of level-sets) image intensities on a scan line Laplacian of intensities Kolmogorov and Boykov 2005 (in the context of graph cuts) flux Laplacian zero crossings (if vectors are gradients of some potential field) The University of Ontario Why flux? Riemannian length + Flux [Kimmel,Bruckstein03] Riemannian length Flux of Riemannian length + Flux The University of Ontario Integrating Laplacian Zero-crossings into Graph Cuts [Kolmogorov&Boykov05] The image is courtesy of David Fleet University of Toronto graph cuts Flux = smart ballooning counteract shrinking bias The University of Ontario Shrinking bias in stereo CVPR05 slides from Vogiatzis, Torr, Cippola The University of Ontario Uniform ballooning CVPR05 slides from Vogiatzis, Torr, Cippola The University of Ontario Our approach: introduce flux for multi-view stereo n Capture some properties of photohull through a novel surface functional photoflux n Photoflux can be combined with regularization combines benefits of space carving and deformable models can recover fine details while keeping the noise low n Photoflux addresses shrinking or over-smoothing bias of standard regularization methods for N-view reconstruction data-driven intelligent ballooning addresses shrinking bias regularized Laplacian zero crossings for boundary alignment The University of Ontario X From photohull to photoflux for all points X right outside photohull X photohull for all points X on photohull binary photoconsistency The University of Ontario for all points X right outside photohull for all points X on photohull X From photohull to photoflux photohull X Gradient of photoconsistency is large for all points X on photohull non-binary photoconsistency The University of Ontario Photoflux X fixed shape S allows to compute global visibility of point X f(S) = outward surface normal at point X The University of Ontario Photoflux photoflux + regularization n Optimization Local via level-sets or PDE-cuts [BKCD, ECCV06] maintaining global visibility could be expensive in practice local patch dS can approximate visibility global graph cuts on a complex [Lempitsky et al. ECCV06] n other options also E(S) = The University of Ontario Photoflux f(S) = visibility vector The University of Ontario Visibility vector 1 2 X photoconsistency The University of Ontario Generalization of photoflux idea f(S) = unknown visibility given point X is on some surface if visibility w at point X is given, e.g. w=w(S) patch based visibility w=w(N) The University of Ontario Photoflux for estimated local photoconsistency gradients Vector field can be computed at every X without fixing any surfce S The University of Ontario Back to example from K&S00 The University of Ontario Back to example from K&S00 gradients and divergence of photoconsistency zoom The University of Ontario Back to example from K&S00 gradients and divergence of photoconsistency photohull [K&S, IJCV00] The University of Ontario Back to example from K&S00 larger photoconsistent surface gradients and divergence of photoconsistency The University of Ontario Back to example from K&S00 4 cameras 16 cameras The University of Ontario Textured and non-textured objects 4 cameras 16 cameras The University of Ontario Regularizing Photoflux photoflux + regularization n Photoflux does not have regularization by itself n Regularized Liplacian zero-crossings E(S) = The University of Ontario Algorithms n Level-sets n Continuous max-flow techniques n Discrete max-flow (graph cuts) implicit explicit The University of Ontario Implicit graph cuts Most current graph cuts technique implicitly use surfaces represented via binary (interior/exterior) labeling of pixels The University of Ontario Explicit graph cuts Except, a recent explicit surfaces representation method - Kirsanov and Gortler, 2004 The University of Ontario Explicit graph cuts n for multi-view reconstruction Lempitsky et al., ECCV 2006 n Explicit surface patches allow local estimation of visibility when computing globally optimal solution Compare with Vogiatzis et al., CVPR05 approach for visibility local oriented visibility estimate The University of Ontario Explicit graph cuts Regularization + uniform ballooning some details are still over-smoothed Lempitsky et al., ECCV 2006 The University of Ontario Regularization + intelligent ballooning Low noise and no shrinking This work The University of Ontario Space carving The University of Ontario Also tried surface based flux E(S) = sometimes not submodular The University of Ontario Photoflux II Camel The University of Ontario Photoflux II Hand data courtesy of K. Kutulakos and S Seitz