using strong shape priors for multiview reconstruction yunda sunpushmeet kohli mathieu brayphilip hs...
TRANSCRIPT
Using Strong Shape Priors for Multiview Reconstruction
Yunda Sun Pushmeet Kohli
Mathieu Bray Philip HS Torr
Department of Computing
Oxford Brookes University
Objective
+
[Images Courtesy: M. Black, L. Sigal]
Parametric Model
Images
Silhouettes
Pose
Estimate
Reconstruction
Outline
Multi-view Reconstruction Shape Models as Strong Priors Object Specific MRF Pose Estimation Results
Outline
Multi-view Reconstruction Shape Models as Strong Priors Object Specific MRF Pose Estimation Results
Multiview Reconstruction
Need for Shape Priors
Multiview Reconstruction No Priors
• Silhouette Intersection• Space Carving
Weak Priors• Surface smoothness
– Snow et al. CVPR ’00
• Photo consistency and smoothness
– Kolmogorov and Zabih [ECCV ’02]
– Vogiatzis et al. [CVPR ’05] [Image Courtesy: Vogiatzis et al.]
Outline
Multi-view Reconstruction Shape Models as Strong Priors Object Specific MRF Pose Estimation Results
Shape-Priors for Segmentation
OBJ-CUT [Kumar et al., CVPR ’05]• Integrate Shape Priors in a MRF
POSE-CUT [Bray et al., ECCV ’06] • Efficient Inference of Model Parameters
Parametric Object Models as Strong Priors
Layered Pictorial Structures
Articulated Models
Deformable Models
Outline
Multi-view Reconstruction Shape Models as Strong Priors Object Specific MRF Pose Estimation and Reconstruction Results
Object-Specific MRF
Object-Specific MRF
Energy Function
Shape Prior
Unary Likelihood
Smoothness Prior
x : Voxel label θ : Model Shape
Object-Specific MRF
Shape Prior
x : Voxel label θ : Model Shape
: shortest distance of voxel i from the rendered model
Object-Specific MRF
Smoothness Prior
x : Voxel label θ : Model Shape
Potts Model
Object-Specific MRF
Unary Likelihood
x : Voxel label θ : Model Shape : Visual Hull
For a soft constraint we use a large constant K instead of infinity
Object-Specific MRF
Energy Function
Shape Prior
Unary Likelihood
Smoothness Prior
Can be solved using Graph cuts
[Kolmogorov and Zabih, ECCV02 ]
Object-Specific MRF
Energy Function
Shape Prior
Unary Likelihood
Smoothness Prior
How to find the optimal Pose?
Outline
Multi-view Reconstruction Shape Models as Strong Priors Object Specific MRF Pose Estimation Results
Inference of Pose Parameters
Rotation and Translation of Torso in X axes
Rotation of left shoulder in X and Z axes
Inference of Pose Parameters
Minimize F(ө) using Powell Minimization
Let F(ө) =
Computational Problem:
Each evaluation of F(ө) requires a graph cut to be computed. (computationally expensive!!) BUT..
Solution: Use the dynamic graph cut algorithm [Kohli&Torr, ICCV 2005]
Outline
Multi-view Reconstruction Shape Models as Strong Priors Object Specific MRF Pose Estimation Results
Experiments
Deformable Models
Articulated Models• Reconstruction Results• Human Pose Estimation
Deformable Models
Four Cameras 1.5 x 105 voxels DOF of Model: 5
Visual Hull
Our Reconstruction
Shape Model
Articulated Models
Articulated Models
Four Cameras 106 voxels DOF of Model: 26
Shape Model
Camera Setup
Articulated Models
500 function evaluations of F(θ) required Time per evaluation: 0.15 sec Total time: 75 sec
Let F(ө) =
Articulated Models
Visual Hull
Our Reconstruction
Pose Estimation Results
Visual Hull
Reconstruction
Pose Estimate
Pose Estimation Results
Quantitative Results• 6 uniformly distributed cameras• 12 degree (RMS) error over 21 joint angles
Pose Estimation Results
Qualitative Results
Pose Estimation Results
Video 1, Camera 1
Pose Estimation Results
Video 1, Camera 2
Pose Estimation Results
Video 2, Camera 1
Pose Estimation Results
Video 2, Camera 2
Future Work
• Use dimensionality reduction to reduce the number of pose parameters.
- results in less number of pose parameters to optimize- would speed up inference
• High resolution reconstruction by a coarse to fine strategy
• Parameter Learning in Object Specific MRF
Thank You
Object-Specific MRF
Energy Function
Shape Prior
Unary Likelihood
Smoothness Prior
+