using strong shape priors for multiview reconstruction yunda sunpushmeet kohli mathieu brayphilip hs...

Using Strong Shape Priors for Multiview Reconstruction

Yunda Sun Pushmeet Kohli

Mathieu Bray Philip HS Torr

Department of Computing

Oxford Brookes University

Objective

+

[Images Courtesy: M. Black, L. Sigal]

Parametric Model

Images

Silhouettes

Pose

Estimate

Reconstruction

Outline

Multi-view Reconstruction Shape Models as Strong Priors Object Specific MRF Pose Estimation Results

Outline

Multi-view Reconstruction Shape Models as Strong Priors Object Specific MRF Pose Estimation Results

Multiview Reconstruction

Need for Shape Priors

Multiview Reconstruction No Priors

• Silhouette Intersection• Space Carving

Weak Priors• Surface smoothness

– Snow et al. CVPR ’00

• Photo consistency and smoothness

– Kolmogorov and Zabih [ECCV ’02]

– Vogiatzis et al. [CVPR ’05] [Image Courtesy: Vogiatzis et al.]

Outline

Multi-view Reconstruction Shape Models as Strong Priors Object Specific MRF Pose Estimation Results

Shape-Priors for Segmentation

OBJ-CUT [Kumar et al., CVPR ’05]• Integrate Shape Priors in a MRF

POSE-CUT [Bray et al., ECCV ’06] • Efficient Inference of Model Parameters

Parametric Object Models as Strong Priors

Layered Pictorial Structures

Articulated Models

Deformable Models

Outline

Multi-view Reconstruction Shape Models as Strong Priors Object Specific MRF Pose Estimation and Reconstruction Results

Object-Specific MRF

Object-Specific MRF

Energy Function

Shape Prior

Unary Likelihood

Smoothness Prior

x : Voxel label θ : Model Shape

Object-Specific MRF

Shape Prior

x : Voxel label θ : Model Shape

: shortest distance of voxel i from the rendered model

Object-Specific MRF

Smoothness Prior

x : Voxel label θ : Model Shape

Potts Model

Object-Specific MRF

Unary Likelihood

x : Voxel label θ : Model Shape : Visual Hull

For a soft constraint we use a large constant K instead of infinity

Object-Specific MRF

Energy Function

Shape Prior

Unary Likelihood

Smoothness Prior

Can be solved using Graph cuts

[Kolmogorov and Zabih, ECCV02 ]

Object-Specific MRF

Energy Function

Shape Prior

Unary Likelihood

Smoothness Prior

How to find the optimal Pose?

Outline

Multi-view Reconstruction Shape Models as Strong Priors Object Specific MRF Pose Estimation Results

Inference of Pose Parameters

Rotation and Translation of Torso in X axes

Rotation of left shoulder in X and Z axes

Inference of Pose Parameters

Minimize F(ө) using Powell Minimization

Let F(ө) =

Computational Problem:

Each evaluation of F(ө) requires a graph cut to be computed. (computationally expensive!!) BUT..

Solution: Use the dynamic graph cut algorithm [Kohli&Torr, ICCV 2005]

Outline

Multi-view Reconstruction Shape Models as Strong Priors Object Specific MRF Pose Estimation Results

Experiments

Deformable Models

Articulated Models• Reconstruction Results• Human Pose Estimation

Deformable Models

Four Cameras 1.5 x 105 voxels DOF of Model: 5

Visual Hull

Our Reconstruction

Shape Model

Articulated Models

Articulated Models

Four Cameras 106 voxels DOF of Model: 26

Shape Model

Camera Setup

Articulated Models

500 function evaluations of F(θ) required Time per evaluation: 0.15 sec Total time: 75 sec

Let F(ө) =

Articulated Models

Visual Hull

Our Reconstruction

Pose Estimation Results

Visual Hull

Reconstruction

Pose Estimate

Pose Estimation Results

Quantitative Results• 6 uniformly distributed cameras• 12 degree (RMS) error over 21 joint angles

Pose Estimation Results

Qualitative Results

Pose Estimation Results

Video 1, Camera 1

Pose Estimation Results

Video 1, Camera 2

Pose Estimation Results

Video 2, Camera 1

Pose Estimation Results

Video 2, Camera 2

Future Work

• Use dimensionality reduction to reduce the number of pose parameters.

- results in less number of pose parameters to optimize- would speed up inference

• High resolution reconstruction by a coarse to fine strategy

• Parameter Learning in Object Specific MRF

Thank You

Object-Specific MRF

Energy Function

Shape Prior

Unary Likelihood

Smoothness Prior

+