modeling 3d deformable and articulated shapes yu chen, tae-kyun kim, roberto cipolla department of...

Modeling 3D Deformable and Articulated Shapes

Yu Chen, Tae-Kyun Kim, Roberto Cipolla

Department of EngineeringUniversity of Cambridge

Roadmap

Brief IntroductionsOur FrameworkExperimental ResultsSummary

Motivation

+

3D Shapes

UncertaintyMeasurements

2D Images

Tasks: – To recover deformable shapes from a single

image with arbitrary camera viewpoint.

Previous Work

Rigid shapes [Prasad’05, Rother’09, Yu’09, etc.]Problems: – Cannot handle self-deformation or articulations.

Category-specific articulated shapese.g., human bodies [Anguelov’05, Balan’07, etc.]Problems: – Requiring strong shape or anatomical knowledge of the

category, such as skeletons and joint angles.– Too many parameters to estimate;– Hard to be generalised to other object categories.

Roadmap

Brief IntroductionsOur FrameworkExperimental ResultsSummary

Our Contribution

A probabilistic framework for:– Modelling different shape variations of

general categories;– Synthesizing new shapes of the category

from limited training data;– Inferring dense 3D shapes of deformable or

articulated objects from a single silhouette;

Explanations on the Graphical Model

Shape Synthesis Matching Silhouettes

Pose Generator

Shape Generator

Joint Distribution:

Generating Shapes

Target: Simultaneous modelling two types of shape variations: – Phenotype variation:

fat vs. thin, tall vs. Short... – Pose variation:

articulation, self deformation, ...Training two GPLVMs:

– Shape generator (MS) for phenotype variation;

– Pose generator (MA) for pose variation.

Shape Generator (MS)– Training Set:

Shapes in the canonical pose.

– Pre-processing: Automatically register each instance with a common

3D template; 3D shape context matching and thin-plate spline

interpolation;Perform PCA on all registered 3D shapes.

– Input: PCA coefficients of all the data.

Generating Shapes

Generating ShapesPose Generator (MA)

– Training Set: Synthetic 3D poses

sequences.

– Pre-processing: Perform PCA on both spatial

positions of vertices and all vertex-wise Jacobian matrices.

– Input: PCA coefficients of all the

data

Shape Synthesis

ZeroShape

V0

Pose Generator

MA

ShapeGenerator

MS

VA VA

VSVS

ShapeSynthesis

V

V

Shape SynthesisModelling the local shape transfer

– Computing Jacobian matrices on the zero shape vertex-wisely.

Ji

Shape SynthesisSynthesizing fully-varied shape V from

phenotype-varied shape VS and pose-varied shape VA.

Probabilistic formulation: a Gaussian Approximation

Matching Silhouettes A two-stage process:

o Projecting the 3D shape onto the image plane

o Chamfer matching of silhouettes

Maximizing likelihood over latent coordinates xA, xS and camera parameters γko Optimizing the closed-form lower bound.o Adaptive line-search with multiple initialisations.

Roadmap

Brief IntroductionsOur FrameworkExperimental ResultsSummary

Experiments on Shape Synthesis

Task: – To synthesize shapes in different phenotypes

and poses with the mean shape μV.

Shape Synthesis: Demo

Shape Generator Pose Generator (Running)

Shape Synthesis: Demo

Shape Generator Pose Generator (Running)

Shape Synthesis: Demo

Shape Generator Pose Generator (Running)

Shape Synthesis: Demo

Shape Generator Pose Generator (Running)

Shape Synthesis: Demo

Shape Generator Pose Generator (Running)

Shape Synthesis: Demo

Shape Generator Pose Generator (Running)

Shape Synthesis: Demo

Shape Generator Pose Generator (Running)

Shape Synthesis: Demo

Shape Generator Pose Generator (Running)

Shape Synthesis: Demo

Shape Generator Pose Generator (Running)

Shape Synthesis: Demo

Shape Generator Pose Generator (Running)

Experiments on Single View Reconstruction

Training dataset:– Shark data:

MS: 11 3D models of different shark species .

MA: 11-frame tail-waving sequence from an animatable 3D MEX model.

– Human data:

MS: CAESAR dataset.

MA: Animations of different 3D poses of Sydney in Poser 7.

Testing: – Internet images (22 sharks and 20 humans in different

poses and camera viewpoints)Segmentation: GrabCut [Rother’04]

Experiments on Single View Reconstruction

Sharks:

Experiments on Single View Reconstruction

Humans:

Experiments on Single View Reconstruction

Examples of multi-modality

Experiments on Single View Reconstruction

Qualitative Results: Precision-Recall Ratios– SF: foreground regions

– SR: image projection of our result

A very good approximation to the results given by parametrical models

Roadmap

Brief IntroductionsOur FrameworkExperimental ResultsSummary

Pros and Cons:

Advantages

Fully data driven; Requiring no strong class-

specific prior knowledge, e.g., skeleton, joint angles;

Capable of modelling general categories;

Compact shape representation and much lower dimensions for efficient optimization;

Uncertainty measurements provided.

Disadvantages

Inaccurate at fine parts, e.g., hands.

Lower descriptive power on poses compared with parametric model, when training instances are not enough;

Training data are sometimes difficult to obtain.

Future Work

A compatible framework which allows incorporating category knowledge

Incorporating more cues: internal edges, texture, and colour;

Multiple view settings and video sequences;

3D object recognition and action recognition tasks.

Thanks!