modeling 3d deformable and articulated shapes yu chen, tae-kyun kim, roberto cipolla department of...
TRANSCRIPT
![Page 1: Modeling 3D Deformable and Articulated Shapes Yu Chen, Tae-Kyun Kim, Roberto Cipolla Department of Engineering University of Cambridge](https://reader036.vdocument.in/reader036/viewer/2022062423/56649d005503460f949d2f56/html5/thumbnails/1.jpg)
Modeling 3D Deformable and Articulated Shapes
Yu Chen, Tae-Kyun Kim, Roberto Cipolla
Department of EngineeringUniversity of Cambridge
![Page 2: Modeling 3D Deformable and Articulated Shapes Yu Chen, Tae-Kyun Kim, Roberto Cipolla Department of Engineering University of Cambridge](https://reader036.vdocument.in/reader036/viewer/2022062423/56649d005503460f949d2f56/html5/thumbnails/2.jpg)
Roadmap
Brief IntroductionsOur FrameworkExperimental ResultsSummary
![Page 3: Modeling 3D Deformable and Articulated Shapes Yu Chen, Tae-Kyun Kim, Roberto Cipolla Department of Engineering University of Cambridge](https://reader036.vdocument.in/reader036/viewer/2022062423/56649d005503460f949d2f56/html5/thumbnails/3.jpg)
Motivation
+
3D Shapes
UncertaintyMeasurements
2D Images
Tasks: – To recover deformable shapes from a single
image with arbitrary camera viewpoint.
![Page 4: Modeling 3D Deformable and Articulated Shapes Yu Chen, Tae-Kyun Kim, Roberto Cipolla Department of Engineering University of Cambridge](https://reader036.vdocument.in/reader036/viewer/2022062423/56649d005503460f949d2f56/html5/thumbnails/4.jpg)
Previous Work
Rigid shapes [Prasad’05, Rother’09, Yu’09, etc.]Problems: – Cannot handle self-deformation or articulations.
Category-specific articulated shapese.g., human bodies [Anguelov’05, Balan’07, etc.]Problems: – Requiring strong shape or anatomical knowledge of the
category, such as skeletons and joint angles.– Too many parameters to estimate;– Hard to be generalised to other object categories.
![Page 5: Modeling 3D Deformable and Articulated Shapes Yu Chen, Tae-Kyun Kim, Roberto Cipolla Department of Engineering University of Cambridge](https://reader036.vdocument.in/reader036/viewer/2022062423/56649d005503460f949d2f56/html5/thumbnails/5.jpg)
Roadmap
Brief IntroductionsOur FrameworkExperimental ResultsSummary
![Page 6: Modeling 3D Deformable and Articulated Shapes Yu Chen, Tae-Kyun Kim, Roberto Cipolla Department of Engineering University of Cambridge](https://reader036.vdocument.in/reader036/viewer/2022062423/56649d005503460f949d2f56/html5/thumbnails/6.jpg)
Our Contribution
A probabilistic framework for:– Modelling different shape variations of
general categories;– Synthesizing new shapes of the category
from limited training data;– Inferring dense 3D shapes of deformable or
articulated objects from a single silhouette;
![Page 7: Modeling 3D Deformable and Articulated Shapes Yu Chen, Tae-Kyun Kim, Roberto Cipolla Department of Engineering University of Cambridge](https://reader036.vdocument.in/reader036/viewer/2022062423/56649d005503460f949d2f56/html5/thumbnails/7.jpg)
Explanations on the Graphical Model
Shape Synthesis Matching Silhouettes
Pose Generator
Shape Generator
Joint Distribution:
![Page 8: Modeling 3D Deformable and Articulated Shapes Yu Chen, Tae-Kyun Kim, Roberto Cipolla Department of Engineering University of Cambridge](https://reader036.vdocument.in/reader036/viewer/2022062423/56649d005503460f949d2f56/html5/thumbnails/8.jpg)
Generating Shapes
Target: Simultaneous modelling two types of shape variations: – Phenotype variation:
fat vs. thin, tall vs. Short... – Pose variation:
articulation, self deformation, ...Training two GPLVMs:
– Shape generator (MS) for phenotype variation;
– Pose generator (MA) for pose variation.
![Page 9: Modeling 3D Deformable and Articulated Shapes Yu Chen, Tae-Kyun Kim, Roberto Cipolla Department of Engineering University of Cambridge](https://reader036.vdocument.in/reader036/viewer/2022062423/56649d005503460f949d2f56/html5/thumbnails/9.jpg)
Shape Generator (MS)– Training Set:
Shapes in the canonical pose.
– Pre-processing: Automatically register each instance with a common
3D template; 3D shape context matching and thin-plate spline
interpolation;Perform PCA on all registered 3D shapes.
– Input: PCA coefficients of all the data.
Generating Shapes
![Page 10: Modeling 3D Deformable and Articulated Shapes Yu Chen, Tae-Kyun Kim, Roberto Cipolla Department of Engineering University of Cambridge](https://reader036.vdocument.in/reader036/viewer/2022062423/56649d005503460f949d2f56/html5/thumbnails/10.jpg)
Generating ShapesPose Generator (MA)
– Training Set: Synthetic 3D poses
sequences.
– Pre-processing: Perform PCA on both spatial
positions of vertices and all vertex-wise Jacobian matrices.
– Input: PCA coefficients of all the
data
![Page 11: Modeling 3D Deformable and Articulated Shapes Yu Chen, Tae-Kyun Kim, Roberto Cipolla Department of Engineering University of Cambridge](https://reader036.vdocument.in/reader036/viewer/2022062423/56649d005503460f949d2f56/html5/thumbnails/11.jpg)
Shape Synthesis
ZeroShape
V0
Pose Generator
MA
ShapeGenerator
MS
VA VA
VSVS
ShapeSynthesis
V
V
![Page 12: Modeling 3D Deformable and Articulated Shapes Yu Chen, Tae-Kyun Kim, Roberto Cipolla Department of Engineering University of Cambridge](https://reader036.vdocument.in/reader036/viewer/2022062423/56649d005503460f949d2f56/html5/thumbnails/12.jpg)
Shape SynthesisModelling the local shape transfer
– Computing Jacobian matrices on the zero shape vertex-wisely.
Ji
![Page 13: Modeling 3D Deformable and Articulated Shapes Yu Chen, Tae-Kyun Kim, Roberto Cipolla Department of Engineering University of Cambridge](https://reader036.vdocument.in/reader036/viewer/2022062423/56649d005503460f949d2f56/html5/thumbnails/13.jpg)
Shape SynthesisSynthesizing fully-varied shape V from
phenotype-varied shape VS and pose-varied shape VA.
Probabilistic formulation: a Gaussian Approximation
![Page 14: Modeling 3D Deformable and Articulated Shapes Yu Chen, Tae-Kyun Kim, Roberto Cipolla Department of Engineering University of Cambridge](https://reader036.vdocument.in/reader036/viewer/2022062423/56649d005503460f949d2f56/html5/thumbnails/14.jpg)
Matching Silhouettes A two-stage process:
o Projecting the 3D shape onto the image plane
o Chamfer matching of silhouettes
Maximizing likelihood over latent coordinates xA, xS and camera parameters γko Optimizing the closed-form lower bound.o Adaptive line-search with multiple initialisations.
![Page 15: Modeling 3D Deformable and Articulated Shapes Yu Chen, Tae-Kyun Kim, Roberto Cipolla Department of Engineering University of Cambridge](https://reader036.vdocument.in/reader036/viewer/2022062423/56649d005503460f949d2f56/html5/thumbnails/15.jpg)
Roadmap
Brief IntroductionsOur FrameworkExperimental ResultsSummary
![Page 16: Modeling 3D Deformable and Articulated Shapes Yu Chen, Tae-Kyun Kim, Roberto Cipolla Department of Engineering University of Cambridge](https://reader036.vdocument.in/reader036/viewer/2022062423/56649d005503460f949d2f56/html5/thumbnails/16.jpg)
Experiments on Shape Synthesis
Task: – To synthesize shapes in different phenotypes
and poses with the mean shape μV.
![Page 17: Modeling 3D Deformable and Articulated Shapes Yu Chen, Tae-Kyun Kim, Roberto Cipolla Department of Engineering University of Cambridge](https://reader036.vdocument.in/reader036/viewer/2022062423/56649d005503460f949d2f56/html5/thumbnails/17.jpg)
Shape Synthesis: Demo
Shape Generator Pose Generator (Running)
![Page 18: Modeling 3D Deformable and Articulated Shapes Yu Chen, Tae-Kyun Kim, Roberto Cipolla Department of Engineering University of Cambridge](https://reader036.vdocument.in/reader036/viewer/2022062423/56649d005503460f949d2f56/html5/thumbnails/18.jpg)
Shape Synthesis: Demo
Shape Generator Pose Generator (Running)
![Page 19: Modeling 3D Deformable and Articulated Shapes Yu Chen, Tae-Kyun Kim, Roberto Cipolla Department of Engineering University of Cambridge](https://reader036.vdocument.in/reader036/viewer/2022062423/56649d005503460f949d2f56/html5/thumbnails/19.jpg)
Shape Synthesis: Demo
Shape Generator Pose Generator (Running)
![Page 20: Modeling 3D Deformable and Articulated Shapes Yu Chen, Tae-Kyun Kim, Roberto Cipolla Department of Engineering University of Cambridge](https://reader036.vdocument.in/reader036/viewer/2022062423/56649d005503460f949d2f56/html5/thumbnails/20.jpg)
Shape Synthesis: Demo
Shape Generator Pose Generator (Running)
![Page 21: Modeling 3D Deformable and Articulated Shapes Yu Chen, Tae-Kyun Kim, Roberto Cipolla Department of Engineering University of Cambridge](https://reader036.vdocument.in/reader036/viewer/2022062423/56649d005503460f949d2f56/html5/thumbnails/21.jpg)
Shape Synthesis: Demo
Shape Generator Pose Generator (Running)
![Page 22: Modeling 3D Deformable and Articulated Shapes Yu Chen, Tae-Kyun Kim, Roberto Cipolla Department of Engineering University of Cambridge](https://reader036.vdocument.in/reader036/viewer/2022062423/56649d005503460f949d2f56/html5/thumbnails/22.jpg)
Shape Synthesis: Demo
Shape Generator Pose Generator (Running)
![Page 23: Modeling 3D Deformable and Articulated Shapes Yu Chen, Tae-Kyun Kim, Roberto Cipolla Department of Engineering University of Cambridge](https://reader036.vdocument.in/reader036/viewer/2022062423/56649d005503460f949d2f56/html5/thumbnails/23.jpg)
Shape Synthesis: Demo
Shape Generator Pose Generator (Running)
![Page 24: Modeling 3D Deformable and Articulated Shapes Yu Chen, Tae-Kyun Kim, Roberto Cipolla Department of Engineering University of Cambridge](https://reader036.vdocument.in/reader036/viewer/2022062423/56649d005503460f949d2f56/html5/thumbnails/24.jpg)
Shape Synthesis: Demo
Shape Generator Pose Generator (Running)
![Page 25: Modeling 3D Deformable and Articulated Shapes Yu Chen, Tae-Kyun Kim, Roberto Cipolla Department of Engineering University of Cambridge](https://reader036.vdocument.in/reader036/viewer/2022062423/56649d005503460f949d2f56/html5/thumbnails/25.jpg)
Shape Synthesis: Demo
Shape Generator Pose Generator (Running)
![Page 26: Modeling 3D Deformable and Articulated Shapes Yu Chen, Tae-Kyun Kim, Roberto Cipolla Department of Engineering University of Cambridge](https://reader036.vdocument.in/reader036/viewer/2022062423/56649d005503460f949d2f56/html5/thumbnails/26.jpg)
Shape Synthesis: Demo
Shape Generator Pose Generator (Running)
![Page 27: Modeling 3D Deformable and Articulated Shapes Yu Chen, Tae-Kyun Kim, Roberto Cipolla Department of Engineering University of Cambridge](https://reader036.vdocument.in/reader036/viewer/2022062423/56649d005503460f949d2f56/html5/thumbnails/27.jpg)
Experiments on Single View Reconstruction
Training dataset:– Shark data:
MS: 11 3D models of different shark species .
MA: 11-frame tail-waving sequence from an animatable 3D MEX model.
– Human data:
MS: CAESAR dataset.
MA: Animations of different 3D poses of Sydney in Poser 7.
Testing: – Internet images (22 sharks and 20 humans in different
poses and camera viewpoints)Segmentation: GrabCut [Rother’04]
![Page 28: Modeling 3D Deformable and Articulated Shapes Yu Chen, Tae-Kyun Kim, Roberto Cipolla Department of Engineering University of Cambridge](https://reader036.vdocument.in/reader036/viewer/2022062423/56649d005503460f949d2f56/html5/thumbnails/28.jpg)
Experiments on Single View Reconstruction
Sharks:
![Page 29: Modeling 3D Deformable and Articulated Shapes Yu Chen, Tae-Kyun Kim, Roberto Cipolla Department of Engineering University of Cambridge](https://reader036.vdocument.in/reader036/viewer/2022062423/56649d005503460f949d2f56/html5/thumbnails/29.jpg)
Experiments on Single View Reconstruction
Humans:
![Page 30: Modeling 3D Deformable and Articulated Shapes Yu Chen, Tae-Kyun Kim, Roberto Cipolla Department of Engineering University of Cambridge](https://reader036.vdocument.in/reader036/viewer/2022062423/56649d005503460f949d2f56/html5/thumbnails/30.jpg)
Experiments on Single View Reconstruction
Examples of multi-modality
![Page 31: Modeling 3D Deformable and Articulated Shapes Yu Chen, Tae-Kyun Kim, Roberto Cipolla Department of Engineering University of Cambridge](https://reader036.vdocument.in/reader036/viewer/2022062423/56649d005503460f949d2f56/html5/thumbnails/31.jpg)
Experiments on Single View Reconstruction
Qualitative Results: Precision-Recall Ratios– SF: foreground regions
– SR: image projection of our result
A very good approximation to the results given by parametrical models
![Page 32: Modeling 3D Deformable and Articulated Shapes Yu Chen, Tae-Kyun Kim, Roberto Cipolla Department of Engineering University of Cambridge](https://reader036.vdocument.in/reader036/viewer/2022062423/56649d005503460f949d2f56/html5/thumbnails/32.jpg)
Roadmap
Brief IntroductionsOur FrameworkExperimental ResultsSummary
![Page 33: Modeling 3D Deformable and Articulated Shapes Yu Chen, Tae-Kyun Kim, Roberto Cipolla Department of Engineering University of Cambridge](https://reader036.vdocument.in/reader036/viewer/2022062423/56649d005503460f949d2f56/html5/thumbnails/33.jpg)
Pros and Cons:
Advantages
Fully data driven; Requiring no strong class-
specific prior knowledge, e.g., skeleton, joint angles;
Capable of modelling general categories;
Compact shape representation and much lower dimensions for efficient optimization;
Uncertainty measurements provided.
Disadvantages
Inaccurate at fine parts, e.g., hands.
Lower descriptive power on poses compared with parametric model, when training instances are not enough;
Training data are sometimes difficult to obtain.
![Page 34: Modeling 3D Deformable and Articulated Shapes Yu Chen, Tae-Kyun Kim, Roberto Cipolla Department of Engineering University of Cambridge](https://reader036.vdocument.in/reader036/viewer/2022062423/56649d005503460f949d2f56/html5/thumbnails/34.jpg)
Future Work
A compatible framework which allows incorporating category knowledge
Incorporating more cues: internal edges, texture, and colour;
Multiple view settings and video sequences;
3D object recognition and action recognition tasks.
![Page 35: Modeling 3D Deformable and Articulated Shapes Yu Chen, Tae-Kyun Kim, Roberto Cipolla Department of Engineering University of Cambridge](https://reader036.vdocument.in/reader036/viewer/2022062423/56649d005503460f949d2f56/html5/thumbnails/35.jpg)
Thanks!