modeling the shape of people from 3d range scans dragomir anguelov ai lab stanford university joint...
TRANSCRIPT
![Page 1: Modeling the Shape of People from 3D Range Scans Dragomir Anguelov AI Lab Stanford University Joint work with Praveen Srinivasan, Hoi-Cheung Pang, Daphne](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cf85503460f949c8f54/html5/thumbnails/1.jpg)
Modeling the Shape of People from 3D Range Scans
Dragomir AnguelovAI Lab
Stanford University
Joint work with
Praveen Srinivasan, Hoi-Cheung Pang, Daphne Koller, Sebastian Thrun, James Davis
![Page 2: Modeling the Shape of People from 3D Range Scans Dragomir Anguelov AI Lab Stanford University Joint work with Praveen Srinivasan, Hoi-Cheung Pang, Daphne](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cf85503460f949c8f54/html5/thumbnails/2.jpg)
The Dataset
Cyberware Scans 4 views, ~125k polygons ~65k points each
Problems Missing surface Drastic pose changes
70 scans
48 scans
![Page 3: Modeling the Shape of People from 3D Range Scans Dragomir Anguelov AI Lab Stanford University Joint work with Praveen Srinivasan, Hoi-Cheung Pang, Daphne](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cf85503460f949c8f54/html5/thumbnails/3.jpg)
Modeling Human ShapePose
vari
ati
on
Body-shape variation
![Page 4: Modeling the Shape of People from 3D Range Scans Dragomir Anguelov AI Lab Stanford University Joint work with Praveen Srinivasan, Hoi-Cheung Pang, Daphne](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cf85503460f949c8f54/html5/thumbnails/4.jpg)
Space of Human Shapes Movie
[scape movie]
![Page 5: Modeling the Shape of People from 3D Range Scans Dragomir Anguelov AI Lab Stanford University Joint work with Praveen Srinivasan, Hoi-Cheung Pang, Daphne](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cf85503460f949c8f54/html5/thumbnails/5.jpg)
Talk outline Data processing pipeline
Registration Recovering an articulated skeleton
Modeling the space of human deformations Pose deformations Body shape deformations
Shape completion Partial view completion Animating motion capture sequences
![Page 6: Modeling the Shape of People from 3D Range Scans Dragomir Anguelov AI Lab Stanford University Joint work with Praveen Srinivasan, Hoi-Cheung Pang, Daphne](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cf85503460f949c8f54/html5/thumbnails/6.jpg)
Talk outline Data processing pipeline
Registration Recovering an articulated skeleton
Modeling the space of human deformations Pose deformations Body shape deformations
Shape completion Partial view completion Animating motion capture sequences
![Page 7: Modeling the Shape of People from 3D Range Scans Dragomir Anguelov AI Lab Stanford University Joint work with Praveen Srinivasan, Hoi-Cheung Pang, Daphne](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cf85503460f949c8f54/html5/thumbnails/7.jpg)
Data Processing Pipeline
![Page 8: Modeling the Shape of People from 3D Range Scans Dragomir Anguelov AI Lab Stanford University Joint work with Praveen Srinivasan, Hoi-Cheung Pang, Daphne](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cf85503460f949c8f54/html5/thumbnails/8.jpg)
Registration - CC Algorithm [Anguelov et al. 2004]
Input:Pair of scans
Output: Correspondences
Correlated Correspondence Algorithm
ZX
ZX
Computes an embedding of mesh Z into mesh X
1. Defines a discrete Markov Net M over correspondence variables of mesh Z
Markov Net potentials enforce• Minimal surface deformation• Similar local surface appearance• Preservation of geodesic distance
2. Embedding of Z into X is computed by performing Loopy Belief Propagation on M
Related work: [Huttenlocher et al 00] [Coughlan 02]
![Page 9: Modeling the Shape of People from 3D Range Scans Dragomir Anguelov AI Lab Stanford University Joint work with Praveen Srinivasan, Hoi-Cheung Pang, Daphne](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cf85503460f949c8f54/html5/thumbnails/9.jpg)
Results: Human poses datasetModel
• 4 markers were used on each scan to avoid the need for multiple initializations of Loopy-BP
Cyberware scans
Registrations
![Page 10: Modeling the Shape of People from 3D Range Scans Dragomir Anguelov AI Lab Stanford University Joint work with Praveen Srinivasan, Hoi-Cheung Pang, Daphne](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cf85503460f949c8f54/html5/thumbnails/10.jpg)
Recovering articulation
Input: models, correspondences
Output: rigid parts, skeleton
![Page 11: Modeling the Shape of People from 3D Range Scans Dragomir Anguelov AI Lab Stanford University Joint work with Praveen Srinivasan, Hoi-Cheung Pang, Daphne](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cf85503460f949c8f54/html5/thumbnails/11.jpg)
Recovering articulation [Anguelov et al’04]
Stages of the process1. Register meshes using Correlated Correspondences
algorithm2. Initialize
1. break template surface into N arbitrary components
3. Cluster surface into rigid parts4. Estimate joints
Related work: [Cheung et al’03]
1 2 3 4
![Page 12: Modeling the Shape of People from 3D Range Scans Dragomir Anguelov AI Lab Stanford University Joint work with Praveen Srinivasan, Hoi-Cheung Pang, Daphne](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cf85503460f949c8f54/html5/thumbnails/12.jpg)
Probabilistic Generative Model
Transformations
T
TransformedModel
y1 yN…
Modela1
x1
aN
xN
…
…
Part labels
Points
zK
bK
…z1
b1Instanc
e
Point corrs
Points
![Page 13: Modeling the Shape of People from 3D Range Scans Dragomir Anguelov AI Lab Stanford University Joint work with Praveen Srinivasan, Hoi-Cheung Pang, Daphne](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cf85503460f949c8f54/html5/thumbnails/13.jpg)
Contiguity Prior Parts are preferably contiguous regions
Adjacent points on the surface should have similar labels
Enforce this with a Markov network:
a1 a2
a3
![Page 14: Modeling the Shape of People from 3D Range Scans Dragomir Anguelov AI Lab Stanford University Joint work with Praveen Srinivasan, Hoi-Cheung Pang, Daphne](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cf85503460f949c8f54/html5/thumbnails/14.jpg)
Clustering algorithm
Algorithm Given transformations , perform min-cut* inference
to get
Given labels , solve for rigid transformations
Objective
Contiguity prior
Data Likelihood
* [Greig et al. 89], [Kolmogorov & Zabih 02]
![Page 15: Modeling the Shape of People from 3D Range Scans Dragomir Anguelov AI Lab Stanford University Joint work with Praveen Srinivasan, Hoi-Cheung Pang, Daphne](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cf85503460f949c8f54/html5/thumbnails/15.jpg)
Results: Puppet articulation
![Page 16: Modeling the Shape of People from 3D Range Scans Dragomir Anguelov AI Lab Stanford University Joint work with Praveen Srinivasan, Hoi-Cheung Pang, Daphne](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cf85503460f949c8f54/html5/thumbnails/16.jpg)
Results: Arm articulation
![Page 17: Modeling the Shape of People from 3D Range Scans Dragomir Anguelov AI Lab Stanford University Joint work with Praveen Srinivasan, Hoi-Cheung Pang, Daphne](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cf85503460f949c8f54/html5/thumbnails/17.jpg)
Results: 50 scans of a human
Tree-shaped skeletonfound
Rigid parts found
![Page 18: Modeling the Shape of People from 3D Range Scans Dragomir Anguelov AI Lab Stanford University Joint work with Praveen Srinivasan, Hoi-Cheung Pang, Daphne](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cf85503460f949c8f54/html5/thumbnails/18.jpg)
Talk outline Data processing pipeline
Registration Recovering an articulated skeleton
Modeling the space of human deformations Pose deformations Body shape deformations
Shape completion Partial view completion Animating motion capture sequences
![Page 19: Modeling the Shape of People from 3D Range Scans Dragomir Anguelov AI Lab Stanford University Joint work with Praveen Srinivasan, Hoi-Cheung Pang, Daphne](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cf85503460f949c8f54/html5/thumbnails/19.jpg)
Modeling Pose Deformationinput
Deformations
output
Regression function
Joint angles
![Page 20: Modeling the Shape of People from 3D Range Scans Dragomir Anguelov AI Lab Stanford University Joint work with Praveen Srinivasan, Hoi-Cheung Pang, Daphne](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cf85503460f949c8f54/html5/thumbnails/20.jpg)
Modeling Pose Deformation
template
Predict independently for each triangle
Reconstructcomplete shape
Related work: [Allen et al ‘03][Sumner+Popovic 2004]
![Page 21: Modeling the Shape of People from 3D Range Scans Dragomir Anguelov AI Lab Stanford University Joint work with Praveen Srinivasan, Hoi-Cheung Pang, Daphne](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cf85503460f949c8f54/html5/thumbnails/21.jpg)
Representation of pose deformation
Pose deformation
Rigid articulateddeformation
Given estimates of R, Q, synthesizing the shape is straightforward :
![Page 22: Modeling the Shape of People from 3D Range Scans Dragomir Anguelov AI Lab Stanford University Joint work with Praveen Srinivasan, Hoi-Cheung Pang, Daphne](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cf85503460f949c8f54/html5/thumbnails/22.jpg)
Learning pose deformation
For each polygon, predict entries of from rotations of nearest 2 joints (represented as twists ).
Linear regression parameters :
Obtaining values of in the first place:
![Page 23: Modeling the Shape of People from 3D Range Scans Dragomir Anguelov AI Lab Stanford University Joint work with Praveen Srinivasan, Hoi-Cheung Pang, Daphne](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cf85503460f949c8f54/html5/thumbnails/23.jpg)
Twists and exponential maps
Twist
From twist to rotation matrix
Joint angles
![Page 24: Modeling the Shape of People from 3D Range Scans Dragomir Anguelov AI Lab Stanford University Joint work with Praveen Srinivasan, Hoi-Cheung Pang, Daphne](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cf85503460f949c8f54/html5/thumbnails/24.jpg)
Pose deformation space
![Page 25: Modeling the Shape of People from 3D Range Scans Dragomir Anguelov AI Lab Stanford University Joint work with Praveen Srinivasan, Hoi-Cheung Pang, Daphne](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cf85503460f949c8f54/html5/thumbnails/25.jpg)
Learning body-shape deformation Include also change in shape due to different
people:
Do PCA over body-shape matrices :
Getting estimates of :
![Page 26: Modeling the Shape of People from 3D Range Scans Dragomir Anguelov AI Lab Stanford University Joint work with Praveen Srinivasan, Hoi-Cheung Pang, Daphne](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cf85503460f949c8f54/html5/thumbnails/26.jpg)
PCA over body shape
![Page 27: Modeling the Shape of People from 3D Range Scans Dragomir Anguelov AI Lab Stanford University Joint work with Praveen Srinivasan, Hoi-Cheung Pang, Daphne](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cf85503460f949c8f54/html5/thumbnails/27.jpg)
Combining pose and body shape spaces
![Page 28: Modeling the Shape of People from 3D Range Scans Dragomir Anguelov AI Lab Stanford University Joint work with Praveen Srinivasan, Hoi-Cheung Pang, Daphne](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cf85503460f949c8f54/html5/thumbnails/28.jpg)
Talk outline Data processing pipeline
Registration Recovering an articulated skeleton
Modeling the space of human deformations Pose deformations Body shape deformations
Shape completion Partial view completion Animating motion capture sequences
![Page 29: Modeling the Shape of People from 3D Range Scans Dragomir Anguelov AI Lab Stanford University Joint work with Praveen Srinivasan, Hoi-Cheung Pang, Daphne](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cf85503460f949c8f54/html5/thumbnails/29.jpg)
Shape completion Find surface Y from our space which matches a set
of markers Z
Y[Z] : completed mesh deforms out of space spanned by , R to match Z
Y’[Z]: predicted mesh constrained to be in space spanned by , R
Target optimized by iteratively solving for , R orY while holding the others fixed
![Page 30: Modeling the Shape of People from 3D Range Scans Dragomir Anguelov AI Lab Stanford University Joint work with Praveen Srinivasan, Hoi-Cheung Pang, Daphne](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cf85503460f949c8f54/html5/thumbnails/30.jpg)
Partial view completion
Process: Add a few markers (~6-8) Run CC algorithm to get >
100 markers Optimize to find Y[Z]
![Page 31: Modeling the Shape of People from 3D Range Scans Dragomir Anguelov AI Lab Stanford University Joint work with Praveen Srinivasan, Hoi-Cheung Pang, Daphne](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cf85503460f949c8f54/html5/thumbnails/31.jpg)
Shape completion from motion capture data
![Page 32: Modeling the Shape of People from 3D Range Scans Dragomir Anguelov AI Lab Stanford University Joint work with Praveen Srinivasan, Hoi-Cheung Pang, Daphne](https://reader036.vdocument.in/reader036/viewer/2022062421/56649cf85503460f949c8f54/html5/thumbnails/32.jpg)
Conclusions Presented a data-driven method of modeling human
deformations induced by Pose Body shape
Extending the model Nonlinear prediction of pose deformation (ongoing)
Shape complete original scans using current model Acquire and learn from the entire pose-bodyshape matrix Prior on likely joint angles, e.g. [Popovic + et al ’04] Enforce temporal consistency in tracking applications
Extending the possible applications Markerless motion capture (shape completion in shape-from-
silhouette data) Modeling other beasts