body shape from video - research.microsoft.com€¦ · michael j. black, iwv 2009, barcelona...
TRANSCRIPT
![Page 1: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/1.jpg)
Michael J. Black, IWV 2009, Barcelona
Body Shape from Video
Alexandru Balan Michael Black
Brown University
![Page 2: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/2.jpg)
Michael J. Black, IWV 2009, Barcelona
What’s constant?
• Video is interesting because it gives us information about the
“structure” of the world.
• What’s constant is important (explanation).
• What changes may be more important (violation of assumptions).
![Page 3: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/3.jpg)
Michael J. Black, IWV 2009, Barcelona
What’s constant?
Constant:
• camera pose and focal length
• identity
• height, weight
• limb lengths
• global scene illumination
• albedo
Changing
• pose (joint angles and spine)
• visibility
• soft tissues
• drape of clothing
• cast shadows and shading
![Page 4: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/4.jpg)
Michael J. Black, IWV 2009, Barcelona
What’s constant?
Approach: Model what’s
constant using a model of
3D body shape.
Problem: Pose changes
shape. Need a pose-invariant
shape model.
![Page 5: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/5.jpg)
Michael J. Black, IWV 2009, Barcelona
Why a graphics model?
Goals
– Provide strong constraints for interpreting video.
– Combine information • Across views
• Across non-rigid pose changes
– Support inference of gender, height, age, etc.
– Explain changes due to illumination.
Problem: must factor changes due to shape and
pose
– Learn statistics of shape variation across people
and poses
– Model what you know, learn the statistics of the
rest.
![Page 6: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/6.jpg)
Michael J. Black, IWV 2009, Barcelona
Factoring pose and shape
Low dimensional
parameterization
learned from examples
We use an “intrinsic”
shape representation invariant to pose
Pose + Shape
[ SCAPE: Anguelov et al. ’05 ]
![Page 7: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/7.jpg)
Michael J. Black, IWV 2009, Barcelona
SCAPE
[ Cyberware ] [ Allen et al. ’03 ]
Shape Completion and Animation of PEople [ Anguelov et al. Siggraph ’05 ]
CAESAR dataset (SAE Int.): 3D mesh
models of over 2000 North American adults
![Page 8: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/8.jpg)
Michael J. Black, IWV 2009, Barcelona
Shape space • Align a “template mesh” to each scan using an
iterative closest point method.
• 25,000 triangles and 12,500 vertices.
Example scans of different people. [ Anguelov et al. ’05 ]
![Page 9: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/9.jpg)
Michael J. Black, IWV 2009, Barcelona
Shape space • All vertices are in correspondence.
• All bodies in the canonical pose.
• Vertices assigned to body “parts”.
Example scans of different people. [ Anguelov et al. ’05 ]
![Page 10: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/10.jpg)
Michael J. Black, IWV 2009, Barcelona
Shape space • Learn low dimensional shape deformation model
using incremental PCA (Brand, ECCV’02). – applied to deformations (i.e. rotations) of the triangles.
Example scans of different people. [ Anguelov et al. ’05 ]
![Page 11: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/11.jpg)
Michael J. Black, IWV 2009, Barcelona
Adam Pat Eve
![Page 12: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/12.jpg)
Michael J. Black, IWV 2009, Barcelona
Eigen-People
![Page 13: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/13.jpg)
Michael J. Black, IWV 2009, Barcelona
Cum
ula
tive R
ela
tive V
ariance
Number of principal components
2124 scans of various people [ CAESAR dataset ]
6 20
![Page 14: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/14.jpg)
Michael J. Black, IWV 2009, Barcelona
Gender from shape
Learn 3 models:
1. Men; 2. Women;
3. Both together.
![Page 15: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/15.jpg)
Michael J. Black, IWV 2009, Barcelona
What about pose changes? For each part, apply a rigid rotation to each triangle.
Rpart
(3x3)
x 15 parts
Preserve desired orientation and scale of
edges in a least-squares sense
[ Anguelov et al. ’05 ]
![Page 16: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/16.jpg)
Michael J. Black, IWV 2009, Barcelona
What about pose changes?
Deformation of each triangle from predicted
model to training example.
Qt
(3x3)
Triangles are all in correspondence.
[ Anguelov et al. ’05 ]
![Page 17: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/17.jpg)
Michael J. Black, IWV 2009, Barcelona
Pose deformation space
Model articulated and non-rigid deformations
– Non-rigid deformations learned as a function of relative
part orientations.
70 example scans in diverse
poses [ Anguelov et al. ’05 ]
![Page 18: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/18.jpg)
Michael J. Black, IWV 2009, Barcelona
Pose deformation space
Model non-rigid deformations.
Linear prediction from relative part
rotations.
[ Anguelov et al. ’05 ]
Learn a matrix At for each triangle
t in the mesh via linear regression.
![Page 19: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/19.jpg)
Michael J. Black, IWV 2009, Barcelona
SCAPE deformations
– part rotations
Articulated Rigid
Deformation
Non-rigid
Deformation
Body Shape
Deformation
– shape parameters
Qt ( ) St ( ) Rp ( )
[ Anguelov et al. ’05 ]
![Page 20: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/20.jpg)
Michael J. Black, IWV 2009, Barcelona
SCAPE deformations
– part rotations ( 37D ) – shape parameters ( 6-20D )
Qt ( ) St ( ) Rp ( )
Parameters (state): s = ( , , )
– global position ( 3D )
[ Anguelov et al. ’05 ]
Shape parameters can be gender specific.
![Page 21: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/21.jpg)
Michael J. Black, IWV 2009, Barcelona
![Page 22: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/22.jpg)
Michael J. Black, IWV 2009, Barcelona
Pose and shape optimization
Recover s = ( , , ) (Initialization: Sigal et al. NIPS
’07)
![Page 23: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/23.jpg)
Michael J. Black, IWV 2009, Barcelona
Minimize
Pose and shape optimization
Recover s = ( , , )
K cameras
Penalize interpenetration.
bi-directional
distance measure
between silhouettes
![Page 24: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/24.jpg)
Michael J. Black, IWV 2009, Barcelona
Minimize
Pose and shape optimization
bi-directional
distance measure
between silhouettes
Recover s = ( , , )
K cameras
Penalize interpenetration.
E(s) = d(Fie (s),Fi
o) + (1 ) d(Fio,Fi
e (s))i=1
K
![Page 25: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/25.jpg)
Michael J. Black, IWV 2009, Barcelona
Optimization – Step by Step
![Page 26: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/26.jpg)
Michael J. Black, IWV 2009, Barcelona
Optimization – Step by Step
![Page 27: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/27.jpg)
Michael J. Black, IWV 2009, Barcelona
Optimization – Step by Step
![Page 28: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/28.jpg)
Michael J. Black, IWV 2009, Barcelona
Optimization – Step by Step
![Page 29: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/29.jpg)
Michael J. Black, IWV 2009, Barcelona
Optimization – Step by Step
![Page 30: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/30.jpg)
Michael J. Black, IWV 2009, Barcelona
Optimization – Step by Step
![Page 31: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/31.jpg)
Michael J. Black, IWV 2009, Barcelona
Optimization – Step by Step
![Page 32: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/32.jpg)
Michael J. Black, IWV 2009, Barcelona
The “naked” case
20 bases.
Subject not in training set.
![Page 33: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/33.jpg)
Michael J. Black, IWV 2009, Barcelona
Problem: Clothing
![Page 34: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/34.jpg)
Michael J. Black, IWV 2009, Barcelona
Problem: Clothing
![Page 35: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/35.jpg)
Michael J. Black, IWV 2009, Barcelona
Can you guess what someone looks like under
their clothes?
– Exploit the factorization of the model to combine
constraints across poses.
Shape under clothing
![Page 36: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/36.jpg)
Michael J. Black, IWV 2009, Barcelona
• Silhouettes are larger when there is clothing
Principle: Shape under clothing
![Page 37: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/37.jpg)
Michael J. Black, IWV 2009, Barcelona
Principle: Shape under clothing • Silhouettes are larger when there is clothing
– Body must fit inside silhouettes
E inside s( ) = d Fk,se ,Fk
o( )
![Page 38: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/38.jpg)
Michael J. Black, IWV 2009, Barcelona
• Silhouettes are larger when there is clothing
– Body must fit inside silhouettes
– Should not try to explain the entire image silhouette
Principle: Shape under clothing
E inside s( ) = d Fk,se ,Fk
o( )
d Fko,Fk,s
e( ) - NO
![Page 39: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/39.jpg)
Michael J. Black, IWV 2009, Barcelona
• Silhouettes are larger when there is clothing
– Body must fit inside silhouettes
– Constraints are tight in regions without clothes
Principle: Shape under clothing
![Page 40: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/40.jpg)
Michael J. Black, IWV 2009, Barcelona
• Silhouettes are larger when there is clothing
– Body must fit inside silhouettes
– Constraints are tight for skin regions
Skin Detection
Principle: Shape under clothing
![Page 41: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/41.jpg)
Michael J. Black, IWV 2009, Barcelona
• Silhouettes are larger when there is clothing
– Body must fit inside silhouettes
– Constraints are tight for skin regions
Eexpand s( ) = d Sko,Fk,s
e( ) + d Fko \ Sk
o,Fk,se( )
Principle: Shape under clothing
skin non-skin
Image Skin / Non-skin
Silhouettes
Model
Silhouette Overlap
model model
![Page 42: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/42.jpg)
Michael J. Black, IWV 2009, Barcelona
• Silhouettes are larger when there is clothing
– Body must fit inside silhouettes
– Constraints are tight for skin regions
• True shape not observable
– Family of human body shapes (known statistics)
Principle: Shape under clothing
[ Allen et al. ’03 ]
![Page 43: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/43.jpg)
Michael J. Black, IWV 2009, Barcelona
• Silhouettes are larger when there is clothing
– Body must fit inside silhouettes
– Constraints are tight for skin regions
• True shape not observable
– Family of human body shapes (known statistics)
Principle: Shape under clothing
Objective Function
![Page 44: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/44.jpg)
Michael J. Black, IWV 2009, Barcelona
Comparison Single-pose Fitting with Clothes Fitting as if Naked
![Page 45: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/45.jpg)
Michael J. Black, IWV 2009, Barcelona
• Silhouettes are larger when there is clothing
– Body must fit inside silhouettes
– Constraints are tight for skin regions
• True shape not observable
– Family of human body shapes (known statistics)
• Body shape constant although pose may vary:
Principle: Shape under clothing
![Page 46: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/46.jpg)
Michael J. Black, IWV 2009, Barcelona
• Silhouettes are larger when there is clothing
– Body must fit inside silhouettes
– Constraints are tight for skin regions
• True shape not observable
– Family of human body shapes (known statistics)
• Combine constraints across pose
Principle: Shape under clothing
“Batch” objective function
![Page 47: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/47.jpg)
Michael J. Black, IWV 2009, Barcelona
Comparison Fitting as if Naked Single-pose Fitting Batch Fitting
![Page 48: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/48.jpg)
Michael J. Black, IWV 2009, Barcelona
![Page 49: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/49.jpg)
Michael J. Black, IWV 2009, Barcelona
Shape under clothing
![Page 50: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/50.jpg)
Michael J. Black, IWV 2009, Barcelona
Quantitative Evaluation
![Page 51: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/51.jpg)
Michael J. Black, IWV 2009, Barcelona
Failure case
![Page 52: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/52.jpg)
Michael J. Black, IWV 2009, Barcelona
![Page 53: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/53.jpg)
Michael J. Black, IWV 2009, Barcelona
What about “regular” video?
• Everything here is multi-camera
– See ICCV’07 for some monocular results (using cast shadows)
• Combine multiple poses to estimate shape
– These poses can come from different cameras or different frames
of the same camera.
• Though pose estimation in a single frame is harder.
• There are many more cues to use beyond silhouettes
• Assume calibration here
![Page 54: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/54.jpg)
Michael J. Black, IWV 2009, Barcelona
Applications
• Surveillance
– Extract body shape measurements from surveillance
video.
– Gender from video (94.3% correct on our clothing
dataset).
• Games
– Avatar creation and animation.
• Fashion
– Extract body shape measurements.
– Virtual try-on.
![Page 55: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/55.jpg)
Michael J. Black, IWV 2009, Barcelona
Summary of approach
• Estimating human shape may be as important as estimating 3D pose
• Detailed graphics model practical for vision – Sometimes complexity makes things easier (or possible).
– Better match to image evidence (e.g. torso shape).
– Initial optimization approx 5hrs (!) per frame; down to about 2min.
• Shape under clothing – Combine constraints on shape constancy and skin with clothing-
appropriate observation model.
• Representation – Supports more than “detection”; is someone tall? fat? male? old?
– 3D model allows us to explain illumination, shading, pose variation, muscle bulging, ….
– This allow us to extract what is constant over time (identity, shape, etc.)
![Page 56: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/56.jpg)
Michael J. Black, IWV 2009, Barcelona
Current & future work
• Monocular estimation and tracking in multiple frames with moving camera – Constraints on shape consistency
• Beyond silhouettes – Internal structure & motion
• Modeling clothing and hair
• Dynamics of soft tissue and cloth
• Evaluation of accuracy – Pose and shape
See Balan et al, ICCV ‘07
![Page 57: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/57.jpg)
Michael J. Black, IWV 2009, Barcelona
The evolution of man
1996
2000 2007
![Page 58: Body Shape from Video - research.microsoft.com€¦ · Michael J. Black, IWV 2009, Barcelona Summary of approach • Estimating human shape may be as important as estimating 3D pose](https://reader030.vdocument.in/reader030/viewer/2022041020/5ecfa826bd702a2768028527/html5/thumbnails/58.jpg)
Michael J. Black, IWV 2009, Barcelona
Acknowledgements Funding Sources:
– NSF IIS-0535075, IIS-0812364
– NIH-NINDS R01 NS 50967-01
• part of the Collaborative Research on Computational Neuroscience
Program
– Intel Corporation
Collaborators: Horst Haussecker (Intel), Leonid Sigal
(Toronto), James Davis (UCSC)
Thanks: Dragomir Anguelov (Google), Alex Weiss (Brown),
Matthew Loper (Brown), Steven Scher (UCSC), Loretta
Reiss (Brown), David Hirshbirg (Brown)