Modeling the Shape of People from 3D Range Scans

Dragomir AnguelovAI Lab

Stanford University

Joint work withPraveen Srinivasan, Hoi-Cheung Pang,

Daphne Koller, Sebastian Thrun

The Dataset Scans

4 views, ~125k polygons ~65k points each

Problems Missing surface Drastic pose changes

Space of Human Shapes movie

[scape movie]

The Modeling Pipeline

Talk outline Scan registration

Rigid Nonrigid Correlated correspondence

Decomposition into approximately rigid parts

Learning a deformable human model

Talk outline 3D Scan registration

Rigid Nonrigid Correlated correspondence

Decomposition into approximately rigid parts

Learning a deformable human model

Rigid Scan Registration Aligning point clouds

X and Z Match each point in Z

to its closest point in X Compute the best

rigid transformation T which aligns the matching pairs

Transform the points of cloud X by T

Repeat

Iterative Closest Point (ICP) algorithm [Besl et al ’92]

ICP matching step Objective

Setting obtains

Substituting the above for s yields a system linear in R Choose a parameterization of R (quaternions q) and solve

Exact solution Overall, ICP converges to a local minimum of the

energy

(using )

Nonrigid Registration Example Deformable template

model X Set of deformable springs

Nonrigid registration Deforms X to match a scan Z Minimizes the deformation of

X

Generative Model

Model mesh X Transformed mesh X’

Deformation / Transformati

on

Goal: Given meshes X and Z, recover transformation and correspondences C

Data mesh Z

Data Generation /

Correspondences

C

Data mesh generation

ZX’

kzix'

),';()',|( Zikikk xzNxiczP

Correspondence ck specifies which point in X’ generates point zk:

DataTransformed

Model

X Z

Nonrigid ICP failure example

Local Minima Poor transformation and correspondence estimates

reinforce each other

Nonrigid ICP attempts to maximize

Correlated correspondences Nonrigid-ICP decorrelates the correspondences

Correspondences are conditionally independent given

If is unknown, correspondences are correlated

E.g. nearby points in Z should be mapped to nearby points in X

The search space is exponential for the general registration problem

c1c

2

Correlated Correspondence

Compute an embedding of mesh Z into model X

The domain of ci is the set of possible (discrete) matches for zi on the surface of X

The solution is a consistent assignment to all correspondence variables C

Z

X

Correlated CorrespondenceZ

X

Edge is generated from some edge in X

Prefer values of c1, c2 according to the deformation of X they induce

Doing this for all edges in Z results in a Markov network over C

c2c1

c3

(c1, c2)

(c2,c3)(c1,c3)

(c2)

(c3)

(c1)

{1, …, K}

Pairwise Markov Networks

Inference is Markov Nets is generally intractable Good approximate algorithms exist (belief

propagation)

yi yjiij

K

…

1

K K

…

1 2

1 1

CC Potentials

Deformation potentials Spin image* potentials

Quantify the similarity of the local surface around two matching points

Geodesic Potentials

)( ick

),( jcic lkG

*[Johnson+Hebert ’97]

c2c1

c3

(c1, c2)

(c2,c3)(c1,c3)

(c2)

(c3)

(c1)

Geodesic potentials

Nearby points in Z must be nearby in X Constraint between each pair of

adjacent points zk, zl

Distant points in Z must be distant in X Constraint between each pair of distant

points zk, zl (farther than 5r)

Z X Z X

otherwise

rxxDistjcic jiG

lk 13),(0

),(

otherwise

rxxDistjcic jiG

lk 12),(0

),(

r resolution of mesh X

Local surface signatures

Use spin-images [Johnson ’97] 2D Histogram of distances from an oriented reference point Rotationally-invariant / Robust under clutter and occlusion / Compressible

(PCA) Potential (ck = i) encodes how well the signature of point zk matches

the signature of point xi in the model:

),;()( Sikk SSNic

Results: Human poses datasetModel

ArtModel I ArtModel II

• 4 markers were used on each scan to avoid the need for multiple initializations of Loopy-BP• ArtModel I found using 20 registered scans• ArtModel II found using 44 registered scans

Cyberware scans

Registrations

CC movie

[movie]

Talk Outline Scan registration

Rigid Nonrigid Correlated correspondence

Recovering the articulated skeleton Learning a deformable human model

Recovering the skeleton

Input: models, correspondences

Output: rigid parts, skeleton

Probabilistic Generative Model

Transformations

T

TransformedModel y1 yN…

Model a1

x1

aN

xN

……

Part labels

Points

aj = Part (xj)

z1

b1

zK

bK

Instance …

Point labels

Points

bk = ModelPoint (zk)

Part Properties Parts are preferably contiguous regions

Adjacent points on the surface should have similar labels

Enforce this with a Markov network: a1 a3

a2

ji

jiji

:1:

),(

If i, j are connected in model mesh

1 > > 0.5

Iterative Optimization

Hard EM Given transformations , perform min-cut

inference to get

Given labels , solve for rigid transformations

Results: Puppet articulation

Results: Arm articulation

Parts movie

[parts.gif]

Talk Outline Scan registration

Rigid Nonrigid Correlated correspondence

Recovering the articulated skeleton Learning a deformable human model

Modeling Human ShapePo

se v

aria

tion

Body-shape variation

Representation of pose deformation

Pose deformation

Rigid articulateddeformation

Given estimates of R, Q, synthesizing the shape is straightforward :

Learning non-rigid pose deformation For each polygon, predict entries of from

rotations of nearest 2 joints (represented as twists ).

Linear regression parameters :

Learning body-shape deformation Include also change in shape due to different

people:

Do PCA over body-shape matrices :

Shape completion Process:

Add a few markers (~6-8)

Run CC algorithm to get >100 markers

Optimize for pose/body-shape parameters and non-rigid surface

Mocap shape completion

Recap Algorithms for registration between two surfaces

ICP algorithm iterates between finding corresponding points between two scans and computing the transformation between them

Rigid variant (6 unknown DOF) Nonrigid variant (6 Number of polygons DOF)

CC explicitly encodes correlations between correspondence variables

Essential when we have poor initial transformation estimate Exponential complexity, approximate algorithm

Modeling humans Decouple effects of pose and different physique on shape

deformation

The end

Geodesic Potentials: closeclose

Nearby points in Z must be nearby in X Constraint between each pair of adjacent points

zk, zl

Z X

Geodesic Potentials: farfar

Distant points in Z must be distant in X Constraint between each pair of distant points

zk, zl

Z X