1 MSRIUniversity of California Berkeley

Recovering Human Body Configurations using Pairwise Constraints between PartsRecovering Human Body Configurations using Pairwise Constraints between Parts

Xiaofeng Ren, Alex Berg, Jitendra MalikXiaofeng Ren, Alex Berg, Jitendra Malik

2 MSRIUniversity of California Berkeley

Finding PeopleFinding People

Challenges: Pose, Clothing, Lighting, Clutter, …

3 MSRIUniversity of California Berkeley

Previous WorkPrevious Work• Related Domains

– Tracking People– Detecting Pedestrians– ... …

• Localizing Human Figures– Exemplar-based:

[Toyama & Blake 01], [Mori & Malik 02], [Sullivan & Carlsson 02], [Shakhnarovich, Viola & Darrell 03], …

– Part-based:[Felzenswalb & Huttenlocher 00], [Ioffe & Forsyth 01], [Song, Goncalves & Perona 03], [Mori, Ren, Efros & Malik 04], …

– … …

• Related Domains– Tracking People– Detecting Pedestrians– ... …

• Localizing Human Figures– Exemplar-based:

[Toyama & Blake 01], [Mori & Malik 02], [Sullivan & Carlsson 02], [Shakhnarovich, Viola & Darrell 03], …

– Part-based:[Felzenswalb & Huttenlocher 00], [Ioffe & Forsyth 01], [Song, Goncalves & Perona 03], [Mori, Ren, Efros & Malik 04], …

– … …

4 MSRIUniversity of California Berkeley

Beyond “Trees”Beyond “Trees”

• A hard problem! More information is needed.

• Important cues that are NOT in the tree model:– Symmetry of clothing/color

– “V-shape” formed by the upper legs

– Distance/smooth connection between arms and legs

– ……

• A hard problem! More information is needed.

• Important cues that are NOT in the tree model:– Symmetry of clothing/color

– “V-shape” formed by the upper legs

– Distance/smooth connection between arms and legs

– ……

?

5 MSRIUniversity of California Berkeley

Our ApproachOur Approach• Preprocessing with Constrained Delaunay Triangulation

• Detecting Candidate Parts from Bottom-up

• Learning Pairwise Constraints between Parts

• Assembling Parts by Integer Quadratic Programming (IQP)

• Preprocessing with Constrained Delaunay Triangulation

• Detecting Candidate Parts from Bottom-up

• Learning Pairwise Constraints between Parts

• Assembling Parts by Integer Quadratic Programming (IQP)

6 MSRIUniversity of California Berkeley

Constrained Delaunay TriangulationConstrained Delaunay Triangulation

– Detect edges with Pb (Probability of Boundary)

– Trace contours with Canny’s hysteresis

– Recursively split contours into piecewise straight lines

– Complete the partial graph with Constrained Delaunay Triangulation

– Detect edges with Pb (Probability of Boundary)

– Trace contours with Canny’s hysteresis

– Recursively split contours into piecewise straight lines

– Complete the partial graph with Constrained Delaunay Triangulation

7 MSRIUniversity of California Berkeley

Detecting Parts using ParallelismDetecting Parts using Parallelism

(L1,1)

(L2,2)T

N

• Candidate parts as parallel line segments (Ebenbreite)

• (Scale-invariant) Features for parallelism:|Pb1+Pb2|/2, |1-2|, |L1-L2|/|L1+L2|, |(C1-C2)T|/|L1+L2|, |(C1-C2)N|/|L1+L2|

• Logistic Classifier

• Candidate parts as parallel line segments (Ebenbreite)

• (Scale-invariant) Features for parallelism:|Pb1+Pb2|/2, |1-2|, |L1-L2|/|L1+L2|, |(C1-C2)T|/|L1+L2|, |(C1-C2)N|/|L1+L2|

• Logistic Classifier

C1

C2

8 MSRIUniversity of California Berkeley

Pairwise Constraints between PartsPairwise Constraints between Parts

• Scale (width) consistency– Use anthropometric data as groundtruth

• Symmetry of appearance (color)

• Orientation consistency

• Connectivity– Short distance between adjacent parts

– “Smooth” connection between non-adjacent parts• short “gaps” on shortest path (on CDT graph)

• small maximum angle on the shortest path

• few T-junctions/turns on the shortest path

• Scale (width) consistency– Use anthropometric data as groundtruth

• Symmetry of appearance (color)

• Orientation consistency

• Connectivity– Short distance between adjacent parts

– “Smooth” connection between non-adjacent parts• short “gaps” on shortest path (on CDT graph)

• small maximum angle on the shortest path

• few T-junctions/turns on the shortest path

C1

C2

9 MSRIUniversity of California Berkeley

Learning Pairwise ConstraintsLearning Pairwise Constraints

15 hand-labeled images from a skating sequence

Empirical distributions of some pairwise features

For simplicity, assume all features are Gaussian (future work here as they are clearly non-Gaussian)

10 MSRIUniversity of California Berkeley

Assembling Parts as AssignmentAssembling Parts as Assignment

Candidates {Ci} Parts {Lj}

(Lj1,Ci1=(Lj1))

(Lj2,Ci2=(Lj2))

Cost for a partial assignment {(Lj1,Ci1), (Lj2,Ci2)}:

assignment

2

)22)(11( 2,1

2,1)22(),11(

kijij jj

k

jjk

ijijkf

H

11 MSRIUniversity of California Berkeley

Assignment by IQPAssignment by IQP• Suppose there are m parts and n candidates, the optimal

assignment minimizes a quadratic function• Suppose there are m parts and n candidates, the optimal

assignment minimizes a quadratic function

Q(x)=xTHxQ(x)=xTHxwhere x is a mn1 indicator vector and H is of size mnmn.where x is a mn1 indicator vector and H is of size mnmn.

• This is a well-formulated Integer Quadratic Programming (IQP) problem and has efficient approximate solutions.

• We choose an approximation scheme which solves mn linear programs followed by gradient descent.

• The approximate scheme produces a ranked list of torso candidates. We consider the top 5 torso candidates and solve the corresponding 5 IQP problems.

• We have m=9 and n~150; the total time is less than a minute.

• This is a well-formulated Integer Quadratic Programming (IQP) problem and has efficient approximate solutions.

• We choose an approximation scheme which solves mn linear programs followed by gradient descent.

• The approximate scheme produces a ranked list of torso candidates. We consider the top 5 torso candidates and solve the corresponding 5 IQP problems.

• We have m=9 and n~150; the total time is less than a minute.

12 MSRIUniversity of California Berkeley

13 MSRIUniversity of California Berkeley

14 MSRIUniversity of California Berkeley

15 MSRIUniversity of California Berkeley

ConclusionConclusion

• To find people under general conditions, we need to go beyond the traditional tree-based model;

• Most important constraints for the human body are between pairs of body parts;

• Pairwise constraints may be learned from a small set of training examples;

• Integer Quadratic Programming (IQP) efficiently finds optimal configurations under pairwise constraints.

• To find people under general conditions, we need to go beyond the traditional tree-based model;

• Most important constraints for the human body are between pairs of body parts;

• Pairwise constraints may be learned from a small set of training examples;

• Integer Quadratic Programming (IQP) efficiently finds optimal configurations under pairwise constraints.

16 MSRIUniversity of California Berkeley