pose–cut simultaneous segmentation and 3d pose estimation of humans using dynamic graph cuts...
TRANSCRIPT
![Page 1: POSE–CUT Simultaneous Segmentation and 3D Pose Estimation of Humans using Dynamic Graph Cuts Mathieu Bray Pushmeet Kohli Philip H.S. Torr Department of](https://reader034.vdocument.in/reader034/viewer/2022051515/5514784d550346b0158b53c0/html5/thumbnails/1.jpg)
POSE–CUTSimultaneous Segmentation and 3D Pose
Estimation of Humans using Dynamic Graph Cuts
Mathieu Bray Pushmeet Kohli Philip H.S. Torr
Department of Computing
Oxford Brookes University
![Page 2: POSE–CUT Simultaneous Segmentation and 3D Pose Estimation of Humans using Dynamic Graph Cuts Mathieu Bray Pushmeet Kohli Philip H.S. Torr Department of](https://reader034.vdocument.in/reader034/viewer/2022051515/5514784d550346b0158b53c0/html5/thumbnails/2.jpg)
Objective
Image Segmentation Pose Estimate
[Images courtesy: M. Black, L. Sigal]
![Page 3: POSE–CUT Simultaneous Segmentation and 3D Pose Estimation of Humans using Dynamic Graph Cuts Mathieu Bray Pushmeet Kohli Philip H.S. Torr Department of](https://reader034.vdocument.in/reader034/viewer/2022051515/5514784d550346b0158b53c0/html5/thumbnails/3.jpg)
Outline
Image Segmentation Problem Pose-Specific Segmentation The Pose Inference Problem Optimization Results Conclusion and Future Work
![Page 4: POSE–CUT Simultaneous Segmentation and 3D Pose Estimation of Humans using Dynamic Graph Cuts Mathieu Bray Pushmeet Kohli Philip H.S. Torr Department of](https://reader034.vdocument.in/reader034/viewer/2022051515/5514784d550346b0158b53c0/html5/thumbnails/4.jpg)
Outline
Image Segmentation Problem Pose-Specific Segmentation The Pose Inference Problem Optimization Results Conclusion and Future Work
![Page 5: POSE–CUT Simultaneous Segmentation and 3D Pose Estimation of Humans using Dynamic Graph Cuts Mathieu Bray Pushmeet Kohli Philip H.S. Torr Department of](https://reader034.vdocument.in/reader034/viewer/2022051515/5514784d550346b0158b53c0/html5/thumbnails/5.jpg)
The Image Segmentation ProblemSegments
Image
![Page 6: POSE–CUT Simultaneous Segmentation and 3D Pose Estimation of Humans using Dynamic Graph Cuts Mathieu Bray Pushmeet Kohli Philip H.S. Torr Department of](https://reader034.vdocument.in/reader034/viewer/2022051515/5514784d550346b0158b53c0/html5/thumbnails/6.jpg)
Problem – MRF Formulation
Notation• Labelling x over the set of pixels• The observed pixel intensity values y (constitute data D)
Energy E (x) = - log Pr(x|D) + constant
Unary term• Likelihood based on colour
Pairwise terms• Prior• Contrast term
Find best labelling x* = arg min E(x)
![Page 7: POSE–CUT Simultaneous Segmentation and 3D Pose Estimation of Humans using Dynamic Graph Cuts Mathieu Bray Pushmeet Kohli Philip H.S. Torr Department of](https://reader034.vdocument.in/reader034/viewer/2022051515/5514784d550346b0158b53c0/html5/thumbnails/7.jpg)
MRF for Image Segmentation
D (pixels)
x (labels)
Image Plane
i
j
xi
xj Unary Potential
i(D|xi)
Pairwise Potential
ij(xi, xj)
xi = {segment1, …, segmentk} for instance {obj, bkg}
![Page 8: POSE–CUT Simultaneous Segmentation and 3D Pose Estimation of Humans using Dynamic Graph Cuts Mathieu Bray Pushmeet Kohli Philip H.S. Torr Department of](https://reader034.vdocument.in/reader034/viewer/2022051515/5514784d550346b0158b53c0/html5/thumbnails/8.jpg)
Can be solved using graph cuts
MRF for Image Segmentation
MAP SolutionPair-wise Terms
Contrast Term
IsingModel
Data (D) Unary likelihood
Unary likelihood
Maximum a-posteriori (MAP) solution x* =
![Page 9: POSE–CUT Simultaneous Segmentation and 3D Pose Estimation of Humans using Dynamic Graph Cuts Mathieu Bray Pushmeet Kohli Philip H.S. Torr Department of](https://reader034.vdocument.in/reader034/viewer/2022051515/5514784d550346b0158b53c0/html5/thumbnails/9.jpg)
MRF for Image Segmentation
Pair-wise Terms MAP SolutionUnary likelihoodData (D)
Unary likelihood
Contrast Term
Uniform Prior
Maximum-a-posteriori (MAP) solution x* =
Need for a human like segmentation
![Page 10: POSE–CUT Simultaneous Segmentation and 3D Pose Estimation of Humans using Dynamic Graph Cuts Mathieu Bray Pushmeet Kohli Philip H.S. Torr Department of](https://reader034.vdocument.in/reader034/viewer/2022051515/5514784d550346b0158b53c0/html5/thumbnails/10.jpg)
Outline
Image Segmentation Problem Pose-Specific Segmentation The Pose Inference Problem Optimization Results Conclusion and Future Work
![Page 11: POSE–CUT Simultaneous Segmentation and 3D Pose Estimation of Humans using Dynamic Graph Cuts Mathieu Bray Pushmeet Kohli Philip H.S. Torr Department of](https://reader034.vdocument.in/reader034/viewer/2022051515/5514784d550346b0158b53c0/html5/thumbnails/11.jpg)
Shape-Priors and Segmentation
OBJ-CUT [Kumar et al., CVPR ’05]– Shape-Prior: Layered Pictorial Structure (LPS)– Learned exemplars for parts of the LPS model– Obtained impressive results
Layer 2Layer 1Spatial Layout
(Pairwise Configuration)
+ =
![Page 12: POSE–CUT Simultaneous Segmentation and 3D Pose Estimation of Humans using Dynamic Graph Cuts Mathieu Bray Pushmeet Kohli Philip H.S. Torr Department of](https://reader034.vdocument.in/reader034/viewer/2022051515/5514784d550346b0158b53c0/html5/thumbnails/12.jpg)
Shape-Priors and Segmentation
OBJ-CUT [Kumar et al., CVPR ’05]– Shape-Prior: Layered Pictorial Structure (LPS)– Learned exemplars for parts of the LPS model– Obtained impressive results
Shape-Prior Colour + ShapeUnary likelihoodcolour
Image
![Page 13: POSE–CUT Simultaneous Segmentation and 3D Pose Estimation of Humans using Dynamic Graph Cuts Mathieu Bray Pushmeet Kohli Philip H.S. Torr Department of](https://reader034.vdocument.in/reader034/viewer/2022051515/5514784d550346b0158b53c0/html5/thumbnails/13.jpg)
Problems in using shape priors
Intra-class variability• Need to learn an
enormous exemplar set• Infeasible for complex
subjects (Humans)
Multiple Aspects?
Inference of pose parameters
![Page 14: POSE–CUT Simultaneous Segmentation and 3D Pose Estimation of Humans using Dynamic Graph Cuts Mathieu Bray Pushmeet Kohli Philip H.S. Torr Department of](https://reader034.vdocument.in/reader034/viewer/2022051515/5514784d550346b0158b53c0/html5/thumbnails/14.jpg)
Do we really need accurate models?
Interactive Image Segmentation [Boykov & Jolly, ICCV’01]• Rough region cues sufficient • Segmentation boundary can be extracted from edges
additional segmentation
cues
user segmentation cues
![Page 15: POSE–CUT Simultaneous Segmentation and 3D Pose Estimation of Humans using Dynamic Graph Cuts Mathieu Bray Pushmeet Kohli Philip H.S. Torr Department of](https://reader034.vdocument.in/reader034/viewer/2022051515/5514784d550346b0158b53c0/html5/thumbnails/15.jpg)
Do we really need accurate models? Interactive Image Segmentation
• Rough region cues sufficient • Segmentation boundary can be extracted from edges
![Page 16: POSE–CUT Simultaneous Segmentation and 3D Pose Estimation of Humans using Dynamic Graph Cuts Mathieu Bray Pushmeet Kohli Philip H.S. Torr Department of](https://reader034.vdocument.in/reader034/viewer/2022051515/5514784d550346b0158b53c0/html5/thumbnails/16.jpg)
Rough Shape Prior - The Stickman Model
26 degrees of freedom• Can be rendered extremely efficiently• Over-comes problems of learning a huge exemplar set• Gives accurate segmentation results
![Page 17: POSE–CUT Simultaneous Segmentation and 3D Pose Estimation of Humans using Dynamic Graph Cuts Mathieu Bray Pushmeet Kohli Philip H.S. Torr Department of](https://reader034.vdocument.in/reader034/viewer/2022051515/5514784d550346b0158b53c0/html5/thumbnails/17.jpg)
Pose-specific MRF Formulation
D (pixels)
x (labels)
Image Plane
i
j
xi
xj Unary Potential
i(D|xi)
Pairwise Potential
ij(xi, xj)
(pose parameters)
Unary Potentiali(xi|)
![Page 18: POSE–CUT Simultaneous Segmentation and 3D Pose Estimation of Humans using Dynamic Graph Cuts Mathieu Bray Pushmeet Kohli Philip H.S. Torr Department of](https://reader034.vdocument.in/reader034/viewer/2022051515/5514784d550346b0158b53c0/html5/thumbnails/18.jpg)
Pose-specific MRFEnergy to be
minimizedUnary term
Shape prior
Pairwise potential
Potts model
distance transform
![Page 19: POSE–CUT Simultaneous Segmentation and 3D Pose Estimation of Humans using Dynamic Graph Cuts Mathieu Bray Pushmeet Kohli Philip H.S. Torr Department of](https://reader034.vdocument.in/reader034/viewer/2022051515/5514784d550346b0158b53c0/html5/thumbnails/19.jpg)
Pose-specific MRFEnergy to be
minimizedUnary term
Shape prior
Pairwise potential
Potts model
+ =
Shape Prior
MAP Solution
Colourlikelihood
Data (D) colour+shape
![Page 20: POSE–CUT Simultaneous Segmentation and 3D Pose Estimation of Humans using Dynamic Graph Cuts Mathieu Bray Pushmeet Kohli Philip H.S. Torr Department of](https://reader034.vdocument.in/reader034/viewer/2022051515/5514784d550346b0158b53c0/html5/thumbnails/20.jpg)
What is the shape prior?Energy to be
minimizedUnary term
Shape prior
Pairwise potential
Potts model
How to find the value of
ө?
![Page 21: POSE–CUT Simultaneous Segmentation and 3D Pose Estimation of Humans using Dynamic Graph Cuts Mathieu Bray Pushmeet Kohli Philip H.S. Torr Department of](https://reader034.vdocument.in/reader034/viewer/2022051515/5514784d550346b0158b53c0/html5/thumbnails/21.jpg)
Outline
Image Segmentation Problem Pose-Specific Segmentation The Pose Inference Problem Optimization Results Conclusion and Future Work
![Page 22: POSE–CUT Simultaneous Segmentation and 3D Pose Estimation of Humans using Dynamic Graph Cuts Mathieu Bray Pushmeet Kohli Philip H.S. Torr Department of](https://reader034.vdocument.in/reader034/viewer/2022051515/5514784d550346b0158b53c0/html5/thumbnails/22.jpg)
Formulating the Pose Inference Problem
![Page 23: POSE–CUT Simultaneous Segmentation and 3D Pose Estimation of Humans using Dynamic Graph Cuts Mathieu Bray Pushmeet Kohli Philip H.S. Torr Department of](https://reader034.vdocument.in/reader034/viewer/2022051515/5514784d550346b0158b53c0/html5/thumbnails/23.jpg)
Formulating the Pose Inference Problem
![Page 24: POSE–CUT Simultaneous Segmentation and 3D Pose Estimation of Humans using Dynamic Graph Cuts Mathieu Bray Pushmeet Kohli Philip H.S. Torr Department of](https://reader034.vdocument.in/reader034/viewer/2022051515/5514784d550346b0158b53c0/html5/thumbnails/24.jpg)
Resolving ambiguity using multiple views
Pose specific Segmentation Energy
![Page 25: POSE–CUT Simultaneous Segmentation and 3D Pose Estimation of Humans using Dynamic Graph Cuts Mathieu Bray Pushmeet Kohli Philip H.S. Torr Department of](https://reader034.vdocument.in/reader034/viewer/2022051515/5514784d550346b0158b53c0/html5/thumbnails/25.jpg)
Outline
Image Segmentation Problem Pose-Specific Segmentation The Pose Inference Problem Optimization Results Conclusion and Future Work
![Page 26: POSE–CUT Simultaneous Segmentation and 3D Pose Estimation of Humans using Dynamic Graph Cuts Mathieu Bray Pushmeet Kohli Philip H.S. Torr Department of](https://reader034.vdocument.in/reader034/viewer/2022051515/5514784d550346b0158b53c0/html5/thumbnails/26.jpg)
Solving the Minimization ProblemSolving the Minimization Problem
Minimize F(ө) using Powell Minimization
To solve:
Let F(ө) =
Computational Problem:
Each evaluation of F(ө) requires a graph cut to be computed. (computationally expensive!!) BUT..
Solution: Use the dynamic graph cut algorithm [Kohli&Torr, ICCV 2005]
![Page 27: POSE–CUT Simultaneous Segmentation and 3D Pose Estimation of Humans using Dynamic Graph Cuts Mathieu Bray Pushmeet Kohli Philip H.S. Torr Department of](https://reader034.vdocument.in/reader034/viewer/2022051515/5514784d550346b0158b53c0/html5/thumbnails/27.jpg)
Dynamic Graph Cuts
PB SB
cheaperoperation
computationally
expensive operation
Simplerproblem
PB*
differencesbetweenA and B
A and Bsimilar
PA SA
solve
![Page 28: POSE–CUT Simultaneous Segmentation and 3D Pose Estimation of Humans using Dynamic Graph Cuts Mathieu Bray Pushmeet Kohli Philip H.S. Torr Department of](https://reader034.vdocument.in/reader034/viewer/2022051515/5514784d550346b0158b53c0/html5/thumbnails/28.jpg)
Dynamic Graph Cuts
20 msec
Simplerproblem
PB*
differencesbetweenA and B
A and Bsimilar
xasolve
xb400 msec
![Page 29: POSE–CUT Simultaneous Segmentation and 3D Pose Estimation of Humans using Dynamic Graph Cuts Mathieu Bray Pushmeet Kohli Philip H.S. Torr Department of](https://reader034.vdocument.in/reader034/viewer/2022051515/5514784d550346b0158b53c0/html5/thumbnails/29.jpg)
Outline
Image Segmentation Problem Pose-Specific Segmentation The Pose Inference Problem Optimization Results Conclusion and Future Work
![Page 30: POSE–CUT Simultaneous Segmentation and 3D Pose Estimation of Humans using Dynamic Graph Cuts Mathieu Bray Pushmeet Kohli Philip H.S. Torr Department of](https://reader034.vdocument.in/reader034/viewer/2022051515/5514784d550346b0158b53c0/html5/thumbnails/30.jpg)
Segmentation Results
Colour +Smoothness
Colour + Smoothness+ Shape Prior
Only Colour
Image
[Images courtesy: M. Black, L. Sigal]
![Page 31: POSE–CUT Simultaneous Segmentation and 3D Pose Estimation of Humans using Dynamic Graph Cuts Mathieu Bray Pushmeet Kohli Philip H.S. Torr Department of](https://reader034.vdocument.in/reader034/viewer/2022051515/5514784d550346b0158b53c0/html5/thumbnails/31.jpg)
Segmentation Results - Accuracy
Information used
% of object pixels correctly
marked
Accuracy(% of pixels correctly
classified)
Colour 45.73 95.2
Colour + GMM 82.48 96.9
Colour + GMM + Shape
97.43 99.4
![Page 32: POSE–CUT Simultaneous Segmentation and 3D Pose Estimation of Humans using Dynamic Graph Cuts Mathieu Bray Pushmeet Kohli Philip H.S. Torr Department of](https://reader034.vdocument.in/reader034/viewer/2022051515/5514784d550346b0158b53c0/html5/thumbnails/32.jpg)
Segmentation + Pose inference
[Images courtesy: M. Black, L. Sigal]
![Page 33: POSE–CUT Simultaneous Segmentation and 3D Pose Estimation of Humans using Dynamic Graph Cuts Mathieu Bray Pushmeet Kohli Philip H.S. Torr Department of](https://reader034.vdocument.in/reader034/viewer/2022051515/5514784d550346b0158b53c0/html5/thumbnails/33.jpg)
Segmentation + Pose inference
[Images courtesy: Vicon]
![Page 34: POSE–CUT Simultaneous Segmentation and 3D Pose Estimation of Humans using Dynamic Graph Cuts Mathieu Bray Pushmeet Kohli Philip H.S. Torr Department of](https://reader034.vdocument.in/reader034/viewer/2022051515/5514784d550346b0158b53c0/html5/thumbnails/34.jpg)
Outline
Image Segmentation Problem Pose-Specific Segmentation The Pose Inference Problem Optimization Results Conclusion and Future Work
![Page 35: POSE–CUT Simultaneous Segmentation and 3D Pose Estimation of Humans using Dynamic Graph Cuts Mathieu Bray Pushmeet Kohli Philip H.S. Torr Department of](https://reader034.vdocument.in/reader034/viewer/2022051515/5514784d550346b0158b53c0/html5/thumbnails/35.jpg)
Conclusions
• Efficient method for using shape priors for object-specific segmentation
• Efficient Inference of pose parameters using dynamic graph cuts
• Good segmentation results
• Pose inference- Needs further evaluation- Segmentation results could be used for silhouette intersection
![Page 36: POSE–CUT Simultaneous Segmentation and 3D Pose Estimation of Humans using Dynamic Graph Cuts Mathieu Bray Pushmeet Kohli Philip H.S. Torr Department of](https://reader034.vdocument.in/reader034/viewer/2022051515/5514784d550346b0158b53c0/html5/thumbnails/36.jpg)
Future Work
• Use dimensionality reduction to reduce the number of pose parameters.
- results in less number of pose parameteres to optimize- would speed up inference
• Use of features based on texture
• Appearance models for individual part of the articulated model (instead of using a single appearance model).
![Page 37: POSE–CUT Simultaneous Segmentation and 3D Pose Estimation of Humans using Dynamic Graph Cuts Mathieu Bray Pushmeet Kohli Philip H.S. Torr Department of](https://reader034.vdocument.in/reader034/viewer/2022051515/5514784d550346b0158b53c0/html5/thumbnails/37.jpg)
Thank You