![Page 1: Multiple Frame Motion Inference Using Belief Propagation](https://reader033.vdocument.in/reader033/viewer/2022042822/56812ced550346895d91b488/html5/thumbnails/1.jpg)
Multiple Frame Motion Inference Using Belief
Propagation
Jiang Gao Jianbo Shi
Presented By: Gilad Kapelushnik
Visual Recognition, Spring 2005, Technion IIT.
![Page 2: Multiple Frame Motion Inference Using Belief Propagation](https://reader033.vdocument.in/reader033/viewer/2022042822/56812ced550346895d91b488/html5/thumbnails/2.jpg)
Abstract
• Find “best fit” upper body joint configuration.
• Input is a 2D video
• Each joint is described by its location on a 2D grid.
S1(X,Y)
S2(X,Y)
S4(X,Y)
S6(X,Y)
S5(X,Y)
S3(X,Y)
Let J be a joint configuration – {S1,S2,S3,S4,S5,S6}
We would like to find:
argmax ( | )J
J P J observations
![Page 3: Multiple Frame Motion Inference Using Belief Propagation](https://reader033.vdocument.in/reader033/viewer/2022042822/56812ced550346895d91b488/html5/thumbnails/3.jpg)
• Step 1: Subtract two sequential frames.
• Step 2: Apply threshold.
Motion Energy Image
![Page 4: Multiple Frame Motion Inference Using Belief Propagation](https://reader033.vdocument.in/reader033/viewer/2022042822/56812ced550346895d91b488/html5/thumbnails/4.jpg)
From #NrgPixels To Probability• Sum the Energy Pixels in the Patch.
• Calculate probability using the following:
S5(10,60)
S6(40,30)
1 exp( # )
NormConstP
NrgPixels
![Page 5: Multiple Frame Motion Inference Using Belief Propagation](https://reader033.vdocument.in/reader033/viewer/2022042822/56812ced550346895d91b488/html5/thumbnails/5.jpg)
0.12
0.19
0.84
0.68
0.02
• Find configuration J with the highest probability.
• Computing all possible probabilities is inefficient.
• a-Priori data give better and faster results.
• removing impossible configurations reduce inference time.
Main Idea
( )JX Y
![Page 6: Multiple Frame Motion Inference Using Belief Propagation](https://reader033.vdocument.in/reader033/viewer/2022042822/56812ced550346895d91b488/html5/thumbnails/6.jpg)
a-Priori Data• A probability table for Each P(Sx,Sy).
• Compute probability at grid crossing.
• Use nearest neighbor for the rest of the image.
Example:
For right arm - P(S2,S3)
Red – Low probability
Green – High probability
P(S2,S3)12…Ns^2
100…0
200.1…0
…………0
Ns^20000
![Page 7: Multiple Frame Motion Inference Using Belief Propagation](https://reader033.vdocument.in/reader033/viewer/2022042822/56812ced550346895d91b488/html5/thumbnails/7.jpg)
• Face is detected using face detection algorithm.
• Initial assumption of Shoulders from face and pose.
• Even using BP there are too many possible states to go through.
• Candidates for elbows from shoulders & Energy Map.
• Candidates for Wrists from skin color model.
Detect Candidate states (1)
![Page 8: Multiple Frame Motion Inference Using Belief Propagation](https://reader033.vdocument.in/reader033/viewer/2022042822/56812ced550346895d91b488/html5/thumbnails/8.jpg)
Detect Candidate states (2)• Many states can be discarded.
• Remove close candidate states.
Pros: Much faster inference.
Cons: Less accurate.
• Note: This is only an option.
Fits skin color and wrist location
Pink for right wristRed for left wrist
Blue for elbow
![Page 9: Multiple Frame Motion Inference Using Belief Propagation](https://reader033.vdocument.in/reader033/viewer/2022042822/56812ced550346895d91b488/html5/thumbnails/9.jpg)
The Markov Model• Empty Circles - States - 2D positions of joints
• Full Circles - Observations - Computed from energy map.
• Each state correspond to an observation.
![Page 10: Multiple Frame Motion Inference Using Belief Propagation](https://reader033.vdocument.in/reader033/viewer/2022042822/56812ced550346895d91b488/html5/thumbnails/10.jpg)
Belief Propagation (1)• Solve inference problem using an algorithm with Linear complexity.
• Each joint has a vector with probabilities for each candidate.
Shoulder
Elbow
Wrist
![Page 11: Multiple Frame Motion Inference Using Belief Propagation](https://reader033.vdocument.in/reader033/viewer/2022042822/56812ced550346895d91b488/html5/thumbnails/11.jpg)
Belief Propagation (2)
1
2
3
m23
m32
m21
m12m14
m41For each iteration:
• Each node sends a message to its neighbor nodes containing the “wanted” probability (for each state).
• Messages are computed according to:
( ) /
( )
( , ) ( | ) ( )
( ) ( | ) ( )
i
ij ij i j i i i ki is k N i j
i i i i i ki ik N i
m P s s P x s m s
b s P x s m s
Sum over all candidates
A-priori Data for each state.
Normalize variable.
Observation (# of Energy pixels in patch) for each state
converted to a probability.
Message from k to i (all messages from the
neighbors). This is actually a vector with a probability for
each state.
Message from i to j.
![Page 12: Multiple Frame Motion Inference Using Belief Propagation](https://reader033.vdocument.in/reader033/viewer/2022042822/56812ced550346895d91b488/html5/thumbnails/12.jpg)
Belief Propagation (3) - Example
21
Message from 1 to 2
4states
2states
![Page 13: Multiple Frame Motion Inference Using Belief Propagation](https://reader033.vdocument.in/reader033/viewer/2022042822/56812ced550346895d91b488/html5/thumbnails/13.jpg)
Belief Propagation (4)
• BP converge after 2-4 iterations (giving the right a-Priori data).
• For every joint there is a probability vector for each candidate state.
![Page 14: Multiple Frame Motion Inference Using Belief Propagation](https://reader033.vdocument.in/reader033/viewer/2022042822/56812ced550346895d91b488/html5/thumbnails/14.jpg)
Multiple Frame Probability
• Multiple frame (8) is proposed for smoother transition between configurations.
• Prevents joints changing their state to a different which is “far away” (Euclidian distance).
• Though BP was designed to work with loopy-free models, the author stated that it worked fine.
21
, 1 12
1( , ) exp
2
t ti it t t t
i i i
s sP s s
And for those who really want to know:
![Page 15: Multiple Frame Motion Inference Using Belief Propagation](https://reader033.vdocument.in/reader033/viewer/2022042822/56812ced550346895d91b488/html5/thumbnails/15.jpg)
2D to 3D
• 2D -> 3D by Taylor (2000).
• Assuming (u1,v1) and (u2,v2) are projections then depth can be retrieved using the following:
1 2 1 2
1 2 1 2
2 22 1 2 1 2
1 2 2
( ) ( )
( ) ( )
( ) ( ){ _ }
co
u u const X X
v v const Y Y
u u v vZ Z arm length
nst
![Page 16: Multiple Frame Motion Inference Using Belief Propagation](https://reader033.vdocument.in/reader033/viewer/2022042822/56812ced550346895d91b488/html5/thumbnails/16.jpg)
Results(1)
![Page 17: Multiple Frame Motion Inference Using Belief Propagation](https://reader033.vdocument.in/reader033/viewer/2022042822/56812ced550346895d91b488/html5/thumbnails/17.jpg)
Results(2)
![Page 18: Multiple Frame Motion Inference Using Belief Propagation](https://reader033.vdocument.in/reader033/viewer/2022042822/56812ced550346895d91b488/html5/thumbnails/18.jpg)
Results(3)
Errors accrue when 2 joints intersect each other.
On some occasions, even when limbs intersect, it was possible to infer correctly.
![Page 19: Multiple Frame Motion Inference Using Belief Propagation](https://reader033.vdocument.in/reader033/viewer/2022042822/56812ced550346895d91b488/html5/thumbnails/19.jpg)
Q?