Inferring Human Upper Body Motion

Jiang Gao, Jianbo ShiRobotics Institute

Carnegie Mellon University

Abstract

We present a new algorithm for automatic inferenceof human upper body motion in a natural scene. Theinitial motion cues are first detected from the capturedvideo. A graph model is proposed for human upper bodymotion, and motion inference is posed as a mappingproblem between state nodes in the graph model and themotion cues in images. Belief Propagation and dynamicprogramming algorithms are utilized for Bayesianinference of upper body motion in this graph model,which captures constraint of human body configurationunder specific view angles. A multi-frame inferencealgorithm is proposed to combine temporal smoothnessconstraints in human upper body motion. The algorithmis applied in a prototype system that can automaticallydetect and track human upper body motion from capturedvideos, without manual initialization of human bodyparts. We present evaluation results of our algorithm onthis system and some further considerations for refine themodel in the future.

1. Introduction

Human motion detection and tracking is useful inmany applications. In human-machine perceptualinterface, human communicate with computer using bodylanguages or gestures. Another application is humanactivity analysis, in which human motion and gestures aredetected and recognized from a video recording or videostreams from surveillance cameras.

Many research activities have been focused on trackingand recognition of human motion, body languages, andgestures. While recognition of routine activities, such aswalking, provides useful information in many aspects,human upper body gestures (including body language,limb movements and hand gestures) play an importantrole in communication, and get influence from differentcountry, culture, and different personalities, which makeit an important measurement for understanding intension,interaction, and personality. In this paper, we describe analgorithm for automatic detection and tracking of humanupper body motion.

Fully automatic detection and tracking of human upperbody motion is a challenging problem. Although inconstraint environments, such as motion capture using

markers, motion can be successfully and reliably detectedand tracked, it is difficult to detect and track human upperbody motion in real world without any restrictions.Illumination, clothing, and background, among others,can easily affect the performance of a vision system.

In fact, one of the most difficult problems is automaticdetection of human configuration (locate human bodyparts) without any prior knowledge or constraints. Theincapability of automatic detection makes it difficult forbuilding a fully automatic human motion tracking,recognition, and understanding system. Furthermore,unlike repetitive routine human activities, such aswalking or running, the motion pattern of human upperbody gesture is largely unknown, making it difficult totake advantage of the dynamics constraints.

The previous research efforts on human body detectionand tracking from video or images fall into severaldifferent areas: (1) Automatic detection of human body orbody parts; (2) Tracking of human motions; (3) Modelingof human motion patterns; (4) Recovery of 3Dconfiguration from 2D contour or point correspondences;(5) Recognition of human gestures.

For detection and tracking, the areas above are notindependent. The more aspects considered, the morelikely the detection and tracking system can performreliably. In this paper we focus on aspects that encompass(1), (2), and (4) above, for detection and tracking ofhuman upper body motion.

There are many works on human motion tracking(Bregler 1998, Gavrila 1996, Isard 1998, Ju, 1996, Rehg1995, Sidenbladh 2000). However, many of thealgorithms need manual initialization of body parts fortracking. Bayesian techniques, such as Condensationtracking, are widely used to take advantage of bothdynamics of human body motion and measurements fromimages. For algorithms with self-initialization ability,only some of them estimate details of body parts.Generally, detection of human body parts is based ondifferent image features and techniques.

Background subtraction and contour matching is aneffective technique for human detection and tracking(Wren 1997, Mittal 2002, Kettnaker 1999, Haritaoglu1998, Felzenszwalb 2001, Darrell 1998, Baumberg 1994),and is widely used in video surveillance applications withstatic cameras.

The Multi-view approach (Mittal 2002, Kettnaker1999, Darrell 1998) makes use of 3D information and can

ConfigurationInference

Temporal-SpatialInference

Feature Extraction

Head & Shoulder

LeftUpper trunk

Right uppertrunk

resolve some of the ambiguities in complex situations,such as occlusions. However, the application ofbackground subtraction and multi-view algorithms maynot always be possible in some applications, such ashuman activity analysis in recorded videos.

Several recent works are focused on using a templatemodel of human (Papageorgiou 1999, Stauffer 2001) ormotion patterns (Fablet 2002) to detect people in acluttered background. Signal processing techniques, suchas wavelets, can be applied to obtain a generic templatefor human body. Optical flow is used for parameterizationof human motion patterns. Both human and backgroundmodels are learned from example images, and likelihoodration between human and background is used to detectthe presence of people in the scene.

Body structure approach is proposed recently based oncomponent models of human body (Ioffe 2001,Felzenszwalb 2001). To find human, an algorithm isdesigned to search the space of possible humanconfigurations, and find the best match with the imageobservations. The hierarchical approach (Bregler 1997)takes advantage of combining mid-level simple motionmodel and high level gesture model with low levelfeatures, to detect and track human motion. Thehierarchical model needs a comprehensive understandingof human routine gaits and gestures to take advantage ofthe mid- and high level motion models.

Both of the above approaches have their advantagesand disadvantages concerning difficulties in modelacquisition, computational complexity of matching, androbustness of the algorithm. Compared with previousworks, we proposed a new framework that can detecthuman upper body motion and label body parts directlyfrom videos, without any manual initializations. We usemulti-frame Bayesian inference to perform detection andtracking at the same time.

We first apply face detection to find approximateposition of people. We then focus on interpreting motioncues and use it to infer human body configurations. Weadopt an articulated human model, the “cardboard person”model (Ju, 1996), and compute motion cues in the wholeareas of each body parts, rather than using local features.Instead of using a kinematics model (Bregler 1998, Rehg1995, Sidenbladh 2000), we capture constraints of humanbody configurations by using a joint probability of bodyjoint positions, which simplified the modeling and bodyconfiguration estimation process.

Using the methods above, we model the detection andtracking problem as Bayesian inference problem in aMarkov network (Jordan 1998, Yedidia 2001). Wepropose a multi-frame Bayesian inference algorithm forcombining both temporal (dynamics) and structuralconstraints in the Markov network. With temporalconstraints, the Bayesian inference algorithm is morerobust, and can even recover from detection errors in asingle frame.

The organization of this paper is as follows: In section2, we describe the algorithm structure and a Markov

network model for human upper body motion detectionand tracking; In section 3, we propose a multi-frameBayesian inference algorithm for upper body motiondetection and tracking. Section 4 describes a prototypesystem that can automatically detect and track humanupper body motion from a captured video, and someexperimental results are given. Section 5 concludes thepaper and gives comments on future works.

2. Detection of Human Upper Body Motion

In this section, we describe our approach for locatingbody parts in human upper body motion.

2.1. Algorithm structure

We utilize a hierarchical algorithm structure (Figure 1).In this three-layer structure, the first layer extracts motionand color features, and detects candidate position of bodyjoints based on these image features.

Figure 1. Algorithm structure for humanupper body motion detection andtracking.

The second Layer estimates human bodyconfigurations. In graphics and animation society, humanbody configuration is defined as the global translation androtation of a root, normally selected as the hips, and therelative angles of body joints all the way to the root.Since we are only interested in human upper bodyconfiguration, we define the root as shoulder girdle. Weobtain the position of shoulder girdle based on poseassumptions and face detection result. Instead of usingjoint angles, we define body configuration by positions ofhuman upper body joints. Considering human body asrigid bones connect together at joints, the positions ofthese joints implicitly define the joint angles. In oursystem, human upper body is represented as in Figure 2.

Detection of human body parts in a single frame isprone to error. In the third layer of our algorithm, wecombine temporal constraints in the algorithm.

Figure 2. Definition of human upper bodyconfiguration based on body joint positions(the empty circles).

FaceDetectio

n

Video Stream

ColorModel

ColorFeature

extraction

Motion Feature Extraction

Detection and Trackingof Body Joints

Figure 3 shows a result of the algorithm, withestimated 2D joint positions and 3D recovery result.

2.2. Feature extraction

The feature extraction part of our algorithm is shownin Figure 4. We apply frame difference to obtain motioninformation for subsequent processing. Since we arefocused on detection of human motions, we are interestedin finding parallel contours. We use a multi-scale contourfilter and apply it to the frame difference image. Contourfiltering is computed as:

),()],([),( yxfyxGaussDyxg *= , (1)

where []⋅D is derivative function, ),( yxGauss is a

Gaussian function. ),( yxf is the input pixel intensity at

position ),( yx , and ),( yxg is result.After contour filtering, threshold detection is

conducted on ),( yxg to find the contours, and if there isa parallel contour exists in the image, the contour ispreserved as motion cues. We control the sensitivity(scale) of the contour detector by the width of theGaussian function ),( yxGauss and the threshold for

contour detection from ),( yxg .For color feature extraction, we apply face detection

algorithm first, and build a skin color model from thedetected face region. We then use this model to findcandidate positions of hands and wrist joints.

(a)

(b)

Figure 3. An upper body motion detectionresult. (a) Estimated 2D joint position; ( b )3D Recovery result of joint coordinates,two views.

Figure 4. Diagram of the feature extractionalgorithm.

Detection of candidate states. Candidate positions ofupper body joints, are used in Markov network methoddescribed in the next section. Rather than uniformlysampling the space of interest as the candidate positions,we first find candidate positions of body joints usingimage features. These positions of body joints are used inthe Markov network to infer the optimal positions ofbody joints.

Candidate positions for wrists are detected based onthe color model obtained from face detection. As shownin Figure 5, distracters in background comprise much ofthe candidate positions.

For elbow and shoulder joints, we use anotherstrategy. We first generate position of shoulder jointsbased on assumption of human pose, then we use motionfeature to generate candidate positions of elbows. Figure 6shows the method used in generating the candidate elbowpositions. We accumulate motion cues in rectangles withwidth approximate the width of upper arm, rotatingaround shoulder joints. For each rectangle at a rotationangle, we cluster motion cues within the rectangle, andfind the major connected component of motion cues. Theborder of the connect component at far end from theshoulder joint is detected as a candidate elbow jointposition. After inference in the Markov network,we use motion feature to optimize the position

Figure 5. Detected candidate elbow andwrist joint positions overlaid on framedifference images. Red (gray) rectanglesrepresent position of elbow joints. Whiterectangles represent wrist joint positions.The detection is conducted in the motionregion enclosed by the blue rectangle.

of shoulders, based on the estimated position of elbows.Since shoulders usually have much less motion,compared with arm joints, we only detect shoulder jointsafter several frames. Figure 5 gives a result of jointdetection.

Figure 6. Detection of elbow joint positions.

200250

300350

400

450500

-100

-50

0-400

-350

-300

-250

-200

-150

-100

200250

300350

400450

500

-100

-50

0

2.3. The Markov network model

The second layer of the algorithm performs bodyconfiguration inference. Using Bayesian rule, given imagefeatures x, body configuration h can be estimated as:

)|(maxarg xhPhh

= , (2)

)()|()|( hPhxcPxhP = . (3)

)(hP is the a priori probability of human upper body

configuration h. )|( hxP is the likelihood, and )|( xhP isthe posterior probability.

Equations (2) and (3) can be solved using a Markovnetwork model. First, we introduce the “cardboardperson” model (Ju, 1996) as shown in Figure 7. Thismodel decompose image plane into image patchescorresponding to body parts.

Figure 7. A cardboard person model.

2.3.1. The Bayesian network model. We define humanbody configuration h as the 2D positions of body jointson the image plane. Later we will use 3D reconstructiontechnique to recover 3D coordinates from 2D positions.

We use Markov network, an undirected graphicalmodel of N nodes with pair-wise potentials, to solve theinference problem for human body configurations. Asshown in Figure 8 (a), a Markov network is a network ofconnected state nodes (the empty circles) andmeasurement nodes (the filled-in circles). Each state nodeis connected with a measurement node, as well as to itsneighbors. Connections between nodes in Markovnetwork indicate statistical dependencies.

In this Markov network model, we denote states atnode i as is , which represents the 2D positions of thebody joints, and the observation at measurement node i

(a) (b)Figure 8. (a) A Markov network. (b) AMarkov network model for upper bodymotion inference. The empty circles arestate nodes, and filled-in circles aremeasurement nodes.

as ix , which in this case represent the motion or colorfeatures at the local image patches. The image patches aredefined by the cardboard person model. The connectionsbetween state and measurement nodes correspond to

)|( iii sxP , and the connections between neighboring state

nodes correspond to ),( jii ssP .

Due to the definition of states, there are no imagepatches corresponding to them. To solve this problem,We use body parts between joints as measurements. Wedefine the image patches (represented by the measurementnodes) corresponding to wrist joint, elbow, and shoulderjoint, as lower arm, upper arm, and shoulder girdle,respectively. This is shown in Figure 8 (b). In this wayeach states will correspond to measurable observations.

We define the likelihood function )|( hxP as:

)exp()|( EchxP -= , (4)where c is a normalization coefficient. E is an energyfunction defined by

)(hE x= . (5)

)(hx is computed by accumulating the motion pixels inthe cardboard regions, which is determined by body jointpositions h.

Markov network decomposes the Bayesian inferenceproblem (2)-(3) locally. In a Markov network, we useprobability dependencies between locally connected nodes

)|( iii sxP , )|( jjj sxP and ),( jiij ssP , to infer body joint

positions ),,,( 21 Nsss L , which best explain the motionand color features in images. Bayesian inference in aMarkov network consists of 2 phases: learning, andinference.

Learning: In learning process, we determine theparameters of this Markov network, which is the jointprobability ),( ji ssP between neighboring state nodes i

and j. Several authors have proposed to learn theparameters of a Markov network using labeled orunlabeled training data. We use a supervised learningmethod.

First, we uniformly sample the 2D space of upperbody motion, as shown in Figure 9. We only estimatejoint probability of states at the sampling points(intersections of the grid). To learn ),( ji ssP , we run the

system through video sequences. For each pair ofneighboring body joints, ),( ji ssP is estimated by:

Â=

ji

jiij

jiij

jiij

ssm

ssmssP

,

),(

),(),( , (6)

where ),( ji ssm is the number of times joint i present at

position is while joint j present at position js . Here

is and js are 2D positions of body joints. If is and /or jsfall between the sampling points, we tie them to thenearest sampling positions.

Figure 9. Learning joint probabilities a tsampling points of a 2D image plane.

Comments: Essentially, this learning process willincrease the probability of routine human upper bodyconfigurations, while the probability of thoseconfigurations that seldom appear will decrease.

Inference: We use inference process to determine thehidden states in this Markov network. The algorithmproceeds as follows:

1. Compute local likelihood probabilities)|( iii sxP for each candidate states, using (4) and

(5);2. Apply Belief Propagation algorithm to obtain the

body joint positions.Belief Propagation (BP) is an iterative algorithm

based on message passing, to infer the hidden states in aMarkov network. The basic iteration is as follows:

’ÂŒ

=jiNk

iki

x

iiijiijjij smsxPssPsmi \)(

)()|(),()( a , (7)

’Œ

=)(

)()|()(iNk

ikiiiiii smsxPsb a . (8)

where ijm refers to the message that node i sends to node j,

and ib is the belief, i.e., marginal posterior probability, atnode i, obtained by multiplying all incoming messages tothe node by the local evidence (likelihood). a is anormalization constant. jiN \)( means all nodes

neighboring node i, except j. All messages )( iij sm are

initialized to 1 before the iterations begin. The beliefswill converge to stable values after several iterations.Some results of BP algorithm are given in section 4.2.3.2. Pose ambiguity in Markov network model. Theflexibility of human body configuration poses a difficultproblem. Under different body poses, the a prioriprobability of body configuration )(hP is different.

Considering body pose u, Eq. (3) becomes:)()|(),|()|,( uPuhPuhxcPxuhP = . (9)

Different pose u will give rise to different probabilitydistribution of h. In general, for K different poses,

)(uP can be represented as a weighted mixture of K basisfunctions:

Â=

⋅=K

k

kk hwhP1

)()( a , (10)

where kw is the mixture weight, representing the

probability for each pose ku = to be present, and )(hka is

the probability distribution of h under pose ku = . Theprobability that the pose k being responsible forgenerating body joint positions h can be computed as:

)(

)()(

hp

hwh kk

k

ag

⋅= . (11)

For general applications, (9)-(11) are useful forcharacterizing the probability each body configuration willbe present. In our system, we trained a separate

),( ji ssP for each different poses and we consider only 3

poses in this stage, namely, frontal, turning to the left,and turning to the right.

We found that it is not easy to estimate pose using thesame likelihood function as in (4)-(5), for each bodyposes. The reason is the likelihood probability )|( hxP isalways higher for configurations with more occlusionsbetween body parts, such as turning left or turning right,since motion cues generated by body trunk can easily beinterpreted as the motion of arms. Considering this fact,we designed special likelihood functions for each differentposes by give penalties to occlusion situations. Thelikelihood function becomes:

)()exp(),|( uPlEcuhxP -= , (12)

where )(uPl is the penalty function for body pose u.The body configuration is estimated by first compute

optimal body configurations under each poses. The finalestimation result is the body configuration with highestbelief (in the case of BP) or posterior probability (in thecase of dynamic programming).

2.4. 3D upper body configuration recovery

We divide the task of estimating body joint positionsinto 2 phases: First, estimate 2D positions of bodyjoints; second, recover 3D positions of body joints.

Recovering 3D configurations is accomplished basedon the algorithm of Taylor (2000). Given 2D jointpositions, we recover 3D coordinates of each body joints.Assume ),( 11 vu and ),( 22 vu are projections of the 3D

points ),,( 111 ZYX and ),,( 222 ZYX on image plane,under orthographic projection we have

)()( 2121 XXsuu -=- , (13))()( 2121 YYsvv -=- . (14)

then

2221

221

221 /))()(( svvuulZZ -+--=- , (15)

where s is a constant, and l is the length between),,( 111 ZYX and ),,( 222 ZYX . For details, refer [22].

3. Detection /Tracking in Multiple Frames

In everyday life, people use knowledge of humanmotion trajectories to help tracking of body parts. Whilepeople may not be able to find body configuration in oneor two frames with enough confidence, they can definitelytrack human body parts after a long sequence of humanperformance. Human body parts not only need to be invalid configuration, but also need to satisfy dynamiclimits which constrain the trajectory of the body joints.

FeatureExtraction

Computeconfigurationprobability ateach frames

BP or DP

3DRecover

y

In this section, we extend the single frame Markovnetwork model of section 2 into a Markov network formultiple frames. The new Markov network structure isshown in Figure 10. This model considers both spatial(structural) and temporal (dynamic) constraints in humanupper body, and each individual frames in image sequenceare treated equally.

In the Markov network, same body joints inconsecutive frames are connected and their jointprobability are defined. Assume t

is is a state of node i in

frame t, and 1+tis is a state of node i in frame t+1. We

impose the smoothness of transition between tis and 1+t

is bymodeling their joint probability as follows:

2

2111, ||

exp2

1),(

ssp

+++ -

=ti

tit

iti

tti

ssssP . (16)

Figure 10. Markov network of upper bodywith temporal constraints. The dashedlines separate consecutive frames. Thesame body joints in each frames a reconnected.

which is a Gaussian of the 2D distance between tis and

1+tis . s is a constant and is determined empirically.

Two different approaches are adopted to solve theBayesian inference problem in this network. The firstapproach is belief propagation (BP). Different from thesingle frame Markov network in section 2, the multi-frame model contains loops. Belief propagation algorithmis exact only in networks without loops, but recent studyshows that it can also converge in many loopy networks.Our experimental results on this network are given insection 5.

The second approach we adopted is dynamicprogramming (DP), widely used in speech recognitionsociety to solve similar sequential inference problems.To apply DP, we approximate the Markov network inFigure 10 as a Markov chain of mega-states ),( tStM , as

shown in Figure 11. Here tS represent a combination ofbody joint positions, Tt

Nttt sssS ),,,( 21 L= , at frame t.

Then all we need compute is the likelihood of each mega-state given image features, )),(|( tStMXP , and thetransition probability between consecutive mega-states.

We approximate each frame of image as one mega-state node, ),( tStM . Given image features and acombination of joint positions, we compute thelikelihood of each frame based on Eqs. (4) and (5):

),,,|,,,()),(|( 2121tN

ttN

t sssxxxPStMxP LL= . (17)The temporal smoothness constraints between each joints for 2 consecutive frames t and t+1 are approximated by aGaussian probability function of average distance betweenall body joints positions in frames t and t+1:

2

11

1

)()(exp

2

1

)),1(),,((

ssp N

SSSS

StMStMPttTtt

tt

++

+

--=

+

(18)

Figure 11. The Markov chain approximationof the Markov network model for multipleframes.

Compared with BP, in dynamic programmingapproach, we compute both likelihood probability (17)and the transition probability (18) approximately. We alsouse beam search to reduce the computational complexityof the DP algorithm. The joint probabilities (16) and (18)correspond to a “random walk” like model for motion,because as a detection problem, we do not have any aprior knowledge of the motion trajectory.

4. Experimental Results

4.1. Algorithm structure

We developed a prototype system that can auto-matically detect and track human upper body motion in anatural environment. It is part of our ongoing effort indeveloping a natural human gesture understandingsystem, in which a vision module is trained to detect,track, and recognize human upper body gestures. Theframework of the detection and tracking algorithm isshown in Figure 12.

Figure 12. The upper body motiondetection and tracking algorithm.

The algorithm proceeds as follows: First, facedetection is conducted, color and motion features areextracted, and candidate positions for body joints aredetected based on the image features (section 2.2). Then,Bayesian inference is conducted in the Markov networkmodel. We use both BP and DP to find the best bodyconfiguration to explain the image features. Specializedlikelihood functions are used to discriminate different

L1

1-tS

12

-tS

13

-tS

M

),1( 1-- tStM

tS1

tS2

tS3

M

),( tStM

),1( ttP - )1,( +ttP

… …

poses (section 2.3). And we use multi-frames Markovnetwork model to combine the temporal constraints(section 3). In this stage, the estimated bodyconfiguration is 2D positions of body joints on images.Then we apply 3D recovery algorithm to recover 3Dcoordinates of body joints (section 2.4).

We tested our algorithm on captured videos withpeople performing meaningful gestures or just stand stilland talking with other people. The videos are recordedwith 5 people, each 5 to 10 minutes. We restrict only onepeople in the scene at one time.

We tested our algorithm on these video segments.BothBP (multi-frame) and Dynamic programming algorithmare tested. In most cases, both algorithms give the sameresults, though implementations of the 2 algorithms aredifferent. In the situation when results of the 2 algorithmsare different, there is only minor differences due toapproximations used in details of the algorithms.

Figure 13 gives a result for convergence of beliefs inBP. In this experiment we use 4 state nodes for eachframe, 7 frames (total of 28 state nodes) to detect andtrack body joint positions. The structure of the Markovnetwork is as shown in Figure 10. Initially, all messagesare given value 1. We show the beliefs obtained after eachiterations, for all candidate states in 2 state nodes in thismulti-frame Markov network model. It can be seen thatafter 3 iterations, the beliefs of all states in the 2 nodesconverged to stable values in this loopy network.

Figure 14 gives a detection and tracking result usingmulti-frame inference. Both BP and DP give the same 2Djoint positions in the experiment.

We also find some errors (comprising about 4% of thetotal frames) in the experiments. Errors occur mostly inocclusion situations or when there is a misjudgment ofbody pose. Figure 15 gives an example of an errorsituation. By incorporating temporal constraints, themulti-frame model avoids many problems that would bedetection errors based on single frame algorithm.

The algorithms can also effectively consider caseswhen there is no enough motion information present,such as people standing still. For those body joints thatstay motionless, the algorithm will give no estimation orgive the previous estimation, until enough motion cuesare detected. In all of our experiments, human upper bodygestures give enough motion information for detectionand tracking of body joints of interest.

(a) (b)Figure 13. Convergence of beliefs for twonodes in BP iterations. (a) Convergenceof beliefs of node A with 9 candidatestates; (b) Convergence of beliefs of 4 8candidate states for node B;

(a)

(b)

Figure 14. An upper body motion detectionresult. (a) Estimated 2D joint position; (b) 3 DRecovery result of joint coordinates, twoviews.

Figure 15. (a) An error case happens forpose with occlusions. (b) The constraintavoids major errors in most frames.

5. Conclusions

In our daily lives, people can easily estimate humanbody part locations and infer human motions. While aroutine task for humans, it is difficult for a state of artcomputer vision system to do the task with similaraccuracy and robustness. Some previous studies show thathumans rely on prior knowledge of human bodyconfiguration and motion dynamics in performing thistask.

In this paper, we approach the problem using aMarkov network model, where both spatial and temporalconstraints are considered. We test belief propagation anddynamic programming algorithms for Bayesian inferencein this Markov network. The multi-frame Bayesianinference algorithms (both BP and DP algorithms) givepromising detection and tracking results.

In the future, we will improve the algorithm in thefollowing aspects. First, we will focus on the occlusionproblem, which causes most of the errors in the presentsystem; Second, in addition to motion features, we willtake other image features into consideration, such asproposed in Sidenbladh 2001; Finally, we plan to modifythe present 2D probability constraints of bodyconfigurations, and model it directly in 3D, soconfiguration probability under an arbitrary view angle

1 2 3 4 5 6 7 8 9 100

0.05

0.1

0.15

0.2

0.25

0.3

0.35

Iterations

Beliefs

1 2 3 4 5 6 7 8 9 100

0.02

0.04

0.06

0.08

0.1

0.12

0.14

Iterations

Beliefs

100 150 200 250 300 350 400 450 500 -100-50

0

-400

-350

-300

-250

-200

-150

-100

100

150

200

250

300

350

400

450

500-100

-50

0

can be generated on the fly. Our goal is to analyze humanupper body gestures and activities in videos automa-tically, using this approach.

AcknowledgementsThis work is partly supported by ONR N00014-00-1-

0915 (HumanID)

References

[1] A. Baumberg and D. Hogg, An effient method for contourtracking using active shape models. Proc. WorkshopMotion of Nonrigid and Articulated Objects. LosAlamitos, 1994.

[2] C. Bregler and J. Malik. Tracking people with twists andexponential maps. Proc. IEEE CVPR, pp. 8-15, 1998.

[3] C. Bregler. Learning and recognizing human dynamicsin video sequences. Proc. IEEE CVPR, pp. 568-574,1997.

[4] T. Darrell, G.Gordon, M.Harville, and J. Woodfill.Integrated person tracking using stereo, color, andpattern detection. Proc. IEEE Conf. CVPR, pp. 601-608,1998.

[5] R. Fablet and M.J. Black. Automatic detection andtracking of human motion with a view-basedrepresentation. ECCV 2002, pp. 476-491, 2002.

[6] P.F.Felzenszwalb. Object recognition with pictorialstructures. MIT AI Technical Report 2001-002, 2001.

[7] W.T. Freeman, E.C. Pasztor, and O.T. Carmichael.Learning Low-Level Vision, International Journal o fComputer Vision, Vol 40, no.1, pp. 24-57, October 2000.

[8] D. Gavrila and L.S. Davis. 3-D model-based tracking ofhuman in action: a multi-view approach. Proc. IEEECVPR, pp. 73-80, 1996.

[9] I. Haritaoglu, D. Harwood, and L.S. Davis. W4: Who,when, where, what: a real time system for detecting andtracking people. 3rd IEEE Int. Conf. Automatic Face andGesture Recognition, Nara, Japan, 1998.

[10] S. Ioffe and D.A. Forsyth. Human tracking with mixturesof trees. Proc. IEEE Int. Conf. on Computer Vision, pp.690-695, July, 2001

[11] M. Isard and A. Blake. Condensation: conditionaldensity propagation for visual tracking. IJCV, 29(1):5-28, 1998.

[12] M.I. Jordan ed. Learning in Graphical Models,Cambridge: MIT Press, 1998.

[13] S.M. Ju, M.J. Black, and Y. Yacoob. Cardboard people: Aparameterized model of articulated motion. Int. Conf. OnAutomatic Face and Gesture Recognition, pp. 38-44,1996.

[14] V. Kettnaker and R. Zabih. Counting people frommultiple cameras. IEEE Int. Conf. MultimediaComputing and Systems, Florence, Italy, pp. 267-271,1999.

[15] A. Mittal and L.S. Davis. M2Tracker: A multi-viewapproach to segmenting and tracking people in acluttered scene using region-based stereo. ECCV, pp. 18-33, 2002.

[16] G. Mori and J. Malik. Estimating human bodyconfigurations using shape context matching. Proc.ECCV, pp. 666-680, 2002.

[17] C. Papageorgiou and T. Poggio. Trainable pedestriandetection. Proc. Int. Conf. Computer Vision, Corfu,Greece, pp. 1223-1228, 1999.

[18] J. Rehg and T. Kanade. Model-based tracking of self-occluding articulated objects. Proc. Int. Conf. ComputerVision, pp. 612-617, Cambridge, 1995.

[19] H. Sidenbladh, M. Black, and D. Fleet. Stochactictracking of 3d human figures using 2d image motion.ECCV, pp. 702-718, 2000.

[20] H. Sidenbladh, M. Black. Learning image statistics forBayesian tracking. Proc. Int. Conf. Computer Vision,pp.709-716, 2001.

[21] C. Stauffer and E. Grimson, Similarity templates fordetection and recognition, Proc. CVPR, Kauai, 2001.

[22] C.J. Taylor. Reconstruction of articulated objects frompoint correspondences in a single image. ComputerVision and Image Understanding, vol.80, no.3, pp.349-363, 2000.

[23] C. Wren, et al. Pfinder: Real-time tracking of the humanbody. IEEE Trans. Pattern Analysis and MachineIntelligence, 19(7):780-785, 1997.

[24] J.S. Yedidia, W.T. Freeman, and Y. Weiss. GeneralizedBelief Propagation, Advances in Neural InformationProcessing Systems (NIPS), Vol 13, pp. 689-695, 2001.