Action Recognition with Motion-Appearance Vocabulary Forest

Krystian MikolajczykUniversity of SurreyGuildford, UK

K.Mikolajczyk@surrey.ac.uk

Hirofumi UemuraKyushu Institute of Technology

Kitakyushu, JapanH.Uemura@surrey.ac.uk

Abstract

In this paper we propose an approach for action recog-nition based on a vocabulary forest of local motion-appearance features. Large numbers of features with as-sociated motion vectors are extracted from action data andare represented by many vocabulary trees. Features froma query sequence are matched to the trees and vote for ac-tion categories and their locations. Large number of treesmake the process efficient and robust. The system is capa-ble of simultaneous categorization and localization of ac-tions using only a few frames per sequence. The approachobtains excellent performance on standard action recogni-tion sequences. We perform large scale experiments on 17challenging real action categories from olympic games1 .We demonstrate the robustness of our method to appear-ance variations, camera motion, scale change, asymmetricactions, background clutter and occlusion.

1. IntroductionSignificant progress has been made in classification of

static scenes and action recognition is receiving more andmore attention in computer vision community. Many exist-ing methods [2, 5, 8, 18, 21, 24, 25] obtain high classifi-cation score for simple action sequences with exaggeratedmotion, static and uniform background in controlled envi-ronment. Example of a category used by these methods isdisplayed in Figure 1(left). It is however hard to make a vi-sual correspondence to the real action of the same categorydisplayed in Figure 1(right) as the appearance, motion andclutter of the scene is very different. Such scenes representa real challenge which is rarely addressed in the literature.Our main goal in this paper is to propose a generic solutionwhich could handle these type of actions but also to demon-strate how the performance for the controlled environmentand the real one can differ.The need for using real world data is argued in image

recognition community [3]. The same direction shouldbe followed in action classification, since the solutionsproposed in both fields start to converge. Recently, a

1Video footage covering olympic games in Barcelona.

Box punchBox punch Box punch Box punch

Figure 1. Examples of an object-action category in different envi-ronments.

boosted space-time window classifier from [8] was appliedto real movie sequences in [10]. However, boosting sys-tems are known to require weak classifiers and large num-ber of training examples to generalize, otherwise the per-formance is low. Other frequently followed class of ap-proaches is based on spatio-temporal features computedglobally [1, 4, 26] or locally [2, 5, 19, 24]. Both meth-ods suffer from various drawbacks. Global methods can-not recognize multiple actions simultaneously or localizethem spatially. In these methods recognition can be done bycomputing similarity between globally represented actionsusing cross-correlation [4] or histograms of spatio-temporalgradients [26]. Spatio-temporal interest points [9] result ina very compact representation but are too sparse to buildaction models robust to camera motion, background clutter,occlusion, motion blur etc. Moreover, local features are of-ten used to represent the entire sequence as a distribution,which results in a global representation at the end. It wasdemonstrated in [25] that as few as 5 to 25 spatio-temporalinterest points give high recognition performance on stan-dard test data. We argue that this number is insufficient forreal actions. The need for more features has been observedin [2], where Harris interest point detector was combinedwith Gabor filter to extract more spatio-temporal points.This argument was also emphasized by [5, 19], which pro-pose a hybrid of spatio-temporal and static features to im-prove the recognition performance. This shifts the attentionfrom motion towards the appearance of objects performingactions. In this context it seems more appropriate to addressobject-action categorization problem rather than action viamotion only.A different class of approaches rely on a strong as-

1

sumption that body parts can be reliably tracked [23], eventhough existing tracking tools often fail in real video data.These methods use relatively large temporal extent and rec-ognize more complex actions often viewed from multiplecameras, thus are less relevant to this work.In this paper we address the problem of recognizing

object-actions with a data driven method, which does notrequire long sequences or high level reasoning. The maincontribution is a generic solution to action classification in-cluding localization of objects performing actions. We drawfrom existing work recently done in recognition and re-trieval of static images [11, 15, 20]. Our approach followsthe standard paradigm, which is the use of local features,vocabulary based representation and voting. Such systemshave been very successful in retrieval and recognition ofstatic images. However, recognition of actions is a distinctproblem and a number of modifications must be proposedto adopt it to the new application scenario. Compared to ex-isting approaches which usually focus on one of the issuesassociated with action recognition and make strong assump-tions, our system can deal with appearance variations, cam-era motion, scale change, asymmetric actions, backgroundclutter and occlusion. So far, very little was done to addressall these issues simultaneously. The key idea explored hereis the use of large number of features represented in manyvocabulary trees in contrast to many existing action clas-sification methods based on a single, small and flat code-book [2, 19, 24]. This message also comes from the staticobject recognition [3], where efficient search methods us-ing many different features from a lot of data provide thebest results. The advantage of using multiple trees has beendemonstrated in image retrieval [22]. In this paper the treesare build various types of features, represent appearance-action models and are learnt efficiently from videos as wellas from static images. Moreover, we use a simple NN clas-sifier unlike the other methods based on SVM [2, 19, 24].Among other contributions, we adopt Linear Discrimi-

nant Projections [7, 16] to the categorization problem. Weimplement an object-action representation which allows tohypothesize an action category, its location and pose froma single feature. We show how to make use of static train-ing data and static features to support action hypothesis. Incontrast to all the other systems our method can simultane-ously classify the entire sequence as well as recognize andlocalize multiple actions within the sequence. Finally, weconsider the use of new action categories and recognitionresults reported in this paper as one of our major contribu-tions.

2. Overview of the recognition systemThe main components of the system are illustrated in

Fig. 2. The representation of object-action categories isbased on multiple vocabulary trees. Training of the trees

starts with feature extraction which includes scale invariantfeature detection, motion estimation, and region descriptiondiscussed in Sec. 3. The dimensionality of features is re-duced with Linear Discriminant Projections [16]. Sec. 4explains how the vocabulary forest is build from subsets oflow dimensional features. Sec. 4.2 discusses the recogni-

�������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������

�������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������

������������������������������������

������������������������������������

Results

reduction

motion

Feature

detection &

Dominant vectorsprojection

clusteringtrainingTree Query data

Training data PartitioningDimensionality

AgglomerativeTree VocabularyDimensionality

Maxima Voting

Matching

of featuresreduction

compensationflowOptical

detection

extractionFeature

description

extractionFeature

Recognition

Learning

Estimation of

Figure 2. Overview of learning and recognition of object-actioncategories.

tion where features and their motion vectors are first ex-tracted from the query sequence. The descriptors are pro-jected in low dimensional spaces and matched to the vo-cabulary trees. The features that match to the tree nodesaccumulate scores for different categories and vote for theirlocations and scales within a frame. The learning and recog-nition process is very efficient due to the use of many treesand highly parallelized architecture discussed in Sec. 4.3.Finally, experimental results are presented in Sec. 5.

3. Motion-appearance representationThis section discusses feature extraction methods, di-

mensionality reduction as well as local and global motionestimation.

3.1. AppearanceLocal features. The central part of our object-action rep-resentation are local features with associated motion vec-tors. Given the frames of action sequences we apply variousstate-of-the-art interest point detectors: MSER [14], Harris-Laplace and Hessian-Laplace [17]. These features provedvery powerful in many recognition systems [11, 15, 20]. In-spired by pairs of adjacent segments from [6] we use similarmethod to extract edge segments but based on more efficientCanny detector. These features are robust to backgroundclutter as only the connected edges are used to compute thedescriptors. To obtain more features from the MSER detec-tor we run it at multiple image scales and on red-green aswell as blue-yellow projected images if color is available.Thus we obtain 5 types of image features which representcomplementary patterns. MSER and Hessian-Laplace ex-tract various types of blobs, Harris-Laplace finds cornersand other junctions, pairs and triplets of edge segments rep-resent contours. Each feature is described by a set of param-eters : (x, y) - location, σ - scale, which determines the sizeof the measurement region, φ - dominant orientation an-gle, which is estimated from gradient orientations within themeasurement region. Given these parameters we compute

GLOH features from [17]. Interest points are describedwith17 bins in log-polar location grid and 8 orientation bins over2π range of angles, thus 136 dimensional descriptor. Thesegment features use 17 bins in log-polar location grid and6 orientation bins over π range of angles, resulting in 102 di-mensions. There are 100s up to 1000s of features per framein contrast to other action recognition methods [2, 24, 25]which extract only 10s of spatio-temporal features but donot deal with sequences containing more than one action,camera motion or complex background.Dimensionality reduction. High dimensional features arevery discriminative, slow to compute the similarity distanceand make data structures for fast nearest neighbor search in-effective. Recently, a dimensionality reduction techniquesmore effective than PCA, yet based on global statistics wasintroduced in [7, 16]. Two global covariance matrices Cand C are estimated for correctly matched and non-matchedfeatures, respectively, on image pairs representing the samescenes from different viewpoints. The matrices are thenused to compute a set of projection vectorsv bymaximizingJ(v) = vT Cv

vTCv . Since correctly matched features are diffi-cult to identify in category recognition we adopt a differentstrategy. Given the features extracted by a combination ofa detector-descriptor (e.g. MSER and GLOH) we performefficient agglomerative clustering [12] until the number ofclusters is equal 10% of the number of features. In otherwords we expect a local image pattern represented by thecluster to occur on 10 training examples on average. Mostof the resulting clusters are very compact and contain fewfeatures with only a few large clusters of indistinctive pat-terns. To prevent the domination of large clusters we useonly those with less than 10 feature members, which is typ-ically more than 90% of all clusters. Next, for each clus-ter member we generate 10 additional features by rotating,scaling and blurring its measurement region and comput-ing new descriptors, which further populate the cluster. Thecovariance matrices are then estimated and we obtain theprojection vectors v. We select a number of eigenvectorsassociated with e largest eigenvalues to form the basis forlinear projections. The parameter e is different for variousfeature types and automatically determined by the sum ofeigenvalues which is equal to 80% of the sum of all eigen-values. This typically results in 10 to 30 dimensions out oforiginal 102 and 136 of GLOH. It leads to great reductionof memory requirements, increase of efficiency and most ofall it makes the tree structures effective.

3.2. MotionMotion maps are computed using standard implementa-

tion of Lucas-Kanade optical flow in image pyramids [13].The motion is represented by velocity maps between pairsof frames. To remove erroneous vectors we run a medianfilter on each map.

Dominant motion compensation. Action sequences areoften shot with significant camera motion or zoom. Wefound that the similarity transformation provides sufficientapproximation for the dominant motion between two con-secutive frames in most of the sequences we dealt with.Our dominant motion estimation starts by sampling pointswithin the frames with the interval of 8 pixels. We select theinterest point nearest to the sample, if there is one within8x8 pixels, otherwise we extract an 8x8 patch centered onthe sample point. These patches in addition to the interestpoints provide good coverage of the image and often con-tain sufficient texture or an edge to verify a match. Stan-dard RANSAC is then applied to find the parameters of theglobal similarity transformation between frames using themotion vectors of the selected points. We accept the domi-nant motion vector if more than 20% of sparsely distributedpoints follow that motion, otherwise the dominant motionis assumed to be zero. Finally, the dominant motion is sub-tracted from the motion maps.Local motion. For each appearance feature, dominant mo-tion orientation angle is estimated within the measurementregion using the motion maps. This is done by buildingmotion orientation histogram and selecting the angle corre-sponding to the largest bin. We found that a single motionorientation angle per feature is sufficient as the interest pointregions usually cover parts moving in the same direction. Ina similar way we estimate the motion magnitude.

3.3. Action representationThe object-action categories are represented by appear-

ance features with associated motion vectors extracted frompairs of consecutive frames. By using only pairs we avoidtracking issues with fast moving objects on complex back-ground. Fig. 3 shows the representation with the parameterswhich allow to recognize and localize an object-action cate-gory. We use a star shape model to capture the global struc-ture of an object and its moving parts. Similar model wassuccessfully used for object detection in [11, 12, 15] and itis adapted here to actions. The training frames are annotatedby bounding boxes which allow to estimate the size and thecenter of the training example. Each feature contains oc-currence parameter vector r = [a, x, y, d, σ, β, γ, φ, μ]; a- label of the action category, (x, y) - feature coordinates,d - distance to the object center, σ - scale (blue circles inFig. 3), β - dominant orientation angle, γ - angle betweenthe vector to the object center and the gradient orientation,φ - angle between the motion direction and the gradient ori-entation, and μ - motion magnitude. Angle γ is invariant tosimilarity transformations. With these parameters we canconstruct a local reference frame for every query featureand hypothesize pose parameters of an object-action cate-gory. A query feature can draw a hypothesis if its appear-ance and motion is similar to the model feature. The center

σ qσ m

x hh ,y

β

m

m

γ

d

φ

m

m

μm

(a)β

(c)

φqq

dm

γ

xq,yq

m

μq

(b)

Figure 3. Object-action representation. Examples of features withmotion-appearance parameters for jogging (a) and boxing (b).(c) Hypothesized object center based on a single feature.

of the hypothesis is computed by:∙xhyh

¸=

∙xqyq

¸+

σqσm

dm

∙cos(βq − γm)sin(βq − γm)

¸(1)

where indexes q and m indicate the query and the modelfeatures, respectively. (xh, yh) is the location of the hypoth-esis within the image, σq/σm is the scale of the hypothesis,and βq − βm is its orientation angle (see Fig. 3(c)).The angle between the dominant gradient orientation and

the dominant motion orientation of a feature is characteris-tic for a given time instance of an object-action categoryand it is used during recognition to validate a match be-tween a query feature and a model feature. Note that somefeatures do not move. These features are labeled static andwill serve for refinement of object-action hypotheses, whichis discussed in Sec. 4.2. In this representation many of theappearance features can be shared among various categories(see Fig. 3).

4. Vocabulary forestThe number of action examples is usually not large

enough to build a generic model, in particular for object cat-egory appearance. To improve that we augment the trainingset by static images of our category objects, which are eas-ier to obtain from the Internet or existing datasets than thevideos. The features extracted from still images are labeledstatic. Once features all examples are extracted we build aset of trees. The features are first separated according to dif-ferent types, which are combinations of detector-descriptor.Note that each type is projected with different set of vec-tors to reduce the number of dimensions and the featurescan be compared only within the same type. We start bypartitioning the features from each type into subsets withkmeans. The argument to use kmeans instead of randomsplits exploited in [22] is that our objective is to cluster andcompress the amount of information within each subset andnot only to search for the nearest neighbors. A tree is con-structed from each subset of features.

4.1. Tree constructionClustering. The kmeans is initialized such that the subsetcontains less than F = 200 000 features. A vocabulary treeis built with the agglomerative clustering which can han-dle this number of features within reasonable time. Initiallyeach feature forms a cluster. Two nearest clusters in thewhole set are merged at each iteration based on their Eu-clidean distance. We continue merging until one large clus-ter remains. This results in a binary tree of clusters whereeach node is represented by the average of its children andthe size. The size of the node is given by the distance fromthe node center to its (0.9Fn)-th leaf child, where Fn isthe number of the node’s leaf children ordered by the dis-tance. Factor 0.9 discards 10% of outliers when estimatingthe node size and makes the tree more compact. Finally,to compress the volume of the tree we remove the smallestclusters from the bottom of the tree until the remailing num-ber of leaf nodes is 10% of the initial number of features F .See Fig. 4(a) for illustration.

......

......

(a) (b)

q qr,

...

......

rn,i

nw

f

Figure 4. Vocabulary tree. (a) Binary clustering tree with cuts re-moving small clusters. (b) Vocabulary tree with node weightswn,parameters rn and the path of query feature fq .Fast matching. The leaf nodes of the tree can be consid-ered a codebook, which is similar to many other approachesbased on codebooks. However, matching with a vocabu-lary tree is much more efficient than with flat codebooksin [2, 11, 18, 19, 24]. A query feature is first compared tothe top node. If the distance from the query descriptor to thenode center is less than the size of the node then the featureis accepted and compared to the children of this node. Thiscontinues until the query feature reaches the leaf nodes ofthe tree. We use this simple technique during training andrecognition.Training. We estimate weightswn = [wn,0, . . . , wn,a, . . .]for each tree node. Weight wn,a indicates how discrimi-native node n is for category a. Weights are estimated bymatching features extracted from all the training examplesto the tree. We first estimate the probability to match noden by features from positive examples pn,a = Fn,a

Fa, which

is the ratio of the number of training features Fn,a from athat matched to this node to the number of all features inthis category Fa. We set Fn,a = 1 if no feature matched tonode n. In addition to the positive training data we also usebackground category b which consists of image examplesof scenes without our object-action categories. Backgrounddata is used to estimate pn,b = Fn,b

Fb. The weight of the node

is then given by wn,a = log(pn,apn,b

). The top nodes usuallyhave small weights as they match to many foreground and

background features. The weights tend to increase towardsthe bottom of the tree and the nodes become more discrim-inative for specific categories. In addition to the weightvector wn, each leaf node contains a list of parameter vec-tors rn,i = [an,i, dn,i, σn,i, βn,i, γn,i, φn,i, μn,i] from thefeatures that matched to this leaf node (see Fig. 4(b)). Theseparameters allow to hypothesize the positions and scalesof object-actions during recognition. The nodes formed byfeatures from static images also represent motion informa-tion. This information is transferred from motion featuresin action sequences that match to the static nodes.

4.2. RecognitionGiven the trees the recognition starts by extracting fea-

tures from the query sequence. To handle asymmetricobject-actions we compute a symmetric copy of each queryfeature. This is done by swapping bins in the GLOH de-scriptor and inverting parameters (xq, βq, γq, φq) with re-spect to the vertical image axis. In this way with very littleoverhead we can handle various actions performed in differ-ent directions e.g. walking, even if the training set containsexamples in one direction only. Next, the number of dimen-sions is reduced with the projection vectors estimated dur-ing training (cf. Sec. 3.1). The features are then matchedto the trees. Each query feature fq from frame t accu-mulates weights for different categories from all the nodesit matches to on its path to the leaf node (cf. Fig. 4(b)):wa,t,f =

Pn ka,φwn,a,f , where ka,φ is the fraction of oc-

currence vectors rn,i of class a for which the motion anglesagree |φq − φn,i| < Tφ. If fq is static or motion anglesare different the weight is labeled static. We keep only theweights accumulated by a query feature on its path to thenearest neighbor leaf node in each tree. Finally, we use5 best paths from all trees. This matching strategy differsfrom the one in [20], where a single tree and a single pathis used. From our observations, using a single path sig-nificantly reduces the number of good matches. Moreover,multiple paths allow to generate more votes for localiza-tion. Using many trees significantly improves the recogni-tion performance compared to a single large tree which isdemonstrated in Sec. 5.2.Sequence classification. To classify the sequence we inte-grate the weights over motion features and frames: wa =P

t

Pf wa,t,f . In contrast to [19] we do not use the static

features here, otherwise they dominate the score and wecannot distinguish between similar categories e.g. runningand jogging. The action is present if its accumulated weightwa exceeds a fixed threshold. Thus, the classification isbased on features in motion only.Action localization. To find the location and size of theobject performing an action we use the occurrence vectorsrn,i stored in the leaf nodes. A 3D voting space (x, y, σ) iscreated for each object-action category. Parameter vector rq

of query feature fq that matches to leaf node n casts a votein the 3D space for each parameter vector rn,i stored in thatnode if the motion angles are similar |φq − φn,i| < Tφ,otherwise the vote is labeled static. The weight wa,t,f isequally distributed among all the motion votes casted bythis feature and the votes are stored in the correspondingbins in the voting space. The coordinates of the bins arecomputed with Eq. 1. Once all the features in motion casttheir votes, the hypotheses are given by local 3D maxima inthe voting space.Refinement. Local maxima in the voting spaces are oftendrawn by only a few features in motion. We use static votesto improve the robustness of the recognition process. Thevoting space bin which corresponds to the local maximumand the neighboring ones are incremented by the weightsof the static votes pointing to these bins. Thus, the mo-tion based hypothesis is supported by the object appearance.This often changes the ranking of the hypotheses and im-proves the localization accuracy. If the action hypothesis isdue to noise in the motion field, there are usually few addi-tional static features that contribute to the score. The scoresare thresholded to obtain the final list of object-actions withtheir positions and scales within the frame. In addition tothat, from all the votes contributing to the local maximumwe can compute a histogram of βq − βn,i and estimate theglobal pose angle.

4.3. Efficient implementationMany recognition systems work sequentially which re-

quires large memory and high computational power. Alter-natively GPU processors are deployed to speed up imageprocessing. We adopt a different strategy to attain high effi-ciency of training and recognition. Our system is designedsuch that many operations can be done independently andsimultaneously. This makes it possible to parallelize thetraining and recognition and run it with multiple processeson a single or many machines if available. The features areextracted and partitioned into subsets and all the trees arethen trained in parallel. For example, 5000 frames of actionsequences give approximately 3M features and result in 18trees. It takes approximately 2h to train the system on eightP4 3GHz machines but running it in a sequential way takes26h. It is also more efficient than the sequential way whenrun in parallel on a single machine. Estimating the trainingtime without separating features into subsets was beyondour time constraints. Recognition takes 0.5s up to 10s perframe but it largely depends on the number of features ex-tracted from the image.

5. ExperimentsIn this section the datasets and the evaluation criteria are

discuss first. The results are then presented and comparedto other methods.

206 17476725311155111181410139113947343611814526650

front side front side ing ingfront side front front inginging ing ride ride salto side front hurdle jump− punch−jump salto lift− pick− ing ingrun

gym gym sprint sprint sprint cycl− cycl− weig. weig.gymswim− row− horse horse box boxhorse

6 25 7 829 3134 61761071645299 100 100100100100

.76 .81 .58 .51 .61 −multi−KTH

.53 .71 .68 .57 .55 .59 .80.47 .82 .42 .44 .47 .63 .21 .19 .44 .43 .35 .78 .31.4148.22

.96 .98 .79 .86 .78.97 .81.80.47.56.58.25.32.79.61.56.53.87.61 .56 .53.65.32

test

action

2570 9682202135422052165

.96.97 .98 .93 .87.88

.93 1.0 .75 .88[2]1.0state of the art

classification

localization

handhand clap− wav− box− jog− walk− run−

ninginggingingingping

#sequences

.85 .91 .89 .65 .76 .73multiple trees17 Object−action categories from Olympic Games ,166 sequences, 5065 annotated frames

.79 .87 .85 .56 .66 .68

.83 .88 .84 .75 .81 .77

.90.68 .68 .73 .73.83.91 .67 .71 .77 .81 .41 .47 .72 .76 .61 .81 .53

static

motion

single tree

#frames annotated

[25] [18] [18] [25] [18]

Figure 5. Action classification and detection results. (left) KTH action categories. (right) Olympic games categories.

5.1. DatasetsWe use several datasets to train and evaluate the perfor-

mance of our system. The KTH action sequences were in-troduced in [24] and frequently used in many action recog-nition papers [2, 18, 21, 25]. We present the results for thisdata and compare to the others methods. However, recog-nition performance for the KTH data has already saturated,we therefore acquire another sequence of actions includedin the KTH set, but performed simultaneously with morecomplex background, occlusion and camera motion.Olympic games are an abundant source of natural actions

with high intra class similarities yet extremely challeng-ing due to background clutter, large camera motion, motionblur and appearance variations. We select 10 different dis-ciplines with various viewpoints and separate them in 17action categories. The categories, the number of sequencesand frames are summarized in Fig. 5. Image examples aredisplayed in Fig. 6. Each sequence contains more than oneperiod of repetitive actions. We annotated every 5th frameof each sequence with bounding boxes using an interactiveinterface supported by color based tracking. In total, weannotated 11464 frames from 599 sequences of 6 KTH cat-egories, 753 frames from multi-KTH sequence and 5065frames from 166 sequences of 17 sport categories. In ad-dition to the sequences we use images from Pascal set [3]:1000 pedestrians, 200 horses and 200 bicycles, to capturelarge appearance variations. Finally, there are 1000 back-ground images containing urban scenes and landscapes.Performance measures. We evaluate the performance ina similar way to [2, 18, 21, 24, 25]. We refer to this testas ’classification’. In addition to that, we report resultsfor detection of individual actions within frames, which wecall ’localization’. The detection is correct if the intersec-tion/union of the detected and the groundtruth boundingboxes is larger than 50% and the category label is correct.The detection results are presented with average precision,which is the area below precision-recall curve, as proposedin [3]. All the experiments are done with leave-one-out test,that is we train on all sequences except one in a given cat-egory, which is used for testing. Within that sequence, weperform recognition for every annotated frame and compare

with the groundtruth. The results are averaged for all framesand all sequences of a given action. The temporal extent forintegrating the votes is 5 frames (cf. Sec. 4.2).

5.2. KTH - Basic actionsClassification. We repeat the classification experimentsfrom [2, 18, 25]. All action categories obtain high recog-nition score which favorably compares to the state-of-theart results, both are displayed in Fig. 5 (classification). Itis interesting to observe that a system based on appearancewith little motion information extracted from few framescan produce results comparable to a method that analyzesentire sequences but is based on very sparse features. Thereare small confusions between clapping and waving, as wellas between walking, jogging and running that other ap-proaches also suffer from. We also investigated the influ-ence of the number of frames over which we integrate thevotes. The results are high even if we use only a pair offrames as a query sequence. The score increases by up to0.05 if we use more frames, which mainly helps in discrim-inating between similar categories e.g. waving - clapping,running - jogging. However, using more than 5 frames doesnot introduce significant improvements. Training on 16 se-quences and testing on remaining 9, as done in [21, 24] pro-duces similar results with a small drop of performance by0.03 for running and jogging.Localization. In addition to the classification we alsopresent the results for recognition and localization in Fig. 5(localization). Given that the KTH object-actions are onuniform background with exaggerated motion, the score forboxing, clapping, is as high as for the classification. A fewmissed detections for running, jogging and walking are dueto incorrect scale estimation.Multiple vs. single tree. In this experiment we demonstratethat multiple vocabulary trees are superior to a single largecodebook. We reduce the number of training frames to 1000by using very short sequences, to train a single tree for eachfeature type. For comparison we train from the same dataa system with 5 vocabulary trees per type, thus 25 in total.We show in Fig. 5 (multiple trees) that the improvementis by up to 0.1 for walking. It is worth to mention that the

training and the recognition speed increases by a factor of 8.Multiple vocabulary trees allow to represent many variantsof similar image patterns at different levels of details, thusthe representation is richer and the probability that a querypattern is represented by some nodes is significantly higherthan for a single tree or a flat codebook.Motion vs. static features. In this test we investigate theimpact of using the static features in addition to the motionones. Some categories can be easily distinguished from asingle frame without motion information due to very spe-cific appearance of objects and background. In the con-text of the KTH data, static features are not discriminativeenough and confusions between similar categories signifi-cantly increase if they are used for classification and initiallocalization (see Fig. 5 (static)). Location and scale estima-tion still works well as there is sufficient information in theappearance. Fig. 5 (motion) shows that the detection im-proves if based on features in motion, which corresponds tothe classification and the initial localization in Sec 4.2. Theresults further improve if the refinement with static featuresfollows the initial localization, which is shown in Fig. 5 (lo-calization). Large number of static features help refine thehypotheses by increasing the score and improving estima-tion of pose parameters.Multi-KTH. Fig. 6 (top row) shows frames from a sequenceof 5 KTH actions performed simultaneously. We used theKTH data to train the system and the detection results aredisplayed in Fig. 5 (multi-KTH). Lower recognition ratethan for KTH data is due to occlusion, camera motion, pan-ning and zooming as well as differences between the back-ground in training and testing sequences. This can be ob-served in the video sequence2.

5.3. Sport actionsWe demonstrated that our system can handle basic ac-

tions on static background or with camera motion. How-ever, the real challenge is to recognize and localize realworld actions filmed in uncontrolled environment.Appearance-motion. Fig. 1 and Fig. 6 (row 2 to 4) showexamples from 17 categories of sport actions. We performthe classification and localization tests as described in theprevious section. Sport actions give more realistic estimatesof recognition capabilities and the scores are significantlylower than for the KTH data. Some categories can be re-liably recognized from static features only e.g. weight lift-ing or rowing. However, for the majority of object-actionsmotion information acts as focus-of-attention and allows todiscard many features from the background. Note that italso excludes the context on which many image classifica-tion methods rely. We found that only 5% to 10% of queryfeatures are correctly matched with respect to both, appear-ance and motion. It is therefore essential to have a large

2Supplementary material

number of features extracted from the query frames suchthat the initial voting is robust to noise. Similarly, static im-ages used for training are essential for capturing large ap-pearance variations. We observed an improvement of 0.09for horse categories by using additional static data for train-ing.Motion vs. static features. This test corresponds to recog-nition based on the appearance only. It confirms the obser-vations from the KTH data that static features tend to dom-inate in the classification and the performance for all cate-gories is low (see Fig. 5 (static)). For example, for horseride and jump, static features draw many false hypothesesdue to significant clutter in these scenes. The motion con-straint improves the results by up to 0.14. Unfortunately,features in motion can be significantly affected by motionblur which occurs in some categories e.g. gymnastics. Ro-bustness to such effects is improved by using large numberof features extracted with different detectors.

ConclusionsIn this paper we proposed an approach for classification

and localization of object-action categories. The systemis capable of simultaneous recognition and localization ofvarious object-actions within the same sequence. It workson data from uncontrolled environment with camera mo-tion, background clutter and occlusion. The key idea hereis the use of a large number of low dimensional local fea-tures represented in many vocabulary trees which capturejoint appearance-motion information and allow for efficientrecognition. We have conducted large scale experiments onunprecedented number of real action categories and demon-strated high capabilities of the system. We have also im-proved state-of-the art results on standard test data.Possible improvements can be made in local motion es-

timation which is very noisy, in particular for small objectsand for scenes with camera motion. Another direction toexplore is tracking of individual features over longer timeperiod to capture complex motion. This would also makethe representation more discriminative and help in resolv-ing ambiguities between similar actions e.g. jogging andrunning. Finally, the proposed system can be extended torecognize static as well as moving objects simultaneouslyusing appearance and motion information when available.

AcknowledgmentThis research was supported by EU VIDI-Video IST-2-

045547 and UK EPSRC EP/F003420/1 grants. We wouldlike to thank Richard Bowden, Falk Schubert and DavidGeronimo for their contribution to this work.

References[1] M. Blank, L. Gorelick, E. Shechtman, M. Irani, and R. Basri.

Actions as space-time shapes. In ICCV, 2005. 1

Cycle sideCycle side

Cycle side

WeightpickHurdlerace

Hurdlerace

HurdleraceHurdlerace Hurdlerace

Hurdlerace

WeightliftSprint front

Sprint frontSprint sideSprint sideSprint side

Salto side

Sprint front

Salto side

Salto front

Salto sideSalto front

Walk

Clap BoxWav Clap Box

WalkWalk

WavWalk

Clap Wav Walk Clap BoxWave Clap BoxJog

JogJog Jog WaveWav

WavWav

Walk

Box

Horse jump frontRowing sideSwiming side

Horse ride side

Rowing sideSwiming side Horse ride front

Horse ride front

Cycle frontCycle front

Figure 6. Examples of object-action categories with correct detections in yellow and false positives in red. (Top row) Frames from multi-KTH sequence. (Row 3) Pose angle was estimated only for this sequence and all recognition results are for fixed orientation. Some of thesegment features that contributed to the score are displayed in yellow.

[2] P. Dollar, V. Rabaud, G. Cottrell, and S. Belongie. Behaviorrecognition via sparse spatio-temporal features. In VS-PETS,2005. 1, 2, 3, 4, 6

[3] M. Everingham et. al. The PASCAL Visual Object ClassesChallenge 2007. 1, 2, 6

[4] A. Efros, A. Berg, G. Mori, and J. Malik. Recognizing actionat a distance. In ICCV, 2003. 1

[5] C. Fanti, L. Zelnik-Manor, and P. Perona. Hybrid models forhuman motion recognition. In ICCV, 2005. 1

[6] V. Ferrari, F. Jurie, and C. Schmid. Accurate object detectionwith deformable shape models learnt from images. InCVPR,2007. 2

[7] G. Hua, M. Brown, and S. Winder. Discriminant Embeddingfor Local Image Descriptors. In ICCV, 2007. 2, 3

[8] Y. Ke, R. Sukthankar, and M. Hebert. Efficient visual eventdetection using volumetric features. In ICCV, 2005. 1

[9] I. Laptev. On space-time interest points. IJCV, 64:107123,2005. 1

[10] I. Laptev and P. Perez. Retrieving actions in movies. InICCV, 2007. 1

[11] B. Leibe, A. Leonardis, B. Schiele Robust object detec-tion with interleaved categorization and segmentation. IJCV,2007. 2, 3, 4

[12] B. Leibe, K. Mikolajczyk, B. Schiele. Efficient clusteringand matching for object class recognition. In BMVC, 2006.3

[13] B.D. Lucas and T. Kanade. An Iterative Image RegistrationTechnique with an Application to Stereo Vision. In IJCAI,1981. 3

[14] J. Matas, O. Chum, M. Urban, and T. Pajdla. Robust widebaseline stereo from maximally stable extremal regions. InBMVC, 2002. 2

[15] K. Mikolajczyk, B. Leibe, B. Schiele. Multiple object classdetection with a generative model. In CVPR, 2006. 2, 3

[16] K. Mikolajczyk and J. Matas. Improving Descriptors for FastTreeMatching by Optimal Linear Projection. In ICCV, 2007.2, 3

[17] K. Mikolajczyk and C. Schmid. A performance evaluationof local descriptors. PAMI, 27(10):1615–1630, 2005. 2, 3

[18] J.C. Niebles, H. Wang, and L. Fei-Fei. Unsupervised learn-ing of human action categories using spatial-temporal words.In BMVC, 2006. 1, 4, 6

[19] J.C. Niebles and L. Fei-Fei. A hierarchical model of shapeand appearance for human action classification. In CVPR,2007. 1, 2, 4, 5

[20] D. Nister and H. Stewenius. Scalable recognition with a vo-cabulary tree. In CVPR, 2006. 2, 5

[21] S. Nowozin, G. Bakir, K. Tsuda. Discriminative subsequencemining for action classification. In ICCV, 2007. 1, 6

[22] J. Philbin, O. Chum,M. Isard, J. Sivic and A. Zisserman. Ob-ject retrieval with large vocabularies and fast spatial match-ing. In CVPR, 2007. 2, 4

[23] D. Ramanan and D.A. Forsyth. Automatic annotation of ev-eryday movements. In NIPS, 2004. 2

[24] C. Schuldt, I. Laptev, and B. Caputo. Recognizing humanactions: A local SVM approach. In ICPR, 2004. 1, 2, 3, 4, 6

[25] S. Wong and R. Cipolla. Extracting spatiotemporal interestpoints using global information. In ICCV, 2007. 1, 3, 6

[26] L. Zelnik-Manor and M. Irani. Event-based analysis ofvideo. In CVPR, 2001. 1