transferable dictionary pair based cross-view action recognition lin hong
Post on 17-Dec-2015
218 Views
Preview:
TRANSCRIPT
Transferable Dictionary Pair based Cross-view Action Recognition
Lin Hong
Outline
• Research Background• Method• Experiment
Research Background
• Cross-view action recognition:– Automatically analyze ongoing activities from an unknown video;– Recognize actions from different views, to be robust to viewpoints
variation ;– Essential for human computer interaction, video retrieval, especially in
activity monitoring in surveillance scenarios.
• Challenging:– Same action looks quite different from different viewpoints;– Action models learned from one view become less discriminative for
recognizing in a much different view;
Research Background
• Transfer learning approaches for cross-view action recognition:1. Split-based features method [1];
2. Bilingual-words based method [2];
3. Transferable dictionary pair (TDP) based method [3];
[1] Ali Farhadi and Mostafa Kamali Tabrizi. Learning to recognize activities from the wrong view point. In ECCV, 2008.
[2] Jingen Liu, Mubarak Shah, Benjamin Kuipers, and Silvio Savarese. Cross-view action recognition via view knowledge transfer. In CVPR, 2011.
[3] Zheng, J., Jiang, Z., Phillips, J., Chellappa,R.. ‘Cross-View Action Recognition via a Transferable Dictionary Pair’. Proc. of the British Mach. Vision Conf., 2012.
Research Background
• Motivation:1. Split-based features method: Exploited the frame-to-frame
correspondence in pairs of videos taken from two views of the same action by transferring the split-based features of video frames in the source view to the corresponding video frames in the target view. Defect: the frame-to-frame correspondence is computationally expensive.
2. Bilingual-words based method: exploit the correspondence between the view-dependent codebooks constructed by k-means clustering on videos in each view. Defect: the codebook-to-codebook correspondence is not accurate enough to guarantee that a pair of videos observed in the source and target views will have similar feature representations.
Research Background
• Motivation 3. Transferable dictionary pair (TDP) based method:
It the most encouraging method now. It learn two dictionaries of source and target views simultaneously to ensure the same action to have the same representation.
Defect:
Although this transfer learning algorithm achieve good performance. However, it still remain hard to transfer action models across views that involves the top view.
Object: forcing two sets of videos of shared actionsin two views to have the same sparse representations. Thus, the action model learned in the source view can be directly applied to classify test videos in the target view.
Method: TDP based method
Flowchart of cross-view action recognition framework
Spatio-temporal interest point (STIP) based feature: Advantage:Capture local salient characteristics of appearance and motion;Robust to spatiotemporal shifts and scales, background clutter and multiple motions;
Local space–time feature extraction: Detector: select spatio-temporal interest points in video by maximizing specific saliency functions Descriptor: capture shape and motion in the neighborhoods of selected points using image measurements
Method
Method
Bag of Words (BoW) feature:STIP features are first quantized into visual words and a video is then represented as the frequency histogram over the visual words.
STIP
STIP
cookbook
cookbook
BoW feature
BoW feature
view1
view2
K-means
K-means
Method
Sparse coding and Dictionary learning The K-SVD is well known for efficiently learning a dictionary from a set of
training signals. It solves the following optimization problem:
where is the learned dictionary, and are the sparse representations of input signals
Method• View-invariant action recognition Object: recognize an unknown action from an unseen (target) view using training data taken from other (source) views. Method: simultaneously learn the source and target dictionaries by forcing the shared videos taken from two views to have the same sparse representations
Ys
Yt
Xs
||
Xt
Method
Ds and Dt are learned by forcing two sets of videos of shared actions in two views to have the same sparse representations. With such sparse view-invariant representations, we can learn an action model for orphan actions in the source view and test it in the target view.
Where,
Then, {Ds, Dt } can be efficiently learned using K-SVD algorithm
• Objective function:
Experiment
• Protocol: leave one action class out Each time we only consider one action class for testing in the target view. This action class is not used to construct a transferable dictionary pair.
• Dataset We test the approach on the IXMAS multi-view dataset; Website: http://4drepository.inrialpes.fr/public/viewgroup/6
Experiment: dataset
The most popular multi-view dataset:• Five views: four side views and one top view;• 11 actions performed 3 times by 10 actors;• Each view contain 330 action videos;• Each action class contain 30 samples under each view;• The actors choose freely position and orientation;
cam0
cam1
cam2
cam3
The four arrows indicate the
directions actors may face
Experiment: dataset
cam0 cam1 cam2 cam3 cam4
IXMAS dataset: exemplar frames
time1
time2
time3
Experiment: dataset
Experiment• Each time, we select one action class (30 samples) for testing in the target
views. Except this action class samples, the rest (300 samples ) in both source view and source view are used for dictionary pair learning.
• The classification accuracy is averaging over all possible combinations for selecting test actions.
Experiment: STIP feature Local feature: Cuboid detector & descriptor [4] Global feature: shape-flow descriptor[5] BoW feature: 1000-dimensional local BoW feature + 500-dimensional
global BoW feature.
Finally, each action video is represented by 1500-dimensional BoW feature.
[4] P. Doll´ar, V. Rabaud, G. Cottrell, and S. Belongie. Behavior recognition via sparse spatio-temporal features. In VS-PETS.
[5] D. Tran and A. Sorokin. Human activity recognition with metric learning. In ECCV, 2008.
Experiment: result
• Cuboid + KNN
C0 96.7 96.7 98.4 97.9 97.9 97.6 84.6 84.9C1 99.1 97.3 96.7 96.4 97.0 89.7 76.1 81.2C2 96.7 92.1 92.4 89.7 97.6 94.9 88.2 89.1C3 95.8 97.0 96.4 94.2 97.0 96.7 81.2 83.9C4 81.5 83.0 69.7 70.6 90.0 89.7 84.2 83.7Ave. 93.3 92.4 88.8 87.8 95.5 95.2 94.2 91.5 82.5 84.8
Source View
cross-view (Cuboids+KNN, unsupervised) 1500dimension (Dictsize:50, Sparse:30)
%Target View
C0 C1 C2 C3 C4
The red numbers are the recognition results in [1]. The black blod numbers are our result with the same method as [1]. This experiment’s setting is follow the [1], using the same feature and classifier.The comparative results can show that our experiment is correct. It demonstrates that we realize the method in [1] successfully.
[1] Zheng, J., Jiang, Z., Phillips, J., Chellappa, R.. ‘Cross-View Action Recognition via a Transferable Dictionary Pair’. Proc. of the British Mach. Vision Conf., 2012.
top related