decision forests and discriminant analysis

Randomised Decision Forests

Tae-Kyun Kim

http://www.iis.ee.ic.ac.uk/icvl/

BMVA 2014

Computer Vision Summer School

Randomised Forests in the field

[Moosmann et al., 06]

[Shotton et al., 08]

boat chair

Visual Words

[Lepetit et al., 06]

Keypoint Recognition

[Gall et al., 09]

Hough Forest

[Shotton et al., 11]

KINECT

Randomised Forests @ ICL

ICCV13 (oral),

CVPR14 (oral) CVPR13

CVPR14 ICCV13

http://www.iis.ee.ic.ac.uk/icvl/publication.htm

Randomised Forests @ ICL

Ongoing, partially at IJCV12

BMVC12 BMVC10

ECCV 10,

ICCV 11

http://www.iis.ee.ic.ac.uk/icvl/publication.htm

CVPRW14

ECCV14

Resource

• ICCV09 Tutorial on Boosting and Random Forest (by T-

K Kim et al)

http://www.iis.ee.ic.ac.uk/icvl/iccv09_tutorial.html

• MVA13 Tutorial on Randomised Forests and Tree-

structured Algorithms (by T-K Kim)

http://www.iis.ee.ic.ac.uk/icvl/res_interest.htm

• ICCV11 Tutorial on Decision Forests (by Criminisi and

Shotton) http://research.microsoft.com/en-

us/projects/decisionforests/

• ICCV13 Tutorial on Decision Forests and Field (by

Nowozin and Shotton) http://research.microsoft.com/en-

us/um/cambridge/projects/iccv2013tutorial/

Resource

• For Random Forest Matlab codes – (i) P. Dollar’s toolbox

http://vision.ucsd.edu/~pdollar/toolbox/doc/

– (ii) http://code.google.com/p/randomforest-matlab/

– (iii) http://www.mathworks.co.uk/matlabcentral/fileexchange/31036-random-forest

– (iv) Karpathy’s toolbox https://github.com/karpathy/Random-Forest-Matlab.

• For open-source visualisation tool for machine learning algorithms, including RF – MLDEMOS http://mldemos.epfl.ch/

Boosting

Cascade

…… t1 tT t2

Boosting

Decision tree Randomised Decision Forest

[IJCV 2012]

[NIPS 2008]

Part I

Split node

……

Tree 1 Tree 2 Tree N

Leaf node

More discriminative split functions

BMVC12

Multiple split criteria /adaptive weighting

CVPR13 ICCV13

Capturing structural info. BMVC10

Probabilistic outputs

ICCV11 ECCV10

Part II

Tutorial Structure

• Part I: Tree-structured Algorithms • Computational Demands

• Boosting

– Speed-up, Multiple-classes

• Committee Machine

• Randomised Decision Forest (Basics)

Criminisi and Shotton, ICCV11 tutorial

• Part II: Randomised Forest • Classification forest

• Regression forest

• Density/Manifold forest

• Semi-supervised forest

Tutorial Structure

Split node

Leaf node

• Part II: Randomised Forests (Case studies) • Capturing structural information

• Probabilistic interpretation

of outputs

• More discriminative

split functions

• Multiple split criteria / adaptive weighting

• Part I: Tree-structured Algorithms • Computational Demands

• Boosting

– Speed-up, Multiple-classes

• Committee Machine

• Randomised Decision Forest (Basics)

PART I

Tree-structured Algorithms

Extremely fast classification by

tree structure algorithms

Object Detection

Number of windows

Number of Windows: 747,666

x # of scales

# of pixels

Time per window

or raw pixels

……

Num of feature

vectors: 747,666

Classification:

Time per window

or raw pixels

……

Num of feature

vectors: 747,666

Time per window (or vector):

0.00000134 sec

In order to finish the task in 1 sec

Neural Network?

Nonlinear SVM?

Real-Time Object Tracking

by Fast (Re-) Detection

Time t to

Detection again

Search region

• Online discriminative feature selection [Collins et al 03], Ensemble tracking [Avidan 07]

Previous object

location

Semantic Segmentation

• Requiring pixel-wise classification

Input Output

• Integrating Visual Cues [Darrell et al IJCV 00]

– Face pattern detection output (left).

– Connected components recovered from stereo range data (mid).

– Flesh hue regions from skin hue classification (right).

More traditionally…

Narrow down the search space

Since about 2001…

Boosting Simple Features [Viola &Jones 01]

• Adaboost classification

• Weak classifiers: Haar-basis like functions (45,396

in total)

Weak classifier

Strong classifier

Boosting Simple Features [Viola and Jones CVPR 01]

• Integral image

– A value at (x,y) is the sum of the

pixel values above and to the left of

(x,y).

• The sum of original image

values within the rectangle can

be computed: Sum = A-B-C+D

Boosting as an optimisation

framework

• Input/output:

• Init:

• For t=1 to T

– Learn ht that minimises

– Set the hypothesis weight

– Update the sample weights

• Break if

AdaBoost [Freund and Schapire 04]

via min

Boosting Iteratively reweighting training samples.

Higher weights to previously misclassified samples.

1 round 2 rounds 3 rounds 4 rounds 5 rounds 50 rounds Slide credit to J. Shotton

Unified Boosting Framework

AnyBoost : unified framework [Mason et al 00]

• Most boosting algorithms have in common that they iteratively update sample weights and select the next hypothesis based on the weighted samples.

• Input: , Loss ftn , Output:

• Init:

• For t=1 to T

– Find that minimises the weighted error

– Set

– Update

AnyBoost an example

• Loss ftn:

• Weight samples:

AnyBoost related

• Multi-component boosting [Dollar et al ECCV08]

• MP boosting [Babenko et al ECCVW08]

• MCBoost [Kim and Cipolla NIPS08]

Noisy-OR boosting for multiple

instance learning [Viola et al

NIPS06]

Boosting as a Tree-structured

Classifier

Boosting (very shallow network)

• The strong classifier H as boosted decision stumps has

a flat structure

– Cf. Decision “ferns” has been shown to outperform “trees” [Zisserman et

al, 07] [Fua et al, 07] 32

c0 c1 c0 c1 c0 c1 c0 c1 c0 c1 c0 c1

…… ……

Boosting -continued

• Good generalisation by a flat structure

• Fast evalution

• Sequential optimisation

A strong boosting classifier

Boosting Cascade [viola & Jones

04], Boosting chain [Xiao et al]

Very unbalanced tree

Speeds up for unbalanced binary problems

Hard to design

Speeding up

A sequential structure of varying

length • FloatBoost [Li et al PAMI04]

– A backtrack mechanism deleting non-effective weak-learners at each round.

• WaldBoost [Sochman and Matas CVPR05]

– Sequential probability ratio test for the shortest set of weak-learners for the given error rate.

Classify x as + and exit

Classify x as - and exit

Making a Shallow Network Deep

T-K Kim, I Budvytis, R. Cipolla, IJCV 2012

Imperial College, Univ. of Cambridge

Decision tree Boosting

• Many short paths for speeding up

• Preserving (smooth) decision regions for good

generalisation

Converting a boosting classifier to a

decision tree

• Many short paths for speeding

• Preserving (smooth) decision

regions for good generalisation

5 times

speed up

Boosting

Super tree

Converting a boosting classifier to a

decision tree

Boolean optimisation formulation

• For a learnt boosting classifier

split a data space into 2m primitive regions by m binary

weak-learners.

• Code regions Ri i=1,..., 2m by boolean expressions.

W1 W2 W3 C

R1 0 0 0 0

R2 0 0 1 0

R3 0 1 0 0

R4 0 1 1 1

R5 1 0 0 1

R6 1 0 1 1

R7 1 1 0 1

R8 1 1 1 x

Boolean expression minimization

• Optimally joining the regions of the same class label or don’t

care label.

• A short tree built from the minimised boolean expression.

W1 W2 W3 C

R1 0 0 0 0

R2 0 0 1 0

R3 0 1 0 0

R4 0 1 1 1

R5 1 0 0 1

R6 1 0 1 1

R7 1 1 0 1

R8 1 1 1 x

R5,R6,R7,R8

W1W2W3 v W1W2W3 v W1W2W3 vW1W2W3 W1 v W1W2W3

Boolean optimisation formulation

• Optimally short tree defined with average

expected path length of data points

where p(Ri)=Mi/M and the tree must duplicate the Boosting

decision regions.

Solutions Boolean expression minimization

Growing a tree from the decision regions

Extended region coding

Examples

generated from

Synthetic data exp1

Face detection experiment

Boosting Fast exit [Zhou 05] Super tree

No. of weak learners

False positive rate

False negativerate

Average path length

False positiverate

False negativerate

Average path length

False positive rate

False negative rate

Average path length

20 501 120 20 501 120 11.70 476 122 7.51

40 264 126 40 264 126 23.26 258 128 15.17

60 222 143 60 222 143 37.24 246 126 13.11

100 148 146 100 148 146 69.28 145 152 15.1

200 120 143 200 120 143 146.19 128 146 15.8

Total data points = 57499

The proposed solution is about 3 to 5 times faster than boosting and

1.5 to 2.8 times faster than [Zhou 05], at the similar accuracy.

ST vs Random forest

• ST is about 2 to 3 times faster than RF at similar accuracy

Experiments on tracking and

segmentation by ST

Segmentation by binary-class

global :

74.50%

average:

79.89%

global:

88.33%,

average:

87.26%

global:

77.46%,

average:

80.45%

global:

85.55 %,

average:

85.24 %

Building Non-building Error

Tree Non-tree Error Car Non-car Error

Road Non-road Error

Multi-Class Boosting

Multi-view and multi-category

object detection

• Images exhibit multi-modality.

• A single boosting classifier is not sufficient.

• Manual labeling sub-categories.

Images from Torralba et al 07

Multiple Classifier Boosting

(MCBoost)

T-K Kim, R Cipolla, NIPS 08

Problem

K-means clustering

Face cluster 1

Face cluster 2 MCBoost

MCBoost: Multiple Classifier Boosting

• Objective ftn:

• The joint prob. as a Noisy-OR:

• By AnyBoost Framework [Mason et al. 00]

– For t=1 to T

For k=1 to K

– Update sample weights

State diagram

Toy XOR classification problem

• Discriminative clustering of positive samples.

• It needs not partition an input space.

Toy XOR classification problem

• Matlab demo

classifier1 classifier2 classifier3

Boosting round 10 20 30

Experiments

• INRIA pedestrian data set containing 1207 pedestrian images and

11466 random images.

• PIE face data set involving 1800 face images and 14616 random

images.

• A total number of 21780 simple rectangle features.

s Random images + simple features

Image cluster centers

Pedestrian detection by MCBoost [Wojek, Walk, Schiele et al CVPR09]

• TUD

Brussels

onboard

dataset

1-precision

0.0 0.0 1.0

Pedestrian detection by MCBoost [Wojek, Walk, Schiele et al CVPR09]

Multi-class Boosting +

Speeding up

Multiclass object detection [Torralba

et al PAMI 07]

• Learning multiple boosting classifiers by sharing features

• Tree structure speeds up the classification

Multiclass object detection [Torralba

et al PAMI 07]

Multiclass object detection [Torralba et

al PAMI 07]

ClusterBoost [Wu et al ICCV07]

• Algorithm – Start with one sub-category

– Select a feature per branch by Boosting

– Divide the branch if the feature is too weak by clustering on the feature value

– Continue to grow while updating the weak-learners of parent nodes

• Cf. VBT [Huang et al ICCV05],PBT [Tu ICCV05]

Feature sharing

←Splitting

ClusterBoost result

AdaTree Result [Grossmann 04]

1% Noise in data

Boosting round

10% Noise in data

Mean computation cost

0 0 90

Multiple Classifier System

Committee Machine

Mixture of Experts [Jordan, Jacobs 94]

Gating network encourages specialization (local

experts) instead of cooperation

output

Input x S

Expert 1

Expert 2

Expert L

Gating Network

Gating ftn

output

Input x S

Expert 1

Expert 2

Expert L

constant g1

Ensemble learning:

Boosting and Bagging

• Cooperation instead of specialization

Committee Machine

The average sum-of-squares error is

The average error by acting individually

The expected error of the committee is

Committee Machine

Randomised Decision Forests

(Basics)

Binary Decision Trees

category c

split nodes

leaf nodes v

10 11 12 13

14 15 16 17

• data vector v

• split functions fn(v)

• thresholds tn

• Classifications Pn(c)

Slide credit to J. Shotton

……

• Train data:

• Recursive algorithm

– set In of training examples that reach node n is split:

• Features f and thresholds t chosen at random

• Choose f and t to maximize

gain in information

Randomized Tree Learning

left split

right split

category c

category c category c 74

Slide credit to J. Shotton

• Forest is an ensemble of decision trees

• Bagging (Bootstrap AGGregatING) – For each set, it randomly draws examples from the

uniform dist. allowing duplication and missing.

Forest of Trees : Learning

…… tree t1 tree tT

DS1 DST

• Forest is an ensemble of decision trees

– Classification is done by

Forest of Trees : Recognition

…… tree t1 tree tT

category c

category c split nodes

leaf nodes

Overfit

Reasonably Smooth

…… tree t1 tree tT tree t2

Decision Tree vs Random Forest

Wrap-up

Boosting

Bagging

Tree hierarchy

A decision stump

Random ferns?

Decision tree

Random forest

Boosted decision trees

Bagged boosting classifiers

Boosted decision stumps

Inspired by Yin, Crinimisi CVPR07

Comparisons in literature

In motion segmentation [Yin, Crinimisi CVPR07],

• RF>GB(Boosting stumps)>=BT(Boosting trees) in accuracy

• RF=GB>BT in speed at their best accuracy

In face detection/recognition [Belle et al ICPR08],

• RF>AB (AdaBoost) at low false positive rate,

• AB>RF at high false positive rate, in face detection accuracy

• RF>AB in speed

• SVM>RF in face recognition

Summary

Pros Cons

Random Forest

• Generalization through random samples/features • Very fast classification • Inherently multi-classes • Simple training

• Inconsistency • Difficulty for adaptation

Boosting Decision Stumps

• Generalisation by a flat structure • Fast classification • Optimisation framework

• Slower than RF • Slow training

PART II

Randomised Forests

(by Case Studies)

Split node

……

Leaf node

BMVC12

CVPR13 ICCV13

ICCV11 ECCV10

Part II (Recap)

Random Forest Codebook

1. Capturing Structural + Semantic

Leaf node

… Capturing structural info. BMVC10

Real-time Action Categorisation

T. Yu, T-K Kim, R. Cipolla, BMVC 2010

Univ. of Cambridge, Imperial College

A novel real-time solution for action

recognition

• Our contribution is three-fold:

Efficient codebook learning

High run-time performance

Local appearance + structural information

Typical Approaches

Feature Encoding

Feature Matching

K-means Clustering

Slow for Large Codebook

The “Bag of Words” (BOW) Model

Lacks Structural Information

Quantisation Error

Our Method

Semantic Texton Forest

Efficient

PSRM Structural Information

Hierarchical Matching

Robust

Comparison with existing

approaches

Overview

Spatiotemporal Semantic Texton Forest

V-FAST Corner

BOST Random Forest Classifier

K-means Forest

Results

Spatio-temporal Cuboids

Feature detection

Feature extraction

Feature matching

Classification

Building a codebook using STF

• Extract small video cuboids at detected keypoints

• Visual codebook using STF:

• Efficient visual codebook

• One feature → multiple codewords.

• Quantisation and partial matching

Random forest based codebook

• Work on pixels directly

• Hierarchical splits

“Textonises” patches recursively

Randomized Forest Codebook

8 9 6 7

4 2 6 1 3

……

tree t1 tree tT

Example Cuboids

… FASTEST Corner

Histogram of Forest Codewords

8 9 6 7

4 2 6 1 3

……

tree t1 tree tT

node index

“bag of words”

……

Extremely Fast Powerful Discriminative Codebook

TREE N

Pyramidal Spatiotemporal Relationship Match

(PSRM)

Multiple Structural Relationship Histograms

Pyramid Match Kernel (PMK)

Pyramidal Spatiotemporal

Relationship Match (PSRM)

Experiments

• Short video sequences (50 frames ~ 2 seconds) are

extracted from the input video.

• Sampling frequency is 5 frames for experiment and 1

frame for the laptop demo. (so it is a frame-by-frame recognition)

• Two datsets are used for performance evaluation:

• The standard benchmark

• Six classes, with viewpoint changes, illumination changes, zoom , etc.

KTH dataset

• Human interactions, 6 classes of actions, cluttered background

UT dataset (for ICPR contest on Semantic Description of Human Activities 2010)

• Intel Core i7 920 (for accuracy and speed tests)

• Core 2 Duo P9400 (for laptop demo)

Hardware specifications

KTH dataset

UT interaction dataset

Experiments: Results (KTH dataset)

90 91 92 93 94 95 96 97 98 99 100

Mined features (ICCV2009)

CCA (CVPR2007)

Neighbourhood (CVPR2010)

Info. Max. (CVPR2008)

Shape-motion tree (ICCV2009)

Vocabulary Forest(CVPR2008)

Point clouds (CVPR2009)

our method (sequence)

our method (snippets)

Comparison with recent state-of-the-art

• Comparable to most state-of-the-art.

• Around ~3% slower than the top performer

• Is it a sensible trade-off?

• Useful for many more practical applications. (surveillance, robotics, etc.)

snippet: subsequence level recognition

sequence: major voting of subsequence labels

leave-of-out-cross-validation

Leave-of-out-cross-validation

Experiments: Results

• Results: UT interaction dataset

• Run time performance

PSRM and BOST gave low accuracies when applied separately.

~20% performance improved by simply combining the class labels!

< 25 fps, but enough for most real-time applications

Can be further optimised (e.g. GPU, mult-core processing)

2. Probabilistic Interpretation

of RF Output

Leaf node

… Probabilistic outputs

ICCV11 ECCV10

using 3D Shape Priors

Y. Chen, T-K Kim, R. Cipolla, ICCV 2011, ECCV 2010

Object Phenotype Recognition

Pose Estimation

• Signal: Poses

• Noise: Shape

variations + camera

view points

Challenges:

Pose and viewpoint variations >> phenotype variations

Phenotype

Recognition vs

• Signal: Shapes

• Noise: Poses +

camera view points

• Shape Generator (MS) by GPLVM

– Training Set: shapes in the canonical pose.

– Input: PCA coefficients of vertex positions.

Learning 3D Shape Priors

• Pose Generator (MA) by GPLVM

– Training Set: Synthetic 3D poses sequence.

– Input: PCA coefficients of vertex positions and

vertex-wise Jacobian matrices.

Learning 3D Shape Priors

The Graphical Model

Shape Synthesis

Pose Generator

Shape Generator

Shape Synthesis: Demo

Shape Generator Pose Generator (Running)

Shape Synthesis

Zero Shape

Pose Generator

Shape Generator

Shape Transfer

Solution: Two-Stage Model • Main Ideas:

Discriminative + Generative

• Two stages:

1. Hypothesising – Discriminative;

– Using random forests;

2. Shape Synthesis and Verification – Generative;

– Synthesising 3D shapes using shape priors;

– Silhouette verification.

• Recognition by a model selection process.

• Use 3 RFs to quickly hypothesize phenotype,

pose, and camera parameters.

• Learned on synthetic silhouettes generated by the

shape priors.

Parameter Hypothesizing

Pose classifier

Camera pose classifier

Phenotype classifier

(canonical pose)

Examples of RF Classifiers

The phenotype

classifier The pose

classifier

Shape Synthesis and Verification

• Generate 3D shapes V

– From candidate parameters

given by RFs.

– Use GPLVM shape priors

[Chen et al.’10].

• Compare the projection of V

with the query silhouette Sq.

– Oriented Chamfer matching

(OCM). [Stenger et al’03] 118

• We infer the label c∗ of the query instance by

maximizing the posteriori probability

• We infer the label c∗ of the query instance by

maximizing the posteriori probability

Experiments

• Testing data:

– Manually segmented silhouettes;

• Current Datasets

– Human jumping jack

• (13 instances, 170 images);

– Human walking

• (16 instances, 184 images);

– Shark swimming

• (13 instances, 168 images).

• Phenotype Categorisation

Comparative Approaches:

• Learn a single RF phenotype classifier;

• Histogram of Shape Context (HoSC) – [Agarwal and Triggs, 2006]

• Inner-Distance Shape Context (IDSC) – [Ling and Jacob, 2007]

• 2D Oriented Chamfer matching (OCM) – [Stenger et al. 2006]

• Mixture of Experts for the shape reconstruction – [Sigal et al. 2007].

– Modified into a recognition algorithm

Comparative Approaches:

• Internal comparisons:

– Proposed method with both feature error

modelling and similarity-aware criteria

function (G+D);

– Proposed method w.o feature error

modelling (G+D–E);

– Proposed method w.o similarity-aware

criteria function (G+D–S)

• Using standard diversity index instead.

Recognition Performance • Cross-validation by splitting the dataset instances.

• 5 phenotype categories for every test.

• Selecting one instance from each category.

Experiments on Single View

Reconstruction Sharks:

Experiments on Single View

Reconstruction

Humans:

3. More discriminative split

functions

Split node

BMVC12

Classification forest: the split function model

Split ftn: axis aligned Split ftn: oriented line Split ftn: conic section

Examples of split functions

See Appendix C for relation with kernel trick.

Slide credit to Criminisi and Shotton, ICCV11 tutorial

Fast Pedestrian Detection by Cascaded

Random Forest with Dominant Orientation

Templates

D. Tang, Y. Liu, T-K Kim, BMVC 2012

Imperial College

Fastest Pedestrian Detection by Cascaded

Random Forest with Dominant Orientation

Templates

Beginning with Hough Forest … [Gall et al 2009]

Random Forest + Implicit Shape Model

Multi-cue features (HOG etc)

Hough voting

Patch input

... ≤ τ > τ ≤ τ > τ

2 pixel test

(axis aligned

splits) i

f(p)=I(i)-I(j)

Experiments - Accuracy

1. Dominant Orientations; 2. 2-Pixel Test; 3. Template Matching;

4. Dominant Colours; 5. Cascade; 6. Clustered Votes.

Experiments - Efficiency

Multi-cue features (HOG etc)

Hough voting

Patch input

... ≤ τ > τ ≤ τ > τ

2 pixel test

(axis aligned splits)

f(p)=I(i)-I(j)

To accelerate run-time speed

Extract gradient information

Discretize dominant orientations into 1 byte

Discard magnitude information

Binarization of HOG…

Dominant Orientation Templates

[Hinterstoisser, et al, CVPR10]

Hough voting

Patch input

... ≤ τ > τ ≤ τ > τ

2 pixel test

(axis aligned

splits) i

f(p)=I(i)-I(j)

Template matching split function

2 pixel test

(axis aligned

splits),

efficient but

incapable… 137

Template

(nonlinear),

capable but

efficient? 138

Template Matching Split Function

using Binary Bit Operations

0 0 1 0 0 1 0 0

0 0 1 0 1 0 0 0

≤ τ > τ

Hough voting

Patch input

... ≤ τ > τ ≤ τ > τ

Template

Matching Split

Hough voting

Patch input

... ≤ τ > τ ≤ τ > τ

Template

Matching Split

Architecture

DOT Feature Extraction from Input

Holistic Random Forest

Patch-based Random Forest

Hough Voting Output

Applied DOT to pedestrian detection, and extended to colour domain

Cascade of detectors

Clustering hough votes (similar to Girshick et al, ICCV11) 144

4. Dominant Colours; 5. Cascade; 6. Clustered Votes. 145

Time per window: 1.07x10-6 sec

5 Frames per sec

Cf. Fastest Pedestrian Detector in the West(FPDW) [Dollar 2010]: 5 fps [Benenson et al 2012]: 50 fps by GPU 146

4. Multiple split criteria

Split node

CVPR13 ICCV13

Unconstrained Monocular 3D Human

Pose Estimation by Action Detection

T. Yu, T-K Kim, R. Cipolla, CVPR 2013

Challenges

Typical techniques for 3D human pose estimation:

• Segmentation

• Multi-camera approach

• Depthmaps

• Inverse kinematics

Multiple cameras with invserse kinematics [Bissacco et al. CVPR2007] [Yao et al. IJCV2012] [Sigal IJCV2011]

Learning-based (regression) [Navaratnam et al. BMVC2006] [Andriluka et al. CVPR2010]

Specialized hardware (e.g. structured light sensor, TOF camera) [Baak et al. ICCV2011] [Ye et al. CVPR2011] [Sun et al. CVPR2012]

Require controlled environments /

special hardware!

Our Approach

• Action recognition + detection

1. Actions are temporally and spatially

structured • Action recognition:

select the best manifold for pose regression. [Yao et al. IJCV2012]

• Action detection:

Detect the occurence of an action in space-time [Yang, ICCV2009]

Actions provides strong prior for pose estimation

Our Approach

2. Unconstrained part detection • Low level feature descriptors are replaced by a deformable part models (DPM) to detect high

level 2D body parts in the video (e.g. Pictorial structure [FelzenswalbIJ and Huttenlocher,

IJCV2005])

2D HPE

•Body parts detection (The pose Is known by detecting body parts)

•A problem of 2D detection and tracking

3D HPE

•Full 3D pose estimation

•A problem of optimization and manifold learning

[Eichner et al. IJCV2012] [Yang and Ramanan.ICCV2011]

[Andriluka et al. CVPR2009]

System overview

Method: Feature extraction

Input: • Action snippet, short, overlapping video subsequences

[Schindler CVPR2009]

• A DPM, using HoG, as an off-the-shelf 2D body part detector.

(Any 2D part detector will do)

Feature representation • Pairwise displacements between parts – 325 dimensions

• Dimensionality reduction: PCA – 6 dimensions (separately per

action)

For each frame • A pose is represented by a bag of feature vectors extracted from

the “action snippet”

Action Detection

Action as an atomic sequence of shape

Each detection provides the

spatiotemporal structure of an action.

Hough Forest: Each frame in the video votes

for the current pose. ACTION: CLAP

ACTION: DANCE

Locate space-time position

Classify action

Reference

Starting Time

Method: Action Detection

•Randomized Hough Forest for action recognition and pose estimation [Gall et al. PAMI2011]

• Action and pose are estimated from a feature vector simultaneously.

• At each split, the quality measure is:

• Ha is the information gain used in standard classification forest

• Where Hp is:

• Which describes the spread of pose in a node

= covariance

Method: Action Detection

•Randomized Hough Forest for action

recognition and pose estimation [Gall et al. PAMI2011]

• ω is the purity of the current tree node

• Each terminal node gives

1. The posterior of the action

P(action class | features)

2. A vote of estimated pose

Action

detection

Action recog.

Method: Action Detection • (Testing)

• Pass down feature vectors extracted from a video snippet

• Action classification: same as typical randomized classification forest.

• Pose: Distributions of votes (joint positions) is modeled as 3D Gaussians.

Pose Regression & Combination

Global:

Action detection Local:

Regression Forests

Final 3D pose estimation:

A mixture of 3D Gaussians

Joint positions are

refined locally by

regression forests

Method: Pose Refinement

•Randomized Regression Forest

for local joint position

refinement •Differences among individual action instance (of the same

class) are corrected using regression forests [Criminisi et al.

MACCAI2010]

• Input:

• Training feature vectors

• Corresponding 3D joint locations

• Each forest refines one joint location

• so there are 15 forests (150 trees!).

• Output: Joint locations in 3D Gaussians

Method: Final Pose Estimation

•Compute the final 3D pose estimation

•Assuming the pose distributions from

(1) Action detection, and

(2) Pose regression

are independent

• the final pose estimation is computed by the joint distribution of Gaussians in (1) and (2)

•Traditional method: single output, global distribution.

•Our approach produces distributions of local "confidence levels" of joint locations.

Experiments

• A relatively new problem • No public dataset that provides both 2D and 3D ground truths

• We collected a new challenging dataset

(APE dataset) • 2D and 3D “ground truths” from Kinect

• 7 action classes, 7 different subjects, 5 sequences / class / subject

• Different environment for each subject

• No segmentation can be applied

• Moving backgrounds and occlusions

• Unseen testing environments --- traditional methods do not work in this dataset!

Bend Box Wave 1 Wave 2 Balance Dance Clap

Experiments • Comparison with the state-of-the-art 2D pose estimator

• A better accuracy is observed: Action detection (ad joint refinement) which gives

strong cues to the 3D pose

• Limitation: the action performed needs to be seen during the training phase.

Per-class localization error (pixels)

Experiments • Demo Video

1. APE dataset

2. Extra challenges: KTH and Weizmann dataset

1. Originally used in action recognition

2. Extremely low resolution for typical 3D HPE KTH dataset: 160 x 120 pixels, each subject is ~80-100 pixels high.

Weizmann dataset: 180 x 144 pixels, each subject is about ~60-75 pixels high.

KTH action dataset Weizmann action dataset

Multiple Split Criteria / Adaptive

Weighting … goes on …

For Hand Pose Estimation

Summary

Boosting

Cascade

…… t1 tT t2

Boosting

Decision tree Randomised Decision Forest

[IJCV 2012]

[NIPS 2008]

Part I

Split node

……

Leaf node

BMVC12

CVPR13 ICCV13

ICCV11 ECCV10

Part II

What other topics @ ICL

Thanks to

Jamie Shotton

Roberto

Cipolla

Wenhan Luo Akis Tsiotsios Yuru Pei Yang Liu

Tsz-Ho

Danhang

A. Doumanoglou Chanki Yu Chao Xiong Mang Shao

Kyuhwa Lee Alykhan Tejani

Ignas Budvytis

Woodley

Bjorn Stenger

Any Questions?

http://www.iis.ee.ic.ac.uk/icvl

decision forests and discriminant analysis

boosting algorithms

shotton http

tree structured algorithms

tree structure algorithms

unified boosting framework

hough forest shotton

treestructured classifier

time t

Software

decision forests with long-range spatial context...

decision trees and random forests reference: leo breiman,...

decision trees & random forests - github pagesdecision trees...

1 lecture 4 linear machine linear discriminant functions...

discriminant analysis

ics 273a intro machine learning decision trees, random...

deep neural decision forests - cv-foundation.org · deep...

decision trees and random forests

deep neural decision forests

1. fisher linear discriminant 2. multiple discriminant

decision trees & random forests x deep neural...

kernelized discriminant analysis and adaptive methods for...

incremental hierarchical discriminant...

oriented edge forests for boundary detection · randomized...

chapter 5: linear discriminant functions introduction ...

from decision trees to random forests

semantic texton forests for image categorization and...

learning representations for axis-aligned decision forests...

linear discriminant functions discriminant functions least...

discriminant analysis. 18-2 similarities and differences...