Mouse Pose Estimation From Depth Images

Ashwin Nanjappa1, Li Cheng∗1, Wei Gao1, Chi Xu1, Adam Claridge-Chang2, andZoe Bichler3

1Bioinformatics Institute, A*STAR, Singapore2Institute of Molecular and Cell Biology, A*STAR, Singapore

3National Neuroscience Institute, Singapore

Abstract

We focus on the challenging problem of efficientmouse 3D pose estimation based on static images,and especially single depth images. We introduce anapproach to discriminatively train the split nodes oftrees in random forest to improve their performanceon estimation of 3D joint positions of mouse. Our al-gorithm is capable of working with different types ofrodents and with different types of depth cameras andimaging setups. In particular, it is demonstrated inthis paper that when a top-mounted depth camera iscombined with a bottom-mounted color camera, thefinal system is capable of delivering full-body pose es-timation including four limbs and the paws. Empiri-cal examinations on synthesized and real-world depthimages confirm the applicability of our approach onmouse pose estimation, as well as the closely relatedtask of part-based labeling of mouse.

1 Introduction

The study of mouse behavior, referred to as be-havioural phenotyping, is an important topic in neu-roscience [17], pharmacology [11] and other relatedfields in biomedical sciences. For example, it plays akey role in studying neurodegenerative diseases [21],is used in modeling human psychiatric disorders [8]

∗Corresponding author with email address: chengli@bii.a-star.edu.sgThe project is partially funded by A*STAR JCO1431AFG120.

and aids in better understanding of the genetics ofbrain disorders [1], due to the known homology be-tween animals and humans. Automated analysis ofmouse behavior has the potential to improve repro-ducibility and to enable new kinds of psychologicaland physiological experiments [17], besides efficiencyand cost concerns. From a computer vision stand-point, it presents exciting challenges including track-ing [3], activity recognition [12], and social behavioranalysis [5], based on color video feeds of mice behav-ior. Existing efforts have mainly focused on analyzingtwo-dimensional color images. On the other hand, re-cent growth of commodity depth cameras, like XBoxKinect, makes it possible to investigate mouse be-havior further into three dimensions. This opportu-nity motivates us to consider efficient 3D mouse poseestimation based on single depth images. By poseestimation of a mouse, we refer to the collective esti-mation of the 3D positions of its major body joints,which is the fundamental problem in mouse behavioranalysis. Once the full 3D poses of mouse over timeare obtained, behavioral and activity related infor-mation which are commonly used can be deduced ina rather straightforward manner.

Our work has four main contributions: First, weintroduce an approach to discriminatively train thesplit nodes of trees in a random forest, one node ata time, to improve their performance. We adapt thisapproach to the regression task of joint position esti-mation of mouse and also to the classification task ofbody part labeling of mouse from depth images. Sec-

1

arX

iv:1

511.

0761

1v1

[cs

.CV

] 2

4 N

ov 2

015

ond, based on our discriminatively trained randomforest, we propose an efficient and effective systemto estimate the 3D pose of a mouse based on singledepth images. To our best knowledge, this is the firstsuch attempt to address this fundamental task. Ad-ditionally, we also propose a method for the relatedtask of labeling mouse body parts based on singledepth images. Third, we demonstrate the flexibilityof our methods: it is designed to work with vari-ous depth cameras such as structured illumination ortime-of-flight (ToF) and with different imaging setupsand with additional cameras, like a bottom-mountedcolor camera to facilitate the estimation of full-bodypose of the mouse. Finally, we introduce a simple 3Dmouse skeletal model, a mouse pose visualization andsynthesis engine and a dedicated cage setup. This isthe first such kind of academic effort to our knowl-edge and might be useful for other researchers in thisfield.

1.1 Related work

There has been substantial progress in mouse track-ing using color cameras. Many commercial softwareused for studying social interaction of mice, suchas HomeCageScan, EthoVision, AnyMaze, Pheno-Tracker and MiceProfiler [7] use this technique totrack up to two mice from color videos. A commonissue with tracking is that it is highly sensitive tosmall color or texture perturbations over time. Thisoften gives rise to failures, which makes a manual re-initialization of the system necessary and inevitableif used to track for long durations. There has beensome recent progress in long-duration tracking, suchas [5, 15, 24], which can work robustly without suchmanual intervention. This is typically achieved withthe help of invasive methods such as attaching RFIDchips or applying dedicated fur dyes to maintain iden-tities reliably over time.

In behavior phenotyping, existing systems rely ondiscriminative learning methods such as SVM andconditional random fields (CRFs) [5, 12, 23] that di-rectly focus on classifying behaviours or interactions.These discriminative methods are usually supportedby extracting visual features from 2D videos, suchas spatio-temporal interest points (STIP) [23] ob-

tained by convolution of video sequence with filtersdesigned to respond to salient points in the video vol-ume, on which the visual bag-of-word features canbe constructed. One key drawback is that these ap-proaches are based on highly sophisticated featuresthat are usually inscrutable by behavioral neurosci-entists. This might lead to undesirable consequenceswhen one tries to interpret the empirical findings,even if the classification performance is acceptable.An alternative scheme is to instead rely on single-frame based pose estimation, which is a lot easier fordomain experts to understand and interpret.

Meanwhile, the lack of three-dimensional informa-tion hinders the ability of existing approaches to ac-cess rich spatial context that is crucial for elucidat-ing mouse behaviors and interactions. As a rem-edy, multi-camera setups have been considered [18],which are relatively expensive, cumbersome, slow,and the results are unreliable due to long-standingissues with multi-view vision such as sensitivity totextures, background clutters and lightings.

The recent advance of commodity depth imagingopens the door to drastic progress in this direction.These depth cameras are currently based on eitherstructured illumination or time-of-flight (ToF) tech-nologies. The most related work in the field [16] isa low-res body pose estimation that involves the 3Dlocation and orientation of the entire mouse basedan overhead Primesense depth camera. Our ap-proach may be viewed as the next step to revealhigh-resolution details of 3D mouse full-body skele-ton. Note the mice dataset of [5] has also been stud-ied in [6] for a similar low-resolution pose estimationpurpose in 2D based on gray-scale images.

Similar trends have been observed recently in com-puter vision problems related to humans, which havealready demonstrated the unique value of utilizingdepth cameras. These include depth-image basedpose estimation of the human body [20], head [10],and hand [25]. Nonetheless, the set of challenges formouse pose estimation are distinct and cannot be ac-complished by a trivial re-implementation of theseexisting methods. A lab mouse is noticeably smallerthan a typical human hand, is highly deformable andis highly agile attaining over 1m/s maximum veloc-ity [2] in a small enclosed space. For any practical

2

(a) (b) (c)

Figure 1: (a) Points randomly sampled from 18 Gaussian distributions in 2D. Blue and red colors signify the two classes.Each distribution contributes to points of a single class. (b) shows the classification results of the base forest on a test dataset.(c) shows the classification results of the discriminatively trained forest on the same test dataset. It can be seen visually thatthe discriminatively trained forest can better differentiate between the two classes.

camera setup, occlusions of body parts such as limbsand paws of mice are common problems. These fac-tors also inspire us to consider dedicated cage andcamera setups which will be described in later sec-tions.

2 Discriminative training of de-cision tree nodes

In this section, we introduce a discriminative learningapproach that improves the performance of randomforests. The basic idea is to discriminatively trainthe nodes of each tree t ∈ T of an existing randomforest, modifying them to improve the performanceon the input training dataset. The resulting forestis termed Td to better differentiate from the originalone. A training dataset Dd, that is different fromthe training dataset Dt used for creating T , is gener-ated separately for the purpose of this discriminativelearning. For each t of Td, a randomly sampled subsetD′d ⊂ Dd is applied to discriminatively train it.

The training process is applied iteratively on thenodes of t, a single node at a time, using D′d as input.D′d is processed through t by splitting it based on thetests stored at each split node in t. For any S ⊆ D′dthat arrives at a split node q, S is split into Sl and Srbased on the test criteria stored in q. Our discrimi-native process tries a random subset of m tests in q,

one at a time. For each test a performance metricE(q, S) → R is used to measure the performance ofthe subtree rooted at q for S. The test delivering thebest performance is stored at q and S is split furtherbased on that test and this process is repeated on therest of the nodes below q. On the other hand if q is aleaf node, then the estimated result that maximizesE(q, S) is stored at the leaf.

Proceeding iteratively in this manner, this discrim-inative training method perturbs nodes of t, whiletrying to maintain the existing structure of the tree.There are two exceptional cases that are handled dif-ferently. Let ln be the maximum number of trainingsamples that are used to create a leaf node, and |S|denotes the size of set S. If |S| ≤ ln at a split nodeq, then the subtree rooted at q is replaced with a leafnode that maximizes E . Alternatively, if |S| > ln at aleaf node q, then a subtree is dynamically grown andreplaces q. Next we investigate the details of this dis-criminative training approach and analyze the effectof its tuning parameters using a simple example.

2.1 A simple running example

Consider points randomly sampled from two Gaus-sian distributions in R2, each representing a differentclass and let their spreads overlap each other. As asimple binary classification problem, 106 points arerandomly sampled from nine such pairs of Gaussian

3

distributions spread out in R2, as shown in Fig. 1(a).Each point belongs to one of two classes and all pointssampled from a distribution belong to the same class.Mean positions (µx, µy) such that µ ∈ (0, 1) and stan-dard deviation σ ∈ (0, 0.2) are used for the distri-butions. Canonical training dataset Dt, discrimina-tive training dataset Dd and evaluation dataset De

are all sampled from these same distributions, with|Dt| = |Dd| = |De| = 106.

2.2 Baseline method

A random forest of classification trees is first trainedusing Dt as the baseline forest. Consider S as thedata points from the training set that arrive at a splitnode q. A set of m threshold real values Γ = {γ}are uniformly sampled from the range (0, 1) andeach threshold γ is used to split S into two subsets{Sl, Sr}. For nodes at even level, γ splits S horizon-tally and for nodes at odd level, γ splits S vertically.To reduce the threshold space, we restrict the realvalue sampled to three decimal places.

The γ that maximizes the gain I(γ) is chosen forthe split node, where I is defined as:

I(γ) = E(S)−( |Sl||S| E(Sl) + |Sr|

|S| E(Sr)). (1)

The entropy E(S) is defined as:

E(S) = − ∑c∈C

p(c) log p(c), (2)

where p(c) is the normalized empirical histogram oflabels for the data points in S, with c indexing aparticular class label and C the set of classes.

When |S| < ln, then a leaf node is created whichstores the class cl:

cl = arg maxc∈C

p(c). (3)

A baseline classification forest is trained this wayand evaluated using evaluation data De. Each 2Dpoint of S arriving at a split node is split using thethreshold γ stored at the node, considered horizontalor vertical split depending on the level of the node.When the point reaches a leaf node, it is assigned theclass label cl stored there. The final class label for a

D′p

m tests

Node q

E

Figure 2: Discriminative training of a node q ∈ t usingtraining data D′p. A random set of mp tests is tried at q, oneat a time, to minimize E(t,D′p).

2D point is picked from its |T | estimated class labelssimilar to Eq. (3).

Fig. 1(b) shows a visual result of classification of106 points in 2D by a baseline forest of |T | = 5 whereeach tree has a maximum of L = 20 levels. It can becompared to the ground-truth labels of the pointsshown in Fig. 1(a). The classification accuracy, com-puted as the fraction of the points in the unseen testdataset that are classified correctly, is found to be0.794 for this experiment.

2.3 Our discriminative trainingmethod

Next we take the baseline trees of T and iterativelytrain each of nodes of each tree as described earlier ina depth-first manner. Specifically, at each split nodeq, a new set of m thresholds is randomly sampledand subsequently examined using E at the related leafnodes. Please refer to Eq. (7) for the exact definitionof E used here.

Fig. 2 illustrates this task at a split node q from abaseline tree. The nodes lying on the path from rootnode to q have already been discriminatively learned.Using the set of points S that has reached q, a newset of m thresholds is randomly sampled as before

4

and the threshold that minimizes E is chosen and setin the split node. This requires processing the pointsof S through the nodes of the subtree rooted at q,as indicated in Fig. 2. The estimated class labelsfor S are gathered from the leaves they reach in thissubtree and E is computed for each threshold γ.

If |S| ≤ ln at a split node q, then the entire sub-tree rooted at q is replaced with a leaf, whose classis computed using Eq. (3). If |S| > ln at a leaf node,then the leaf is removed and a subtree is grown us-ing baseline method and attached to the tree. Thesesteps allow dynamic modification of a baseline treestructure while training using D′d.

Fig. 1(c) shows a visual result of the classificationof De for such a discriminatively trained forest of|Td| = 5 trees, where the training is performed byscanning over all the tree nodes one time (i.e. oneiteration). By visual inspection, it can be easily no-ticed that this classification has finer classificationboundaries that are closer to the ground-truth whencompared to Fig. 1(b). The classification accuracyis found to be 0.812, a clear improvement over thebaseline forest.

2.4 Empirical studies of the runningexample

In what follows, a series of experiments are conductedon the aforementioned simple running example to ex-amine the performance changes when varying the in-ternal parameters of our discriminative training pro-cess.

2.4.1 Effect of discriminative training withsecond dataset

We devise two experiments to ascertain the effective-ness of the combination of our discriminative trainingmethod with its second dataset Dd. In the first at-tempt, we intend to find the effect of using a seconddataset for the baseline method. To do this, baselinetrees are trained with the dataset Dt∪Dd. To ensurethat trees of similar height as before are created, thevalue of ln is doubled. This forest is found to havea classification accuracy of 0.796. This shows thatmerely applying more training data does not provide

3 5 7 9 11 13 15 17 19 21 23 25 27 29Number of trees in forest

0.790

0.795

0.800

0.805

0.810

0.815

0.820

Cla

ssifi

catio

n ac

cura

cy

TTd

Figure 3: Effect of forest size on classification accuracy ofbaseline and discriminatively trained forests.

an on par improvement as with our discriminativetraining.

In the second attempt, we measure how much ac-curacy can be improved with discriminative trainingusing Dd with a small change: to use I(γ) insteadof E on the leaf nodes as the internal error metric.The classification accuracy of such a discriminativelytrained forest on De is found to be 0.804, clearly bet-ter than the baseline, but not as accurate as using E .This shows that applying a second training datasetalong with use of E that tests the performance of theentire subtree rooted at a node are both crucial forbetter accuracy of this method. Thus it is this type ofdiscriminative training that will be discussed in therest of this paper.

2.4.2 Effect of forest size

In Fig. 3, we investigate the effect of forest size onclassification accuracy comparing both the baselineand its discriminatively trained forests. It can beclearly seen that no matter how many trees are used,the discriminatively trained forest results in higherclassification accuracy. We observe that the baselineforest accuracy improves slowly with the number oftrees, with the improvement lessening with increas-ing forest size. The discriminatively trained forestaccuracy remains almost stable after |Td| = 13. Thisalso indicates that with discriminative training, the

5

5 10 20 40 80 160 320 640 1000m: Number of thresholds used at split node

0.78

0.79

0.80

0.81

0.82

0.83

Cla

ssifi

catio

n ac

cura

cy

Figure 4: Effect of feature size on classification accuracy.

forest size can be considerably smaller without muchloss in performance.

2.4.3 Effect of m

In Fig. 4, we investigate the effect of m, the numberof thresholds picked for evaluation at each split nodeduring discriminative training. A baseline forest of|Td| = 5 and m = 50 is used for this experiment. Asmentioned earlier, we restrict the real value of thethreshold to three decimal places, so there are 104

thresholds to uniformly sample from. We can observethat the classification accuracy increases with m, butlevels away after m = 320 with an accuracy of 0.82.This may be attributed to the increasing correlationbetween the trees of a forest as m increases.

2.4.4 Effect of ln

In Fig. 5, we look at the effect of ln, the maxi-mum number of data points required to create aleaf, on discriminative training. The baseline forestis the same as used in Sec. 2.4.3. We can see thatas ln increases, the classification accuracy decreasesdrastically, with accuracy decreasing to 0.78 whenln = 10000. We note that the accuracy of discrimi-native training now decreases below that of its base-line forest. This decrease in accuracy is attributed tothe increasing number of subtrees of the baseline that

10 500 1000 1500 5000 10000ln : Maximum number of samples to create leaf node

0.76

0.77

0.78

0.79

0.80

0.81

0.82

Cla

ssifi

catio

n ac

cura

cy

Figure 5: Effect of maximum number of samples in leaf onclassification accuracy.

0 2 4 6 8 10 12Start level of discriminative training

0.800

0.802

0.804

0.806

0.808

0.810

0.812

0.814

Cla

ssifi

catio

n ac

cura

cy

Figure 6: Effect of start level for discriminative trainingon classification accuracy.

undergo the shrink operation, being replaced with aleaf.

2.4.5 Effect of node level

In Fig. 6, we look at the effect of discriminative train-ing of nodes at different levels on the classificationaccuracy. In this experiment, we begin the processof discriminative training from nodes at a specifiedlevel in the tree. When the starting level is 0, this isequivalent to discriminative training applied on thefull tree, which we have described earlier. We increase

6

0 1 2 3 4 5Iterations of discriminative training

0.76

0.77

0.78

0.79

0.80

0.81

0.82

0.83

0.84

Cla

ssifi

catio

n ac

cura

cy

Figure 7: Effect of multiple iterations of discriminativetraining on classification accuracy.

the starting level of training in increments of 2, sincealternate level split nodes hold thresholds of X and Yaxis respectively. We see that as the starting level ofdiscriminative training is lowered along the tree, theclassification accuracy also decreases gradually. Sinceevery level n typically has twice the number of nodesof level n− 1, this indicates that that the discrimina-tive training at higher level nodes has a bigger effectper node than lower levels.

2.4.6 Effect of multiple training iterations

In Fig. 7, we investigate into the effect of performingmultiple iterations of discriminative training. Iter-ation 0 is the baseline forest and iteration 1 appliesdiscriminative training on the baseline with a new Dp

dataset. In iteration n = {2, 3, 4, 5}, the discrimina-tively trained forest of iteration n − 1 is used as thebaseline and a new Dp is created for training. We ob-serve that though the accuracy increase by repeatediterations of training is quite less after iteration 2.This is attributed to the tree structure and thresh-olds that stabilize after a few iterations and no furthertraining helps after this.

In what follows, this discriminative training ap-proach is applied to our regression forests for poseestimation and classification forests for part-based la-beling of mouse from depth images as described inSec. 4 and Sec. 5.3 respectively.

Depth camera

Color camera

Open-field cage withtransparent bottom

White shroud(gives bright background forbottom-camera color image)

Black shroud(prevents reflections onglass of cage bottom)

(mounted at an angle to avoidIR from top depth camera)

Bright LED light(illuminates paws of mousefor bottom color camera)

Figure 9: Our experimental setup consisting of a cage, atop depth camera and a bottom RGB camera.

3 Overview of our mouse poseestimation system

Our system uses a setup of a top-mounted depth cam-era and a bottom-mounted color camera which deliv-ers a synchronized pair of depth and color images.Simple image preprocessing is used to segment themouse patch from the depth image, as illustrated inthe left image of step 1 in Fig.3. Step 1 of the systemestimates the 3D locations of main body joints froma depth image using a regression forest. It is difficultto uncover the limbs that are largely occluded from atop view. To address this issue, step 2 takes as inputa color image captured from below the glass floor ofthe cage, to determine locations of the paws by mak-ing use of a cascade detection procedure. Finally,step 3 delivers the final full-body pose estimation byintegrating these separate estimation results using 3Dregistration.

We next introduce our experimental setup andthen our 3D mouse skeletal model and our 3D synthe-sis engine that generates 3D virtual mice. Followingthat, we describe our base system and the applicationof discriminative training on it to reduce the error.

7

Input depth image Joints prediction

Top-view

Step 1

Step 2

Step 3

Boosted cascade

Regression forest

Input color image

Bottom-view

Paws prediction

Depth camera

Color camera

3D calibration and

registration

Side view

Top view

Final prediction

Figure 8: The three steps of our approach: From top-view depth image, Step 1 estimates 3D positions of main bodyjoints using a learned regression forest model. Step 2 uses a bottom-view color image to predict the locations of the four paws.Output of these two steps are fused in Step 3 to estimate the final full-body pose.

3.1 Experimental setup

Our setup consists of a custom-built open-field ap-paratus (50cm×45cm×30cm) with transparent glassfloor, a depth camera fixed 60cm above and a colorcamera placed 60cm below the cage, as shown inFig.9. The bottom camera is mounted at a smallangle deviating from the vertical axis to avoid directincoming infrared light from the top-mounted depthcamera. In principle, the top-view depth image isused to capture the main body pose, while locationof the paws are obtained from the bottom-view colorimage. When working with both top- and bottom-mounted cameras, both are synchronized to ensure apair of images is observed simultaneously at a time.

A grid of LEDs is placed near the bottom camerato illuminate the paws and improve the quality of im-ages of this camera. To prevent reflections appearingon glass floor bottom, a black cloth is used to encirclethe bottom of the setup. To prevent effect of externallight sources, a white cloth is used to encircle the topof the setup.

3.2 3D mouse model

We propose a 3D skeletal model of the mouse,containing 24 joints (J = 24), as illustrated inFig.10(b). It has been designed based on mouse skele-

(a)

1

2

3

4

5

6

7

8

9 10

13

15

21

14

16

22

11

17

1823

12

19

2024

(b) (c)

(d)

Figure 10: (a) Skeletal anatomy of mouse. (b) Our kine-matic model of mouse. The spine is colored in red, ears ingreen and the limbs and paws in blue. (c) Our mouse modelrendered with skin mesh. (d) Part-based labeling of our modelwith 6 distinct colors: head in green, front-right in blue, front-left in yellow, rear-right in pink, rear-left in cyan and tail inred.

8

tal anatomy [9, 13] shown in Fig.10(a), with few mod-ifications: (1) Joints: Mouse skeleton has more than100 joints and we have simplified these to 24 in ourmodel. (2) Ears: To help differentiate head and tailends of a mouse, the ears are explicitly considered astwo joints connected to main body. (3) Tail: Twojoints are assigned to approximately characterize thetail. This approximation of tail is to account for thelow resolution and noise of current consumer depthcameras, which make it difficult to detect thin andlong objects such as mouse tail in depth images.

To differentiate the limbs and the main body spinein visualization, bones are highlighted in unique col-ors. Bones of the spine from head to tail are coloredin red. Bones of ear joints {9, 10} are colored in green.Bones connecting the rest of the joints 11-24 of thefour limbs and paws are in blue. Fig.10(c) shows a3D rendering of applying the proposed skeletal modelwith surface mesh and skin texture mapping.

3.3 Synthesis engine

This skeletal model is animated to simulate most ofthe mouse full-body poses using a pose synthesis en-gine. This is a GUI program we have developed tofacilitate the easy visualization and modification ofbody joint angles and to efficiently generate sufficientamount of ground-truth synthetic depth images fortraining and testing purposes. The GUI of this en-gine is shown at the top of Fig.11. Using this engineand with the aid of an animation expert, 800 uniqueposes of typical mouse activity like standing, walking,running, turning and a combination of these poses arecreated. A few exemplar poses and their depth im-ages can be seen at the bottom of Fig.11. Randompitch, roll and in-plane rotations and small variationsin scale and bone lengths are applied to these poses torender the depth images for training. To our knowl-edge, this is the first such reported academic effort insynthesizing and visualizing mouse poses based on akinematic model.

Figure 11: Top: UI of our 3D virtual mouse engine, whichis used to synthesize and visualize ground-truth mouse depthimages. Bottom: Exemplar mouse poses (in part labels) andtheir rendered depth images.

9

4 Our mouse pose estimationsystem

Now we are ready to present the three-step pipelineof our system as follows.

Step 1: Estimation of Main Body Jointsfrom Top-view Depth Image

The main body joints 1-12 in Fig.10(b) consist of thejoints of spine, ear and hip. Starting with a set of sim-ple depth features, this estimation is accomplished bydeveloping a dedicated regression forest model thatis influenced by the work of [4, 20]. In our context,a regression forest contains an ensemble of |T | bi-nary decision trees. Each tree t ∈ T is independentlyconstructed by recursively growing the split nodesstarting from the root split node, and is trained ona dataset of synthesized depth images obtained byrunning our 3D mouse engine that renders the meshof Fig. 10(c) in various poses, rotations and scales.

For depth features, a simple version of the populardepth features in [20, 25] are adapted. At a givenpixel 2D location x ∈ R2 of an image I, denote itsdepth value as a mapping dI(x), and construct a fea-ture function φ by considering a 2D offset positionu deviating from x. Following [4], a binary test isdefined as a pair of elements, (φ, γ), with φ being thefeature function, and γ being a real-valued threshold.When an instance x passes through a split node ofour binary trees, it will be sent to the left branch ifφ(x) > γ, and to the right branch otherwise.

Regression forest: Tree and Split Nodes

Similar to existing regression forests in [20], at a splitnode, we randomly select a small set of m distinctfeatures Φ := {φi}mi=1. At every candidate featuredimension, a set of candidate thresholds Γ is uni-formly selected over the range defined by the empiri-cal set of training examples in the node. The best test(φ∗, γ∗ ∈ Γ

)is chosen from these features and accom-

panying thresholds, by maximizing the gain functiondefined next. This procedure is then repeated untilthere are L levels in the tree or once the node con-tains fewer than ln training examples.

The split test is obtained by

(φ∗, γ∗) = arg maxφ∈Φ,γ∈Γ

I(φ, γ), (4)

where the gain I(φ, γ) is defined as in Eq.(1). Inour context, a training example refers to a pixel inthe depth image, as well as its 3D location in theunderlying 3D virtual mouse. Therefore, any set oftraining examples S naturally corresponds to a pointcloud in 3D space. Denote oi→j ∈ R3 the offset of

example i to the 3D location of joint j. Let Sj :={i ∈ S | ‖oi→j‖ < ε

}be the subset of examples

in S that are sufficiently close to joint j. Denoteoj := 1

|Sj |∑i∈Sj

oi→j the mean offset of the set to

j-th joint location. The function E is defined over aset of examples that evaluates the sum of deviationsfrom the mean joint estimations:

E(S) =∑Jj=1

∑i∈Sj

∥∥oi→j − oj∥∥

2, (5)

with the set of main body joints visible from top-mounted depth camera. For a point cloud S, asmaller E(S) suggests a more compact cluster. Inthis context, for S ⊆ D′d that arrives at a node q hav-ing test (φq, γq), a set of m tests (φ, γ) is attemptedto maximize E(φ,γ)(t,D

′d). E is adapted to regression

as the reciprocal of the joint estimation error for thesamples in S:

E(t,D′p) = (∑i∈D′

p

∑Jj=1 ‖pj − pj‖2)−1, (6)

where the Euclidean distance between true positionpj and the estimated position pj by tree t is the errorfor joint j. Again we denote as Td the forest createdby this discriminative training.

Regression forest: Leaf Nodes

Denote as S the set of training examples arriving atleaf node l, let i ∈ Sj ⊆ S indexes over the subsetSj containing training examples that are sufficientlyclose to joint j in 3D space, and let i ∈ S \Sj indexesover the complementary subset. Let oli→j representthe offset of a particular training example i of leafnode l to joint j.

For each joint j of the mouse visible from top-mounted camera, the mean µj of the offset vectors

10

oli→j is computed and stored in the leaf. Theremight be few cases in training when the offsets aredistributed too widely or are multi-modal and mightnot be useful as predictor in leaf. One solution tothis problem is to compute the meanshift of the off-set vectors. Another alternative is to use the largesteigenvalue λ1. When λ1 < λmax, where λmax is abound on the eigenvalue, we mark the offset vectorstored in the leaf as a low confidence estimate. Atruntime, these estimates are used only if no otherhigh confidence estimates available for any sampledpixel from the depth image from any other tree in theforest.

At test run every depth pixel sampled from themouse patch will be processed through each of thetrees in Td from root to a certain leaf node l. For eachjoint j visible from top camera, the offset estimationsfrom the leaf nodes are collected and their mean isused as the estimated offset position of joint j fromdepth pixel location.

Discriminative training of TThe discriminative training approach introduced inSec. 2 can be adapted to this regression forest witha few modifications. In practice, for S ⊆ D′d that ar-rives at a node q having test (φq, γq), a set of m tests(φ, γ) is attempted to maximize E(φ,γ)(t,D

′d). Our

discriminative training method changes the nodes it-eratively in this manner while attempting to maintainthe existing structure of the tree if that gives betterperformance. There are two exceptional cases as be-fore. If |S| ≤ ln at a split node q, then the subtreerooted at q is replaced with a leaf node that maxi-mizes E . In the second case, if |S| > ln at a leaf nodeq, then a subtree is dynamically grown and it is re-placed in the place of q. We denote as Td the forestcreated by this discriminative training.

Step 2: Paw Detection from Bottom-view Color Image

From top-mounted depth camera, we are now ableto estimate the main body joints 1-12, as displayedin Fig.10(b). However, this is insufficient for full-body joint estimation since the limbs and paws are

typically occluded below the main body. This issuewould persist with alternate setups such as a side-mounted depth camera. This inspires us to place anadditional color camera below the cage to explicitlyaid the estimation of lower limbs and paws.

Our paw detector utilizes a cascade Adaboost clas-sifier with Haar wavelet features, similar to [22]. Ourcascade contains 20 stages that have been trained us-ing mice images captured from bottom camera, withregions of paws being manually cropped and used asforeground bounding boxes. Background images areobtained by randomly cropping from the static back-ground images with diverse lighting conditions andthe paw regions blurred out. Empirical evidence dis-cussed in later section also demonstrates the effec-tiveness of the proposed paw detector in our context.

Step 3: Fusion by 3D Calibration andRegistration

Step 3 integrates the intermediate outputs of Steps1 and 2 to deliver the full body pose of mouse. Aglobal 3D coordinate system and associated transfor-mations are required to align the intermediate out-puts. Chessboard is used to calibrate the intrinsicparameters of the two camera separately. Extrinsicparameters are computed by solving the Perspective-n-Point (PnP) problem [14]. These parameters areused to transform the 1-12 joints estimated from top-view and the paw joints 21-24 from the bottom viewto the same global coordinate system.

The paws are assumed to be on or close to thefloor. This is usually true, except in grooming orrearing poses, where the front paws can be observedfrom the top view. If the front paws cannot be ob-served from either camera, a default configuration ofthe missing joints will be imposed by executing thekinematic chain model.

What remains to be computed are the limb jointsthat connect the main body to the paws. The fore-limb joints 13-16, seen in Fig.12(a), can be estimatedby local inverse kinematics, using the paw locationsand the lengths of limb bones. The left hind joints,seen in Fig.12(b), are solved by setting the angle be-tween 17, 18, and 23 to 90◦, and computing the loca-tion of 17 using local inverse kinematics. The right

11

3

13

15

21

14

16

22

6

11

17

1823

12

19

2024

Figure 12: Left: Fore limb structure. Right: Hind limbstructure.

hind joints are computed similarly.

5 Experiments

The proposed pose estimation approach has been ap-plied to a variety of synthetic and real data, to dif-ferent types of rodents, to different types of depthcameras and imaging setups. In addition, to demon-strate the applicability of our approach to work onrelated problems, our method is used for the task ofpart-based labeling from single depth images, whereit is shown to produce satisfactory results.

The training and testing are performed on a com-puter with Intel Core i7-4790 CPU and 16 GB ofmemory. Our discriminative training stage is multi-threaded and the rest of the steps use single CPUcore. The full-body pose estimation from a pair ofdepth and color images on real data is currently un-optimized and performs at a speed of 2 frame-per-second (FPS).

Unless mentioned otherwise, all the regressionforests are trained with |T | = 7 and |Td| = 7. A setof 240, 000 synthetic images generated using commonmouse poses together with pitch, roll, and in-planerotations and scale variations of the mouse model isused as training data. The depth pixels for train-ing, discriminative training and testing are randomlysampled from these images. At each split node, asubset of m = 50 tests are randomly selected. ε = 10is used for paws, ε = 15 for limb joints and ears,ε = 50 for tail joints and ε = 25 for the rest. τ = 10is used for weights at leaf nodes. Trees are grownto a maximum of L = 20 levels or a node has lessthan ln = 60 training examples. For paw detection,a cascaded classifier with 20 stages is trained using

1 2 3 4 5 6 7 8 9 10|T|: Number of trees in forest

3.50

3.55

3.60

3.65

3.70

3.75

3.80

3.85

3.90

Aver

age

join

t erro

r (m

m)

Figure 13: Effect of forest size in T on average joint error.

4200 images with paws manually marked and 3000negative background images.

5.1 Synthetic datasets for mouse poseestimation

To evaluate our method we use joint error as definedin Sec. 2. We scale the model and the joint error as-suming a mouse length of 100mm, which is about thelength of an adult mouse. As test data, 700 imagesare generated with variations in mouse size, pose, andorientation.

5.1.1 Effect of forest size

In Fig. 13 we look at the effect of forest size in Ton the average joint error for the 12 joints estimatedfrom top depth image. The values presented are theaverage difference in Euclidean distance between es-timated and true joint positions in 3D over all joints.We can see that the joint error decreases as the num-ber of trees in T increases. However, the joint errordoes not decrease much after |T | = 7, where the aver-age joint error is 3.62mm. We will use this forest sizefor the rest of our experiments with pose estimation.

5.1.2 Effect of m

In Fig. 14 we examine the effect of m, the numberof tests applied at split nodes in T . We see that

12

25 50 75 100 125 150 175 200m: Number of tests applied at split nodes

3.50

3.55

3.60

3.65

3.70

3.75

3.80

3.85

3.90

Aver

age

join

t erro

r (m

m)

Figure 14: Effect of m in Td on average joint error.

1 2 3 4 5 6 7 8 9 10 11 12Joints

0

1

2

3

4

5

6

Aver

age

join

t erro

r (m

m)

TTd

Figure 15: Comparison of joint error between baseline T andTd trees.

the average joint error over all estimated joints de-creases with increasing values of m until m = 100,after which it increases. This can be attributed tothe increasing correlation between the split nodes ofdifferent trees in the forest as the sampling size froma fixed number of tests increases. We use m = 100for the rest of our experiments with pose estimation.

5.1.3 Effect of discriminative training

In Fig. 15 we examine the effect of applying discrim-inative training as described in Sec. 4 with natu-rally the same number of trees |T | = |Td| = 7 and

13 14 15 16 17 18 19 20 21 22 23 24Joint

0

1

2

3

4

5

6

7

8

9

Aver

age

join

t erro

r (m

m)

Figure 16: Average error of joint positions estimated frombottom image.

m = 100. The graph shows a comparison of the errorsfor the 12 joints estimated from top camera syntheticdepth image for both T and Td. We can see that thisdiscriminative learning process has substantially de-creased the error on every joint. For example, joint 1which has the highest error of 5.90mm with T is re-duced to 4.72mm with Td. The average error over alljoints for T is 3.62mm and it is substantially reducedto 2.96mm by Td. For all the following experimentswe use this discriminatively trained Td.

Fig. 16 shows the average joint error for the limband paw joints estimated using the paw locations de-tected in bottom image. It can be observed that theseerrors are relatively higher compared to top joints.This is due to the higher DoF of limb joints comparedto spine and our method of limb joint estimation isonly a relatively coarse approximate.

Fig.17 presents several results of utilizing Td formouse pose estimation with diverse gestures and ori-entations. The figure shows the actual 3D positions ofboth the ground-truth and estimated joints. Visuallyour estimation results are shown to be well-alignedwith the ground-truth poses.

Fig. 18 shows a few exemplar results comparingthe pose estimation results on synthetic data usingforests without and with discriminative training. Thejoint positions, drawn as spheres, are colored basedon whether they are ground-truth, estimated withand without discriminative training. It can be ob-

13

Estimated joint position

True joint position

Spine bone

Limb bone

Ear boneDistance fromtrue to estimate

Depth pixel in 3D

(a)

(b)

Figure 17: Top: Four pose estimates are obtained by applying our method on depth-color test-image pairs containing posesof standing, walking, running and bending while walking respectively. The estimated and true joint positions are shown as redand cyan disks respectively. The depth pixels converted to 3D positions are rendered as grey dots. Bones of the skeleton arecolored following the protocol of Fig.10(b). Bottom: A diverse range of estimated poses as seen from various viewpoints in 3D.

Estimate using discriminative trainingEstimate without discriminative trainingGround truth

Joint position

Figure 18: Comparison of pose estimation results using forests without discriminative training and with discriminativetraining. Results show that with discriminative training, the estimated joint positions are closer to the ground-truth and havelesser error than without discriminative training.

14

Joint1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Gaussian noise

σ=0

σ=1

σ=2

σ=3

σ=4

Joint error (mm

)

0

5

10

15

20

Figure 19: Average joint error with increasing amountof Gaussian noise σ = {0, 1, 2, 3, 4, 5} added to input depthimages.

served that discriminative training helps in reducingthe joint position error and the estimated joints liecloser to ground-truth.

5.1.4 Effect of signal noise

Fig.19 shows the robustness of our method to noisein depth image. Gaussian random noise with levelsσ = {0, 1, 2, 3, 4, 5} is applied to synthetic depth im-ages. We see that there is only a slight increase inaverage joint error in proportion to the noise. Thisdemonstrates the robustness of our random forestmodel for joint position estimation. The tail jointis highly sensitive to noise, which can be attributedto its appearance as a thin strip of pixels in the depthimage and its topology can be warped easily by noise.

5.2 Real datasets for mouse pose esti-mation

Real data have been captured with two rodents:lab mouse (mus musculus) and hamster (cricetidae).Two consumer depth cameras are employed for top-view: structured illumination (Primesense Carmine)and ToF (SoftKinetic DS325). Though the noisecharacteristics of the two depth cameras are signif-icantly different, our approach utilizing Td is able toreliably deliver good results. The color images are allfrom a bottom-view Basler CMOS camera.

5.2.1 Pose estimation from a pair of depthand color images

Using the two-camera setup of Fig. 3, a large numberof image pairs are captured of both lab mouse andhamster. Primesense depth camera is used for top-view and a Basler color camera is used for bottom-view. Exemplar results on the image pairs of labmouse (mus musculus) is illustrated in Fig.20. Sim-ilarly, Fig.21 shows exemplar results for our hamsterdata set. These results show that our approach candeliver visually meaningful full body skeletal poseoutputs that match well with the camera observa-tions. This estimation is far from being trivial, con-sidering the difficult contexts where many interme-diate joints such as limb joints 13-20 are usually oc-cluded from both cameras, and the input signals arehighly noisy.

5.2.2 Pose estimation from single depth im-ages

To demonstrate the robustness of the proposed ap-proach, a simpler setup of a single top-mounted depthcamera is considered. Using only the top-view depthimages, the joints of the spine can be estimated byour random forest using only Step 1 of our system.

Fig.22 shows our pose estimation results for top-view depth images obtained from a Primesense struc-tured illumination camera. In this case, only the es-timated 3D positions of main body joints obtainedfrom our discriminatively trained forest is used.

Fig.23 shows our pose estimation results for top-view depth images obtained from a SoftKineticDS325 ToF camera. The resulting skeleton formsseem to be visually appealing, while remaining well-aligned with the inputs from both the input top-view as well as a novel side-view. These resultsalso demonstrate the general applicability of our sys-tem to depth cameras using different technologies likestructured illumination and time-of-flight (ToF).

Fig. 24 shows examples of cases where paw detec-tion has errors. Sometimes the mouth or tail of themouse is mistaken for a paw.

15

Figure 20: Exemplar results of full-body pose estimation of lab mouse (mus musculus) from image pairs captured fromPrimesense depth and Basler color cameras. Left column shows pairs of cropped regions from depth and color input imageswith the detected paws marked in red circles. Middle column shows a side-view 3D rendering of the estimated mouse poseskeleton while right column is the corresponding top view rendering of the pose. Bones of the skeleton are colored followingthe protocol of Fig. 10(b) in our paper. The paws are rendered as yellow spheres. The depth pixels (i.e. 3D cloud of points)are rendered in light-grey and floor is rendered in pink color.

16

Figure 21: Exemplar results of full-body pose estimation of hamster (cricetidae) from image pairs captured from Primesensedepth and Basler color cameras. Left column shows pairs of cropped regions from depth and color input images with the detectedpaws marked in red circles. Middle column shows a side-view 3D rendering of the estimated mouse pose skeleton while rightcolumn is the corresponding top view rendering of the pose. Bones of the skeleton are colored following the protocol of Fig. 10(b)in our paper. The paws are rendered as yellow spheres. The depth pixels (i.e. 3D cloud of points) are rendered in light-greyand floor is rendered in pink color.

17

Figure 22: Two examples of pose estimation based on single depth images from top-view Primesense structured illuminationcamera. The estimated 3D positions of spine joints are shown in top and side view. The white point cloud is the set of depthvalues of the mouse pixels obtained from the input depth image. Please see our supplementary video for more results on realdata.

18

Figure 23: Exemplar results of pose estimation based on single depth images from top-view SoftKinetic ToF camera. Top:Single depth images. Middle: Projection of the estimated joints 3D locations onto the top-view (i.e. the same viewing positionas of the depth camera). Bottom: Projection onto a novel side view. The white point cloud is the set of depth values of themouse pixels obtained from the input depth image.

19

Figure 24: Paw detection errors in depth images fromPrimesense camera. Detected paws are circled in red, wronglydetected region (usually mouth) is indicated with blue arrowand undetected paw is shown with green arrow.

5.3 Part-based labels from singledepth images

Experiments are also carried out on the closely re-lated task of part-based labeling of mouse using depthimages captured from a top-mounted depth camera.The body of the mouse is partitioned into six distinctclasses with distinct colors as shown in Fig.10(d). Askin texture with these colors is used by our mouseengine to render poses to depth images. During test-ing stage, every pixel of a test depth image is pro-cessed using the learned random forest model to ob-tain the averaged class histogram. The estimatedclass for a depth pixel is the class with highest num-ber of votes in the histogram.

5.3.1 Synthetic datasets

To generate training data, 360 poses are renderedwith random rotations applied on each pose. A Gaus-sian noise of σ = 16 is applied on the depth imagesand a total of 496, 800 depth pixels are sampled. Arandom forest T of 7 trees is trained using 20000tests, L = 13 and ln = 60.

To further improve the results of T on noisy depthimages, we adapt the discriminative training ap-proach described in Sec. 2 to perturb the nodes ofT using a second training dataset Dd. The nodesof these classification trees can be perturbed as de-scribed in Sec. 4 for regression trees, including growand shrink actions as needed. For classification, wetry to maximize E(t,Dd) defined as:

E(t,Dd) =∑i∈Dd

L(yi, yi), (7)

where yi is the predicted class for training examplei and yi is its true class. L = 1 when yi = yi and 0

Tm=30 Tdm=25 Tdm=50 Tdm=100 Tdm=200

m: Number of tests used at split node

0.760

0.765

0.770

0.775

0.780

0.785

0.790

Cla

ssifi

catio

n ac

cura

cy

Figure 26: Part-based labeling classification accuracy ofbase T with m = 30 compared with discriminatively trainedTd with increasing values of m.

otherwise.

Fig. 25 shows results of our part-based labelingmethod for a few exemplar inputs where the mouse isstanding, walking, running, rearing and bending. Wecan see that the estimated results match the groundtruth for most of the pixels.

Fig. 26 shows the classification accuracy results forthe forest before and after discriminative training ofits nodes. First we note that applying discriminativetraining gives better accuracy no matter what valueof m is used. Further we notice that increasing m in-creases accuracy, but only until m = 100 after whichthe accuracy reduces. This can be attributed to theincreased correlation between trees which decreasesthe performance of the forest.

Fig. 27 shows several exemplar results comparingthe part-based labeling results on synthetic data us-ing forests without discriminative training (m = 30)and with discriminative training (m = 100). Theseresults can be compared to the ground-truth labeledimage shown in leftmost column. It can be observedthat discriminately trained forest improves the re-sults in a few problematic areas in the head regionand middle region of mouse.

Fig.28 presents the confusion matrix results forpart-based labeling of single depth images with noiseσ = 16 with the final forest. All classes, except for therear-right class, have an accuracy of 89% or higher.

20

Figure 25: Exemplar results of part-based labeling method. Top row is input depth image, middle row is ground truth andbottom row is estimated result from our discriminatively trained forest.

Head Front-R Front-L Rear-R Rear-L Tail

Predicted class

Head

Front-R

Front-L

Rear-R

Rear-L

Tail

True

cla

ss

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Figure 28: Confusion matrix of our part-based labelingresults on a mouse body partitioned into six classes.

(a)

(b)

Figure 29: Part-based labeling results on single depthimages from (a) Primesense camera (b) SoftKinetic camera.

The rear-right class has an accuracy of 52.5%, as itspixels are sometimes misclassified as that from front-right.

21

Ground truthWithout discriminative

trainingWith discriminative

training

Figure 27: Comparison of part-based labeling results using forests without and with discriminative training. Resultsdemonstrate that with discriminative training, fewer pixels in head region of mouse are mislabled as tail (red color). It can alsobe observed that pixels in the middle of mouse body have fewer disconnected components.

22

(a)

(b)

Figure 30: (a) Failed part based labeling results for depthimages of Primesense camera. (b) Failed part-based labelingresults for depth images of SoftKinetic camera. Some of theerrors are mouse labels being reversed from head to tail.

5.3.2 Real datasets

To demonstrate the applicability of our method onreal data, we perform part-based labeling on singledepth images of a mouse captured from a top depthcamera. The real data results show a bit more noise,but is within expected bounds of part-based label-ing for pose estimation as explained in in [19]. Tofurther demonstrate the versatility of the proposedapproach, two types of depth cameras are consid-ered. For experiments, the depth images are prepro-cessed to remove shadows and noise and the patchcontaining mouse is recognized using contour detec-tion. Fig. 29(a) and Fig. 29(b) shows the results ofour part-based labeling method on test data capturedfrom a SoftKinetic and Primesense camera respec-tively.

Since the tail is hardly visible in depth images, ourlabeling usually misses the tail class. Similar to thesynthetic results, the rear right label region is oftenquite small comparing to the rest part labels. Over-all our approach is shown to be capable of providingsatisfactory results robustly in this context.

Fig. 30(a) shows examples of failures in part-based labeling on images from Primesense camera.

Fig. 30(b) shows examples of failures in part-basedlabeling on images from SoftKinetic camera. Otherthan tiny regions being mislabeled, another cause offailures is that sometimes the head is mistaken forthe tail region.

6 Conclusion and future work

In this paper we presented a discriminative trainingapproach that is demonstrated to reduce errors inboth regression and classification random forests. Wealso introduced an approach for mouse 3D pose es-timation based mainly on single depth images. Theproposed approach is capable of producing reliable3D full-body pose predictions when incorporatingadditional color image information from a differentview. It also works well with various types of depthcameras. We have also demonstrated satisfactoryperformance when addressing the related problem ofpart-based labeling. Future work includes extensionsto deal with mice tracking and group behavior anal-ysis using depth cameras.

References

[1] Abbott, A.: Novartis reboots brain division. Nature502, 153–154 (2013)

[2] Bono, J.D., Adlam, D., Paterson, D., Channon, K.:Novel quantitative phenotypes of exercise trainingin mouse models. Am J Physiol Regul Integr CompPhysiol 290(4), 926–34 (2006)

[3] Branson, K., Belongie, S.: Tracking multiple mousecontours (without too many samples). In: CVPR(2005)

[4] Breiman, L.: Random forests. Machine Learning45(1), 5–32 (2001)

[5] Burgos-Artizzu, X., Dollar, P., Lin, D., Anderson,D., Perona, P.: Social behavior recognition in con-tinuous videos. In: CVPR (2012)

[6] Burgos-Artizzu, X., Hall, D., Perona, P., Dollar, P.:Merging pose estimates across space and time. In:BMVC (2013)

23

[7] Chaumont, F., Coura, R., Serreau, P., Cressant, A.,Chabout, J., Granon, S., Olivo-Marin, J.: Comput-erized video analysis of social interactions in mice.Nat Methods 9(4), 410–7 (2012)

[8] Cryan, J., Holmes, A.: The ascent of mouse: ad-vances in modelling human depression and anxiety.Nat Rev Drug Discov. 4(9), 775–90 (2005)

[9] Dogdas, B., Stout, D., Chatziioannou, A., Leahy, R.:Digimouse: a 3D whole body mouse atlas from ctand cryosection data. Phys Med Biol. 52(3), 577–87(2007)

[10] Fanelli, G., Dantone, M., Gall, J., Fossati, A., Gool,L.V.: Random forests for real time 3D face analysis.IJCV 101(3), 437–58 (2013)

[11] Giacomotto, J., Segalat, L.: High-throughputscreening and small animal models, where are we?British J Pharmacol 160(2), 204–16 (2010)

[12] Jhuang, H., Garrote, E., Mutch, J., Poggio, T.,Steele, A., Serre, T.: Automated home-cage be-havioural phenotyping of mice. Nat Commun. 1(68)(2010)

[13] Khmelinskii, A., Baiker, M., Kaijzel, E., Chen, J.,Reiber, J., Lelieveldt, B.: Articulated whole-bodyatlases for small animal image analysis: constructionand applications. Mol Imaging Biol. 13(5), 898–910(2011)

[14] Li, S., Xu, C., Xie, M.: A robust O(n) solution tothe perspective-n-point problem. IEEE Trans. PAMI34(7), 1444–1450 (2012)

[15] Ohayon, S., Avni, O., Taylor, A., Egnor, R., Per-ona, P.: Automated multi-day tracking of markedmice for the analysis of social behavior. J NeurosciMethods 219(1), 10–9 (2013)

[16] Ou-Yang, T., Tsai, M., Yen, C., Lin, T.: An infraredrange camera-based approach for three-dimensionallocomotion tracking and pose reconstruction in a ro-dent. J Neurosci Meth. 201(1), 116–23 (2011)

[17] Schaefer, A., Claridge-Chang, A.: The surveillancestate of behavioral automation. Curr Opinion inNeurobiology 22(1), 170–6 (2012)

[18] Sheets, A., Lai, P., Fisher, L., Basso, D.: Quanti-tative evaluation of 3D mouse behaviors and motorfunction in the open-field after spinal cord injury us-ing markerless motion tracking. PLOS One 8(9),1–13 (2013)

[19] Shotton, J., Fitzgibbon, A., Cook, M., Sharp, T.,Finocchio, M., Moore, R., Kipman, A., Blake, A.:Real-time human pose recognition in parts from sin-gle depth images. In: CVPR (2011)

[20] Shotton, J., Girshick, R., Fitzgibbon, A., Sharp, T.,Cook, M., Finocchio, M., Moore, R., Kohli, P., Cri-minisi, A., Kipman, A., Blake, A.: Efficient humanpose estimation from single depth images. IEEETPAMI 35(12), 2821–40 (2013)

[21] Steele, A., Jackson, W., King, O., Lindquist, S.:The power of automated high-resolution behavioranalysis revealed by its application to mouse modelsof huntington’s and prion diseases. PNAS 104(6),1983–8 (2007)

[22] Viola, P., Jones, M.: Robust real-time face detection.International Journal of Computer Vision 57, 137–154 (2004)

[23] Wang, H., Ullah, M., Klaser, A., Laptev, I., Schmid,C.: Evaluation of local spatio-temporal features foraction recognition. In: British Machine Vision Con-ference (2009)

[24] Weissbrod, A., Shapiro, A., Vasserman, G., Edry, L.,Dayan, M., Yitzhaky, A., Hertzberg, L., Feinerman,O., Kimchi, T.: Automated long-term tracking andsocial behavioural phenotyping of animal colonieswithin a semi-natural environment. Nat Commun.4, 2018 (2013)

[25] Xu, C., Nanjappa, A., Zhang, X., Cheng, L.: Esti-mate hand poses efficiently from single depth images.International Journal of Computer Vision pp. 1–25(2015)

24