given video streams from several cameras, multi-person tracking … · 2012. 10. 1. ·...

MULTI-CAMERA MULTI-TARGET TRACKING

FRANCOIS FLEURET

OCTOBER 1, 2012

Joint work with H. Benshitrit, J. Berclaz,

R. Lengagne, E. Turetken, and P. Fua

ÉCOLE POLYTECHNIQUEFÉDÉRALE DE LAUSANNE

INTRODUCTIONWHAT IS TRACKING ?

Given video streams from several cameras, multi-person trackingconsists of estimating where individuals are located on the groundplane in each frame.

Original sequence [video]

Francois Fleuret 2 / 45

INTRODUCTIONAPPLICATIONS

Detecting, localizing, and tracking multiple pedestrians is the corecomponent of multiple applications:

- Preprocessing for behavioral studies.- Hazardous area control.- People counting.- Activity surveillance.


PEDESTRIAN DETECTION

Monocular

. M. Andriluka, S. Roth, and B. Schiele. Pictorial structures revisited: People detectionand articulated pose estimation. In Conference on Computer Vision and PatternRecognition, 2009

. Lubomir Bourdev, Subhransu Maji, Thomas Brox, and Jitendra Malik. Detecting peopleusing mutually consistent poselet activations. In European Conference on ComputerVision, 2010. O. Barinova, V. Lempitsky, and P. Kohli. On the detection of multiple object instancesusing Hough transforms. In IEEE Conference on Computer Vision and PatternRecognition, 2010

Multi-camera

. A. Ess, B. Leibe, and L. Van Gool. Depth and appearance for mobile scene analysis. InInternational Conference on Computer Vision, 2007

. S. M. Khan and M. Shah. Tracking multiple occluding people by localizing on multiplescene planes. IEEE Transactions on Pattern Analysis and Machine Intelligence,31(3):505–519, 2009

. S. Sternig, T. Mauthner, A. Irschara, P. M. Roth, and H. Bischof. Multi-cameramulti-object tracking by robust hough-based homography projections. In ICCV Workshopon Visual Surveillance, pages 1689–1696, 2011


TRACKINGPAIRING AND FILTERING

. A. G. A. Perera, C. Srinivas, A. Hoogs, G. Brooksby, and Hu W. Multi-object trackingthrough simultaneous long occlusions and split-merge conditions. In Conference onComputer Vision and Pattern Recognition, 2006

. M. Isard and J. MaxCormick. Bramble: A bayesian multiple-blob tracker. In InternationalConference on Computer Vision, 2001

. K. Smith, D. Gatica-Perez, and J-M. Odobez. Using particles to track varying numbersof interacting people. In Conference on Computer Vision and Pattern Recognition, 2005

. M. D. Breitenstein, F. Reichlin, B. Leibe, E. Koller-Meier, and L. J. V. Gool. Robusttracking-by-detection using a detector confidence particle filter. In International Conferenceon Computer Vision, 2009


TRACKINGGRAPH-BASED

Adaptive Graph

. H. Jiang, S. Fels, and J. Little. A linear programming approach for multiple objecttracking. In Conference on Computer Vision and Pattern Recognition, 2007

. L. Zhang, Y. Li, and R. Nevatia. Global data association for multi-object tracking usingnetwork flows. In Conference on Computer Vision and Pattern Recognition, 2008

. B. Yang and R. Nevatia. An online learned CRF model for multi-target tracking. InConference on Computer Vision and Pattern Recognition, 2012

Pre-defined Graph

. F. Fleuret, J. Berclaz, R. Lengagne, and P. Fua. Multi-camera people tracking with aprobabilistic occupancy map. IEEE Transactions on Pattern Analysis and MachineIntelligence (TPAMI), 30(2):267–282, 2008

. A. Andriyenko and K. Schindler. Globally optimal multi-target tracking on a hexagonallattice. In European Conference on Computer Vision, 2010

. A. Alahi, L. Jacques, Y. Boursier, and P. Vandergheynst. Sparsity driven peoplelocalization with a heterogeneous network of cameras. Journal of Mathematical Imagingand Vision, 41(1-2):39–58, 2011


PROBABILISTIC OCCUPANCY MAP

(POM)

OBJECTIVESUMMARY

Given the images from multiple cameras taken synchronously, wewant to compute a probability of occupancy for every location onthe ground.

We use a real-time background subtraction to pre-process theimage. It estimates a background image and which parts of thecurrent frame differ from it.

Segmented sequence [video]


OBJECTIVEDESIRED PROPERTIES

Our estimation of the occupancy should

- Work on isolated temporal frames.- Combine several views.- Provide probabilistic estimates.- Handle noisy background subtraction.- Take into account silhouette sizes for distance.- Handle occlusions consistently.


MODELINGGROUND DISCRETIZATION

The ground plane is regularly divided into N locations. HereN = 1332, corresponding to a 20cm accuracy.

Top-view From camera 0 From camera 1


MODELINGCAMERA CALIBRATION

The calibration of the C cameras is provided by a collection ofrectangles

Ac1, . . . ,A

cN , c = 1, . . . ,C

of human size.

Collection of “human shapes” [video]


MODELINGNOTATION

We work on one frame and introduce the following randomvariables

X = (X 1, . . . ,X N) on {0, 1}N the grid occupancy (unknown).

B = (B1, . . . , BC) on({0, 1}w×h)C the images (known).

The quantity of interest is P(X |B).


MODELINGINDEPENDENCE ASSUMPTIONS

We make the following independence assumptions

P(X 1, . . . , X N) =∏

n

P(X n)

P(B1, . . . , BC |X 1, . . . , X N) =∏

c

P(Bc |X 1, . . . , X N)


MODELINGCRUDE SYNTHESIS

Let Ac denote a synthetic picture obtained by putting rectanglesAc

n where X n =1.

X 1, . . . ,X N A0 A1


MODELINGMODELING P(B |X)

We introduce a pseudo-distance Ψ between images

Ψ(Bc ,Ac) =1σ

‖Bc − Ac‖2

‖Ac‖2

and use the following model

P(B |X) =∏

c

P(Bc |X)

=1z

∏c

e−Ψ(Bc ,Ac)


ESTIMATION OF THE qn

APPROXIMATION OF P(X |B)

Let Q denote a product law approximating the true posteriordensity P( · |B)

Q(X = x) =∏

n

Q(X n =xn)

We want to compute the marginals qn of Q

qn = Q(X n =1)

... /...



PROOF (CONT.)

=∂

∂qn EQ

[log

Q(X n)

P(X n)− log P(B |X)

]=

∂

∂qn qn(

logqn

ε− EQ

[log P(B |X) |X n =1

])+

∂

∂qn (1−qn)

(log

1−qn

1−ε− EQ


])= log

qn

ε+ 1− EQ


]− log

1−qn

1−ε− 1 + EQ


]= − log

(1qn − 1

)+ log

1−εε

− EQ[log P(B |X) |X n =1

]+ EQ


]�



LOCAL LINEARIZATION OF Ψ

Under our model

P(B |X) =1z

∏c

e−Ψ(Bc ,Ac),

we have

EQ(log P(B |X) |X n =ξ

)= −

∑c

EQ(Ψ(Bc ,Ac) |X n =ξ) + cst

Computing EQ(Ψ(Bc ,Ac) |X n = ξ) is intractable. However, sinceAc is highly picked around Bc for X ∼ Q, we use ∀n,∀ξ

EQ(Ψ(Bc ,Ac) |X n =ξ) ' Ψ(Bc , EQ(Ac |X n =ξ))



AVERAGE IMAGE

The synthetic image Ac is a function of X, thus EQ(Ac) is welldefined, and a function of the qn:

EQ(Ac(x , y)) = Q(Ac(x , y) = 1)

= 1−∏

n:Acn(x ,y)=1

(1− qn)

Averaging images [video]



SUMMARY

Finally, our algorithm solves, ∀n

qn =1

1 + exp(λ+

∑c Ψ(Bc , EQ(Ac |X n =1))−Ψ(Bc ,EQ(Ac |X n =0))

)

Where Ψ is the pseudo-distance and λ is the prior log-ratio.



INTUITIVELY

The algorithm updates qn according to the fit between theEQ(Ac |X n =ξ) and the real images Bc , with the current estimatesof the qls for Q

EQ(Ac) EQ(Ac |X n =0) EQ(Ac |X n =1)



ALGORTIHM

Algorithm 1 Probabilistic Occupancy Map1: for n = 1, . . . ,N do2: q0

n ← ε3: end for4: repeat5: for n = 1, . . . ,N do6: for c = 1, . . . ,C do7: A

c0 ← EQk (Ac |X n =0)

8: Ac1 ← EQk (Ac |X n =1)

9: end for10: qk+1

n ← 11+exp

(λ+∑

c Ψ(Bc ,Ac1)−Ψ(Bc ,A

c0))

11: end for12: until Convergence


RESULTSCONVERGENCE

The algorithm chooses the qns so that the average syntheticimages match the result of the background subtraction.

Two cameras (106) [video]




Four cameras (1563) [video]

Four cameras (3258) [video]

Four cameras, kids [video]


RESULTSFULL SEQUENCES

This procedure can then be applied on every single frame of asequence:

Two person sequence [video]

EPFL Exterior [video]

FIBA match [video]


RESULTSALGORITHM COMPLEXITY

Since EQ(Ac) and EQ(Ac |X n =ξ) differ only in Acn, the

computation of Ψ(Bc ,EQ(Ac |X n =ξ)) can be done at constanttime with integral images.

With N = 1332, two cameras and one hundred iterations toconverge, it takes 150ms per time frame on a 2.4Ghz Pentium.

Real speed [video]


MULTI-TRACKED PATHS

(MTP)

GRAPH-BASED TRACKINGTIME CONSISTENCY ALLOWS TO FILTER MISTAKES

Time (T steps)S

pace

(Npo

sitio

ns)


GRAPH-BASED TRACKINGABSTRACTING A BIT OUR GOAL

IfM⊂ {0,1}N×T is the set of Binary maps corresponding tophysically possible presences of people, we are looking for

arg max(x1

1,...,xNT )∈M

log P(∀n, t , X nt = xn

t |B11 , . . . ,B

CT )

= arg max(x1

1,...,xNT )∈M

∑t ,n

log P(X nt = xn

t |B1t , . . . ,B

Ct )

= arg max(x1

1,...,xNT )∈M

∑t ,n

logP(X n

t = 1 |B1t , . . . ,B

Ct )

P(X nt = 0 |B1

t , . . . ,BCt )

xnt

= arg max(x1

1,...,xNT )∈M

∑t ,n

(log

qnt

1− qnt

)xn

t︸︷︷︸S(x1

1,...,xNT )

The score S to maximize is linear.


GRAPH-BASED TRACKINGUNION OF DISJOINT TRAJECTORIES

The setM contains all maps which are the unions of disjointcontinuous trajectories.

Since the value S is linear, if x is an union of disjoint trajectories,S(x) is the sum of the values of the trajectories composing x .

A straight-forward strategy to compute arg maxx S(x) is to addtrajectories one after another.


GRAPH-BASED TRACKINGGREEDY OPTIMIZATION

Time (T steps)

Spa

ce(N

loca

tions

)

1

-1

-1

1

1

-1

-1

1

-1

1

-1

1

-1

-1

1

1


GRAPH-BASED TRACKINGTYPICAL PROBLEM

Time (T steps)

Spa

ce(N

loca

tions

)10

-1

-1

1

1

-1

-1

1

1

-1

-1

1

1

-1

-1

10


GRAPH-BASED TRACKINGKSP

Given a graph, with a source and a sink, there is an algorithm tocompute the family of K disjoint paths from source to sink withminimal total length.

It consists of iteratively finding the shortest path, after inverting allthe edges occupied by the paths found in the previous iterations.

The computed family is globally optimal.



SS T

2

4

4

2

1



0 5

2

3

2

4

4

2

1



0 7

3

4

-2

4

4

-2

-1



S T

2

4

4

2

1



S T

2

4

4

2

1


KSPPSEUDO-CODE

Algorithm 2 k -shortest Paths1: for k = 1, . . . ,K do2: Compute distances to source3: Find best path4: Invert occupied edges5: Use distances to make all lengths positive6: end for


MTPPSEUDO-CODE

Algorithm 3 Multi-tracked Paths1: repeat2: Compute distances to source3: Find best path4: Invert occupied edges5: Use distances to make all lengths positive6: until Best path is of positive (original) length


MTPEXAMPLE

Time (T steps)

Spa

ce(N

loca

tions

)

10

-1

-1

1

1

-1

-1

1

1

-1

-1

1

1

-1

-1

10


MTPEXAMPLE

Time (T steps)

Spa

ce(N

loca

tions

)-10

1

1

-1

-1

1

1

-1

-1

1

1

-1

-1

1

1

-10


MTPEXAMPLE

Time (T steps)

Spa

ce(N

loca

tions

)

-10

1

1

-1

-1

1

1

-1

-1

1

1

-1

-1

1

1

-10

s t


MTPEXAMPLE

Time (T steps)

Spa

ce(N

loca

tions

)

-18

-10

1

1

-1

-1

1

1

-1

-1

1

1

-1

-1

1

1

-10

s t


MTPEXAMPLE

Time (T steps)

Spa

ce(N

loca

tions

)

-18

-8

-10

1

1

-1

-1

1

1

-1

-1

1

1

-1

-1

1

1

-10

s t


MTPEXAMPLE

Time (T steps)

Spa

ce(N

loca

tions

)-10

1

1

-1

-1

1

1

-1

-1

1

1

-1

-1

1

1

-10

s t


MTP GPL IMPLEMENTATIONEXAMPLE

// Initialize the tracker

MTPTracker *tracker = new MTPTracker();

tracker->allocate(nb_time_steps, nb_locations);

// Define the allowed target motions

for(int l = 0; l < nb_locations; l++) {

for(int m = 0; m < nb_locations; m++) {

tracker->allowed_motion[l][m] = abs(l - m) <= motion_amplitude;

}

}

// Define the entrances (first frame and location 0) and exists

// (last frame and location nb_locations-1)

for(int t = 0; t < nb_time_steps; t++) {


tracker->entrances[t][l] = (t == 0 || l == 0);

tracker->exits[t][l] = (t == nb_time_steps - 1 || l == nb_locations-1);

}

}

// Build the graph to us the modified KSP

tracker->build_graph();


MTP GPL IMPLEMENTATIONEXAMPLE

// Fill the detection map

for(int t = 0; t < nb_time_steps; t++) {


tracker->detection_scores[t][l] = detect(t, l);

}

}

// Performs the tracking per se

tracker->track();

// Prints the detected trajectories

for(int t = 0; t < tracker->nb_trajectories(); t++) {

cout << t;

for(int u = 0; u < tracker->trajectory_duration(t); u++) {

cout << " " << tracker->trajectory_location(t, u);

}

cout << endl;

}

delete tracker;


MTP GPL IMPLEMENTATIONRESULT GRAPH

0

1

0

2

0

3

0

4

0

5

0

6

0

7

0

15

029

0 43

0 57

0 71

0 85

0

99

0

113

8-10

0

16

0

9-1

0

0

17

0

10

10

0

18

0

11

1 0

0

19

0

12

1 0

0

20

0

13

1

0

0

21

0

14

1

0

0

0

221

0

30

0

23-10

0

0

31

0

241

0

0

32

0

251

0

0

33

0

261

0

0

34

0

27-1

0

0

35

0

281

0

0

0

361

0

44

0

371

0

0

45

0

38-10

0

0

46

0

391

0

0

47

0

40-1

0

0

48

0

411

0

0

49

0

421

0

0

0

501

0

58

0

511

0

0

59

0

52-10

0

0

60

0

531

0

0

61

0

54-1

0

0

62

0

551

0

0

63

0

561

0

0

0

641

0

72

0

651

0

0

73

0

66-1

0

0

74

0

671

0

0

75

0

68-10

0

0

76

0

691

0

0

77

0

701

0

0

0

781

0

86

0

791

0

0

87

0

80-1

0

0

88

0

811

0

0

89

0

82-10

0

0

90

0

831

0

0

91

0

841

0

0

0

921

0

100

0

93-1

0

0

101

0

941

0

0

102

0

95-1

0

0

103

0

961

0

0

104

0

97-10

0

0

105

0

981

0

0

0

106

-1

0

107

1

0108

1

0109

1

0110

1

0111

1

0

112-10

0

L = 7 and T = 8


RESULTS

This algorithm has been tested on the tracking of basketballplayers in real-world situation:

Basketball POM+KSP [video]

We have recently put an appearance model back by adding anadditional “identity” variable. KSP can not optimize it anymore, sowe run LP on the locations selected without appearance by KSP.

Basketball POM+KSP+LP [video]


REFERENCES

. F. Fleuret, J. Berclaz, R. Lengagne, and P. Fua. Multi-camera people tracking with aprobabilistic occupancy map. IEEE Transactions on Pattern Analysis and MachineIntelligence (TPAMI), 30(2):267–282, 2008

. J. Berclaz, E. Turetken, F. Fleuret, and P. Fua. Multiple object tracking using k-shortestpaths optimization. IEEE Transactions on Pattern Analysis and Machine Intelligence(TPAMI), 33(9):1806–1819, 2011

. J. Berclaz, F. Fleuret, and P. Fua. Multi-camera tracking and atypical motion detectionwith behavioral maps. In Proceedings of the European Conference on Computer Vision(ECCV), pages 112–125, 2008

. J. Berclaz, F. Fleuret, and P. Fua. Multiple object tracking using flow linear programming.In Proceedings of the 12th IEEE International Workshop on Performance Evaluation ofTracking and Surveillance (Winter-PETS), pages 1–8, 2009

. H. Ben Shitrit, J. Berclaz, F. Fleuret, and P. Fua. Tracking multiple objects under globalappearance constraints. In Proceedings of the IEEE International Conference onComputer Vision (ICCV), pages 137–144, 2011


OPEN-SOURCE IMPLEMENTATIONS

You can get the implementation of POM, under GPL3, at

git clone http://www.idiap.ch/~fleuret/git/pom/

and the implementation of MTP, under GPL3, at

git clone http://www.idiap.ch/~fleuret/git/mtp/

The reference implementations, additional information, and videoscan be found at

http://cvlab.epfl.ch/research/playground/


THANK YOU!

[email protected]

http://www.idiap.ch/~fleuret/


http://www.idiap.ch/~fleuret/git/pom/

http://www.idiap.ch/~fleuret/git/mtp/

http://cvlab.epfl.ch/research/playground/

given video streams from several cameras, multi-person tracking … · 2012. 10. 1. ·...

Documents