Michel AMAT PROBAYES SAS

3rd year ENSIMAG 345, rue Lavoisier - Inovallée

July 6th to Sept 15th 2009 38330 MONTBONNOT

WORK EXPERIENCE REPORT

TITLE: Design & Integration of computer vision algorithms in a video surveillance application.

ABSTRACT:

Open source softwares such as Zone Minder (http://www.zoneminder.com/) constitute complete video

surveillance systems but lack of any artificial intelligence component. On the contrary, Probayes SAS owns

many artificial intelligence libraries employed for computer vision, as well as for Bayesian analysis.

The topic of my training period was to plug existing softwares developed by Probayes in a complete video

surveillance solution: ZoneMinder.

First, I had to install and test ZoneMinder, the video surveillance software Probayes would like to employ.

For this purpose, a specific environment made of several USB & IP cameras was provided.

Second, existing libraries from Probayes allowing to do image and video analysis have been integrated in

this platform.

1

Contents

1 INTRODUCTION 3

1.1 About Probayes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.2 Work Experience Development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2 Background Subtraction Algorithms 4

2.1 Problematics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2.2 Background Modelling with One Gaussian . . . . . . . . . . . . . . . . . . . . . . . . 4

2.3 Background Modelling with a Mixture of Gaussian . . . . . . . . . . . . . . . . . . . 5

2.4 Post processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.5 Software architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

2.6 Utilization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

3 ZONEMINDER 15

3.1 Installation of ZoneMinder . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

3.2 Integration of the background subtraction algorithms in ZoneMinder . . . . . . . 17

4 CONCLUSION 18

Bibliography 19

2

1 INTRODUCTION

1.1 About Probayes

Probayes headquarters is located in France, 10km away from Grenoble. Grenoble is often referred as the

French capital of Alps and winter sports. The company is part of the 300 companies and laboratories of

the technopolis Inovallée. Probayes is a software editor specialized in Bayesian modelling, inference, and

learning. The experience of Probayes allows to propose:

• an unsurpassed know-how in implementing Bayesian solutions;

• an exclusive methodology to design such applications;

• a robust, fast, and easy-to-use computation engine;

• a set of patented software components ready to be integrated in any application.

The core technology ProBT c© extends the Bayesian Networks framework by providing a structured

programming language allowing the developers to increase their applications capabilities and robustness by

easily integrating Bayesian models.

Built upon a strong and rigorous theory, Bayesian techniques allow modelling complex real-world prob-

lems in a unified framework.

Bayesian inference and learning provide a powerful mathematical tool for:

• merging inaccurate/incomplete data with expert priors in a unified way;

• on-line learning to fit data evolution.

1.2 Work Experience Development

I worked within the R&D team, involved in important R&D projects like ProBT (http://www.probayes.com/products/

modelling-development-toolkits/probt/) or Prometheus (https://www.informationsystems.foi.se/∼prometheus-

fp7) whose responsible is my supervisor, Julien ROS, R&D engineer. However, my work was independent

of all the other projects and actually was pretty free in the directions to follow. It allows me to choose

to develop new competences interesting for my career. Julien ROS was very available, this gave me the

opportunity to learn very fast. Yet, he wasn’t at Probayes on the first week, and left on holidays for 2 weeks

in August. So, I also had to be autonomous in order to solve problems on my own.

3

2 Background Subtraction Algorithms

2.1 Problematics

Identifying moving objects in a video sequence is a fundamental and critical task in many computer vision

applications. Background subtraction techniques are commonly used to separate foreground moving objects

from the background [2].

Although different, most Background Subtraction techniques share a common framework: they make the

hypothesis that the observed video sequence I is made of a fixed background B in front of which moving

objects are observed. It is also based on the assumption that a moving object at time t has a color (or a

color distribution) different from the one observed in B.

In the algorithms presented in this report, the background B is modeled by a Gaussian (or a mixture of

K Gaussian) for each pixel. For each new frame It, the algorithm builds a mask of the new frame showing

which pixels are part of the foreground. Some post processing steps based on this mask permit to obtain a

better vision of the objects in the foreground (i-e: morphological steps, shadows removing, blob detection).

2.2 Background Modelling with One Gaussian

Principle Each pixel of the background is modeled separately by its average µ and its standard deviation

σ. For each new frame, those parameters of the background are updated as follow:

µt+1 = α ∗ Ft + (1 − α)µt (2.1)

σ2t+1 = α(Ft − µt)2 + (1 − α)σ2t (2.2)

The mask M is then obtained by comparison between the current background B and the current frame

It:

Mt = {|It − Bt| > Th ∗ σt} (2.3)

This algorithm has been implemented twice: once using RGB images, another time using grey images.

The results are equivalent except the program is quicker for grey images.

This algorithm has been applied on different video sequences and a result is shown on figure 1.

4

figure 1: Modelling of the background with one Gaussian

2.3 Background Modelling with a Mixture of Gaussian

In a lot of cases, the Background is not stable (camera vibrations, fluids, moving trees...). Consequently, the

distributions of the background pixels are not unimodal. In order to prevent the algorithm from considering

those pixels in the foreground, a mixture of Gaussian is used to model the background [4, 1].

Principle Each pixel of the background is modeled separately by a mixture of K Gaussian. The back-

ground is updated, before the foreground is detected, as follows:

1. If It matches component i, i.e., It is within λ standard deviations of µi,t, then the ith component is

updated as follows:

ωi,t = ωi,t−1 + α (2.4)

5

µi,t = µi,t−1 ∗ (1 −α

ωi,t) +

α

ωi,t∗ It (2.5)

σ2i,t = σ2i,t−1 ∗ (1 −

α

ωi,t) +

α

ωi,t∗ (It − µi,t)

2 (2.6)

2. If It does not match any component, then the least likely component is replaced with a new one initialized

with µi,t = It , σ0 large, and ωi,t low. After the update, the weights ωi,t are re-normalised. The foreground

is detected as follows. All components in the mixture are sorted into the decreasing order of ωi,t and the

components of higher importance are assumed to be the background.

If we consider

B = argminb(Σbi=1ωi,t > (1 − cf)) (2.7)

for some threshold cf. Then components 1. . .B are assumed to be background and if pixel does not match

one of these components, the pixel is marked as foreground.

This algorithm has been applied on different video sequences and a result is shown on figure 2.

figure 2: Modelling of the background with a mixture of Gaussian

6

2.4 Post processing

Mathematical morphology operations In order to improve the detection of the shapes of the object,

or to reduce noise, we use mathematical morphology operations:

• reducing noise is obtained by an opening (which consists in eroding and then dilating the mask),

• improving the shapes of the objects is obtained by a closing (which consists in dilating and then eroding

the mask).

Some results are shown on figure 3 and we can see that:

• the opening has reduced most of the noise of the mask.

• the closing has improved the shapes of the mask.

figure 3: Result of an opening

7

figure 4: Result of a closing

Shadows removing Foreground is commonly perturbed by shadow which need to be detected and

removed [3] . The shadows detection is based on the c1c2c3 model:

c1 = arctan(R

max(G,B)) (2.8)

c2 = arctan(G

max(R,B)) (2.9)

c3 = arctan(B

max(R,G)) (2.10)

This model is invariant to illumination moving and so allow to detect pixels of the foreground which are

not part of an object but of its shadow.

It is easy to see on figure 5 that this algorithm works quite well.

8

figure 5: Result of shadows removing

Blobs detection Blob detection permits to separate different objects of the foreground. It can be useful

to track, count or analyse behaviors. The blob detection algorithm we employed use connected component

analysis and we can see on figure 6 that it permits to effectively isolate objects thanks to the foreground

segmentation.

9

figure 6: Result of blob detection

2.5 Software architecture

The software developed has been written in C++ under Eclipse and the OpenCV and cvBlob libraries. It

is important to note that tools like valgrind and kcachegrind have been used to optimize it. Each algorithm

correspond to a sub-class of the CGeneric class allowing the library to be more generic.

10

figure 7: UML modelling of the software

11

2.6 Utilization

Both algorithms have been also implemented with grey images. The results are similar but the computing

times are smaller.

The main problem encountered with such algorithms concerns parameters tuning. It is strongly dependent

on the video to be processed. Launching the program with ”-h” displays the different parameters to tune as

shown on figure 8.

figure 8: Console

12

For saving computing, one can:

• use the grey one Gaussian algorithm

• reduce the video resolution using the -R option

• do not use the post processing options

• increase the threshold

Computing times for the station1.avi videoParameters: Time:-A 0 -R 4 -m 2 -S 1 -B 1 -T 1.8 44.387s-A 1 -R 4 -m 2 -S 1 -B 1 -T 1.8 56.088s-A 1 -R 4 -m 2 -S 1 -B 1 -T 4.0 30.854s-A 2 -m 0 -S 0 -R 4 -I 500 -T 1.0 -B 1 57.776s-A 2 -m 0 -S 0 -R 4 -I 500 -T 4.0 -B 1 26.222s-A 2 -m 0 -S 0 -B 0 -R 4 -I 500 -T 1.0 19.029s-A 2 -m 0 -S 0 -B 0 -R 4 -I 500 -T 4.0 19.025s-A 2 -m 0 -S 0 -B 0 -R 4 -I 500 -T 5.0 18.765s-A 2 -m 0 -S 0 -B 0 -R 4 -I 500 -T 5.0 -I 1000 18.001s-A 3 -R 4 -m 2 -S 1 -B 1 -cf 0.01 -K 3 -I 500 1m12.673s-A 3 -R 4 -m 0 -S 0 -B 0 -cf 0.01 -K 3 -I 500 1m01.728s-A 3 -R 4 -m 0 -S 0 -B 0 -cf 0.01 -K 5 -I 500 1m20.625s-A 4 -R 4 -m 0 -S 0 -B 0 -cf 0.01 -K 5 -I 500 54.699s-A 4 -R 4 -m 0 -S 0 -B 0 -cf 0.01 -K 3 -I 500 42.871s-A 4 -R 4 -m 0 -S 0 -B 0 -cf 0.05 -K 3 -I 500 44.531s-A 4 -R 4 -m 3 -S 1 -B 1 -cf 0.01 -K 3 -I 500 0m52.027s

13

figure 9: Result of a too low threshold

In the case of the figure 9, we can note that a lot of pixels of the background have been detected like

foreground. To improve the result, one can:

• increase the threshold Th (or cf in Mixture of Gaussian Algorithms) to detect less pixels in the fore-

ground;

• add more Gaussian to reduce the noise due to illumination moving;

• apply an opening to reduce the little isolated spots which shouldn’t be part of the foreground.

14

3 ZONEMINDER

ZoneMinder is one of the most employed software for video surveillance under Linux. Intended for use in

single or multi-camera video security applications, it supports capture, analysis, recording, and monitoring

of different cameras attached to a Linux system.

My work on ZoneMinder consisted of two steps:

• Installing and learning how to use ZoneMinder;

• inspect the code to integrate my background subtraction algorithms within of it;

• try to add parameters tuning.

3.1 Installation of ZoneMinder

The installation of ZoneMinder is different according the OS you are using. I followed the Ubuntu Installation

recommendations found on the wiki of the official ZoneMinder website.

First, I had to install an Apache server, a mySQL server, and a lot of others that ZoneMinder needs.

Then, I downloaded the SVN version of ZoneMinder (current development version of the software) before

compiling and installing the software.

ZoneMinder runs under a web interface allowing to see videos from the cameras and of the events recorded

as shown on figures 10 & 11.

15

figure 10: ZoneMinder main interface

figure 11: Events visualization

16

3.2 Integration of the background subtraction algorithms in ZoneMinder

For Probayes, being able to plug easily algorithms in such a software with a great interface could be very

interesting for demonstration purpose.

Though ZoneMinder code is pretty important, it has been rather quick to figure out how the software

was organized. So, I could quickly find where and how to plug my algorithms. Thus, I managed to apply

Background Subtraction to the frames captured by the cameras used by ZoneMinder, and the result was

convincing.

Among other things, I have been able to add different functions to the software like:

• using my Background Subtraction algorithms like detection mode;

• setting the score of alarmed frames to the number of blobs detected by my algorithms;

• allowing to resize the camera captures to accelerate the analysis;

• allowing the grey camera captures;

• allowing the use of inactive zone, and the writing of alarmed image.

figure 12: Background subtraction under ZoneMinder

17

4 CONCLUSION

This work experience was great. I acquired experience in several fields and for now on, I’ll have a

significant luggage to begin my professional life. Though very different than the student life, I immediately

fit in the working life, which is captivating, and, I have to acknowledge, absorbing.

More precisely, I have been able to

• improve my knowledge in C++ (polymorphism, architecture, templates);

• discover the integrated development environment of eclipse (how to install it, and use it with C++

plug-in);

• use the OpenCV Library (C++ library for image processing);

• increase my knowledge of Ubuntu (bash, installing new packages);

• using C++ libraries (installing them and use the right compilation options);

• employ valgrind and kcachegrind to optimize algorithms;

• design softwares with bouml;

• use cvs (version-management system);

• use cmake (to automate the creation of Makefiles);

• discover how works a company (and working with other people).

I wish to thank my instructor, Julien ROS, for his availability, his interest in my study and all his help.

And at last, I also thank all the Probayes team for its welcome and sympathy.

18

References

[1] Alan M. McIvor. Background subtraction techniques. pages 128–221.

[2] M. Piccardi. Background subtraction techniques: a review. In Proc. of IEEE SMC 2004 International

Conference on Systems, Man and Cybernetics, volume 4, pages 3099–3104, The Hague, The Netherlands,

Oct 2004.

[3] E. Salvador, A. Cavallaro, and T. Ebrahimi. Shadow identification and classification using invariant color

models. In ICASSP ’01: Proceedings of the Acoustics, Speech, and Signal Processing, 2001. on IEEE

International Conference, pages 1545–1548, Washington, DC, USA, 2001. IEEE Computer Society.

[4] Zoran Zivkovic. Improved adaptive gaussian mixture model for background subtraction. Pattern Recog-

nition, International Conference on, 2:28–31, 2004.

19