Background MoDepartment of Electronics Eng

E-mail: alex0

Abstract— This paper mainly focuses on crbackground model of a video sequence using together with the RGB pictures. The first keynear objects block the scenes at the back. Wdepth information, we can identify the closer Secondly, we develop a recursive algorithmbetween the depth map and color pictures. Coexisting schemes, our proposed method can quality background images and it improves tdepth map at the same time.

I. INTRODUCTION

A general foreground detection usinmodeling needs to overcome a few commonillumination changes including changes suddenly, (2) slowly moving background omovement of audience or the presence of din background, and (3) changing background

In the previous studies, several aconstructing the background model have bGenerally, people use the color and motioand the other characteristics of the backgrouThen, a model is developed for the backgThe most commonly used method is the Gamodel [1][2], the nonparametric model [3]background model [4], and others.

II. BACKGROUND MODELING USING DEPTH I

A. Main concept and problems

3DV has an advantage that it cinformation compared with the ordinary viHence, there might be a way for us background using simpler or faster algorithmdepth information. Typically, the stationary bthe largest depth value or the smallest dispaThus, we first collect the minimum disparitwhole scene over the entire sequence. We fiwise min filter applied to every pixel to fbackground disparity map,

, min , , ,

where is the minimum background diis the total number of frames of a video sequthe original disparity sequence. Fig. 1 shmap of Poznan_Street sequence.

At the first look, the overall disparity macceptable. But with a close look, we can findisparity values, which are farther than theyto be. We enlarge some of these areas in Fig.

odeling Using Depth InYu-Lun Liu and Hsueh-Ming Hang gineering, National Chiao-Tung University, Hsinchu, 04072000@hotmail.com, hmhang@mail.nctu.edu.tw

reating a global the depth maps y concept is the With the aid of moving objects.

m that iterates omparing to the produce better

the background

ng background n problems: (1)

gradually or objects such as dropping leaves d.

approaches of been proposed. on information, und as features. ground images. aussian mixture , the codebook

INFORMATION

contains depth deo sequences. to model the

ms based on the background has arity in a scene. ty values of the irst try the time form the initial

(1)

isparity map, N uence, and D is hows the

map seems to be nd some wrong y are supposed . 2.

Fig. 1 of the Po

Fig. 2 Enlarged

These artifacts occur estimation algorithm or active noises in the disparity mapdisparity pixels do not occusequence. Hence, how to ctexture image and depth map wpixels is our goal.

B. Proposed depth-basedalgorithm

We develop an iterative aelements: 1) extract the minimupdate the region of interest background region. The floalgorithm is shown in Fig. 3.

The proposed algorithm construct an initial backgroundinformation. (2) Create/updainitial/updated color backgroubackground depth map based ocolor background image basedare repeated iteratively to presults.

nformation Taiwan, R.O.C.

oznan_Street sequence.

d disparity errors. due to inaccurate depth

depth sensors, which contain ps. Typically, the incorrect ur very often in the entire create a robust background with reduced erroneous depth

d background modeling

algorithm which contains two mum disparity values, and 2)

(ROI), which indicates the ow chart of the proposed

mainly contains 4 steps: (1) d color image using the depth ate the ROI based on the und image. (3) Construct the on the ROI. (4) Construct the d on the ROI. The last 3 steps roduce the final and better

978-616-361-823-8 © 2014 APSIPA APSIPA 2014

Fig. 3 Flow chart of the proposed depth-based backalgorithm.

1. INITIALIZE COLOR BACKGROUND We make assumptions that: (1) the bac

should have the relatively larger depth vincorrect depth values do not appear oconstruct the initial color background image

, ∑ , , , , ,∑ , , ,where is the initial background color imin the denominator of the weight to prevenfrom infinite. The basic concept of this equthe color pixel having deep depth values tobackground image. If the wrong depth pismall, the result will be dominated by thsample of of the Poznan_Street sequenFig. 4.

Fig. 4 of the Poznan_Street sequen

The result above shows that weigaccording to reciprocal of difference betwedisparity and time-wise minimum disreasonably well. However, there appears a ga car on the left portion of the background due to its long stay in the same position. Fthis process, we like to create a good displeads to a good color background image.

2. ROI UPDATING In this step, we decide which pixels c

into the background region. The initial ROImatrix with all 1, and the dimension is image)*(width of the image)*(number ofsequence). That is,

kground modeling

ckground pixels values, and (2) ften. We first by:

, (2)

mage. We add 1 nt the dividend uation is to use o construct the ixel number is he majority. A

nce is shown in

nce.

ghting average een the current sparity works ghost artifact of

image. This is Furthermore, in parity map that

can be included I is set as a 3D (height of the

f frame in the

, , 1, Next, update ROI. The c

that are close to the true backgthe true background is approxvalue. Thus, for each frame, wvalue that is close to , and precisely, we use the sum of measure the similarity.

, , ∑ , , ,, ,Now, we need to find a th

ROI pixels if the difference threshold. We simply use the mcurrent ROI.

,∑ , ,∑ , ,We update the ROI indices usi

, , 0, , ,, , .

Based on our notion of Rthe pixels belonging to ROI. AROI, we can eventually reach shows the flow chart of this blo

Fig. 5 Flow char

3. GENERATE DEPTH

After updating ROI (whicpixels), the next stage is usingenerate the background depth

,where denotes the

stands for the first iteration

4. BACKGROUND COL

In this stage, we use thelast stage to produce the weighThe procedure of this stage isbut becomes the depth and only the pixels in the ROI

, , . (3)

concept is to select the pixels round image. At the moment, ximated by the current mean we compare every color pixel then update the ROI. More squared difference (SSD) to

, , , , . (4)

hreshold that is used to decide in (4) is greater than this

mean of the difference in the

, ,, , , , . (5)

ing the following process.

, , (6)

ROI, we only need to consider And by removing the pixels in a stable result. Fig. 5 below ock.

rt of updating ROI.

H MAP

ch are mostly the background g the time-wise min filter to

h map in ROI, min, , , , , , (7)

iteration index, for example, n.

OLOR IMAGE GENERATION e depth map produced in the hted average of color images. s the same as the first stage, produced in the last stage, are calculated.

,∑ , , , , , , ∑ , , , , , , .

The above process is repeated iteratdepth map becomes stable.

III. POST PROCESSING Some example of the resultant depth ma

the proposed iterative process is enlarged anin Fig. 6.

Fig. 6 Corrected depth maps and images and Post-procthe left is , the second is , the third is , a

As you can see, most of the wrong defixed. The road sign post in was brokit is clearly much improved using our propoHowever, there are still some wrong depthround road sign board. Assuming that the dmost of the frames are correct, we apply a tfilter (most frequent value) for every pixel in

, mode , , , , ,After applying the mode filter, the results areFig. 7.

The depth values on the signboard andthe car are closer to what we would expect. map of this example is shown in Fig. 7.

IV. EXPERIMENTAL RESULTS

We apply our proposed algorithm to test sequences and show the results below. Tan example is the selected images from a seqsecond row is their associated depth mapsrow is the background image and its depth m

,, (8)

tively until the

ap produced by nd shown below

cessed depth maps: and the right is

epth values are ken, but in , osed algorithm. h values on the depth values in time-wise mode n ROI. . (9)

e also shown in

d the antenna of The final depth

several MPEG The first row of quence, and the s, and the third

map.

Fig. 7 of the P

Book_A

Balle

Poznan_C

Poznan_

Poznan_Street sequence

Arrival

et

CarPark

_Street

We also compare the results of our propwith some other background modeling metthe help of the depth information and the proalgorithm, our background images are clearly

Proposed method

Static Frame

Weighted Moving Mean

Adaptive BLearn

Fuzzy Sugeno Integral

Fuzzy Choq

Fuzzy Gaussian

Simple G

Gaussian Mixture Model of Zivkovic

Gaussian MixtLaurence

VuMeter

Adaptiv

Fuzzy Adaptive SOM

posed algorithm thods [5]. With oposed iterative y superior.

e Difference

Background ning

quet Integral

Gaussian

ture Model of e Bender

ve SOM

V. CONCL

As the depth sensor becothe RGB video, we also have on the collected depth maps, wat every pixel or every framscheme that utilizes the deptbetter background color imageas well. Our proposed algoriproperties. (1) It iteratively remeliminate the influence of nonon-parametric model; that is,on the probabilistic models obackground images. (3) Combackground modeling methoproduces a better backgroundprocess, we also obtain a goomap, which is critical foapplications. But clearly, this quality of depth maps.

VI. ACKNOW

This work was supportedunder Grant NSC 102-2221-Ethe Top University Project University, Taiwan.

REFERE

[1] C. Stauffer and E. Grimson, Models for Real-time TrInternational Conference on Recognition, pp. 246-252, 199

[2] D. S. Lee, J. J. Hull, and B. EGaussian Mixture BackgrounInternational Conference on 2003.

[3] A. Elgammal, D. Harwood, Model for Background SubFRAME-RATE Workshop, 199

[4] K. Kim, T. H. Chalidabhongs“Real-Time Foreground-BacCodebook Model”, Real-Time

[5] A. Sobral, “BGSLibrary: subtraction library,” IX Work(WVC 2013). Rio de Janeiro, at <http://code.google.com/p/b

LUSIONS omes popular, in addition to the depth information. Based which may not be all correct

me, we propose an iterative th information to produce a e and its associated depth map ithm has the following nice moves wrong depth values to oisy depth map. (2) It is a we have no pre-assumptions

or parametric models on the mpared with the conventional

ods, our method typically d image. In addition, in this od quality background depth or virtual view synthesis approach relies on the good

WLEDGEMENT d in part by the NSC, Taiwan -009-123 and by the Aim for

of National Chiao Tung

ENCES “Adaptive Background Mixture

racking”, in Proceedings of Computer Vision and Pattern

99.

Erol, “A Bayesian Framework for nd Modeling”, in Proceedings of

Image Processing, pp.973-976,

and L. Davis, “Non-parametric btraction”, Proc. of ICCV '99 99.

e, D. Harwood, and L. S. Davis, ckground Segmentation using e Imaging, pp. 172-185, 2005.

an openCV c++ background kshop de Visão Computacional Brazil, 2013, Software available

bgslibrary/>.