j. vis. commun. image r.drivsco/papers/ralli_yjvci09.pdfuncorrected proof 1 2 short communication 3...

8
UNCORRECTED PROOF 1 2 Short Communication 3 A method for sparse disparity densification using voting mask propagation 4 J. Ralli * , J. Díaz, E. Ros 5 Departamento de Arquitectura y Tecnología de Computadores, Escuela Técnica Superior de Ingenierías y de Telecomunicacíon, Universidad de Granada, 6 Calle Periodista Daniel, Saucedo Aranda s/n, E-18071 Granada, Spain 7 9 article info 10 Article history: 11 Received 31 October 2008 12 Accepted 20 August 2009 13 Available online xxxx 14 Keywords: 15 Disparity Q1 densification 16 Voting propagation 17 18 abstract 19 We describe a novel method for propagating disparity values using directional masks and a voting 20 scheme. The driving force of the propagation direction is image gradient, making the process aniso- 21 tropic, whilst ambiguities between propagated values are resolved using a voting scheme. This kind 22 of anisotropic densification process achieves significant density enhancement at a very low error cost: 23 in some cases erroneous disparities are voted out, resulting not only in a denser but also a more accu- 24 rate final disparity map. Due to the simplicity of the method it is suitable for embedded implemen- 25 tation and can also be included as part of a system-on-chip (SOC). Therefore, it can be of great 26 interest to the sector of the machine vision community that deals with embedded and/or real-time 27 applications. 28 Ó 2009 Elsevier Inc. All rights reserved. 29 30 31 1. Introduction 32 Disparity is originally defined as being the horizontal difference 33 of a 3D point being projected on two adjacent imaging devices (e.g. 34 stereo-rig) and if both intrinsic- and extrinsic parameters of the 35 imaging system are known, complete 3D-reconstruction of the 36 scene is possible. However, even if we do not know all the neces- 37 sary parameters to do a complete 3D-reconstruction, disparity still 38 conveys relative information of the 3D structures of the scene 39 which can also be useful. 40 Disparity extraction models are based on local or global opti- 41 mization methods that minimize (or maximize) matching cost of 42 image features between two or more images. Practically all the 43 sparse models have some kind of threshold or other parameter, 44 either implicit or explicit, which affects density and at the same 45 time error in the derived disparity map [1,2]. On the other hand, 46 the global methods that minimize energy functions within the 47 whole scene, through local operations, usually derive a disparity 48 map that is typically 100% dense (sometimes detecting occlu- 49 sions as well). Such global minimization can be done using dif- 50 ferent approaches such as variational methods [3–7] or graph 51 cuts [8,9]. Typically, depending upon the sparse method used, 52 as density increases, after a certain limit error also starts to in- 53 crease concomitantly. Therefore, it is worth calculating a less 54 dense, high-confidence disparity map and afterwards increasing 55 the density by propagating the correct disparity values. In this 56 way we achieve better accuracy vs. density trade-off than by di- 57 rectly reducing the reliability threshold and thus increasing den- 58 sity at the expense of higher error. Many global, dense, disparity 59 calculation methods have built-in mechanisms for propagating/ 60 diffusing disparity [4,10] but sparse methods usually lack this 61 capacity. To the best of our knowledge there are very few inde- 62 pendent propagation methods, apart from interpolation [11] and 63 diffusion [12], that can be applied as a post-processing step. By 64 independent we mean that the propagation method does not de- 65 pend upon the algorithm used to derive the initial disparity map. 66 This work proposes a new densification method that is able to 67 arrive at denser and more accurate results than the standard 68 one-stage disparity algorithms such as dynamic programming, 69 block-matching and so on that only slightly affects the error 70 rate. Since the scheme is based on very simple operations, it 71 can be considered suitable for efficient implementation. Our 72 method resembles image driven anisotropic diffusion, used for 73 instance by Alvarez et al. [4] in variational disparity calculation, 74 in the sense that the propagation direction is based on the image 75 gradient. Instead of using a set of equations for defining the dif- 76 fusion model as it is done in [4] our approach (VMP) uses a bank 77 of predefined masks and a voting process to define the local 78 interactions driving the diffusion process. 79 2. Method 80 The first step is to calculate a sparse, high-confidence, stereo 81 map. Many feature-based disparity calculation methods match 82 edges present in stereo-images, since these can be considered rel- 1047-3203/$ - see front matter Ó 2009 Elsevier Inc. All rights reserved. doi:10.1016/j.jvcir.2009.08.005 * Corresponding author. Fax: +34 958 248993. E-mail addresses: jarno@ralli.fi, [email protected] (J. Ralli), [email protected] (J. Díaz), [email protected] (E. Ros). J. Vis. Commun. Image R. xxx (2009) xxx–xxx Contents lists available at ScienceDirect J. Vis. Commun. Image R. journal homepage: www.elsevier.com/locate/jvci YJVCI 843 No. of Pages 8, Model 5G 19 September 2009 ARTICLE IN PRESS Please cite this article in press as: J. Ralli et al., A method for sparse disparity densification using voting mask propagation, J. Vis. Commun. (2009), doi:10.1016/j.jvcir.2009.08.005

Upload: others

Post on 06-Nov-2020

3 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: J. Vis. Commun. Image R.drivsco/Papers/Ralli_YJVCI09.pdfUNCORRECTED PROOF 1 2 Short Communication 3 A method for sparse disparity densification using voting mask propagation 4 J

1

2

3

4

56

7

9

10111213

1415 Q11617

1 8

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

J. Vis. Commun. Image R. xxx (2009) xxx–xxx

YJVCI 843 No. of Pages 8, Model 5G

19 September 2009ARTICLE IN PRESS

Contents lists available at ScienceDirect

J. Vis. Commun. Image R.

journal homepage: www.elsevier .com/ locate/ jvc i

OF

Short Communication

A method for sparse disparity densification using voting mask propagation

J. Ralli *, J. Díaz, E. RosDepartamento de Arquitectura y Tecnología de Computadores, Escuela Técnica Superior de Ingenierías y de Telecomunicacíon, Universidad de Granada,Calle Periodista Daniel, Saucedo Aranda s/n, E-18071 Granada, Spain

a r t i c l e i n f o

1920212223242526

Article history:Received 31 October 2008Accepted 20 August 2009Available online xxxx

Keywords:Disparity densificationVoting propagation

2728

1047-3203/$ - see front matter � 2009 Elsevier Inc. Adoi:10.1016/j.jvcir.2009.08.005

* Corresponding author. Fax: +34 958 248993.E-mail addresses: [email protected], [email protected]

Díaz), [email protected] (E. Ros).

Please cite this article in press as: J. Ralli et adoi:10.1016/j.jvcir.2009.08.005

DPR

Oa b s t r a c t

We describe a novel method for propagating disparity values using directional masks and a votingscheme. The driving force of the propagation direction is image gradient, making the process aniso-tropic, whilst ambiguities between propagated values are resolved using a voting scheme. This kindof anisotropic densification process achieves significant density enhancement at a very low error cost:in some cases erroneous disparities are voted out, resulting not only in a denser but also a more accu-rate final disparity map. Due to the simplicity of the method it is suitable for embedded implemen-tation and can also be included as part of a system-on-chip (SOC). Therefore, it can be of greatinterest to the sector of the machine vision community that deals with embedded and/or real-timeapplications.

� 2009 Elsevier Inc. All rights reserved.

29

E T 56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

UN

CO

RR

EC

1. Introduction

Disparity is originally defined as being the horizontal differenceof a 3D point being projected on two adjacent imaging devices (e.g.stereo-rig) and if both intrinsic- and extrinsic parameters of theimaging system are known, complete 3D-reconstruction of thescene is possible. However, even if we do not know all the neces-sary parameters to do a complete 3D-reconstruction, disparity stillconveys relative information of the 3D structures of the scenewhich can also be useful.

Disparity extraction models are based on local or global opti-mization methods that minimize (or maximize) matching cost ofimage features between two or more images. Practically all thesparse models have some kind of threshold or other parameter,either implicit or explicit, which affects density and at the sametime error in the derived disparity map [1,2]. On the other hand,the global methods that minimize energy functions within thewhole scene, through local operations, usually derive a disparitymap that is typically 100% dense (sometimes detecting occlu-sions as well). Such global minimization can be done using dif-ferent approaches such as variational methods [3–7] or graphcuts [8,9]. Typically, depending upon the sparse method used,as density increases, after a certain limit error also starts to in-crease concomitantly. Therefore, it is worth calculating a lessdense, high-confidence disparity map and afterwards increasingthe density by propagating the correct disparity values. In this

ll rights reserved.

(J. Ralli), [email protected] (J.

l., A method for sparse dispari

way we achieve better accuracy vs. density trade-off than by di-rectly reducing the reliability threshold and thus increasing den-sity at the expense of higher error. Many global, dense, disparitycalculation methods have built-in mechanisms for propagating/diffusing disparity [4,10] but sparse methods usually lack thiscapacity. To the best of our knowledge there are very few inde-pendent propagation methods, apart from interpolation [11] anddiffusion [12], that can be applied as a post-processing step. Byindependent we mean that the propagation method does not de-pend upon the algorithm used to derive the initial disparity map.This work proposes a new densification method that is able toarrive at denser and more accurate results than the standardone-stage disparity algorithms such as dynamic programming,block-matching and so on that only slightly affects the errorrate. Since the scheme is based on very simple operations, itcan be considered suitable for efficient implementation. Ourmethod resembles image driven anisotropic diffusion, used forinstance by Alvarez et al. [4] in variational disparity calculation,in the sense that the propagation direction is based on the imagegradient. Instead of using a set of equations for defining the dif-fusion model as it is done in [4] our approach (VMP) uses a bankof predefined masks and a voting process to define the localinteractions driving the diffusion process.

80

81

82

2. Method

The first step is to calculate a sparse, high-confidence, stereomap. Many feature-based disparity calculation methods matchedges present in stereo-images, since these can be considered rel-

ty densification using voting mask propagation, J. Vis. Commun. (2009),

Original text:
Inserted Text
Therefore
Original text:
Inserted Text
hand
Original text:
Inserted Text
vs
jarno
Inserted Text
1. Disparity densification (existing) 2. Voting propagation (existing) 3. Voting based disparity disambiguation and densification (new)
Page 2: J. Vis. Commun. Image R.drivsco/Papers/Ralli_YJVCI09.pdfUNCORRECTED PROOF 1 2 Short Communication 3 A method for sparse disparity densification using voting mask propagation 4 J

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

123123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

2 J. Ralli et al. / J. Vis. Commun. Image R. xxx (2009) xxx–xxx

YJVCI 843 No. of Pages 8, Model 5G

19 September 2009ARTICLE IN PRESS

atively robust features [13,14]. For the reasons set out in Section 1we use here a simplified dynamic programming technique basedon image edges [11]. The rational for using dynamic programmingis that it has been shown to be both computationally efficient [15]and capable of producing highly accurate results [2]. Nevertheless,as mentioned earlier, our densification approach does not dependupon the method used for producing the initial sparse disparitymap. The second step is to apply voting mask propagation (VMP)for propagating disparity in the direction where estimations areexpected to be similar and to use voting for resolving ambiguities.Local support of the voting process is based on directional masks:for each image position for which disparity is known a mask from apre-determined bank is chosen, depending upon the underlyingimage structure (gradient). The properties of the chosen filter de-fine how many votes each of the neighborhood positions willreceive.

T

O

140

141

142

143

144

145

146

147

149149

150

151

152

153

154

2.1. Propagation direction

Since without further image analysis we cannot be sure ofwhich object an image pixel for which the disparity is known be-longs to, and since we assume that inside objects disparity changesgradually, image gradient is used as a driving force of the propaga-tion direction. We assume that two different objects will almostcertainly have two different disparity levels. By propagating in adirection perpendicular to the image gradient we reduce errorssince different objects have varying disparities divided by an edge.This assumption of local maximum gradient separating differentobjects is also the basis of anisotropic diffusion, where diffusiondirection is driven by the gradient [12,16]. In this work we concen-trate on the case where the disparities for the edges are known andthe disparities are propagated in an edge-wise direction. There is,however, no reason why the disparities not residing at the edgescould not be propagated as well. The tangent-to-edge direction isapproximated by calculating image gradient.

C 155

156

158158

159

160

161

162

163

164

RR

E2.2. Bank of masks

A bank of masks is designed using a 2D multivariate Gaussiandistribution which is rotated in order to generate masks corre-sponding to different propagation directions. The basic mask, cor-responding to orientation 0� (i.e. the horizontal axis), is calculatedas per the following equation:

z ¼ G i; j;li ¼ lj ¼ 0;X� �

; i ¼ �N . . . N; j ¼ �P . . . P ð1Þ

UN

CO

Fig. 1. A bank of 7 � 7 voting masks corresponding to different orientat

Please cite this article in press as: J. Ralli et al., A method for sparse disparidoi:10.1016/j.jvcir.2009.08.005

F

where GðÞ denotes multivariate Gaussian, ði; jÞ are the coordinates ofthe mask, ðli ¼ lj ¼ 0Þ are the mean,

Pis the covariance, z is the

number of votes that each position receives and N and P definethe mask size. Fig. 1 shows a voting mask corresponding to severaldifferent orientations (rotations).

The use of Gaussian distribution is motivated by the fact that itreflects the probabilistic nature of our approach: the underlyingimage structure drives the propagation direction and thus reflectsour belief on how the disparity is distributed. Other authors haveused similar approaches for image denoising [17]. Furthermore,Gaussian multivariate distribution allows a smooth transition fromisotropic to anisotropic cases, depending on the certainty of theimage structure, which can be used in more elaborated schemesby further analyzing the image. Besides a Gaussian distributioncan be implemented as a separable convolution thus making it effi-cient computationally.

ED

PR

O2.3. Choosing the mask

Once the orientations of the edge tangents have been approxi-mated, propagation is carried out for each disparity value usingthe mask whose orientation best matches the tangent of the edge.The most closely corresponding masks centre is placed on top ofthe disparity value of interest and each pixel within mask size re-ceives as many votes for the disparity value as defined by the mask.This is shown in the following equation:

Vxþi;yþjðDx;yÞ ¼ Gx;y;Dðxþ i; yþ jÞ; i ¼ �N . . . N;

j ¼ �P . . . P ð2Þ

where Vx;y indicates votes received by position ðx; yÞ for disparity,D;Gx;y;Dðxþ i; yþ jÞ denotes how the Gaussian voting mask, chosenas per gradient D, with a size of ð2N þ 1;2P þ 1Þ, placed at ðx; yÞvotes for each mask position. As a final step, after the disparitieshave been propagated for each of the original disparity values, eachpixel position assumes the disparity that receives most votes, as de-fined in the following equation:

Dx;y ¼maxVððVx;y;Dx;yÞÞ ð3Þ

where Dx;y indicates the final chosen disparity value for a positionðx; yÞ and maxV ððV ;DÞÞ returns the disparity value that has re-ceived the most votes for a set of vote-disparity tuples ðV ;DÞ.Due to the spatial support of the voting mechanism erroneous val-ues are in certain cases effectively voted out: if within a certainneighborhood there are more correct values than erroneous val-

ions. Intensity codifies the number of votes each position receives.

ty densification using voting mask propagation, J. Vis. Commun. (2009),

Original text:
Inserted Text
the Introduction
Original text:
Inserted Text
(1).
Original text:
Inserted Text
Furthermore
Original text:
Inserted Text
futher
Original text:
Inserted Text
(2).
Original text:
Inserted Text
,
Original text:
Inserted Text
(3).
Original text:
Inserted Text
and
Page 3: J. Vis. Commun. Image R.drivsco/Papers/Ralli_YJVCI09.pdfUNCORRECTED PROOF 1 2 Short Communication 3 A method for sparse disparity densification using voting mask propagation 4 J

UN

CO

RR

EC

TED

PR

OO

F

165

166

167

168

169

170

171

Fig. 2. Results for the test images: the left-hand column contains left-hand images of the original stereo-pairs, the middle column shows disparity maps calculated bydynamic programming and the right-hand column disparity maps densified using VMP. C denotes the percentage of correct disparities (±1 disparity level) and D, density.

J. Ralli et al. / J. Vis. Commun. Image R. xxx (2009) xxx–xxx 3

YJVCI 843 No. of Pages 8, Model 5G

19 September 2009ARTICLE IN PRESS

ues, the erroneous values receive fewer votes and are discarded.Interestingly enough this effect allows the densification processto arrive at a denser but at the same time more accurate disparitymap than the original.

Please cite this article in press as: J. Ralli et al., A method for sparse disparidoi:10.1016/j.jvcir.2009.08.005

2.4. Pseudocode

For the sake of clarity, below we have included a pseudocode ofthe propagation process.

ty densification using voting mask propagation, J. Vis. Commun. (2009),

Page 4: J. Vis. Commun. Image R.drivsco/Papers/Ralli_YJVCI09.pdfUNCORRECTED PROOF 1 2 Short Communication 3 A method for sparse disparity densification using voting mask propagation 4 J

173174175176177178179180181182183184185186187188189190191192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

Fithusinto

4 J. Ralli et al. / J. Vis. Commun. Image R. xxx (2009) xxx–xxx

YJVCI 843 No. of Pages 8, Model 5G

19 September 2009ARTICLE IN PRESS

Pd

//I input image driving the voting process//D disparity map//M bank of masks (mask size (N,P))//V a matrix for storing the votes (initialize to 0)

//- -VOTE- -//Obtain coordinates for disparities(x y) = coords(notEmpry(D))for i = 1:numel(x)

//Choose the closest mask corresponding to gradient normalDI = calculateImageGradient(I(x(i),y(i)))mask = chooseMask(M, DI)//Vote using the chosen maskV = vote(x(i), y(i), mask, D(x(i),y(i)), N, P)

end//- -CHOOSE WINNERS- -(x y) = coords(notEmpty(V))for i = 1:numel(x)

Out(x(i),y(i)) = MAX(V(x(i),y(i)))end

213

214

215

216

217

218

The actual voting process can be seen as superimposition of thechosen mask on top of the disparity value to be propagated whereeach neighboring pixel (defined by the mask) receives as many

UN

CO

RR

EC

T219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

g. 3. Densification results for several initial densities obtained using differentresholds for dynamic programming. DP refers to results calculated by directlying dynamic programming and VMP refers to results densified from correspond-g DP results using voting mask propagation. (A) TS refers to Tsukuba and V refersVenus images and (B) C refers to Cones and TE refers to Teddy images.

lease cite this article in press as: J. Ralli et al., A method for sparse disparioi:10.1016/j.jvcir.2009.08.005

ED

PR

OO

F

votes for the disparity as defined by the weight of the mask as eachposition. For each pixel position the number of votes for each dis-parity needs to be stored so that a winner can be chosenaccordingly.

3. Experiments and results

We benchmarked the method using well known stereo-imagesavailable at http://www.vision.middlebury.edu/stereo/. In order tostudy how the VMP densification process behaves when dealingwith different initial densities and/or errors, we have used two dif-ferent methods for generating the initial disparity maps providedto the VMP model. The different methods used were dynamic pro-gramming (DP) [11] and a phase-based method [1,18]. Further-more, we have used different thresholds and interleave factorsfor DP in order to generate initial disparity maps with differentdensities and errors. Computational complexity of the our methodwas approximated by comparing it with execution times of the DPmethod. We also introduce a sample application that clearly bene-fits from a more dense disparity map as input. In the experiments,size of the propagation masks was 7 � 7 pixels. Density is given interms of a ratio between the number of pixels for which disparityhas been defined and the total number of pixels in the image. Over-all accuracy is measured as the percentage of correct pixels (±1 dis-parity level) calculated against the ground-truth values.

3.1. Results

Fig. 2 shows the original stereo-pair images and results calcu-lated directly using DP and densified by VMP.

Fig. 3 demonstrates the results for four different initial mapswith different densities. The initial maps are calculated using dif-ferent thresholds for occlusion detection with the effect of increas-ing density at the expense of accuracy. Thus it can be observed thatafter certain reasonable limit, in order to obtain even more densemap, the error starts to increase. In such a case it is better to calcu-late more reliable initial map and then densify.

Fig. 4 shows the results for the Venus case only. It can be clearlyseen that as the cost for occlusions gets higher (threshold from oneto four) density of the resulting disparity map increases slightly atthe expense of accuracy. On the other hand, as the error increases

Fig. 4. Densification results for several initial densities obtained using differentthresholds in dynamic programming. DP refers to dynamic programming and VMPto voting mask propagation. Density refers to the density of the obtained disparitymap and correct refers to the percentage of correct disparities (±1 disparity level).

ty densification using voting mask propagation, J. Vis. Commun. (2009),

Original text:
Inserted Text
i=1:numel(x)
Original text:
Inserted Text
I=calculateImageGradient( I(x(i),y(i) )
Original text:
Inserted Text
chooseMask( M, ΔI )
Original text:
Inserted Text
V=vote( x(i),
Original text:
Inserted Text
P )
Original text:
Inserted Text
coords( notEmpty(V) )
Original text:
Inserted Text
i=1:numel(x)
Original text:
Inserted Text
Out( x(i),y(i) )
Original text:
Inserted Text
MAX( V( x(i),y(i)) )
Original text:
Inserted Text
Furthermore
Original text:
Inserted Text
7x7
Original text:
Inserted Text
(
Original text:
Inserted Text
hand
Page 5: J. Vis. Commun. Image R.drivsco/Papers/Ralli_YJVCI09.pdfUNCORRECTED PROOF 1 2 Short Communication 3 A method for sparse disparity densification using voting mask propagation 4 J

UN

CO

RR

EC

TED

PR

OO

F

Fig. 5. Phase refers to phase-based method and VMP to voting mask propagation. C denotes the percentage of correct disparities (±1 disparity level) and D, density.

Fig. 6. Densification results for several initial densities obtained using differentinterleaves for dynamic programming. DP refers to results calculated by directlyusing dynamic programming and VMP refers to results densified from correspond-ing DP results using voting mask propagation. (a) TS refers to Tsukuba and V refersto Venus images and (b) C refers to Cones and TE refers to Teddy images.

Fig. 7. Densification factor results for different initial disparities. Y-axis shows thedensity of the initial map while X-axis displays the obtained densification factor.

Fig. 8. Computation times for both the dynamic programming and the voting maskpropagation methods. DP refers to dynamic programming and VMP to voting maskpropagation.

J. Ralli et al. / J. Vis. Commun. Image R. xxx (2009) xxx–xxx 5

YJVCI 843 No. of Pages 8, Model 5G

19 September 2009ARTICLE IN PRESS

Please cite this article in press as: J. Ralli et al., A method for sparse disparity densification using voting mask propagation, J. Vis. Commun. (2009),doi:10.1016/j.jvcir.2009.08.005

jarno
Sticky Note
Densification factor results for different initial disparities. X-axis shows the density of the initial map while Y-axis displays the obtained densification factor
Page 6: J. Vis. Commun. Image R.drivsco/Papers/Ralli_YJVCI09.pdfUNCORRECTED PROOF 1 2 Short Communication 3 A method for sparse disparity densification using voting mask propagation 4 J

UN

CO

RR

EC

TED

PR

OO

F

Fig. 9. Down-scaling results using a median filter: the left-hand column contains results of down-scaling Tsukuba disparity calculated using dynamic programming, whilstthat on the right shows the results of applying VMP on the first scale and then down-scaling. Sc denotes the scale, while C and D denote percentage of correct disparities (±1disparity level) and density.

6 J. Ralli et al. / J. Vis. Commun. Image R. xxx (2009) xxx–xxx

YJVCI 843 No. of Pages 8, Model 5G

19 September 2009ARTICLE IN PRESS

Please cite this article in press as: J. Ralli et al., A method for sparse disparity densification using voting mask propagation, J. Vis. Commun. (2009),doi:10.1016/j.jvcir.2009.08.005

Page 7: J. Vis. Commun. Image R.drivsco/Papers/Ralli_YJVCI09.pdfUNCORRECTED PROOF 1 2 Short Communication 3 A method for sparse disparity densification using voting mask propagation 4 J

T

RO

OF

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305306307

Fig. 10. Dense-disparity calculation results: (A) without fusion; (B) fusion using sparse-disparity with 15% density (C) fusion using sparse-disparity with 60% density and (D)the difference between (B) and (C). C denotes the percentage of correct disparities and D, density.

J. Ralli et al. / J. Vis. Commun. Image R. xxx (2009) xxx–xxx 7

YJVCI 843 No. of Pages 8, Model 5G

19 September 2009ARTICLE IN PRESS

UN

CO

RR

EC

the voting mechanism starts to vote out some of the erroneous val-ues thus increasing the accuracy.

Fig. 5 displays densification results using an initial disparitymap calculated by a phase-based method [18]. Density and errorof the resulting densified map is similar to the rest of the resultseven though the starting density with this technique is signifi-cantly higher than in the previous experiments based on dynamicprogramming.

Figs. 6 and 7 demonstrate the results of different initial dispar-ities upon the densified disparity and the obtained densificationfactor. Different initial density disparities were obtained using DPwith different vertical line interleaves (interleave 1 = disparity cal-culated for all the vertical lines; interleave 2 = disparity calculatedfor every second line, etc.). This experiment demonstrates bothrobustness of the VMP method in relation to initial density andwhat kind of densification factors can be expected. As the densityof the input map increases the densification factor decreases whichis due to overlapping of the propagation filters.

Fig. 8 displays the computational times of both the dynamicprogramming and the voting mask propagation methods, imple-mented in Matlab. The four different cases correspond to the fourdifferent thresholds already seen in Fig. 3. This kind of a compari-son is approximate since it depends on implementation issues but,however, it does give a valuable hint of the relative computationalcomplexities.

In applications where less resolution is needed, further densifi-cation can be achieved by down-scaling. Fig. 9 shows the results ofdown-scaling disparity maps by a factor of two, using a 2 � 2 med-ian filter and discarding those pixels that do not have a disparityestimation. In order to calculate the percentage of correct dispari-ties, each of the downscaled disparity maps was upscaled to thesame size as the ground-truth.

3.2. Sample application

We present here a specific application which benefits from adenser sparse-disparity map: fusion of sparse- and dense-disparitystereo [19] for disambiguation of dense disparity estimations. Inthe framework of DRIVSCO [19] Ralli et al. have demonstratedthe benefits of using symbolic, reliable sparse-disparity to disam-biguate unclear cases in dense-disparity calculation. We tested

Please cite this article in press as: J. Ralli et al., A method for sparse disparidoi:10.1016/j.jvcir.2009.08.005

ED

Pthe fusion scheme, with different sparse-disparity densities, upona hardware simulation of a phase-based [1,18] disparity calcula-tion method with and without fusion. The hardware simulationapproximates a realtime FPGA implementation (currently beingimplemented at the University of Granada) of the phase-basedmethod. Due to limited on-chip computational resources theimplementation requires a trade-off between accuracy and effi-ciency. In such a scheme where external approximations are avail-able, these can be used to guide the dense method and thus theaccuracy can be restored as shown in Fig. 10. As can be seen inFig. 10, the fusion process benefits clearly from a denser sparse-disparity map used to guide the dense disparity calculationmethod.

4. Conclusion

We have shown that our novel method of propagating sparsedisparity information based on directional masks and a votingscheme is capable of significantly increasing density with a veryminor increase in overall error, thus considerably enhancing theinitial sparse disparity map. Further densification can be achievedby down-scaling, by active interpolation [20–22] or by diffusion[12]. Even though in this study we have used VMP for propagatingbinocular visual information based on monocular cues, VMP canalso be used for propagating other visual cues, such as optical flowand others. The simplicity of the method facilitates its efficientimplementation.

Acknowledgments

This work was supported by the EU research Project DRIVSCO(IST-016276-2) and the Spanish Grants DINAM-VISION (DPI2007-61683), RECVIS (TIN2008-06893-C03-02), P06-TIC-02007 andTIC-3873. The authors thank A.L. Tate for revising their Englishtext.

References

[1] F. Solari, S. Sabatini, G. Bisio, Fast technique for phase-based disparityestimation with no explicit calculation of phase, Electronics Letters 37 (23)(2001) 1382–1383.

ty densification using voting mask propagation, J. Vis. Commun. (2009),

Original text:
Inserted Text
but however
Original text:
Inserted Text
2x2
Original text:
Inserted Text
Acknowledgements
Original text:
Inserted Text
project
Original text:
Inserted Text
(DPI2007-61683),
Original text:
Inserted Text
(TIN2008-06893-C03-02),
Page 8: J. Vis. Commun. Image R.drivsco/Papers/Ralli_YJVCI09.pdfUNCORRECTED PROOF 1 2 Short Communication 3 A method for sparse disparity densification using voting mask propagation 4 J

308309310311312313314315316317318319320321322323324325326327328329330331332333334335

336337338339340341342343344345346347348349350351352353354355356357358359360361362363

8 J. Ralli et al. / J. Vis. Commun. Image R. xxx (2009) xxx–xxx

YJVCI 843 No. of Pages 8, Model 5G

19 September 2009ARTICLE IN PRESS

[2] M. Brown, D. Burschka, G. Hager, Advances in computational stereo, PatternAnalysis and Machine Intelligence 25 (8) (2003) 993–1008.

[3] B. Horn, B. Schunck, Determining optical flow, Artificial Intelligence 17 (1981)185–203.

[4] L. Alvarez, R. Deriche, J. Sánchez, J. Weickert, Dense disparity map estimationrespecting image discontinuities: a PDE and scale-space based approach,Journal of Visual Communication and Image Representation 13 (1) (2002) 3–21.

[5] J.W.L. Alvarez, J. Sánchez, Reliable estimation of dense optical flow fields withlarge displacements, International Journal of Computer Vision 39 (1) (2004)41–56.

[6] T. Brox, From pixels to regions: partial differential equations in image analysis,Ph.D. Thesis, Saarland University, 2005.

[7] O. Faugeras, R. Keriven, Level set methods and the stereo problem, in:Proceedings of the First International Conference on Scale-Space Theory inComputer Vision, 1997, pp. 272–283.

[8] V. Kolmogorov, Graph based algorithms for scene reconstruction from two ormore views, Ph.D. Thesis, Cornell University, 2003.

[9] V. Kolmogorov, R. Zabin, What energy functions can be minimized via graphcuts?, Pattern Analysis and Machine Intelligence 26 (2004) 147–159

[10] D. Scharstein, R. Szeliski, Stereo matching with non-linear diffusion,International Journal of Computer Vision 28 (2) (1998) 155–174.

[11] J. Ralli, F. Pelayo, J. Díaz, Increasing efficiency in disparity calculation, in:BVAI2007, vol. 4729, 2007, pp. 298–307.

[12] J. Weickert, Anisotropic diffusion in image processing, Ph.D. Thesis, UniversitätKaiserslautern, 1998.

[13] X. Jiangjian, S. Mubrak, Two-frame wide baseline matching, in: Proceedings ofthe ICCV’03, 2003, 2003, pp. 603–609.

UN

CO

RR

EC

T364

Please cite this article in press as: J. Ralli et al., A method for sparse disparidoi:10.1016/j.jvcir.2009.08.005

RO

OF

[14] N. Krüger, M. Felsberg, An explicit and compact coding of geometric andstructural information applied to stereo matching, Pattern Recognition Letters25 (8) (2004) 849–863.

[15] G. Minglun, Y. Yee-hong, Real-time stereo matching using orthogonalreliability-based dynamic programming, IEEE Transactions on ImageProcessing 16 (3) (2007) 879–884.

[16] P. Perona, J. Malik, Scale-space and edge detection using anisotropic diffusion,IEEE Transactions on Pattern Analysis and Machine Intelligence 12 (1990)629–639.

[17] D. Cho, T.D. Bui, Multivariate statistical modeling for image denoising usingwavelet transforms, Signal Processing: Image Communication 20 (1) (2005)77–89.

[18] S. Sabatini, G. Gastaldi, F. Solari, J. Diaz, E. Ros, K. Pauwels, M.V. Hulle, N.Pugeault, N. Krüger, Compact and accurate early vision processing in theharmonic space, in: Proceedings of VISAPP, vol. 1, 2007, pp. 213–220.

[19] J. Ralli, Disparity disambiguation by fusion of signal- and symbolic-levelinformation, Tech. Rep., University of Granada, DRIVSCO EU-Project (FP6-IST-FET, Contract 016276-2), Learning to Emulate Perception-Action Cycles in aDriving School Scenario, 2008. Project web-page available at http://www.pspc.dibe.unige.it/drivsco/.

[20] H. Yoo, Closed-form least-squares technique for adaptive linear imageinterpolation, Electronics Letters 43 (4) (2007) 210–212.

[21] L. Zhang, W. Xiaolin, An edge-guided image interpolation algorithm viadirectional filtering and data fusion, IEEE Transactions on Image Processing 15(2006) 2226–2238.

[22] H.C. Ting, H.M. Hang, Edge preserving interpolation of digital image usingfuzzy inference, Journal of Visual Communication and Image Representation 8(4) (1997) 338–355.

ED

P

ty densification using voting mask propagation, J. Vis. Commun. (2009),

jarno
Sticky Note
[5] The first author should be: L. Alvarez. There seems to have been a mistake in my bibtex file.