1

Weighted Joint Bilateral Filter with Slope Depth Compensation Filter

for Depth Map Refinement

Takuya Matsuo, Norishige Fukushima and Yutaka Ishibashi

VISAPP 2013 International Conference on Computer Vision Theory and

Application

2

Outline

• Introduction• Related Works• Proposed Method–Weighted Joint Bilateral Filter – Slope Depth Compensation Filter

• Experimental Results• Conclusion

3

INTRODUCTION

4

Introduction

• Goal : Using two filters to get more accurate disparity map in real-time.

• Consideration– Noise reduction – Correct edges – Blurring control

Goal

5

RELATED WORKS

6

Left Image Right Image

Related Works

• Stereo Matching

Related Works

Local Global

Estimate accuracy Low High

Calculation cost Low High

Methods Pixel matching Block matching

(Optimization methods) Graph cuts

Belief propagation

Example

Related Works

• Flow Chart (Local)

1• Matching Cost Computation

2• Cost Aggregation

3• Disparity Map Computation/Optimization

4• Disparity Map Refinement• Disparity Map Refinement

9

Related Works

• Depth map refinement with filter– Median filter– Bilateral filter

Input depth map Output depth map Filter

10

Related Works

• Bilateral filter – Space weight:Near pixels has large weight – Color weight:Similar color pixels has large weight

• Smoothing – Keep edges –Weak in spike noise

•

11

Related Works

• Joint bilateral filter – Add in the reference image – Color weight is calculated by the reference – Keep object edges of the reference

Reference : Low noise Target : High noise Filtered image

12

Related Works

• Joint bilateral filter– Noise reduction O– Correct edge O– Blurring X • Mixed depth values • Spreading error regions

• Multilateral filter– Space + Color + Depth– Boundary recovering X

13

PROPOSED METHOD

14

Proposed Method

• Weighted joint bilateral filter – Noise reduction – Edge correction

• Slope depth compensation filter – Blurring control

15

Weighted Joint Bilateral Filter

•

– 𝐷: Depth value – 𝑝: Coordinate of current pixel – 𝑠: Coordinate of support pixel – 𝑁: Aggregation set of support pixel – 𝑤(), (): Space/color weight 𝑐– 𝜎𝑠,𝜎𝑐: Space/color Gaussian distribution – 𝑅𝑠: Weight map

16

Weighted Joint Bilateral Filter

• Add in the weight map – Controlling amount of influence on a pixel –Weight of the edge and error is small

Joint bilateral filter 𝜎- Mixed depth values 𝜎- Spreading error regions

17

Weighted Joint Bilateral Filter

• Making weight map – Space/color/disparity weight – Sum of nearness of space,

color, and disparity between center pixel and surrounding pixels.

•

18

Weighted Joint Bilateral Filter

• Mask image is made by Speckle Filter– Detecting speckle noise –Weight of speckle region is 0

Red region: speckle noise Weight = 0

19

Weighted Joint Bilateral Filter

20

Slope Depth Compensation Filter

• Weighted joint bilateral filter– Remaining small blurring – Difference between foreground and background

color is small

• Slope depth compensation filter – Reason of blurring is mixed depth value – Convert mixed value to non-blurred candidate

using initial depth map

Removing remaining blur

21

Slope Depth Compensation Filter

•

– X in Dx {INITIAL;WJBF;SDCF}∈

22

Slope Depth Compensation Filter

23

Proposed Method

Initial disparityStereo matching

Noise reduction/ edge correction Weighted Joint Bilateral F.

Blurring control Slope Depth Compensention F.

24

EXPERIMENTAL RESULTS

25

Experimental Results

• Evaluating accuracy improvement for various types of depth maps – Block Matching (BM) – Semi-Global Matching (SGM) – Efficient Large-Scale (ELAS) – Dynamic Programing (DP) – Double Belief Propagation (DBP)

26

Experimental Results

27

Experimental Results

28

29

30

31

Experimental Results

• Comparing proposed method with cost volume refinement(Teddy).

Yang, Q., Wang, L., and Ahuja, N. A constantspace belief propagation algorithm for stereo matching.In Computer Vision and Pattern Recognition(2010).

32 times slower

32

Experimental Results

33

Experimental Results

• Device : Intel Core i7-920 2.93GHz• Comparing running time (ms) of BM plus proposed

filter with selected stereo methods.

34

Experimental Results

35

Experimental Results

• Use the proposed filter for depth maps from Microsoft Kinect.

36

37

CONCLUSION

38

Conclusion

Contribution• The proposed methods can reduce depth noise and

correct object boundary edge without blurring. • Amount of improvement is large when an input

depth map is not accurate.

Future Works• Investigating dependencies of input natural images

and depth maps.