Improved Census Transforms for Resource-Optimized Stereo Vision

Wade S. Fife, Member, IEEE, James K. Archibald, Senior Member, IEEE

IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. 23, NO. 1, JANUARY 2013

2

Outline• Introduction• Related Work• Proposed Algorithm

• Sparse Census Transform• Generalized Census Transform• Hardware Implementation

• Experimental Results• Conclusion

3

Introduction

4

Introduction• The challenges:

• The enormous amount of computation required to identify the corresponding points in the images.

• It is critical to…

• maximize the accuracy and throughput of the stereo system • while minimizing the resource requirements

5

Objective• Propose the sparse census transforms :

• Reduce the resource requirements of census-based systems• Maintain correlation accuracy

• Propose the generalized census transforms :

• A new class of census-like transforms • Increase the robustness and flexibility

6

Related Work

7

Related Work• Census Transform :

• Color• Gradient

8

Related Work• After aggregation step:

Census on colors Census on gradients

9

Related Work• Sparse census[6] :

• Half of the bits

X

[6] C. Zinner, M. Humenberger, K. Ambrosch, and W. Kubinger, “An optimized software-based implementation of a census-based stereo matching algorithm,” in Proc. 4th ISVC, 2008, pp. 216–227.

The computation costs for the hamming distances are quite large.

10

Related Work• Mini-census[8] :

X

[8] N.-C. Chang, T.-H. Tsai, B.-H. Hsu, Y.-C. Chen, and T.-S. Chang,“Algorithm and architecture of disparity estimation with mini-census adaptive support weight,” IEEE Trans. Circuits Syst. Video Technol., vol. 20, no. 6, pp. 792–805, Jun. 2010.

11

Related Work• Mini-census[8] :

• Mini-census adaptive support weight

[8] N.-C. Chang, T.-H. Tsai, B.-H. Hsu, Y.-C. Chen, and T.-S. Chang,“Algorithm and architecture of disparity estimation with mini-census adaptive support weight,” IEEE Trans. Circuits Syst. Video Technol., vol. 20, no. 6, pp. 792–805, Jun. 2010.

12

Related Work• Mini-census[8] :

13

ProposedAlgorithm

14

Sparse Census Transform• Definition :

• N: the set of points within a T T window around p• : a new set of N•

P’

P

15

Transform Point Selection• Goal : minimize the size of the census transform vector

• Challenge: Must quantify how much each point in the transform window contributes to overall correlation accuracy

• Test correlation accuracy:

• Define a sparse census transform consisting of a single point (| | = 1)• Determine how consistently this point leads to correct correlation• 13 13 correlation window (aggregation)

16

Transform Point Selection• Go

Tsukuba Venus Average

Teddy Cones

Bright: Higher correlation accuracy

25 25 neighborhood

18

Transform Point Selection• Further from the center : value decreasing

• Very near the center : less effective

• It is best to choose points that are neither too far from nor too close to the center pixel.

• Optimal distance : 2 pixels• If the image is noisy should be slightly further

from the center

19

Transform Point Selection•

Tsukuba Venus Average

Teddy Cones

Bright: Higher correlation accuracy

37 37 neighborhood

Tsukuba Venus

Teddy Cones

With Gaussian noise( = 5.12)

20

Proposed Sparse Census Transform• Very good correlation accuracy can be achieved using very sparse transforms.

16-point 12-point 8-point

4-point 2-point 1-point

21

Experimental Results

22

Generalized Census Transform• Goal : greater freedom in choosing the census transform design

• Definition : redrawing the transform as a graph

3 3 census

3 3 correlation(aggregation)

23

Generalized Census Transform• As..

• (1)transform neighborhoods become more and more sparse• (2)fewer pixels are used in the correlation process

• selection of points to include in the transform becomes more critical

2-point 2-edge

Horizontal + Vertical

24

Generalized Census Transform

symmetric

25

Proposed Generalized Census Transform• Benefits :

• Often require a smaller census transform window (memory)• Increased robustness under varying conditions (noise)

16-edge 12-edge 8-edge

4-edge 2-edge 1-edge

26

Experimental Results

27

Experimental Results

28

Hardware Implementation• Pipelining : to increase throughput in an FPGA implementation

(Field Programmable Gate Array)

Range : 0~3

3 2 1 0

3 2 1 0

3 2 1 0

3 2 1 0

3 2 1 0

One input pixel per clock cycle &Output one disparity result per clock cycle

29

Hardware Implementation• Correlation window sum (Aggregation) :

30

ExperimentalResults

12-edge 4-edgeFull 7x7 censusGround TruthLeft Image

12-edge 4-edgeFull 7x7 censusGround TruthLeft Image

12-edge 4-edgeFull 7x7 censusLeft Image

34

Experimental Results

LUTs (look-up tables) : the amount of logic required to implement the methodFFs : the number of 1-bit registers (the amount of pipelining used)RAMs : the number of 18-kbit block memoriesFreq. : the maximum operating frequency reported by synthesis

𝟖𝟖% ↓ 𝟔𝟏% ↓

35

Conclusion

36

Conclusion• Proposed and analyzed in this paper:

• A range of sparse census transforms

• reduce hardware resource requirements• attempting to maximize correlation accuracy.• often better than or nearly as good as the full census

• Generalized census transforms

• increased robustness in the presence of image noise