Abstract— This paper proposes a noise-resilient road surface

and free space estimation method using dense stereo. The

proposed road surface estimation method selects 3D points

expected to compose a road surface using YZ-plane

accumulation, and then finds the road surface by sequentially

estimating a piece-wise linear function based on a RANSAC

framework. This makes our method insensitive to 3D points on

obstacles and stereo matching errors on textureless road regions.

The proposed free space estimation method is based on the fact

that disparities from roads and obstacles should be equal at the

free space boundary. This method calculates disparity

consistency between road and obstacle surfaces, and finds free

space that gives the best disparity consistency and depth

smoothness using dynamic programming. This approach

achieves robustness against stereo matching errors on obstacle

surfaces and objects located in the air since its estimation

process is independent of disparity accumulation unlike

previous occupancy grid-based method. The experimental

results show that the proposed method is able to estimate road

surfaces and free spaces in various severe situations.

I. INTRODUCTION

Stereo cameras have been widely used for driving environment analysis since it can provide both image and depth information. A system for this task typically starts with the extraction of the road surface and free space boundary. Road surface estimation is used for pitch angle compensation [1] and improving the accuracy of obstacle detection [2], and free space estimation is utilized for vehicle navigation [3] and region of interest selection for pedestrian [4] and vehicle detection [5].

Road surfaces have been estimated in the v-disparity domain by using linear [6] or piece-wise linear functions [1]. Since v-disparity has a drawback in that the resolution decreases with increasing distance [2], many methods that estimate road surface in the YZ-plane domain using multivariate polynomial [7] and B-spline functions [2] have been proposed. Even though the models that have high degrees of freedom are able to describe complex road surfaces, their performances could be affected by 3D points produced by obstacles and stereo matching errors on textureless road regions. To alleviate this drawback, this paper proposes a noise-resilient road surface estimation method. This method

* This research was supported by Basic Science Research Program

through the National Research Foundation of Korea (NRF) funded by the Ministry of Education, Science and Technology (2012-0002467).

J. K. Suhr is with the Research Institute of Automotive Electronics and

Control at the Department of Automotive Engineering, Hanyang University, Seoul, Korea.

H. G. Jung is with the Department of Automotive Engineering, Hanyang

University, Seoul, Korea (email: hogijung@hanyang.ac.kr).

first samples the 3D points expected to compose a road surface to reduce the effect of 3D points produced from obstacle surfaces. Then, a Random Sample Consensus (RANSAC)-based line estimator is applied to the sampled points for robust estimation of the piece-wise linear function. To achieve robustness against situations where stereo matching partially fails on textureless road surfaces, the interval of the piece-wise linear function is adaptively changed according to the number of inliers found by RANSAC. Finally, the location of the last inlier is determined as a limit of the reliable road surface.

Free space has been computed using inverse perspective mapping [8] and disparity space image [9] without generating a dense disparity map to reduce computational cost. As efficient stereo matching algorithms and their hardware implementations have achieved real-time processing [10], dense disparity map-based methods have been suggested [7], [10]. Among them, a polar occupancy grid-based method [10] has been widely used in various applications [5], [11]. This method works properly with well-estimated disparity maps, but it could produce unstable results when stereo matching fails on obstacle surfaces and objects (e.g. tree branches) are located higher up. This is because this approach depends on disparity accumulation. To overcome this problem, this paper proposes a disparity consistency-based free space estimation method utilizing the fact that disparities from roads and obstacles should be equal at the free space boundary. This method measures disparity consistency between road and obstacle surfaces, and determines an optimal path that gives the best disparity consistency and depth smoothness as a free space using dynamic programming. This approach is robust against stereo matching errors because it does not depend on the number of correctly matched disparity pixels. Since this measure is calculated on obstacle areas that are attached to the road surface, its performance is not affected by objects that are higher up.

In experiments, the proposed method was tested using a publicly available stereo database [12] produced under urban situations. The results show that this method reliably estimates both road surface and free space even in various complex road and obstacle environments, and in particular, achieves better free space estimation results compared with the previous occupancy grid-based method in [10].

II. DENSE STEREO MATCHING

This paper utilizes a dense disparity map for road surfaces and free space estimation. Although a dense disparity map requires a high computational cost, its real-time embedded implementation is available thanks to recent research on hard-wired stereo matching. For instance, Daimler

Noise-resilient Road Surface and Free Space Estimation

Using Dense Stereo

Jae Kyu Suhr, Member, IEEE, and Ho Gi Jung, Senior Member, IEEE

2013 IEEE Intelligent Vehicles Symposium (IV)June 23-26, 2013, Gold Coast, Australia

978-1-4673-2754-1/13/$31.00 ©2013 IEEE 461

implemented semiglobal matching (SGM) using a field-programmable gate array (FPGA) and achieved a frame rate of 25Hz [10]. For this paper, dense disparity maps are calculated using an SGM algorithm as implemented in [13]. Figs. 1(a) and (b) show a left image acquired by a stereo camera and a dense disparity map calculated by SGM, respectively. It can be seen that stereo matching fails in a large portion of road surfaces due to strong reflections even though SGM is one of the most reliable stereo matching algorithms.

III. ROAD SURFACE ESTIMATION

A. Point Sampling

One of the major factors that hinder road surface estimation is 3D points produced from obstacle surfaces. This may not be a major problem in highway situations where wide road regions and few obstacles are present. However, it becomes a serious factor when dealing with urban environments that include narrow roads and various obstacles. To prevent 3D points on obstacles from disturbing road surface estimation, we utilize a sampling method that selects the 3D points that the road surface is expected to be composed of.

The proposed sampling method first calculates the 3D positions of all pixels in a dense disparity map using parameters estimated during the stereo camera calibration procedure. Then, all 3D points are accumulated on the YZ-plane. Fig. 2(a) shows an accumulation result of 3D points produced by the dense disparity map in Fig. 1(b). This is depicted in a log-scale for visualization purposes. In this figure, the horizontal axis indicates Z-coordinates from 0m (left) to 100m (right), and the vertical axis indicates Y-coordinates from -10m (bottom) to +10m (top). The camera is located at

(0m, 0m). Bin size is set to 10cm×10cm. In Fig. 2(a), it can be

seen that there are many 3D points produced from obstacles such as parked cars and buildings.

Once 3D points are accumulated onto the YZ-plane, those points expected to be a part of the road surface are selected. For this sampling task, we select a location that gives an accumulated value larger than a predetermined threshold (Tsample) and the least Y-coordinate for each column (Z-coordinate). Since the accumulated value is inversely proportional to the Z-coordinate, the threshold (Tsample) is adaptively selected as (1).

sample

fT Z W

Z (1)

where W and f indicate a minimum width of the valid road surface and the focal length in terms of pixel dimensions, respectively. W is set to 0.5m, which means that this method samples a location accumulated by properly matched 0.5m road surfaces. The minimum value of (Tsample) is set to 10 to discard erroneous locations far away from the camera. Fig. 2(b) shows a sampling result of Fig. 2(a). In this figure, yellow dots indicate the locations sampled by the proposed approach. It can easily be noticed that most sampled locations are from road surfaces. Although this sampling method is similar to the method in [14] that selects a location of maximum accumulated value for each column, the proposed method produced better performance in our experiments.

B. Piece-wise Linear Function Estimation

Since real world roads usually include non-flat surfaces, it is more appropriate to model them using multivariate polynomial and B-spline functions rather than a simple straight line. However, the road surface estimation performances of these functions could be affected by stereo matching errors due to their high degrees of freedom. For instance, a B-spline function with high degrees of freedom could give an unexpected fitting result in a range where there is no or are few sampled points in the middle of Fig. 2(b). Thus, this paper models a road surface with a piece-wise linear function that consists of multiple lines since each line is defined by only one or two parameters and robust estimators can easily be applied.

A robust estimator (i.e. RANSAC [15]) is sequentially applied to estimate multiple lines that compose a piece-wise linear function since the sampling result includes points from both road surfaces and obstacles as shown in Fig. 2(b). When estimating a line in each interval, the line estimation interval and slope constraint are adaptively adjusted according to the number of inliers counted by RANSAC. In other words, if the number of inliers is less than a predetermined value (MIN_INLIER), the estimated line is discarded and a line is

(a)

(b)

(c)

Figure 2. (a) Accumulated 3D points on YZ-plane in a log-scale. (b) Sampled

points (yellow dots). (c) Estimated road surface (red line) and inliers (yellow dots). Horizontal axis indicates Z-coordinate from 0m (left) to 100m (right),

and vertical axis indicates Y-coordinate from -10m (bottom) to +10m (top).

(a) (b)

Figure 1. (a) Left image acquired by a stereo camera. (b) Dense disparity map

calculated by SGM. Stereo matching fails in a large portion of road surfaces

due to strong reflections.

462

re-estimated by increasing the line estimation interval (INTERVAL) and slope constraint (ANGLE). This procedure is used to achieve robustness against situations where stereo matching fails on textureless or reflective road regions as shown in Fig. 1(a). Finally, the location of the last inlier is determined as a limit of reliable road surface. In this paper, INTERVAL, ANGLE, and MIN_INLIER are set to 5m, 1°, and 50% of the bin number within INTERVAL, respectively. This procedure is summarized in Algorithm I. Fig. 2(c) shows the road surface estimation result. In this figure, a red line and yellow dots indicate the estimated road surface and sampled points determined as inliers, respectively. It can be seen that a road surface is properly estimated despite points from obstacles and ranges with no sampled points. The estimated road surface ends at the location of the farthest inlier.

IV. FREE SPACE ESTIMATION

A. Cost Matrix Calculation

The proposed free space estimation method utilizes a disparity consistency (DC) to measure how a certain location is likely to be a free space boundary. DC is calculated by measuring the absolute difference between the disparity of the road and that of the obstacle. This is based on the fact that the disparity of the road surface (d) and potential obstacle disparity (POD(u,d)) at the free space boundary should be equal by assuming that obstacles stand perpendicularly on the road surface. In other words, DC should be zero at the free space boundary in an identical case. This can be described as (2).

, ,

, , :

DC u d d POD u d

where POD u d median D u v d h d v d

(2)

where u and d are the horizontal coordinates of the image and

disparity value, respectively, and POD(u,d) indicates a potential obstacle disparity at (u,d). A function, POD(u,d), is calculated by a median operator from the disparity map (D) within an obstacle region whose horizontal location is determined at u and the vertical range is determined from v(d)-h(d) to v(d). v(d) indicates a road surface function which links between disparity (d) and the vertical coordinates of the image (v). This function is estimated during the road surface estimation procedure. h(d) is a height of the obstacle in terms of pixel dimensions that depends on the disparity value (d) and can be calculated as (3).

obstacle

dh d H

B (3)

where Hobstacle indicates a real world obstacle height, and is set to 1.0m. This means that the proposed method is able to detect obstacles approximately taller than 0.5m since the median operator is used to extract potential obstacle disparity. This setting is enough to detect pedestrians, vehicles, and other obstacles that can appear in front of the ego-vehicle. Vertical red lines in Fig. 3(a) show several obstacle regions at various image locations. It can easily be noticed that the heights of obstacle regions are adaptively selected according to distances from the camera. If there is an insufficient number of valid disparity pixels in an obstacle region (e.g. less than 30% of the obstacle region), a default value (=2) is assigned. Fig. 3(b) shows a cost matrix calculated by (2) at every (u,d) location. Horizontal and vertical axes of the cost matrix are a horizontal coordinate of image (u) and disparity (d), respectively.

Compared with the previous occupancy grid-based method in [10], the proposed disparity consistency-based method has the following advantages. The occupancy grid-based method estimates free space using accumulated values. Thus, it could produce unstable results when an insufficient number of disparities are accumulated due to stereo matching errors caused by reflective vehicle surfaces and vegetation. However, the proposed method is more robust against the same situation since it relies on a potential obstacle

(a)

(b)

Figure 3. (a) Obstacle regions at various image locations. (b) Calculated cost

matrix. Horizontal and vertical axes of cost matrix are horizontal coordinate

of image (u) and disparity (d), respectively.

ALGORITHM I. PIECE-WISE LINEAR FUNCTION ESTIMATION

Z = 0;

INTERVAL = DEFAULT_INTERVAL;

ANGLE = DEFAULT_ANGLE;

while Z+INTERVAL <= MAX_DISTANCE

Select points between Z and Z+INTERVAL;

Estimate a line using RANSAC with ANGLE;

If number of inliers >= MIN_INLIER

Save the estimated line parameters;

Z = Z+ INTERVAL;

INTERVAL = DEFAULT_INTERVAL;

ANGLE = DEFAULT_ANGLE;

else

INTERVAL = INTERVAL + DEFAULT_INTERVAL;

ANGLE = ANGLE + DEFAULT_ANGLE;

end

end

Select the farthest inlier as the road end;

463

disparity value rather than an accumulated value. The median operator used to extract the potential obstacle disparity value makes it even more robust against outliers produced by stereo matching errors. An average operator has been tested for the same purpose, but it gives a less reliable result than the median operator. Furthermore, the proposed method is not affected by obstacles higher up because potential obstacle disparity values are calculated only in the obstacle regions attached to the road surface. Example results on these situations will be presented in the experimental section.

B. Free space computation using dynamic programming

To find a free space that minimizes the cost matrix with spatial smoothness, dynamic programming [16] is utilized. The cost of dynamic programming consists of three terms: data term L(u,di), smoothness term S(di,dj), and background term B(di) as in (4).

, , , ,i j i i j iC u d d L u d S d d B d (4)

In (4), data term (L(u,di)) is defined by a cost matrix calculated by DC as in (5). If DC(u,di) is larger than TL, L(u,di) is assigned by TL which is empirically set to 10.

, , ,

,,

i i L

i

L

DC u d if DC u d TL u d

T otherwise

(5)

The smoothness term (S(di,dj)) is established in terms of the metric depth distance between di and dj (dist(d,dj)). If dist(d,dj) is larger than Ts, S(di,dj) is set to ws•Ts. ws and Ts are set to 2 and 5m exactly the same as in [10] for better performance comparison.

, , ,

,,

i j i j S

i j S

S

dist d d if dist d d TS d d w

T otherwise

(6)

Background term (B(di)) is used to select the nearest location at each column (u) when multiple locations have similar costs. This is because free space is a boundary between the road surface and the first obstacle. In the previous occupancy grid-based method, this task is achieved by using the background subtraction procedure which marks all occupied cells behind the first maximum cell as free. Since the threshold used to find the first maximum cell is sensitive to stereo matching error, this paper conducts this procedure as a part of dynamic programming as in (7).

max

max

1i

i B

d dB d w

d

(7)

where dmax indicate the maximum disparity value. wB is empirically set to 0.1. The red lines in Figs. 4(a) and (b) show an optimal path calculated by dynamic programming and the final free space boundary depicted on left image, respectively. It can easily be found that free space is correctly estimated even under serious stereo matching errors.

V. EXPERIMENTAL RESULTS

The proposed road surface and free space estimation method was tested using a publicly available stereo database

[12]. This database was acquired by a stereo camera whose baseline length and height are 0.25m and 1.17m, respectively,

and consists of 640×480 pixels rectified grayscale stereo

images mostly acquired in urban environments.

Figs. 5(a), (b), and (c) show road surface and free space estimation results of the proposed method in various environments. In these figures, the first, second, and third columns show the disparity map, estimated road surface (lower) with sampled points (upper), and estimated free space (red lines) on the left image, respectively. Fig. 5(a) shows an uphill situation. In this case, road surface estimators that have no mechanism to determine the end of the road tend to wrongly recognize the area beyond the top of the hill as the road surface because obstacles are located not at the top of the hill but far beyond it. However, the proposed method correctly estimated the location of the end of the road by using the farthest inlier point as shown in the second column of Fig. 5(a). This makes the free space end at the location of the top of the hill as shown in the third column of Fig. 5(a). Fig. 5(b) shows a situation where the road slope severely changes and stereo matching fails in half of the road surface, and Fig. 5(c) shows a situation where only a small portion of the road surface is visible due to vehicles and buildings located near the camera. It can be seen that the proposed method successfully estimates the road surface and free space even in these unfavorable situations.

Figs. 6(a), (b), and (c) show a qualitative performance comparison between the occupancy grid-based method [10] and the proposed method. The method in [10] was implemented by us for this comparison. In these figures, the first, second, and third columns show the disparity map, free space estimated by the occupancy grid-based method, and free space estimated by the proposed method, respectively. In both cases, the road surfaces are estimated by the proposed method. Fig. 6(a) shows a case where the disparity map is fairly well-estimated. In this situation, both methods produce similar and reliable free space estimation results. Figs. 6(b) and (c)

(a)

(b)

Figure 4. (a) Optimal path calculated by dynamic programming. (b)

Estimated free space.

464

show situations where severe stereo matching errors are presented on vehicle surfaces and vegetation, respectively. In these cases, the proposed method outperforms the occupancy grid-based method. Unlike the proposed method, the occupancy grid-based method miss-estimates free space in the middle of the road in Fig. 6(c). This is because it wrongly recognizes tree branches located higher up as obstacles.

VI. CONCLUSION

This paper proposes a road surface and free space

estimation method that is robust against various noise factors

presented in dense stereo. Point sampling and

RANSAC-based sequential calculation of a piece-wise linear

function enable a robust estimation of road surface against 3D

points on obstacles and stereo matching errors on textureless

road regions. The disparity consistency measure based on the

median operator makes free space estimation insensitive to

stereo matching errors on reflective vehicle surfaces and

vegetation and objects located higher up. In the future, we are

planning to extend the proposed method by adopting

temporal information, and to conduct quantitative

performance evaluation.

REFERENCES

[1] R. Labayrade, D. Aubert, and J.-P. Tarel, “Real Time Obstacle

Detection in Stereovision on Non Flat Road Geometry Through v-disparity Representation,” In Proceedings of the Intelligent Vehicle

Symposium, 2002, pp. 646-651.

[2] A. Wedel, H. Badino, C. Rabe, H. Loose, U. Franke, and D. Cremers, “B-Spline Modeling of Road Surfaces With an Application to

Free-Space Estimation,” IEEE Transactions on Intelligent

Transportation Systems, vol. 10, no. 4, pp. 572-583, Dec. 2009. [3] M. Muffert, T. Milbich, D. Pfeiffer, and U. Franke, “May I Enter the

Roundabout? A Time-To-Contact Computation Based on

Stereo-Vision,” In Proceedings of the Intelligent Vehicle Symposium, 2012, pp. 565-570.

[4] J. K. Suhr, K. M. Kang, and H. G. Jung, “Dense stereo-based critical

area detection for active pedestrian protection system,” Electronics Letters, vol. 48, no. 19, pp. 1199-1201, Sep. 2012.

(a)

(b)

(c)

Figure 5. Results of road surface and free space estimation. (a) Uphill situation. (b) Road slop changes and stereo matching fails. (c) Small portion of road

surface is visible. The first, second, and third columns show disparity map, estimated road surface (lower) with sampled points (upper), and estimated free

space (red lines) on the left image, respectively.

465

[5] M. Enzweiler, M. Hummel, D. Pfeiffer, and U. Franke, “Efficient

Stixel-Based Object Recognition,” In Proceedings of the Intelligent

Vehicle Symposium, 2012, pp. 1066-1071.

[6] R. Benenson, R. Timofte, and L. V. Gool, “Stixels estimation without

depth map computation,” In Proceedings of the International

conference on Computer Vision Workshops, 2011, pp. 2010-2017. [7] S. Nedevschi, R. Danescu, D. Frentiu, T. Marita, F. Oniga, C. Pocol, T.

Graf, and R. Schmidt, “High accuracy stereovision approach for

obstacle detection on non-planar roads,” In Proceedings of the Intelligent Engineering Systmes, 2004, pp. 211–216.

[8] P. Cerri and P. Grisleri, “Free Space Detection on Highways using

Time Correlation between Stabilized Sub-pixel precision IPM Images,” In Proceedings of the International Conference on Robotics and

Automation, 2005, pp. 2223-2228. [9] S. Kubota, T. Nakano, and Y. Okamoto, “A Global Optimization

Algorithm for Real-Time On-Board Stereo Obstacle Detection

Systems,” In Proceedings of the Intelligent Vehicle Symposium, 2007, pp. 7-12.

[10] D. Pfeiffer and U. Franke, “Modeling Dynamic 3D Environments by

Means of The Stixel World,” IEEE Interlligent Transportation Systems Magazine, vol. 3, no. 3, pp. 24-36, Fall 2011.

[11] F. Erbs, A. Barth, and U. Franke, “Moving Vehicle Detection by

Optimal Segmentation of the Dynamic Stixel World,” In Proceedings

of the Intelligent Vehicle Symposium, 2011, pp. 951-956.

[12] C. Keller, M. Enzweiler, and D. M. Gavrila, “A New Benchmark for

Stereo-based Pedestrian Detection,” In Proceedings of the Intelligent

Vehicle Symposium, 2011, pp. 691-696. [13] OpenCV 2.1, http://opencv.willowgarage.com/, 2012.

[14] A.D. Sappa, F. Dornaika, D. Ponsa, D. Gerónimo, and A. López, “An

Efficient Approach to Onboard Stereo Vision System Pose Estimation,” IEEE Transactions on Intelligent Transportation Systems,

vol. 9, no. 3, pp. 476-490, Sep. 2008.

[15] M.A. Fischler and R.C. Bolles, “Random sample consensus: A paradigm for model fitting with applications to image analysis and

automated cartography,” Communications of the ACM, vol. 24, no. 6, pp. 381-395, Jun. 1981.

[16] R. Bellman, Dynamic Programming. New Jersey: Princeton University

Press, 1957.

(a)

(b)

(c)

Figure 6. Performance comparison between occupancy grid-based method and proposed method. (a) Disparity map is fairly well-estimated. (b) Stereo

matching fails on vehicle surfaces. (c) Stereo matching fails on vegetation and tree branches are located in the air. The first, second, and third columns show

disparity map, free space estimated by occupancy grid-based method, and free space estimated by proposed method, respectively.

466