a stairway detection algorithm based on vision

1

A Stairway Detection Algorithm based on

Vision for UGV Stair Climbing

Yang Cong Xiaomao Li Ji Liu and Yandong Tang 1

'Robotic Laboratory, Shenyang Institute ofAutomation, Chinese Academy ofScience 110016, China

2Graduate School ofthe Chinese Academy ofScience, Beijing 10038, China

Abstract-In the paper, we present a vision based algorithmused to guide the Unmanned Ground Vehicles (UGV) forautonomous stairways climbing and implement it on UGVsuccessfully. The reliability of guiding UGV to climb stairsrequires evaluating two offset parameters: the position of vehicleon stairs and the orientation angle to stairs. The intention of ouralgorithm is to estimate these two parameters through extractingthe stair edges robustly. To achieve this goal, we apply the Gaborfilter to eliminate the influence of the illumination and keep edges,and propose a fast method to remove small lines. Finally we linkstair edges, and estimate the offset parameters used to steer thevehicle by RANSAC algorithm.

Experiments on various stairways including indoor andoutdoor are given in various light conditions. The results validateour algorithm.

Index Terms- computer vision, UGV, stair climbing, Gaborfilter, RANSAC

I. INTRODUCTION

A s the mobile robot technology making remarkableprogress, it has become the preferred solution for many tasks.In order to navigate autonomously in unknown environments,mobile robots, such as UGV, must be equipped with thefollowings: sensors for measuring the terrain, software forinterpreting the data, path planning to decide a safe route ofaction, and control to oversee the plan.

The ability to detect and climb the stairways automaticallyis of a great challenge to UGV. For detecting the stairwaysbased on vision automatically, Stephen Se[1] used monocularto detection curbs and stairways for the partially sightedhuman; in [2] JS Gutmann introduced stereo vision toguided the human robot to climb up and down the stairways;xiaoyu[3] and R.Turchetto[4] proposed methods to detect andlocate curbs and stairways using stereo vision by segmentation

Manuscript received October 1, 2007. This work was supported in part byShenyang institute of Automation (SIA) of Chinese Academic of Science(CAS).

All authors are with Robotics Laboratory, Shenyang Institute ofAutomation, Chinese Academy of Sciences Shenyang 110016, China(congyang8 1 0gmail.com)

images and reconstructing the 3D environment through linematching. In order to make the UGV climb the stairwaysautomatically, two parameters are very important and play keyroles: the angle between the vehicle's heading direction andthe stair's horizontal direction, and the position of the vehicleon the stairs. The angle is used to guide the vehicle to the rightdirection, and the position to avoid the vehicle colliding to thewalls. For evaluating these two offset parameters accurately astairway detection algorithm based on vision for UGV stairclimbing is presented in this paper.

For validating our algorithm, a visual servo system isintroduced to control the UGV stair climbing by these twoparameters and to guide the UGV to go forward along thelines bisecting the stairs ( to see Fig. 1). A vision algorithm forthe visual servo system is proposed to extract a set of long,horizontal lines robustly in any conditions and to estimate theparameters by the set of lines. In practice, the UGV will beconfronted with many different conditions, such as the stairsof different styles and materials, different applicationenvironments including indoors and outdoors, illuminationchanges, shadows, etc. In order to deal with these situations

i1;.i 11in uveraii Ulisign U1 11n U V, L11nC LWU L1raCK aSIuw Ln1 lUUUL 1 USCU

for driving and turning and the other two tracks in front of the robot is usedfor climbing.

1806

Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY MADRAS. Downloaded on March 20,2010 at 06:18:51 EDT from IEEE Xplore. Restrictions apply.

2

robustly, Gabor filter is applied to eliminate the influence ofthe illuminate. Furthermore, a fast method is proposed to filterout small and vertical lines after the edges are detected. Thisstep improves the efficiency and the accuracy of the algorithmin the stair edge detecting and linking processing. Then, arobust edge linking algorithm is used to extract the set ofparallel lines. Finally, we estimate the two offset parametersbased on the theory of RANSAC algorithm [10][11][12]. Theexperimental results validate our algorithm.

The paper is organized as followed: in section 2, wedescribe the whole flow of our algorithm. In section 3, weestimate the two parameters by the RANSAC and describe thestrategy of steering the UGV. In section 4, the results of ourexperiments are shown. We end this paper with conclusionsand discussions in section 5.

II. THE WHOLE FLOW OF OUR ALGORITHM

Stairways are of different styles, different materials, andunder different lighting conditions. The only feature ofstairways is that their profile includes a set of parallel lines in3D space. When the UGV is creeping on the stairways, theprojection of stair edges on the 2D image plane are nearly agroup of parallel lines.

The intention of our vision algorithm is to detect long,horizontal lines in an image, and to extract the most similarones among these lines which should be the stair edges. Theseextracted lines can be used to estimate the current offset angle

N

y

and the current offset position of the UGV on the stairways,which make it possible to control the UGV to creep the stairsautomatically. Fig.2 illustrates the whole flow of ouralgorithm.Our algorithm includes the following steps:Firstly, the Gabor function is used to filter the image in

order to eliminate the influence of the illuminate and keep thestair edges.

Secondly, the generic edge detector is applied to the filteredimage. With our proposed fast algorithm the most of the small,vertical edges are removed. It can improve the efficiency andaccuracy of the linking algorithm in the next step.

Thirdly, the remainder adjacent edges are linked into a long,horizontal edge (which should be the stair edges) according tosome basic constraints.

Finally, the two offset parameters are estimated by theRANSAC method. And the UGV can be steered by the twoparameters to climb the stairs. When the vehicle arrives at thelinking between two staircases, it restarts the Stair DetectionProgram to detect the stairs automatically. If the vehicledetects the stair, it will go on climb the stair based on vision.

A. The GaborfilterThe Gabor filter can eliminate the influence of the

illumination to an image efficiently and reserve the multi-scaleand multi-direction information. The Gabor filter has alsobeen shown to be benefit to the receptive fields of simple cellsin the striate cortex and hence their relationship to models forthe early vision of manuals [7,8,9]. The Gabor filter is appliedto eliminate the influence of variety illumination and shadowseffects on an image. The appropriate selection of the scale andthe frequency of the Gabor filter to maintain the level of thestaircase edges will be a precondition for the followingprocessing.

Equation 1 shows the Gabor function:1 I1X2 y2ga ,, (x, Y) exp[- (- + 2 )] exp[jo (x +y)] (1)

where (x, y) is the point position in the horizontal and vertical

Fig.2 The whole flow of our algorithms

1807

.L

v

i


3

direction, ax, a is the standard deviation of the x direction

and y direction respectively, and o0 is the space frequency of

the complex plane-wave.By selecting the appropriate parameters ofax, (y, and o

to maintain the level of the stair edges, it will be conducive tofurther processing. For our current work, we use themulti-channel filtering approach with a set of predeterminedfilter parameters. In our experiments, we assume that all thestair edges are horizontal or nearly horizontal (the slope is lessthan 0.5). So in order to get the appropriate parameters ofGabor filters, we experimentally apply a series of Gabor filtersto stair images in order to choose the filter which has thelargest response in vertical direction and the lowest responsein horizontal direction as the most suitable one. Throughexperiments, the parameters are determined, which direction isvertical and the band width equal to 1. And these parametersare used in all of our experiments. Fig.3 shows the originalimage, and Fig.4 shows the result of the image convolved bythe Gabor filter.

The main idea of our template operation is that if a line getsthrough or partial get through the template boundaries, it maybe a line preserved. If a line does not get through, it would beeliminated. We define the radius of the template as r, the widthof the template as W=2*r+1 and the image point as img (i, j),

the boundary point of the template as Edge{i, j}where i = 1 or i = W or j = 1 or j = W . The set ofneighbor points of the Edge{ i, j} are denoted as

adj{fi, j} . The method to compute the sum of the neighborpoints of the edge points in the template is as follows:

sum = E adj(i, j)xEdge(k,l)edge point adj i_j

If sum > 1,There would be a line pass through the template;

Else if sum < 1,There would be no line, and so we elim inate allof the edge pixels in the template.

This method can remove all of the isolate points and most ofthe short discontinuous lines, and save much time in edgelinking. The result of the method on synthesize image isillustrated in Fig.6. And Fig.7 shows the results of this methodon the stair binary image.

Fig. 4 The result of Gabor filter

B. Eliminate small LinesThe Gabor filtered image is segmented by the generic edge

detector, and there are still a lot of small lines in the image.Fig.5 shows their results respectively. It is inefficient if weapply Hough transform, RANSAC or other edge linkalgorithms directly on the output binary edge images fordetecting the stair lines. In order to remove most of small,vertical lines which are not wanted, we propose a new fast

'The synthesize image 'The result of the eliminate small linesFig.6 The result of the eliminate small lines on synthesize image

Fig.7 The result of eliminating small lines on real image(a) The edge of no Gabor filtered (b) The edge of Gabor filtered

Fig.5 The results of the edge detector

1808


4

C. Edge LinkingAfter eliminating the small lines, the binary image, some

basic constraints are set to find the stair edges in the image:1) Stairway edges are always long and straight lines;2) Stairways edges are parallel to each other;3) The direction ofthe stairway is nearly horizontal: so weassume the slope the line less than a threshold.

Let us define some parameters about the ith line segment orregion in the binary edge image:

ai: the area of the region i;

x;: the x coordinate of its left endpoint;

x: the x coordinate of its right endpoint;

yI: the y coordinate of its left endpoint;

yx: the y coordinate of its right endpoint;

k: the slope of its fitted line;

b': the intercept of its fitted line.

The ith region is fitted as a line. The equation of the line

segment is y = k'x + b' . Then the parameters k' and b'are computed by equations (2) and (3):

kiarea iS' -_'S'ZSarea SX - Sx'

k =

area I lxI- s I s I

xx x x

S>S1y x=area'S> - Si Si

(2)

(3)

Where S' is the sum of all its x coordinates, S' is thesum of all its y cy

sum of all its y coordinates, S'. is the sum of the square of

all its x coordinates, S'y is the sum of the square of all its y

coordinates, and Syis the sum of the product of all its x

Fig.8 The result of extracting parallel stair edges

coordinates and y coordinates.A series of rules are designed to link the edges, and the

details of the linking processing are listed below:1. We sort the line segments by their area ai in descend

order, and denote the set of lines segments as L:

L.{1.Ill.2..N. .where N is the total number of the lines.

2. The merging process is:Fo-r i=l :X

For j=1A4, where j-i

If II k' - kj II> thresholdlContinue;

Else If xl _x > thresholdxIJ.or Xl -X > thresholdx

Continue;

Else If y' -Yr > thresholdyI Jr

or y -yY > thresholdyContinue;

ElseMerge the line ith and linejth into line ijth.Recompute the kj, bWj and the otherparameters about line ith.If k1' does not close to zero

Reject the line ijth,Continue.

ElseInsert line ijth into L,N=N+1.

For i=l: total number ofthe merged linesIfthe length ofthe line i is less than width/4,

Delete.

Else If x;> width/2 or xl <width/2

Delete.

The remaining linked lines are most likely the staircaseedges. And in the experiments, threshold] equals to 0.5,thresholdx and thresholdy are both 5.

Fig.8 shows the set of extracted parallel lines from the 2Dimage.

III. ESTIMATE THE PARAMETERS AND STEER THEVEHICLE

A. Estimate the parametersNot all the detected parallel line segments from the image

belong to the stairways, and some line segments which belongto the stairways may be only a part of the stair edges. This iscaused by the influence of different illumination. The offsetparameters, the offset angle AO and offset position Ax, can

1809


5

be calculated from these parallel lines using RANSACalgorithm.

The RANSAC algorithm is an algorithm for robust fittingof models. It was first introduced by Fischler and Bolles in1981 [10]. It is robust in the sense of good tolerance to outliersin the experimental data and is capable of interpreting andsmoothing data containing a significant percentage of grosserrors. The RANSAC algorithm has been applied to a widerange of model parameters estimation problems in computervision, such as feature matching, registration or detection ofgeometric primitives[ I1] [ 12]. In this paper, we use RANSACto estimate the offset parameters, and the detail of theRANSAC algorithm is described below:

Set of data points:* Pj=xj, yj} as the center point of each line

wherej=tl ...N},N as the total number ofthedetectedparallel lines.

* 64=kj, bj} as the line parameters which isfitted by the center points of Pj, whereyj=kj*xj+bj.

* h(®j) as the accumulate of the distance ofpoint i from line 64

* S as the largest number of circulation

Forj 1, ,SSelect a minimal subset {pj,l,;*,pj,M} fromthe {pl,;-,pN} points. M equals to 2.Generate an estimate ®j from the minimalsubset, using the equation (2) and (3) in thelast page. We fit the center points as a line.Evaluate h(®j) on the N-M points not in theminimal subset.Retain Oj as the best if h(Oj) is theminimum thus far. Denote the best as ®j*.

* Gather the inliers to the best estimate, Oj8, and refineusing (weighted) least-squares.

Fig.9 shows the process of estimating the offset parameters.The red lines is the detected parallel lines, and the white pointsmeans the center point of each line. The blue line shows theheading direction of the vehicle, the orientation of the UGV isin the vertical direction, and the position of the UGV is inwidth/2 of the image. The white line computed by theRANSAC algorithm shows the estimated center line of thestairways.

The direction of the stairway is 0 = arctan(k*) and the

average of x coordinates is x xi / N So we can

calculate the offset angle AO and offset position Ax asequation (4),

AO =T /2-O

Ax= w idth /2 - x (4)These two estimated parameters will be used to steer the

vehicle.

Fig.9 Estimate the offset parameter

B. Steer the vehicle and turn the cornerOur algorithm is used to guide the vehicle to climb the stair

automatically. The ideal objection of our algorithm is to makethe vehicle stay in the center of the stairways and steer it goahead along the stairways. In practice it is no use to make theUGV always stay in the precision center of the stair. In fact, ifthe vehicle is not too close to the sides of the stairways and theheading orientation angle of the vehicle is almost along thestairways, the UGV will climb the stairways safely. Throughexperiments we find out that the offset angle AO and offsetposition Ax are change slightly in the two continuous frames.And this phonemena is mainly caused by changing ofviewpoints and the influence of illumination. So a set ofthresholds are preset. If both of AO and Ax are less than thepreset thresholds, the UGV will run along the orientation ofthe last operation; but if eitherAO or Ax is great than thethreshold, we will change the direction of the vehicle to makeit along the right orientation. Through experiment we presetthe thresholds of AO and Ax as 0.25zT and 20 pixels.When the UGV get to the landing plane, the stair detection

program will be restarted to recognize the stairs, planed thepath, and navigated the vehicle to the next stairautomatically. So the UGV can turn the corner and go upstairsor downstairs autonomously.

IV. EXPERIMENTS

We test our system and algorithm under various lightingconditions on various style and materials of stairs which areindoors and outdoors.

Fig. 10 shows the results under various conditions, thedetected stair edges are labeled by red lines. In the outdoorcondition, all the stairways are recognized regardless of lightconditions. In the indoor condition, when facing to the sunonly a portion of stair edges are not labeled that is because theimage is saturated by the sunshine.

Fig.11 illustrates a steering processing when the UGV ismounted on the side of the stair or inclined on the stair, ouralgorithm can still detect the stair edges (red lines), guides thevehicle to the center of the stairways, and steer the vehicle up

1810


6

to the stairway. The white lines indicate the estimated centerof the stairwells. Fig. 11(a) shows the vehicle is mounted onthe side of the stair. The offset parameters marked by thewhite line are estimated successfully. Figi 1(b) shows that ourvision system is guiding the vehicle to the center of thestairways. Fig.11(c) shows the vehicle inclines on thestairwells and our algorithm can still detect the stair edges andestimate the parameters robustly. At last, Fig. 11(d) shows thevehicle is adjusted in the center of the stair by our algorithm.

The indoor condition fqce to the, -un

The outdoor conditionFig. 10 The results under various conditions

Finally, implemented on a laptop computer (Pentuium42.4GHZ) using C language, the over time of the systems isless than 60ms, including extracting the edges, estimating thetime, and steering the UGV.

V. CONCLUSION

We have demonstrated a vision system based on our ownalgorithm which can guide the UGV to climb the stairsautomatically under various conditions. The results proved thevalidity of our algorithm.

REFERENCES

[1] S. Se and M. Brady, "Vision-based Detection of Kerbs and Steps",Proc. BMVC97, 1997

[2] JS Gutmann, M Fukuchi, M Fujita "Stair climbing for humanoidrobots using stereo vision" Intelligent Robots and Systems,2004.(IROS 2004)

[3] Xiaoye Lu, Manduchi, R. " Detection and Localization of Curbs andStairways Using Stereo Vision" Proc.ICRA 2005 4648- 4654

[4] R. Turchetto and R. Manduchi, "Visual Curb Localization forAutonomous Navigation", Proc. IEEE/RSJ IROS'03, Las Vegas,October2003.

[5] Y.Xiong and L.Matthies, "Vision-Guided Autonomous StairClimbing",Proc. ICRA, 2000.

[6] Anastasios I. Mourikis, Nikolas Trawny, Stergios I. Roumeliotis,etc"Autonomous Stair Climbing for Tracked Vehicles" The InternationalJournal of Robotics Research, Vol. 26, No. 7, 737-758 (2007)

[7] G.Daugman. "Uncertainty relation for resolution in space,spatial-frequency, and orientation optimized by two-dimensional visualcortical filters". J.OPT.Soc.Am., 2(7):1160-1169,1985

[8] Tai Sing Lee. "Image Representation Using 2D Gabor Wavelet" IEEETrans. on Pattern Analysis and Machine Intelligence, 1996,18(10):959-971

[9] J. K. Kamarainen, V. Kyrki and H. Kalviainen. "Invariance Properties ofGabor Filter-Based Features" Overview and Applications. IEEE Trams.On Image Processing, 2006, 15(5):1088 1099

[10] M.A.Fischler and R.C.Bolls. Random sample consensus: A paradigm formodel fitting with application to image analysis and automatedcartography. Communication of the ACM, 24(6):381-395,1981

[11] D.R.Myatt, P.H.S.Torr, S.J.Nasuto, J.M.Bishop, andR.Craddock.Napsac :High noise, high dimensional robust estimation-it's in the bag. British Machine Visioin Conference, Cardiff, UK, pages458-467,2002.

[12] 0. Chum and J. Matas. Randomized ransac with t(d,d) test. BritishMachine Vision Conference, Cardi*), UK, pages 448 {457, 2002.

Fig.11 A steering processing when the UGV are mounted on the side of thestair

1811


a stairway detection algorithm based on vision

Documents