Autonomous Robots 18, 81–102, 2005c© 2005 Springer Science + Business Media, Inc. Manufactured in The Netherlands.

Obstacle Detection and Terrain Classification for AutonomousOff-Road Navigation

R. MANDUCHIUniversity of California at Santa Cruz, Santa Cruz, CA 95064, USA

manduchi@soe.uesc.edu

A. CASTANO, A. TALUKDER AND L. MATTHIESJet Propulsion Laboratory, California Institute of Technology, Pasadena, CA 91109, USA

Abstract. Autonomous navigation in cross-country environments presents many new challenges with respect tomore traditional, urban environments. The lack of highly structured components in the scene complicates the designof even basic functionalities such as obstacle detection. In addition to the geometric description of the scene, terraintyping is also an important component of the perceptual system. Recognizing the different classes of terrain andobstacles enables the path planner to choose the most efficient route toward the desired goal.

This paper presents new sensor processing algorithms that are suitable for cross-country autonomous navigation.We consider two sensor systems that complement each other in an ideal sensor suite: a color stereo camera, and asingle axis ladar. We propose an obstacle detection technique, based on stereo range measurements, that does notrely on typical structural assumption on the scene (such as the presence of a visible ground plane); a color-basedclassification system to label the detected obstacles according to a set of terrain classes; and an algorithm for theanalysis of ladar data that allows one to discriminate between grass and obstacles (such as tree trunks or rocks),even when such obstacles are partially hidden in the grass. These algorithms have been developed and implementedby the Jet Propulsion Laboratory (JPL) as part of its involvement in a number of projects sponsored by the USDepartment of Defense, and have enabled safe autonomous navigation in high-vegetated, off-road terrain.

Keywords: obstacle detection, terrain classification, color classification, ladar classification, autonomousnavigation

1. Introduction

Robots that can drive autonomously in off–road en-vironments have received a good deal of attentionin recent years. The US Department of Defensethrough its various agencies has been the ma-jor sponsor of research in this field. Notable ex-amples include the DEMO I, II and III projects(Shoemaker and Bornstein, 1998) and the DARPAGrand Challenge (http://www.grandchallenge.org). In-deed, in the 2001 Defense Authorization Act,the U.S. Congress set a goal that by 2015 onethird of the operational ground vehicle be un-manned (http://www.gao.gov/new.items/d01311.pdf).

Autonomous off-road robots will be employed not onlyin military operations, but also in civilian applicationssuch as wide-area environment monitoring, disaster re-covering, search-and-rescue activities, as well as plan-etary exploration.

Possibly the single biggest technological challengefor these systems is the ability to sense the environ-ment and to use such perceptual information for con-trol. Indeed, even if a robot is equipped with a GlobalPositioning System (GPS) and an Inertial NavigationUnit (IMU), reliable environment sensing is neces-sary for autonomous operation beyond the line of sightfrom the operator. Lacking perception capabilities, therobot would have to rely solely on self-localization (for

82 Manduchi et al.

example, using GPS) and on prior environment maps.Unfortunately, the resolution of GPS is too low fortasks such as obstacle avoidance. Furthermore, mapsnormally have too low a resolution for small obsta-cles to be included, lack accurate elevation informa-tion, and become obsolete. Thus, environment sensingis essential for the task of efficient navigation over longdistances.

It should be clear that driving in outdoor,non-urban environments is more challenging thandriving indoors or in urban scenarios (as high-lighted by the 2004 DARPA Grand Challenge(http://www.grandchallenge.org), whereby all compet-ing vehicles failed within the first 7 miles of the 142miles route). In the latter case, the environment is highlystructured, which typically translates into simpler sens-ing and action strategies. For example, in urban situ-ations one may expect the ground surface in front ofthe robot be planar, which helps detecting obstacles(anything sticking out of the ground plane should beavoided). In contrast, on a bumpy dirt road the robotshould constantly determine which bumps and holesare small enough to be negotiated (possibly by slow-ing down) and which ones should be avoided. Vege-tated terrain introduces one more degree of freedomto the problem: what is considered an “obstacle” froma purely geometric point of view, may not represent adanger for the vehicle if it is composed of compressiblevegetation (for example, a tuft of tall grass or a smallbush). Other challenging situations include the pres-ence of negative obstacles (such as ditches), elementssuch as water, mud or snow, and adverse atmosphericconditions such as fog.

This paper addresses a set of perception tasks thatare at the core of any control system for efficient au-tonomous navigation in cross-country environments.More precisely, we introduce new algorithms for (1)obstacle detection and (2) terrain cover classification.The problem of obstacle detection and avoidance iswell studied in robotics, however, existing algorithmsapply mostly to urban or indoor environments and don’twork well in off-road conditions. This is because typi-cal assumptions about the scene, such as the existenceof flat ground surface, do not hold in this case. Ournew algorithm analyzes the slant of surface patches infront of the vehicle, and identifies patches that are steepenough to represent a hurdle for the vehicle. The anal-ysis is carried out on the range data produced by a laserrangefinder (ladar) or, as in the examples in this paper,by a stereo system. Besides detecting visible obstacle

points, our algorithm clusters such points into distinctsurface segments, which helps building a symbolic rep-resentation of the scene in terms of an obstacle map.

While geometric scene description and obstacle de-tection are essential tasks, they are not sufficient toensure efficient navigation on cross-country terrain.In these environments, there are several possible ter-rain types (such as soil, grass, mud) that should benegotiated by the vehicle in different ways. An ef-fective description of the outside world should con-sist of the combination of geometric and terrain typeinformation, coupled with adaptive control strategies(Bellutta et al., 2000; Hebert et al., 2002). The DEMOIII project (Shoemaker and Bornstein, 1998) pioneeredthis concept, resulting in possibly the most advancedautonomous vehicle for cross-country navigation. Wepresent two approaches to terrain cover perception, onebased on stereo and color analysis (from color cam-eras), and the other based on range data processing(from a ladar). These two systems are somewhat com-plementary in the cues they use to discriminate betweendifferent terrain cover classes: surface reflectivity in thefirst case, local range statistics in the second one. Colorclassification allows one to recognize a certain num-ber of distinctive classes such as grass and foliage, dryvegetation and bark, soil and rocks. Since stereo data isco-registered with color data, it is possible to label theobstacle detected on the basis of range information withthe terrain type estimated by color analysis. The mainchallenges of color-based classification are the inherentambiguity of the reflectivity spectrum for some classesof interest (such as dry grass and soil), the effect of theillumination spectrum on the perceived color (the so-called color constancy problem), and chromatic shiftdue to atmospheric effects. The analysis of local rangestatistics from ladar data, which is at the core of oursecond technique, can be used to successfully discrim-inate grass or foliage from other smooth surfaces suchas rocks or tree trunks. This “range texture” approachis very promising, allowing one to detect surfaces evenwhen they are partially hidden by grass.

The two systems for terrain typing (color-stereo andladar) discussed in this paper have complementaryfunctionalities in the context of off-road autonomousnavigation, and ideally should both be part of the sen-sor suite of a robot for such application, as shown in1. A color stereo camera is most effective for detect-ing and characterizing isolated obstacles, as well as forcapturing elevation profiles of the scene. A single-axisladar placed in the lower portion of the front of the

Obstacle Detection and Terrain Classification 83

vehicle could be used for safe navigation in tall grass,where obstacles may be partially hidden by vegeta-tion and therefore may not be detected by stereo anal-ysis. Together, these two systems may enable a newlevel of autonomy in a wide variety of environmentalconditions.

The algorithms presented in this paper have been de-veloped and implemented by the Jet Propulsion Lab-oratory (JPL) as part of its involvement in a num-ber of projects (ARL Demo III, DARPA Mobile Au-tonomous Robot Software (MARS), DARPA TacticalMobile Robotics (TMR), and DARPA Perceptor).

This article is organized as follows. Section 2presents our geometry-based obstacle detection andsegmentation algorithm. Section 3 describes our twostrategies for terrain cover classification: color analysis(Section 3.1) and range texture analysis (Section 3.2).Section 4 has the conclusions.

2. Obstacle Detection

Path planning for autonomous vehicles requires thatthe map of all potential obstacles be produced in realtime using available sensor data. The obstacle-free can-didate paths leading to the desired position may thenbe compared in terms of their hindrance (measured, forexample, by the amount of steering involved (Lacazeet al., 1998; Badal et al., 1994).

For navigation indoors or in structured environments(such as roads), obstacles are simply defined as sur-face elements that are higher then the ground planeby some amount. Thus, assuming that elevation in-formation is available (by means of a stereo cam-era or a ladar), obstacle detection (OD) algorithms

Figure 1. Concept figure for our considered sensor suite.

normally rely on the “flat world” assumption (Zhanget al., 1994; Williamson and Thorpe, 1998; Baten etal., 1998; Broggi et al., 2000; Badal et al., 1994; Singhand Keller, 1991)). While this assumption works wellfor urban/indoor situations, it does not apply to off-road, cross-country environments, where the geome-try of the terrain can hardly be modeled as a planarsurface (Batavia and Singh, 2002; Hong et al., 2000),see also Fig. 7). If the geometry of the terrain in frontof the vehicle is available (from stereo or ladar), it isstill possible to assess whether a candidate path canbe traversed without damage to the robot. In princi-ple, one could determine the traversability of a givenpath by simulating the placement of a 3-D vehiclemodel over the computed elevation map, and veri-fying that all wheels are touching the ground whileleaving the bottom of the vehicle clear (Lacaze et al.,2002). This procedure, however, besides being compu-tationally heavy, requires a high-resolution elevationmap.

Indeed, solutions for obstacle detection that use el-evation maps (Lacaze et al., 2002) or voxel-basedrepresentations (Hebert et al., 2002) of the world, re-quire a change in the input data structure, which intro-duce additional data transformation steps prior to vi-sual obstacle detection. This could produce samplingartifacts at object boundaries, and requires substan-tial computational power and storage space. In con-trast to these approaches, our OD algorithm works di-rectly with measured 3-D point clouds. Obstacles aredetected by measuring slope and height of visible sur-face patches, an operation that can be carried out di-rectly in the range image domain. An earlier version ofour algorithm (Matthies and Grandjean, 1994; Matthieset al., 1996, 1998; Bellutta et al., 2000) was based on the

84 Manduchi et al.

Figure 2. The viewing geometry for the column-based obstacledetector. q is the slope angle. The origin of the axes correspondswith the cameras focal point.

analysis of 1-D profiles corresponding to range valuesin each pixel column in the image plane.1 These mea-surements sample the trace left by the visible surfaceon the slicing planes � defined by each pixel columnand the focal point of the camera (see Fig. 2). Assumethat a world reference frame has been defined, with xand y axes parallel to the image plane, origin coinci-dent with the focal point, and y axis pointing vertically.Assume further that the y axis is parallel to the imagecolumns (this can be obtained by rectifying the imageafter camera calibration). One may then use the slopeof the 1-D range profile with respect to the horizontalplane to reason about obstacles. For example, Fig. 2shows an instance of a 1-D range profile, where slant(θ ) and height (H ) are shown for two different surfacepatches. Obstacles correspond to ramps with a slopeabove a certain threshold and spanning a minimumheight. The rationale behind this approach is simple:if a surface patch has limited slope, we may assumethat it belongs to the ground surface (for example, aspart of a path leading to a plateau), and therefore it istraversable. If a patch is steep but of small size, it cor-responds to an obstacle short enough to be negotiableby the vehicle. For example, the lower patch in 2 maybe considered traversable, while the higher patch maybe considered an obstacle. This approach leads to fastimplementations, because image columns are analyzedindependently. Unfortunately, the slope θ1−D of the 1-D range profile is, in general, smaller than the true slopeθ of the visible surface, defined by the angle betweenthe surface normal and the vertical axis. Indeed, it iseasy to show that the two are related as

θ1−D = arccos1√

1 + tan2 θ cos2 ϕ(1)

where ϕ is the angle between the surface normaland the slicing plane �. This means that, depend-ing on the relative viewing geometry, a sloped sur-face may not be detected if the surface normalpoints away from the camera. This characteristic iscommon to all algorithms that work in a column-by-column fashion using stereo data (Matthies andGrandjean, 1994) or in a scan-by-scan fashion us-ing ladar data (Hong et al., 2000; Batavia and Singh,2002).

The algorithm described in this paper overcomes thisdeficiency and computes actual 3-D slopes, while re-taining most of the simplicity and computational effi-ciency of the early approach. We present our detectiontechnique in Section 2.2, after providing in Section 2.1a simple but rigorous definition of obstacle points thatmake sense for off-road environments, formalizing andextending the intuitive notion in Matthies et al. (1996).Once the obstacle points have been identified in the im-age, they can be clustered together (or segmented) intoisolated obstacles. We show in Section 2.3 that obstaclesegmentation (OS) corresponds to finding connectedcomponents in a suitable graph built by the OD proce-dure. Our OS algorithm uses full 3-D information, andthus produces more meaningful results than the simpleimage-based segmentation procedure of Matthies et al.(1996).

It is instructive to compare our approach with otherstereo techniques that detect obstacles by finding sur-face elements that are above a reference plane (horopter(Baten et al., 1998)). Whereas horopter-based obsta-cle detection is very fast (as it does not require anypost-processing of the stereo data), the definition of a“ground plane” is problematic for cross-country envi-ronments, as discussed earlier. In addition, the orienta-tion of the horopter with respect to the camera’s refer-ence system should be continuously adapted to the cur-rent vehicle’s attitude. In contrast, our algorithm looksonly at local surface slopes, and is more resilient tothe vehicle’s tilt and roll, as discussed in Section 2.2.1.

Figure 3. An example of 1-D range profile.

Obstacle Detection and Terrain Classification 85

Also note that our algorithm detects obstacles basedon a single frame of range data, and therefore does notrequire integration over multiple frames (with possibleadditional errors due to image mis-registration and/orinaccurate localization).

2.1. Obstacle Points: Definition

As mentioned in the previous section, a surface rampis considered part of an obstacle if its slope is largerthan a certain value θ and if it spans a vertical intervallarger than some threshold H . Unfortunately, surfacesin cross-country environments can seldom be modeledas ramps (i.e., linear patches). This section extends ourobstacle definition to arbitrarily shaped surfaces. Weassume here that a world reference frame has been de-fined as in the previous section, and that the range mea-surement system computes coordinates (pi,x , pi,y, pi,z)of visible surface points pi with respect to this refer-ence frame. We will define obstacles in terms of twodistinct points in space as follows:

Definition 1. Two surface points p1 and p2 are calledcompatible with each other if they satisfy the followingtwo conditions:

1. Hmin < |p2,y − p1,y | < Hmax (i.e., their differencein height is larger than Hmin but smaller than Hmax);

2. |p2,y−p1,y |‖p2−p1‖ > sin θmax (i.e., the line joining them

forms an angle with the horizontal plane larger thanθmax);

where Hmin, Hmax and θmax are constant parameters.

Definition 2. Two points p1 and p2 belong to the sameobstacle if:

1. They are compatible with each other, or:2. There exists a chain of compatible point pairs link-

ing p1 and p2.

Note that these definitions identify obstacles only in-directly: we will say that p is an “obstacle point” ifthere exists at least one other visible surface pointthat is compatible with p. One potential drawback ofthis definition is that isolated points protruding fromthe ground may not be classified as obstacles (if theyadmit no compatible points). However, such isolatedpoints are usually measurement errors, which would

Figure 4. The two truncated cones Up and L p for the determinationof compatible points.

probably be discarded anyway. The condition involv-ing Hmax enforces separation of two obstacles in thosecases where pairs of points exist, one for each obstacle,satisfying the slope condition but located far apart (seeSections 2.2.1 and 2.3). The compatibility relationshipcan also be expressed as follows (see Fig. 4):

Equivalent Definition 1. The points compatible witha surface point p are those belonging to the two trun-cated cones Up and L p with vertex in p, axis orientedvertically, and limited by the two planes of equationy = Hmin and y = Hmax respectively.

2.2. Obstacle Detection (OD) Algorithms

To find all obstacle points in a range image, we needto look for pairs of compatible points. A naı̈ve strat-egy would examine all point pairs, resulting in N 2 − Ntests, where N is the number of points in the imagefor which range was computed. In fact, it is not neces-sary to test all point pairs, but only a much smallersubset of them. For if p̄ is the projection of a 3-Dpoint p onto the image plane, then the points com-patible with p must project2 onto two truncated trian-gles (Up̄ and L p̄) with vertex in p̄ (assuming zero tiltand roll; for non-zero tilt/roll the triangle projectionschange, as discussed in Section 2.2.1). These are ap-proximately isosceles triangles (see Fig. 5), with verti-cal symmetry axis, height equal to Hmax f /pz (where fis the camera’s focal length), and base approximatelyequal to 2Hmax f

tan θmax pz cos υwhere υ = arctan px/pz is the

86 Manduchi et al.

Figure 5. The projection of the truncated cones onto the imageplane (only upper cone shown)

azimuth angle of p measured on the x-z plane fromthe z axis (which can be derived directly from p̄ andthe intrinsic camera parameters). This observation sug-gests the following algorithm, with complexity linearin N :

OD Algorithm 1.

• Initialization: Classify all pixels as non-obstacle.• For each pixel p̄i :

• Scan the points in the two truncated triangles Up̄i

and L p̄i until a pixel p̄ j compatible with p̄i isfound, in which case classify pi as an obstaclepoint and move on to the next pixel.

• If no pixel in Up̄i ∪ L p̄i compatible with p̄i isfound, pi is not an obstacle point.

One drawback of this algorithm is that it may test thesame point pair (pi , p j ) twice. The overall number oftests can be estimated as follows. Let α be the expectedproportion of obstacle points in the image, and let Kbe the average number of points in each projected trun-cated triangle on the image plane. The average numberof tests for each non-obstacle point is thus 2K . If oneassumes that there is a proportion α = L/K of obsta-cle points within each double truncated triangle, thenthe probability of finding the first obstacle point afterexactly n tests is

Pn = P(obstacle point found at n - th test)

={

(1 − α)n−1 2L

2K − n − 1, 1 ≤ n ≤ 2(K − L) + 1

0, otherwise

(2)

No simple expression can be found for the expectationof n in (2). Instead, we may approximate it with a ge-ometrically distributed variable:3 P̄n = (1 − α)n−1α,which has expectation equal to 1/α.

Thus, the expected overall number of tests is approx-imately N (1 + 2K (1 − α)).

Let us now introduce a second strategy, which doesnot require duplicate tests:

OD Algorithm 2.

• Initialization: Classify all pixels as non-obstacle.• Scan the pixels in the image from bottom to top and

from left to right. For each pixel p̄i :• Examine all points in U p̄i, and determine the set Si

of points compatible with p̄i .• If Si is not empty, classify all points of Si as well as

pi as obstacle points.

It is easy to see that each pixel p̄i is tested only onceagainst all other pixels in Up̄i ∪ L p̄i . With referenceto the quantities introduced earlier, now NK tests mustbe performed over the image. Thus, if the expectedratio of obstacle points αgn the image is less than0.5(1 + 1/K ), the second algorithm results in highercomputational efficiency.

2.2.1. Parameter Selection. Our OD algorithms relyon a set of parameters (the slant angle threshold θ andthe minimum/maximum height thresholds Hmin andHmax). This section provides some general guidelinesfor selecting suitable parameter values.

The slant threshold should correspond to the maxi-mum slant that the vehicle can safely negotiate. All theexperiments of this paper use θ = 40◦. The minimumheight threshold Hmin should be less than or equal to thevertical clearance under the vehicle (Hmin = 0.2 m inour experiments). Note that, whereas Hmin determinesthe minimum size of a detected obstacle, two points inthe same obstacle may be separated by a vertical dis-tance smaller than Hmin, as long as they are linked bya chain of compatible point pairs.

Differently from Hmin, the maximum height thresh-old Hmax is determined by the quality (density) of thestereo range estimates. The choice of Hmax should bebased on the following two observations. A small valuefor Hmax increases the risk of over-segmentation (a sin-gle obstacle gets identified as two distinct obstacles ver-tically stacked) if there is a “hole” in the measured pointcloud. On good quality stereo data the point cloud will

Obstacle Detection and Terrain Classification 87

be dense and therefore in such cases Hmax can be se-lected to be equal to Hmin. This will signficantly speedup the run-time of the obstacle detection algorithm. Ondata sets with medium-poor quality stereo, on the otherhand, a large value for Hmax is necessary to handle miss-ing segments in the point cloud. However, this increasesthe computational load of the algorithm, since Hmax de-termines the size of the search window. Furthermore,according to Definition 1, Hmax determines the maxi-mum horizontal distance Dmax for two points to be com-patible (it is easy to see that Dmax = 2Hmax cot θmax).By limiting Dmax to a value smaller than or equal to thevehicle’s width, one ensures that the algorithm does notmistakenly merge two obstacles when there is enoughhorizontal clearance to drive between them. In our tests,we used Hmax = 1 m.

It should also be noted that our “canonical” hypoth-esis that the y axis of the chosen reference frame al-ways points vertically is not very realistic. Normally,the camera reference frame is attached to the vehicle,hence when the vehicle itself is tilted our original hy-pothesis fails. If the actual vertical direction can berecovered (based on IMU information), it is in princi-ple possible to adaptively re-adjust the camera refer-ence frame to account for the new attitude. This simplyamounts to pre-multiplying the values of coordinates ofthe measured 3-D points by a suitable rotation matrix.Note that the search windows of OD Algorithms 1 and2 (the truncated projected triangles discussed earlier)will also change when the vehicle tilts. If the vehicletilts by angle θ , the height of each projected trianglewill change by a factor of approximately 1

cos γ± Hmaxpz

sin γ,

while the base of the triangle projection will be approx-imately unchanged. For a pure rotation φ, the projectedtriangles will rotate by the same amount φ on the im-age plane. For combined rotation and tilt, the projectedtriangles will rotate and their lengths and heights willchange. We compensate for small vehicle tilt and roll byusing a larger projected triangle search window than thetheoretically computed parameters. This obviates theneed to estimate vehicle state information from IMUor visual odometry.

2.3. Obstacle Segmentation

Besides detecting obstacle points, it is useful to identifyall distinct obstacles in the image. This facilitates fur-ther reasoning about obstacle size and type, and inhibitsundesired “gap filling” if the measured 3-D points areinterpolated.

In our earlier work (Matthies et al., 1996; Belluttaet al., 2000), obstacle segmentation corresponded tofinding the connected components of the obstacle pixelmap on the image plane. This approach is computation-ally simple, but may incorrectly link two distinct obsta-cles when their projections on the image plane overlap,even though they are well separated in space (under-segmentation). Additionally, this prior solution may as-sign more than one label to a single obstacle when ithas gaps or missing range points (over-segmentation).This can be avoided by using the equivalence relation-ship stated in Definitions 1 and 2, which makes full useof 3D spatial information.

It is useful to associate each measured point withone node of an undirected graph (point graph), wheretwo nodes are linked by an edge if the correspondingpoints are compatible. Thus, according to Definition 2,two points p1 and p2 belong to the same obstacle if andonly if there exists a path in the graph from p1 to p2.We can extend this notion to define a single obstacle asa maximal connected subgraph (i.e., a connected com-ponent) of the point graph. The connected componentsof the point graph may be computed online at the sametime of graph creation (i.e., within OD Algorithms 1 or2) using the Union—Find algorithm (Mehlhorn, 1984).

Note that our obstacle segmentation algorithm doesnot require pixels colored with the same label to bespatially connected (in 2D image space) to each other.Indeed, the algorithm is able to correctly assign onlyone label to a single obstacle even when there are gapsin the range measurement. This is an important advan-tage in scenes with sparse stereo range data.

2.3.1. Shape-Based Validation. Our obstacle detec-tion algorithm identifies a number of distinct obstaclesas distinct sets of image pixels. It is often the case that,due to measurement noise, a number of such clustersneed to be rejected as spurious. Previous work (Belluttaet al., 2000) used 2-D area information (such as regionareas) of pixel clusters in the image plane as a crite-rion for rejection, the rationale being that very smallconnected components are likely to be “outliers” andtherefore should be discarded. However, it should beclear that, due to perspective, 2-D region area is not avery significant measure. Given that the spatial coordi-nates of the obstacle points are known, it makes moresense to define meaningful 3-D attributes of shape andvolume. Our current approach builds simple attributesfrom the raw point-cloud data, enabling real-time anal-ysis. Some of the features we have experimented with

88 Manduchi et al.

are the volume of the 3-D bounding box around an ob-stacle, the average and maximum slope, as well as theobstacle height. A simple rule-based system (assign-ing thresholds to all such attributes) rejects obstacleswith small bounding volume, small average/maximumslopes, or small average/maximum relative height. Forexample, all detected obstacles with an average slopelesser than 2.5, or maximum slope lesser than 5.0 areconsidered false positive.

2.3.2. Experimental Results. Our obstacle detectionalgorithm was implemented and prototyped in C/C++.Initial implementation of the code ran at 1.5 Hz to0.5 Hz on 320×240 images, on a 900 Mhz Pentium IIIPC. The variable speed of the algorithm resulted fromdifferences in density (quality) of stereo-range data inthe image, the window size used in the algorithm fornearby and distant pixels, and camera orientation andplacement on the vehicle (which determines the num-ber of visible ground points).

Some detection and segmentation results for typi-cal cross-country situations are shown in Figs. 6 and7 (the images shown in the figures correspond to the

Figure 6. The measured range map (b) and elevation map py (c) for the image in (a). The elevation profile for the image column highlightedin (b) and (c) is shown in (d). More precisely, (d) shows the elevation as a function of the distance from the camera for one slicing plane (seeFig. 2). The computed obstacle points and ob-stacle segmentation are shown in (e) and (f) respectively.

left camera of the stereo pair). Stereo can only com-pute disparities where the left and right images in thepair overlap; hence, the leftmost columns of the imagecontain no range measurements (see Fig. 6(b) and (c)).Note that our technique does not compute the groundplane, and thus can be used successfully with casessuch as the last image of Fig. 7, where no ground planecan be defined.

The images in Figs. 6 and 7(a) and (c) contain slantedsurfaces pointing away from the camera, which rep-resents a challenge for the columnwise algorithm ofMatthies et al. (1996) and Bellutta et al. (2000) but notfor our 3-D based algorithm. The image in Fig. 7(b)contains a “negative obstacle” (a ditch across the road).A method to detect negative obstacles based on rangeinformation is discussed in Bellutta et al. (2000). Ouralgorithm correctly detects the visible face of the ditchas an obstacle even though it is below the ground level.

3. Terrain Classification

As mentioned in the Introduction, terrain classifica-tion is a required capability for efficient navigation in

Obstacle Detection and Terrain Classification 89

Figure 7. Examples of obstacle detection and segmentation. Different obstacles are shown with different gray values in the right column.

vegetated environments (Bellutta et al., 2000; Hebertet al., 2002). Information about the terrain composition,in conjunction with the geometric description of visi-ble “obstacles”, can be used for selecting the optimalpath as well as the optimal velocity for traversal. Moreprecisely, given the elevation profile along a candidatepath, and some information about the compressibilityof terrain cover, one may predict the dynamic behaviorof the vehicle as it drives through that path at a givenspeed. Vegetation such as grass or bushes is compress-ible, therefore the visible surface does not correspondto the “load bearing” surface (where the vehicle’s tiresactually touch the ground). Simple models for the ve-hicle’s dynamics in vegetated terrain of known com-pressibility characteristics were presented in Talukderet al. (2002), based on the earlier model for bare (un-

compressible) soil of Rankin et al. (1998). In this pa-per we concentrate on the perception problem, usingsensors normally found on autonomous robots: colorcameras and laser rangefinders (ladars). Both sensorshave advantages and disadvantages. Color cameras arepassive, which has tactical value. They are cheap, smalland light, produce full frame data at an acceptable rate,and can be used to compute range by stereopsis. Un-fortunately, regular cameras cannot be used at night,and the quality of stereo data is not always satisfactory,especially in the case of grass patches and dense veg-etation. Ladars, on the other hand, can be used by dayand night, and usually produce very accurate and denserange data. The interpretation of range profiles for ter-rain typing is perhaps less intuitive than color analysis,but is still feasible, as we show in Section 3.2 (see

90 Manduchi et al.

also Macedo et al., 2000; Hong et al., 2000). However,ladars (especially imaging ladars) are bulkier, heavierand more expensive than cameras.

We discuss techniques for terrain classification usingcolor images and ladar data in the next two sections.Note that the chosen class taxonomy depends mainlyon two factors: which classes are useful for autonomousnavigation, and which classes can actually be detectedwith the available sensor information. The right bal-ance must be struck between the desire for high de-scriptiveness (which would lead to dense taxonomies)and the need for robust classification (which favorsfewer classes). In the case of color-based classification(Section 3.1), we normally consider the following setof classes: soil/rock, green (photosynthetic) vegetation,dry (non-photosynthetic) vegetation (which includestree bark), as well as a “none of the above” class. Forthe case of range analysis from a ladar (Section 3.2),we consider discrimination of “obstacles” character-ized by relatively smooth surfaces (such as rocks ortree trunks) from grass or foliage.

3.1. Color-Based Classification

The apparent color of a surface is a function of theirradiance spectrum on the camera, which, in clear at-mospheric conditions, is a combination of the radiancespectrum of the illuminant(s), the reflectance spectrumof the surface, and the relative geometry of the illu-minant and of the observer with respect to the surface(Horn, 1986; Elachi, 1987). Of all these components,it is the surface reflectance that carries informationabout the material type. Examples of reflectivity spec-tra within the visible and Near Infrared (NIR) bandfor three materials of interest (green or photosyntheticvegetation, dry or non-photosynthetic vegetation, andsoil) are shown4 in Fig. 8. Visible wavelengths coverthe interval from approximately 400 nm to 700 nm.NIR wavelengths up to about 1 µm are not visible tohuman eye but are within the range of CCD sensors(although almost all cameras have an optical filter toblock such wavelengths out). It can be noted from Fig. 8that green vegetation is easily separable from the othertwo classes, both in the visible and in the NIR band. Inparticular, its reflectance has a characteristic feature atapproximately 750 nm (beyond the visible spectrum),sometimes called the “red edge” (Elachi, 1987). Soil re-flectance curves generally exhibit a gentle increase withincreasing wavelength. In particular, soil reflectancedepends upon the chemical and physical properties of

Figure 8. Typical reflectance curves for photosynthetic vegetation(solid line), soil (dotted line), and non-photosynthetic vegetation(dashed line) (from [17]).

the components, moisture content (wetter soils appeardarker) and texture, and surface roughness. From Fig. 8,one also notes that the spectrum of dry vegetation ap-pears very similar to that of soil. This makes color-based discrimination between the two classes ratherchallenging in some cases, and represents a limitationof this method (Roberts et al., 1993).

3.1.1. Normalization and Calibration. Ideally, onewould like to derive from the measured [r, g, b] color,the “normalized” surface color (that is, the color ofthe surface under canonical illumination and observa-tion conditions, which is a function only of the sur-face reflectance). Brightness normalization (Healey,1992), which transforms the vector [r, g, b] into[r, g]/(r + g + b), is a simple and popular approach.The new vector is invariant to scalar transformationsof the type α[r, g, b], which can model the effect ofa number of variables: the intensity of the illuminant,the angle between the surface normal and the illumi-nant direction (under the Lambertian hypothesis (Horn,1986), and the iris aperture of the camera. However,the brightness component conveys information aboutthe surface reflectance that is useful for discrimina-tion, and in practical cases this procedure does moreharm than good, at least when enough data is availableto train the system. A more useful type of normaliza-tion is white point calibration, a standard procedure(usually already implemented in the camera hardware)that yields a diagonal transformation of the color vec-tors (i.e. of the type [αr, βg, γ r ], where α, β, γ arepositive coefficients). White point calibration attemptsto normalize the perceived colors with respect to the

Obstacle Detection and Terrain Classification 91

spectral composition of the illuminant. The coefficientsare computed by imaging a white surface, and forcingthe transformed color vector into a canonical triplet.The procedure can be repeated periodically, or everytime the illuminant changes. Apart from the burden ofpresenting a white surface patch to the camera, whitepoint calibration has two main drawbacks. First, thediagonal transformation is only an approximation ofthe actual change of perceived color of surfaces undera changing illuminant (Maloney and Wandell, 1986).Second, white point calibration assumes that only oneilluminant type is present in the scene. This is not truein normal outdoor environments, which usually con-tain at least two illuminants (direct sunlight and dif-fuse light or shade). Despite these shortcomings, whitepoint calibration is a viable pre-processing stage thathelps improve classification.

Color constancy techniques that use more complexmodels of color formation for illuminant compensa-tion have been proposed (for a review, see Buluswarand Draper, 2002). However, these methods often relyon unrealistic assumptions about the scene or requirelaborious procedures to estimate the reflectance andillumination parameters. In contrast to model-basedtechniques, we adopt an exemplar-based approach, andlearn color distributions of terrain classes by trainingover a large number of images taken under widely dif-ferent illumination conditions. In typical field experi-ments, we collected training data containing surfacesof interest under direct sunlight and in the shade, and atdifferent times of the day. Extensive testing has shownthat this method works very well for outdoor images,provided that (1) a representative training data set canbe collected, covering all expected environmental con-ditions, and (2) the classifier is able to adequately repre-sent the variability of perceived color within each singleclass. In the next section, we discuss a simple Maxi-mum Likelihood classifier that has given good resultsfor this task, based on Mixture of Gaussians modeling.

3.1.2. Classification Using Mixtures of Gaussians.We use a maximum likelihood (ML) strategy for pixel-wise classification. For a given class k, the perceivedcolor c is modeled as a random vector with density(class-conditional likelihood) f (c | k). Each pixel isthus assigned to the class that maximizes f (c | k). Notethat if prior probabilities P(k)for the classes are set,then ML classification is easily converted into MAPclassification, by choosing the class k that maximizesP(k) f (c | k). Our classifier models the conditional

class likelihood for a given class k with a Mixture ofGaussians:

f (c | k) =M(k)∑j=1

αk, j G(c; µk, j , k, j ) (3)

where G(c; µk, j , k, j ) is a Gaussian density with meanµk, j and covariancek, j , andαk, j are the mixing coeffi-cients (with αk, j ≥ 0 and

∑j αk, j = 1) (Bishop, 1995).

The quantity M(k) represents the number of Gaussianclusters (or modes) within the given class k. The modelparameters {αk, j , µk, j , k, j } can be learned from la-beled training images5 using the Expectation Maxi-mization (EM) algorithm (Bishop, 1995). The startingpoint for the EM algorithm is obtained by an initialk-means clustering (Bishop, 1995), itself initialized byrandom seeds. The ability of the model to representmulti-modal densities is indeed very important for out-door images. For example, for the same terrain type, thecolor distributions under direct light and under diffuselight (shade) usually correspond to different Gaussianmodes. There exist a number of standard techniques inthe literature to select the “correct” number of modesM(k) for each class (McLachlan and Peel, 2000, chap-ter 6), none of which, in the authors’ experience, isreally satisfactory. Choosing too few modes leads toinaccurate representation, while too many modes mayoverfit the data. In practice, a number of modes between3 and 5 proved a good compromise in most cases.

In addition to the chosen set of terrain classes, itis important to also provide a “none of the above”(or “outlier”) class, to which all of the pixels that donot correspond to any of the selected classes shouldbe assigned. Intuitively, these are colors that are notwell explained by the chosen model. As explained inRipley (1996), we select outliers by analyzing the totallikelihood of the color:

f (c) =K∑

k=1

P(k) f (c | k) (4)

where K is the number of classes, and P(k) is the priorprobability of class k (for example, in the case of uni-form priors, P(k) = 1/K ). Thus, a color c is assignedto the outlier class if f (c) < f0, where f0 is a thresholdconstant. The choice of f0 may be problematic: the totallikelihood is a probability density function, and thereis no intuitive criterion to guide the threshold selection.Following (Ripley, 1996), we link the threshold f0 tothe probability P0 that a color pixel will be assigned to

92 Manduchi et al.

Figure 9. The pdf of a mixture of 3 Gaussians, and the outlierthreshold f0. The area of the shaded region is equal to P0 in (5).

the outlier class based on our color model (see Fig. 9):

P0 =∫

f (c)< f0

f (c) dc (5)

P0 (a monotone function of c) is a more intuitive param-eter to use than f0, since it represents an actual proba-bility value. Given P0, f0 can be found by solving (5).For any candidate value of f0, the integral on the righthand side of (5) is computed by standard Monte Carlosampling. Then, standard nonlinear 1-D root findingtechniques can be used to solve the integral Eq. (5).

Figure 10 shows the classification result on an imagecontaining objects (the red poles) that never appeared inthe training data.6 The classifier correctly assigns thosesurfaces to the outlier class (color-coded in black in theclassification map). The threshold f0 was determinedby solving Eq. (5) for P0 = 0.99. Note that also thepixels corresponding to the sky (which was not usedfor training) are classified as outliers.

Figure 10. Color-based classification. The color code for the classification map in the right image is the following: Light gray: soil/rock;medium gray: dry vegetation; dark gray: green vegetation; black: outlier.

Figure 11 shows two examples of color-based classi-fication in outdoor scenes. Only those pixels for whichthe range (as measured by stereo) was less than 50 me-ters were classified. It is not particularly useful to tryto analyze terrain at longer distances, and chromaticshift may occur due to atmospheric effects (Nayar andNarasimhan, 2000). These images did not belong tothe training data set. It should be noted that in the firstimage, the system correctly classified the scene underdirect sunlight as well as in the shade. The second imagein Fig. 11 highlights the inherent color ambiguity be-tween the “soil” and the “dry grass” classes, discussedin Section 3.1. Indeed, most of the surface of the moundof ground to the right of the image was incorrectly clas-sified as dry grass. Unfortunately, this ambiguity cannotbe removed based on color alone (Roberts et al., 1993).Some other features (such as visual texture (Castano etal., 2001; Shi and Manduchi, 2003) or thermal multi-spectral analysis (Abedin et al., 2003)) could be usedjointly with color to enhance the classifier performancein these cases.

3.1.3. Open Problems. In our experience, color is avery effective feature for terrain typing. Still, there are anumber of open problems related to color that requirefurther investigation. Two such problems, which arethe object of current research in our laboratories, arebriefly summarized in the following:

1. Illuminant compensation. As mentioned earlier, tra-ditional color constancy algorithms are not veryeffective for automatic illuminant compensationin outdoor scenes. Our exemplar-based approachtrains the system with a plethora of images takenof the scene under as many illumination conditions

Obstacle Detection and Terrain Classification 93

Figure 11. Color classification examples. The classification maps in the right image use the same grayscale code as in Fig. 10. Only points forwhich the range was less than 50 meters were classified.

as possible. While this approach has proven veryreliable in our experiments, it requires a substantialtraining data collection and labeling effort that canmake it unpractical. In Manduchi (2004) we presentpreliminary results using a technique that allowsone to train the system with only one (or a few)images containing the different types of terrain ofinterest under just one “canonical” illuminant. Theknowledge of the color statistics of each class un-der the canonical illuminant enables the system toestimate the illuminant (or illuminants, such as di-rect sunlight and shade) contained in a new scene atthe same time as classification. A Maximum Like-lihood strategy is used to assign illuminant labelsto points in the scene, as well as to determine theilluminants’ parameters. Prior knowledge of the il-luminant parameters’ distribution can be also usedin a Bayesian framework, in order to improve therobustness of the system. The interested reader isencouraged to consult (Manduchi, 2004) for moredetails.

2. Color shift in atmospheric conditions. It is wellknown that, under some specific adverse atmo-spheric conditions (such as fog or haze), the per-ceived color of a surface changes according to thedistance of the surface to the observer. The physics

of this phenomenon is well understood (McCartney,1976; Nayar and Narasimhan, 2000). Indeed, colorshift can be a significant nuisance for outdoor colorclassification; our current research is investigatingstrategies for recovering the original surface colorbased on the estimated atmospheric parameters andthe distance to the visible surfaces (which can becomputed, for example, by stereopsis).

3.2. Ladar-Based Classification

The accurate range profile produced by a ladar canbe used to discriminate between certain classes of ter-rain. In this section we use ladar data to discrimi-nate between grass and other types of surfaces that arelikely to belong to obstacles. Although OD is usuallystudied as a geometric identification problem (as inSection 2), we show here that in environments charac-terized by tall and dense vegetation OD also requiresterrain cover classification. Indeed, geometry-basedobstacle detection cannot be used directly in these en-vironments, because grass tufts would be perceived asobstacles. Our goal is thus to discriminate grass patchesfrom non-traversable obstacles such as rocks and treetrunks.

94 Manduchi et al.

Figure 12. (a) The “Urbie” robot facing two rocks at 45◦ and 315◦. (b) The measured range points (the axis units are in millimeters). Thefan-shaped clusters of lines correspond to directions with no range estimation due to self-occluding structures on the robot.

Range data is well-suited to this type of recognition.Intuitively, one would expect that range data should bespatially scattered in areas that correspond to grass andbushes, and should exhibit a more regular and smoothpattern on bare soil or rock (Macedo et al., 2000).Thus, local analysis of range profiles should provideenough information to discriminate between these twocases. An example of a range profile taken by a ladarsystem rotating around the vertical axis is shown inFig. 12(b). Some ladars are capable of returning bothrange estimates and the intensity of the returns associ-ated with each range estimate; we do not use intensityinformation as our interest is to develop a range-onlyapproach that could be extended to the analysis of cloudof points not necessarily obtained via ladar. Our ladar,described in detail in the next section, was mounted onthe Urbie robot, a small Packbot designed for urbanenvironments, shown in Fig. 12(a). The dots shown inFig. 12(b) correspond to the laser hits for the sceneshown in (a). The fan-shaped clusters of lines indicatedirections that were not analyzed in the scan becausethere was a permanent occlusion in the robot that pre-vented the laser from being fired in that direction (suchas a camera mounted to the left of the laser occludingthe ladar’s view to the left of the robot).

In the following sections, we present an algorithmthat can reliably discriminate between grass and ob-

stacle surfaces based in the analysis of the range froma single-axis ladar (Castano and Matthies, 2003). Anattractive feature of this algorithm is that it is able to de-tect surfaces when they are partially hidden by the veg-etation, such as in Fig. 12. We describe in Section 3.2.1the characteristics of the ladar used in our system. Sta-tistical features of range data in grass are discussed inSection 3.2.2. The properties derived in Section 3.2.2are instrumental in the analysis of our obstacle detec-tion algorithm, described in detail in Section 3.2.3.

3.2.1. The Ladar System. We consider here a ladarcollecting range measurements as it rotates around avertical axis. In particular, the examples in this paperare based on data from a customized Acuity Accu-Range 4000 (Bergh et al., 2000; Matthies et al., 2000).This ladar operates at a wavelength of 780 nm, in theNIR domain, rotates at a rate of 5 Hz and can estimaterange up to distances of 15 m (50 ft). The ladar is ca-pable of acquiring up to 1024 samples per revolutionbut we set the sampling interval to 10 mradians (i.e.,α = 0.57◦) to relax a constraint on the minimum sizeof the grass blades, as explained later.

The accuracy of each range estimate is affected byparameters such as distance, environmental tempera-ture and color and pose of the target (Adams, 2000).The standard deviation σd of the estimation error is a

Obstacle Detection and Terrain Classification 95

function of the measured range d , as well as of the angleof incidence. For our experiments, we set the operativerange to [0.3 m, 2.0 m].

3.2.2. Statistics of Range in Grass. To better justifyour detection algorithm, we introduce some statisticalmodels for the measured range in situations of inter-est. For simplicity’s sake, our analysis assumes that themeasurements are noiseless.

We begin with the simple case of a linear surfacepatch, corresponding to an “obstacle” (Note that in con-text of this section, “obstacle” is any object that is notgrass.) Consider a small angular span α ∈ [0, A], andassume that the surface is at angle γ with the laser beamat α = 0 (see Fig. 13), where the measured range isd(0) = D. The measured range at angle α is thus

d(α) = D sin(γ )/ sin(α + γ ) (6)

For small angular spans, we can linearize Eq. (6) andwrite

d(α) = d(0) + αdd

dα

∣∣∣∣α=0

= D(1 − αK ) (7)

where K = cot γ . If the angle α is a random variableuniformly distributed in [0, A], one easily sees that themeasured distance, being a linear function of a uniformrandom variable, is itself uniformly distributed within[D, D(1 − AK )].

Our goal here is to determine the distribution of laserreturns in a situation with uniform grass. To simplifyour treatment, we will make the following assump-tions, which don’t significantly affect the validity ofour model:

Figure 13. Overhead view of a slanted linear patch, correspondingto an obstacle.

1. The laser beam is infinitesimal, with null diver-gence;

2. No two laser beams hit the same grass blade;3. The grass blades in a homogeneous patch have iden-

tical circular sections of diameter h, and their cen-ters are distributed as a uniform Poisson field withintensity λ̄.

Let us briefly comment on such assumptions. As-sumption 1 is not consistent with the fact that a laserbeam has non-null divergence, and that it has non-nullinitial diameter. However, finite divergence has a smalleffect on our analysis, as discussed in Macedo et al.(2000), therefore we can safely neglect it. Assumption2 is acceptable if the grass blades are thin enough, i.e.,if the blade diameter h is smaller than d · tan(α) forall distances d within the operative range. In our case,for an operative range of [0.3 m, 2.0 m] and a sam-pling interval of α = 10 mradians, the constraint issatisfied as long as the blades’ diameter h is less than3 mm in the near field and less than 2 cm in the farfield. Assumption 3 is a working hypothesis that wefound empirically to be reasonably accurate in typicaloutdoor situations. A similar hypothesis was used inLeung and Malik (1997), Huang et al. (2000), albeit ina different context.

Based on these assumptions, it can be shown (Leungand Malik, 1997; Huang et al., 2000; Macedo et al.,2000) that the measured noiseless range for the grasscase is distributed as an exponential random variablewith parameter λ = λ̄h (as shown in Fig. 14(a), solidline). Figure 15(b) shows the histogram of actual rangemeasurements over a number of revolutions of the ladarin the case of tall homogeneous grass. The grass patchwas at a distance of approximately 1.5 meters from therobot, as shown in Fig. 15(a). The best-fitting exponen-tial (superimposed on the histogram in Fig. 15(b)) hasparameter λ = 3.7/m. From visual inspection, it canbe noted that the exponential curve approximates thehistogram tail rather well.

Suppose now a linear surface patch, such as the onedepicted in Fig. 13, is partially hidden by grass. In thiscase, the maximum possible range at a particular angleα is equal to the distance to the surface d(α). However,the laser beam may be intercepted by a blade of grassbefore reaching the surface. The analytical expressionof the pdf of range within an angular sector A, assum-ing that the surface is linear and partially occluded byuniform grass, is derived in the Appendix. The shapeof this density is shown in Fig. 14(a) (dashed line). It is

96 Manduchi et al.

Figure 14. (a) The pdf of the range in the case of grass (solid line) and of a partially hidden obstacle (dashed line). (b) The pdf of the samevariables after maximum-value filtering (n = 5).

Figure 15. Normalized histogram of range measured over 30 revolutions (b). The curve represents the best-fitting exponential density. Therobot was placed about 1.5 m from a patch of tall grass (a).

seen that for values smaller than the minimum distanceD to the surface, the range behaves as an exponentialvariable (like in the uniform grass case); as expected,its pdf vanishes for values beyond the maximum dis-tance Dmax to the surface within the sector A (in Fig. 13,Dmax = d(A)).

3.2.3. The Detection Algorithm. In the previous sec-tion it was shown that the presence of a surface, albeitpartially hidden, may change the distribution of themeasured range in grass. Based on this observation, anearlier detection algorithm (Macedo et al., 2000) used

higher order statistics of range, computed over smallangular sectors, as discriminative features. However,this algorithm did not exploit the “spatial coherence”property of typical smooth surfaces. In other words,it did not use the observation that if a measurementd(α) comes from an object surface, the subsequentmeasurement d(α + α) is likely to produce a similarrange value. This is not true of measurements comingfrom grass. This property is not self-evident from themarginal range distributions discussed in Section 3.2.2,because the order of measurements is lost when build-ing such statistics. For example, in the case of a slanted

Obstacle Detection and Terrain Classification 97

surface patch (large γ ), range measurements have largemarginal variance, and their distribution may be con-fused with that of a patch of grass if the sampling orderis neglected.

The spatial coherence hypothesis, however, is weak-ened when the surface is partially hidden by vegeta-tion, because laser beams have a certain probabilityof being intercepted by grass blades before hitting thesurface. To minimize this effect, we can pre-processthe measured data with a non-linear filter that out-puts the maximum value of range within a windowof contiguous samples. This filter exploits the factthat, facing a (partially hidden) surface, no measure-ment of points beyond such surface can be taken (seeFig. 12). Thus, if a laser beam within a small win-dow hits the surface of an obstacle, then the maxi-mum range measured within this window of samplesis likely to correspond to an obstacle surface measure-ment. In a sense, the maximum-value filter acts as a“grass remover”. This concept can be better formalizedby looking at how the filter transforms the distributionof the input data. Let {d1, d2, . . . , dn}be a window of nequidistributed and mutually independent6 range sam-ples. Then, the cumulative distribution function (cdf)of d̂ = max{d1, d2, . . . , dn} is equal to:

Fd̂ (d) = P(d̂ < d) = P(d1 < d, d2 < d, . . . , dn < d)

= Fd (d)n (8)

where Fd (d) is the marginal cdf of the range measure-ments. By differentiation, we find the pdf of d̂ to be

fd̂ (d) = nFd (d)n−1 fd (d) (9)

where fd (d) is the marginal pdf of the range measure-ments. Figure 14(b) shows the probability densities offiltered range measurements for the case of pure grass(solid line) and of a partially hidden surface (dashedline). It is seen that in the second case, the effect of themaximum-value filter is to concentrate mass around thevalues corresponding to the surface (thereby reducingvariance), while, in the first case (pure grass), varianceremains high.

Our surface detection algorithm works as follows.First, the data is subdivided into windows of n non-overlapping samples, and the maximum value d̂ amongthe samples in each window is extracted. If n is large(in our case, greater than 3), the set of non-overlappingwindows can be replaced by a n-sample sliding win-dow. Then, the resulting sequence {d̂} is analyzed, to

check for segments with high spatial coherence thatprovide evidence for surface patches. Spatial coher-ence can be expressed in terms of the differential char-acteristics of the data. In particular, we compute themagnitude of the second derivative of d̂:

y = |d2d̂/dα2| (10)

This has the advantage of producing a very small ornull output value when the input is a linear ramp (cor-responding to a locally flat surface, in any orientation).Thus, small values of y should indicate possible surfacepatches.

The identification of surface hits is not done on ascan-by-scan basis but on a sequence of scans. Let’sassume that the analysis of a scan at time t indicatesthat a return from a laser beam fired in direction α, thatyielded a range estimate d(α), is likely to have hit anobstacle surface. This hypothesis can be verified by ob-serving the range estimate in directions adjacent to α

at time t +t . Thus, hits are classified as surfaces onlyafter they are compared against hits in previous scans.In essence, this methodology applies consistency of thescene through time, i.e., if t is sufficiently small, therelative positions of ladar and surface can change verylittle regardless of the relative motions and speeds ofboth ladar and surfaces. Thus, a hit in a direction withrange estimate d(α) obtained at time t is likely to bean obstacle surface if there is a hit in direction α + α

with a range estimate d(α)+d at time t +t for suf-ficiently small α, d and t . This is the primary cri-terion for selecting candidate surface hits. Additionalheuristic criteria can be added to remove false positivesfrom such candidates. For example, candidate hits areremoved when there are no other candidate hits nearby.We also remove candidate hits that lie outside the oper-ative range [0.3 m 2.0 m]. As a reference, for our exper-iments t = 0.2 s is given by the 5 Hz rotation rate ofthe laser. Likewise, the value ofd is determined by themaximum distance that the robot can move in t , i.e.,given a maximum speed of 1 ms, the robot can move int = 0.2 s by a distance of d = 0.2 m, that is, 10% ofthe operational range. Thus, the robot is able to observean approaching object (either because of the motion ofthe object or the motion of the robot) up to 10 timesbefore reaching it, which gives the algorithm time toevaluate the scene. Finally, in our setup we have α ≈30◦. There are many situations in which the relativechange of orientation between the robot and obstaclein a t lapse is larger than α including a fast in-place

98 Manduchi et al.

rotation of the robot, a fast straight motion of the robotpassing by a stationary obstacle and any combinationof these, among others (there are corresponding con-ditions when the robot is not moving but the obstacleis in motion, and when both are in motion). In thesecases, the absence of a sufficiently small α does notmean that the obstacles cannot be detected but only thatthey cannot be tracked; when the value of α becomessufficiently small, the obstacle will be detected, as anew different obstacle.

3.2.4. Experimental Results. Our algorithm wastested on scenes taken by the Acuity laser whilemounted on the Urbie robot, as shown in Fig. 12. Theexperiments were run on eight scenes for periods oflength between 15 and 90 seconds. The scenes includedenvironments with sparse and dense grass, with andwithout obstacles (where the rocks were both trees androcks), with obstacles in the clear and partially hiddenand with the robot both stationary and on the move atvarious speeds. From the 1313 scans produced by thesescenes we obtained 4 false alarms (i.e., four cases inwhich grass was misclassified as an obstacle surface);all the obstacles in the sequences were identified cor-rectly, at some point of the run, i.e., not every obstacleis identified in every single frame in which it appears

Figure 16. (a) The “Urbie” robot facing two rocks at 45◦ and 315◦ (repeated from Fig. 12 for reference). (b) Obstacle surface detection on therange data of Fig. 12.

but every obstacle is identified as such, in at least oneframe of the sequence. This classification is consistentwith the hazard detection and avoidance objective ofthe task, in contrast to a target tracking task that wouldpenalize the loss of the target during any frame of thesequence The right half of Fig. 16 shows the obsta-cle classification that corresponds to the scene in lefthalf of Fig. 12, repeated in Fig. 16 as a reference. Thelines shown in light gray indicate hits that are asso-ciated with obstacles. Note that both rocks (includingthe partially hidden one) have been detected correctly.Running times for the algorithm in a Pentium 4, 2.2 Ghzmachine, are in the order of 450 µs and 215 µs per rev-olution (512 samples) for scenes with and without ob-stacles, respectively. Thus, the algorithm can be easilyintegrated into a real-time navigation system. Finally,the sequence of images in Fig. 17 shows some resultsof the obstacle detection and avoidance in tall grassusing Urbie. In this case, the algorithm was ported toC++ and ran under VxWorks. At the time of the port-ing, the customized Acuity ladar of Urbie had beenreplaced by a Sick ladar with a sampling interval of8.7 mradians (0.5 degrees), a scan span of 180◦ and anexpected range error of 70 mm at 4 m. The sequencein Fig. 17 shows Urbie following a path that has an ob-stacle hidden by tall grass; the robot correctly detects

Obstacle Detection and Terrain Classification 99

Figure 17. Obstacle avoidance in tall grass using laser-based foliage classification.

the obstacle and steers to the left to avoid it. Imagesand movies of the results of these tests are available attelerobotics.jpl.nasa.gov/people/andres/{tmr,mars}.

Note that other techniques for discriminating vege-tation from soil and obstacles using ladar data from amoving robot have been proposed (Hebert et al., 2002;Lacaze et al., 2002). These approaches, which are ap-propriate for use with a 2-axes ladar, are voxel-basedrepresentations of the 3-D data. This representation al-lows for the easy integration of measurements as thevehicle moves ahead (if accurate information about thevehicle’s position and attitude is available). With re-spect to voxel-based techniques, our algorithm has amuch lower computational complexity and does not re-quire integration of data over multiple scans (althoughit does use multiple scans to rule out false positives),while providing excellent results with a single-axisladar.

4. Conclusions

We have presented novel algorithms for obstacle de-tection and terrain classification for cross-country envi-ronments. Our techniques, which use data from a stereocolor cameras and from a single-axis ladar, are at thecore of JPL’s perceptual system for autonomous off-road navigation and have been tested in the context ofthe several projects funded by the US Department ofDefense. Our experimental result in the course of theseprojects have shown the viability and robustness of ouralgorithms for obstacle detection, color-based terraintyping and obstacle labeling, and ladar-based obstacledetection in grass.

Our more recent work is looking at further extend-ing the terrain perception capabilities of a vehicle by

using new sensors and new visual analysis algorithms.For example, color analysis can be made more robustby taking atmospheric effects into account (Nayar andNarasimhan, 2000). Texture features (Castano et al.,2001) could be used to either complement color classi-fication or for terrain typing at night, when color cannotbe used. Possible additional sensors include multispec-tral cameras in the thermal IR (Abedin et al., 2003),which also could be used for terrain classification atnight, and polarization cameras, which may help de-tecting water bodies in the scene (Wolff, 1996).

Appendix

In this Appendix we compute the distribution of therange in the case of an obstacle partially hidden bygrass. The geometry of the obstacle surface is repre-sented in Fig. 13; we are interested in the distributionwithin the angular sector [0, A]. To simplify our treat-ment, we will henceforth assume that K = cot γ < 0,as in Fig. 13 (the case K > 0 is derived straightfor-wardly). It is assumed that the area between the ladarand the obstacle is uniformly covered by grass withspatial density λ̄ and constant blade diameter h (thenλ = hλ̄, as defined in Section 3.2.2).

We first compute the cumulative distribution func-tion (cdf) of the range d:

Fd (d) =∫ A

α=0P(d(α) < d) fα(α) dα

= 1

A

∫ A

α=0P(d(α) < d) dα (A.1)

where we used the total probability theorem, condition-ing with respect to the measuring angleα, and assumingthat the pdf fα(α) of α is uniform within [0, A]. Using

100 Manduchi et al.

approximation (7), and remembering that K < 0, wecan write

P(d(α) < d) =

0, d < 0

1 − e−λd , 0 ≤ d ≤ d(α)

1, d > d(α)

=

0, d < 0

1 − e−λd , d ≥ 0 , α ≥ D − d

DK1, α <

D − d

DK(A.2)

Combining (A.1) with (A.2), one obtains the cdf of therange within the sector of angle A:

Fd (d) =

0, d < 0

1 − e−λd , 0 ≤ d < D

1

A

( ∫ D−dDK

0dα +

∫ A

D−dDK

(1 − e−λd ) dα

),

D ≤ d < D(1 − AK )

1, d ≥ D(1 − AK )

=

0, d < 0

1 − e−λd , 0 ≤ d < D

e−λd

(D − d

ADK− 1

)+ 1 ,

D ≤ d < D(1 − AK )

1, d ≥ D(1 − AK )

(A.3)

and, by differentiation, its pdf:

fd (d) =

0, d < 0

λe−λd , 0 ≤ d < D

λe−λd

(1 − D − d + 1/λ

ADK

),

D ≤ d < D(1 − AK )

0, d ≥ D(1 − AK )

(A.4)

This pdf is represented in Fig. 14 (left, dashed line)for λ = 1/m, A = 6◦, γ = 107◦ (and therefore K =cot γ = −3), and D = 1.5 m.s

Acknowledgments

This research in this paper was carried out at the JetPropulsion Laboratory, California Institute of Technol-ogy, under a contract with the National Aeronauticsand Space Administration. This work was supportedby DARPA, Mobile Autonomous Robot Software Pro-gram, under contract NAS7-1407; DARPA, TacticalMobile Robotics Program, under contract NAS-1407;and the Army Research Laboratory, DEMO III Exper-imental Unmanned Vehicle Program.

Notes

1. A similar OD algorithm was used with ladar measurements inHong et al. (2000).

2. Since there is a one-to-one correspondence between visible 3-Dpoints p and their projections p̄ onto the image plane, we will saythat two pixels p̄1 and p̄2 are compatible when the corresponding3-D points p1 and p2 are.

3. This approximation is acceptable when α is not too small.4. The curves in Figure 8 were built using data from the ASTER

Spectral Library (Hook, http://speclib.jpl.nasa.gov).5. We use a simple graphical user interface that allows one to quickly

hand-label images in a training data set.6. The original color images can be obtained by sending email to

manduchi@soe.ucsc.edu.7. Statistical independence is an acceptable hypothesis given our

assumption that no two laser beams can hit the same grass blade,and assuming dense grass.

References

Abedin, M.N. et al. 2003. Multicolor focal plan array detector tech-nology: A review. In SPIE Int. Symp. on Optical Science andTechnology, San Diego.

Adams, M. 2000. Lidar design, use, and calibration concepts forcorrect environmental detection. IEEE Trans. Robotics Automat.,16:753–761.

Badal, S., Ravela, S., Draper, B., and Hanson, A. 1994. A practicalobstacle detection and avoidance system. In 2nd IEEE Workshopon Application of Computer Vision.

Batavia, P. and Singh, S. 2002. Obstacle detection in smooth high cur-vature terrain. In Proceedings of the IEEE Conference on Roboticsand Automation (ICRA ’02).

Baten, S., Mandelbaum, R., Luetzeler, M., Burt, P., and Dickmanns,E. 1998. Techniques for autonomous, off-road navigation. IEEEIntelligent Systems Magazine, 57–65.

Bellutta, P., Manduchi, R., Matthies, L., Owens, K., and Rankin,A. 2000. Terrain perception for Demo III”. Intelligent VehiclesConference.

Bergh, C., Kennedy, B., Matthies, L., and Johnson, A. 2000. A com-pact and low power two-axis scanning laser rangefinder for mobilerobots. In Seventh Mechatronics Forum International Conference,Atlanta, Georgia.

Obstacle Detection and Terrain Classification 101

Bishop, C.M. 1995. Neural Networks for Pattern Recognition.Oxford University Press.

Broggi, A., Bertozzi, M., Fascioli, A., Guarino LoBianco, C., andPiazzi, A. 2000. Visual perception of obstacles and vehicles forplatooning. IEEE Trans. Intell. Transport. Sys., 1(3).

Buluswar, S.D. and Draper, B.A. 2002. Color models for out-door machine vision. Computer Vision and Image Understanding,85(2):71–99.

Castano, A., and Matthies, L. 2003. Foliage discrimination using arotating Ladar. In Proc. IEEE Intl. Conf. Robotics and Automation,Taipei, Taiwan.

Castano, R., Manduchi, R., and Fox, J. 2001.Classification experi-ments on real-world textures. Workshop on Empirical Evaluationin Computer Vision, Kauai, HI.

Elachi, C. 1987. Introduction to the Physics and Techniques of Re-mote Sensing. John Wiley and Sons,

Healey, G. 1992. Segmenting images using normalized color.IEEE Transactions on Systems, Man, and Cybernetics, 22(1):64–73.

Hebert, M., Vandapel, N., Keller, S., and Donamukkala, R.R. 2002.Evaluation and comparison of terrain classification techniquesfrom LADAR data for autonomous navigation. Army Science Con-ference, Orlando.

Hong, T., Abrams, M., Chang, T., and Shneier, M.O. 2000. An in-telligent world model for autonomous off-road driving. ComputerVision and Image Understanding,

Hook, S. ASTER Spectral Library. http://speclib.jpl.nasa.gov.Horn, B.K.P. 1986. Robot Vision. MIT Press: Cambridge,

Massachusetts.Huang, J., Lee, A.B., and Mumford, D. 2000. Statistics of range

images. In IEEE Conf. Computer Vision and Pattern Recognition,Hilton Head.

Lacaze, A., Moscovitz, Y., DeClaris, N., and Murphy, K. 1998. Pathplanning for autonomous vehicles driving over rough terrain. InIEEE ISIC/CIRA/ISAS Joint Conference.

Lacaze, A., Murphy, K., and DelGiorno, M. 2002. Autonomous mo-bility for the DEMO III Experimental Unmanned Vehicle. Asso-ciation for Unmanned Vehicle Systems—Unmanned Vehicle.

Leung T. and Malik, J. 1997. On perpendicular texture or: Why dowe see more flowers in the distance? In IEEE Conf. ComputerVision and Pattern Recognition, San Juan, Puerto Rico.

Macedo, J., Manduchi, R., and Matthies, L. 2000. Ladar-based dis-crimination of grass from obstacles for autonomous navigation.In Proc. Intl. Symposium on Experimental Robotics, pp.111–120,

Maloney, L.T. and Wandell, B.A. 1986. Color constancy: A methodfor recovering surface spectral reflectance. J. Opt. Soc. Amer. A,3:29–33.

Manduchi, R. 2004. Outdoor color classification using only one train-ing image. In European Conference on Computer Vision 2004,Prague.

Matthies, L., Xiong, Y., Hogg, R., Zhu, D., Rankin, A., and Kennedy,B. 2000. A portable, autonomous, urban reconnaissance robot.In International Conference on Intelligent Autonomous Systems,Venice, Italy.

Matthies, L. and Grandjean, P. 1994. Stochastic performance mod-eling and evaluation of obstacle detectability with imaging range

sensors. IEEE Transactions on Robotics and Automation, SpecialIssue on Perception-based Real World Navigation, 10(6).

Matthies, L., Kelly, A., Litwin, T., and Tharp, G. 1996. Obstacle de-tection for unmanned ground vehicles: a progress report. RoboticsResearch 7, Springer-Verlag.

Matthies, L., Litwin, T., Owens, K., Rankin, A., Murphy, K.,Coombs, D., Gilsinn, J., Hong, T., Legowik, S., Nashman, M.,and Yoshimi, B. 1998. Performance evaluation of UGV obstacledetection with CCD/FLIR stereo vision and LADAR. In IEEEISIC/CIRA/ISAS Joint Conference .

McCartney, E.J. 1976. Optics of the Atmosphere: Scattering byMolecules and Particles. John Wiley and Sons: New York.

McLachlan, G. and Peel, D. 2000. Finite Mixture Models. John Wileyand Sons.

Mehlhorn, K. 1984. Data Structures and Efficient Algorithms.Springer Verlag.

Nayar, S.K., and Narasimhan, S.G. 2000. Vision in bad weather. InIEEE Conf. Computer Vision and Pattern Recognition.

Rankin, A., Owens, K., Matthies, L., and Litwin, T. 1998. Terrain-adaptive gaze and velocity control for ugv obstacle detection. InAssociation for Unmanned Vehicle Systems International AnnualSymposium.

Ripley, B. 1996. Pattern Recognition and Neural Networks.Cambridge University Press.

Roberts, D., Smith, M., and Adams, J. 1993. Green vegetation, non-photosynthetic vegetation, and soils in AVIRIS data. Remote Sens.Environ., 44:255–269.

Shoemaker, C.M. and Bornstein, J.A. 1998. The Demo III UGV pro-gram: A testbed for autonomous navigation Research. In IEEEInternational Symposium on Intelligent Control, Gaithersburg,MD.

Shi, X. and Manduchi, R. 2003. A study on Bayes feature fusion forimage classification. In IEEE Workshop on Statistical Analysis inComputer Vision, Madison, WI.

Singh, S., and Keller, P. 1991. Obstacle detection for high speedautonomous navigation. In IEEE Int. Conference on Robotics andAutomation, Sacramento, CA, pp. 2798–2805.

Talukder, A., Manduchi, R., Castano, R., Owens, K., Matthies, L.,Castano, A., and Hogg, R. 2002. Autonomous terrain characteri-zation and modeling for dynamic control of unmanned vehicles.In IEEE/RSJ International Conference on Intelligent Robots andSystems, Lausanne, Switzerland.

Talukder, A., Manduchi, R., Rankin, A., and Matthies, L. 2002. Fastand reliable obstacle detection and segmentation for cross-countrynavigation. In IEEE Intelligent Vehicles Symposium, Versailles,France.

Williamson, T. and Thorpe, C. 1998. A specialized multibaselinestereo technique for obstacle detection. In IEEE Conf. ComputerVision and Pattern Recognition, pp. 238–244.

Wolff, L. 1996. Polarization phased-based method for material clas-sification and object recognition in computer vision. In IEEE Conf.Computer Vision and Pattern Recognition.

Zhang, Z., Weiss, R., and Hanson, A.R. 1994. Qualitative obstacledetection. In IEEE Conf. Computer Vision and Pattern Recogni-tion, pp. 554–559.

http://www.gao.gov/new.items/d01311.pdfhttp://www.grandchallenge.org

102 Manduchi et al.

Roberto Manduchi is an Assistant Professor of Computer Engineer-ing at the University of California, Santa Cruz. Before joining UCSC,he was a Senior Staff Member at the Jet Propulsion Laboratory, anAdjunct Assistant Professor at the University of Southern Califor-nia, and a Senior Engineer at Apple Computer, Inc. He received a“Dottorato di Ricerca” from the University of Padova, Italy, in 1993.He works in the area of computer vision and sensor processing, withapplications to robotics, communications, and assistive technologyfor the blind. His research has bee funded by NSF, DARPA, andNASA.

Andres Castano received the B.S. and M.E.E. degrees from theUniversity of Los Andes, Bogota, Colombia, and the M.S. andPh.D. degrees from the University of Illinois at Urbana-Champaign,all in Electrical Engineering. At Urbana-Champaign he worked atthe Beckman Institute in visual servo-control of manipulators andpanoramic cameras with very large depth-of-field. Later he joinedthe Information Sciences Institute at the University of SouthernCalifornia where he was the lead roboticist of the Conro project onreconfigurable robots. Currently he is with the NASA/Jet PropulsionLaboratory at the California Institute of Technology. His researchinterests are robotics, computer vision and graph theory.

Ashit Talukder obtained a Ph.D. degree in ECE from CarnegieMellon University (CMU). He is currently a senior researcher at

Jet Propulsion Laboratory. He has extensive experience in signal andimage processing, computer vision, machine learning, pattern recog-nition, controls and optimization. His current research interests are inrobotics, intelligent sensor networks, remote sensing, biomedical sig-nal processing, and product inspection. He has led several DARPA,NIH, and NASA sponsored projects. He has a provisional patent forthe design of software architecture and algorithms for biomedicaldata analysis and visualization, which was nominated for the NASAsoftware of the year. He has over 40 publications, has chaired severalSPIE conference sessions & given 3 invited conference talks, and ison the technical program committee of the Optical Pattern Recogni-tion Conference in the SPIE Defense and Security Symposium. Heis a reviewer on IEEE Trans Image Processing, Neural Networks,Optical Engineering, and Neurocomputing journals.

Larry Matthies received his Ph.D. in Computer Science fromCarnegie Mellon University (CMU) in 1989 and has been at theJet Propulsion Laboratory since then, where he is now supervisor ofthe Machine Vision Group. His research interests are in perceptionfor autonomous navigation of robotic ground and air vehicles. Hedeveloped the first successful real-time stereo vision system for au-tonomous, cross-country obstacle avoidance by mobile robots anddemonstrated the first accurate, stereo vision-based visual odome-try algorithm. Both of these algorithms were used in the 2003/2004Mars Exploration Rover (MER) mission. His group developed theDescent Image Motion Estimation System (DIMES), which usedfeature tracking in down-looking imagery to estimate the horizon-tal velocity of the MER landers during terminal descent. He alsoconducts research on multi-sensor terrain classification and scenemotion analysis for mobile robots for Earth-based applications. Heis an Adjunct Professor in the Computer Science Department at theUniversity of Southern California and is a member of the editorialboard for the journal “Autonomous Robots”.