the terramax autonomous vehicle - unipr.it · vehicle to negotiate tighter turns without needing...

The TerraMax AutonomousVehicle

Deborah BraidRockwell CollinsCedar Rapids, Iowa 52498e-mail: [email protected]

Alberto BroggiVisLabUniversity of Parma43100 Parma, Italye-mail: [email protected]

Gary SchmiedelOshkosh Truck CorporationOshkosh, Wisconsin 54902e-mail: [email protected]

Received 22 March 2006; accepted 20 June 2006

The TerraMax vehicle is based on Oshkosh Truck’s Medium Tactical Vehicle Replacementtruck platform and was one of the five vehicles able to successfully reach the finish lineof the 132 miles DARPA Grand Challenge desert race. Due to its size �30 000 pounds,27� 0� long, 8� 4� wide, and 8� 2� high� and the narrow passages, TerraMax had to travelslowly, but its capabilities demonstrated the maturity of the overall system. Rockwell Col-lins developed, integrated, and installed the intelligent Vehicle Management System,which includes vehicle sensor management, navigation, and vehicle control systems. TheUniversity of Parma provided the vehicle’s vision system, while Oshkosh Truck Corp.provided project management, system integration, low level controls hardware, modelingand simulation support, and the vehicle. © 2006 Wiley Periodicals, Inc.

1. INTRODUCTION

TerraMax™, a completely autonomous vehicle, wasdeveloped by Oshkosh Truck Corporation in coop-eration with its partners Rockwell Collins and Uni-

versity of Parma in response to Congress’ goal thatone third of military vehicles be unmanned by 2015.The Oshkosh TerraMax™ was one of only five ve-hicles that successfully completed the 132-mileDARPA Grand Challenge course in October 2005 �5th

FIELD REPORT

• • • • • • • • • • • • • • • • • • • • • • • • • • • • • • •

Journal of Field Robotics 23(9), 693–708 (2006) © 2006 Wiley Periodicals, Inc.Published online in Wiley InterScience (www.interscience.wiley.com). • DOI: 10.1002/rob.20140

place�, and it was the only vehicle whose mission isto provide medium- to heavy-payload logistic sup-port to the battlefield. During the race, the fully au-tonomous vehicle was successful in demonstratingobstacle avoidance, negotiating tunnels, narrowroads and cliffs, GPS waypoint following and 28 h ofnonstop continuous operation—all applicable to mili-tary missions.

2. THE VEHICLE

The TerraMax vehicle shown in Figure 1 is based onOshkosh’s Medium Tactical Vehicle Replacement�MTVR� MK23 truck platform. The MTVR was de-signed with a 70% off-road mission profile. It cancarry a 7-ton payload off-road or a 15-ton payload on-road. All-wheel drive, TAK-4™ independent suspen-sion, and central tire inflation make rocks, dips, holes,and crevasses easier to handle. And the truck canhandle 60% grades and 30% side slopes. A 425-hp CatC-12 engine powers the truck. This kind of vehiclewas chosen for the DARPA Grand Challenge �DGC�because of its proven off-road mobility, as well as forits direct applicability to potential future autonomousmissions. Team TerraMax also participated in the2004 DGC �Ozguner, Redmill & Broggi, 2004� withthe same vehicle. Two significant vehicle upgradesfor the 2005 DGC were the addition of rear-wheelsteering and integrated sensor structure/roll cage.Rear steer has been added to TerraMax to give it atighter 29-ft turning radius. Although this allows thevehicle to negotiate tighter turns without needing fre-quent backups, the backup maneuver is required to

align the vehicle with narrow passages. The sensormounting structure/roll cage provided added protec-tion to the sensors as well as key vehicle components.

2.1. Autonomous System Integration

The autonomous system consists of computers, com-munication network, sensors, vehicle control inter-face, and the supporting mounting and protectionstructures. The autonomous system utilized in the2004 DGC was completely removed and upgradedfor the 2005 DGC.

2.2. Computers and Communication NetworkIntegration

The computers and communication network hard-ware was packaged in a modular shock absorbingrack located inside the base of the passenger seat asshown in Figure 2. The video monitor, keyboard,and mouse were securely mounted on the dash-board. This arrangement allowed the sensitive com-puter equipment to survive the high-G shock andvibration experienced on the trail.

2.3. Sensor Installation

The sensors were mounted to modular adjustablemounts that were integrated into the roll cage. Theroll cage also serves as a protective conduit, throughwhich, the vital sensor communication and powercables are routed. The adjustable mounts used wereselected for their ability to retain the set position re-gardless of the pounding taken from the trail. Thelocation of each sensor was optimized for function-ality while maintaining a high level of protectionfrom the environment, i.e., rain, sun, engine heat,brush, and, yes, even bridge supports. A sensorcleaning system was also developed to keep thelenses of the TerraMax sensors free of debris such asdust, water, and mud. The main components of thissystem are: cleaning controller, valve array, andwasher tank. The cleaning controller controls the se-quence and duration the sensors are dusted,washed, and dried. The valve array has electricallycontrolled valves that pass pressurized water and airthrough pattern nozzles to the sensor lenses.

Figure 1. The TerraMax vehicle.

694 • Journal of Field Robotics—2006

Journal of Field Robotics DOI 10.1002/rob

2.4. Vehicle Control Actuator Integration

The vehicle control integration was comprised offour key areas: brake, throttle, gear selection, andsteering. The brake was controlled via Ethernet by aproportional voltage to pressure valve. This methodwas utilized, in-part, because of safety concerns ofmechanical systems interfering with a driver whileon the road. The throttle was controlled via an I/Ocard in the vehicle manager computer. A pulse widthmodulated �PWM� signal allowed precise control ofengine throttle level. The gear selection was con-trolled via a relay card in the vehicle manager com-puter. A binary pattern applied to the transmissioncontrol harness allowed the ability to select the de-

sired gear required by the trail conditions. The steer-ing was controlled via serial communications to aservodrive and motor. The servomotor is connectedin parallel with the steering wheel shaft through agearbox. The servomotor has an integrated high-resolution encoder that allows precise control ofwheel angle.

3. TERRAMAX MODELING AND SIMULATION

Modeling and simulation efforts supported the con-trols development by providing information such asunderbody clearance, steer angles, and lateral stabil-ity. A full vehicle model of the truck was created inAdvanced Dynamic Analysis of Mechanical Systems�ADAMS� by assembling subsystem models of sus-pensions, steering, chassis, and tires. A typical NATOReference Mobility Model �NRMM� obstacle coursewith over 70 different obstacles of different sizes andshapes was used to evaluate the underbody clearance�see Figure 3�. The results of this simulation gave anidea about the truck’s capability to maneuverthrough different obstacles at low speeds.

The steering model was used to predict the frontand rear steer angles �see Figure 4� for a given steer-ing wheel input. The rear steer model included adwell and had different gear ratios than the front.

The lateral stability of the truck was evaluatedthrough constant-radius tests. Tire forces were moni-tored to detect tire lift-offs. The results of these simu-lations as shown in Figure 5 were used to evaluate thecapability of the truck to take a particular turn at dif-ferent speeds without rolling over.

4. VEHICLE MANAGEMENT SYSTEM

The Rockwell Collins intelligent Vehicle ManagementSystem �iVMS� �Braid & Johnson, 2006� consists ofhardware and software components that together

Figure 2. Computers mounted under passenger seat: �a�Computer aided design �CAD� simulation; �b� realinstallation.

Figure 3. ADAMS model of the MTVR negotiating a simu-lated obstacle.

Braid et al.: The TerraMax Autonomous Vehicle • 695


provide an extensive set of autonomous capabilities.In order to accomplish this, the iVMS interfaces withthe vehicle systems and all onboard sensors. The pri-mary commands to the vehicle interface are throttle,brake, steering, and transmission.

The general architecture for the iVMS software isa set of applications that communicate to each otherover a 100BaseT Ethernet network utilizing transmis-sion control protocol �TCP� and user datagram pro-tocol protocols and a commercial Ethernet switch.The iVMS software has the key role of performing allautonomous behavior and interfacing to numerousline replaceable units and the key vehicle systems.The software applications are as follows:

• Vehicle control—controls and receives feed-back from the throttle, brakes, and steering inorder to control the vehicle while in autono-mous mode.

• Real time path planner—computes the realtime path utilizing the desired path whileavoiding the obstacles along the desired path.

• Obstacle detection—uses LIDAR and Visionto detect positive and negative obstacles. Ob-stacle data coming from the various sensorsare merged into a single obstacle databaseused by the real-time path planner.

• Behavior management—decides what modethe vehicle should be in based on the currentconditions of the other functions

• Navigation—computes present position andprovides a dead reckoning function.

Figure 4. Steering angle behavior of each wheel endthroughout a 360° rotation of the steering wheel originat-ing from a straight ahead position.

Figure 5. Lateral stability simulation of a MTVR traveling on a constant radius path with increasing speed. Graphdepicts weight transfer on each of the six wheel ends. Simulation was conducted at the maximum gross vehicle weightrating of 58 000 lb.



A graphical user interface �GUI� provides mul-tiple functions to the user including data visualiza-tion, recording, and playback. The GUI is primarily adevelopment tool and is not considered to be an in-tegral part of the real-time iVMS system. Figure 6

shows the GUI displayed on a monitor in the cab.A system management function is also imple-

mented that provides a user interface for executioncontrol and status display for the iVMS applications.Once the system has been initialized, the system man-ager performs a health management function thatcontinuously monitors the status of the applicationand automatically stops and restarts applications asnecessary to maintain normal functionality. TheiVMS can continue to operate normally without thesystem manager once initialized so it is not includedas one of the iVMS applications. The system architec-ture can be viewed in Figure 7.

The following sections of this paper will go intofurther detail on each of the iVMS functions.

4.1. Vehicle Control

The vehicle control function of the iVMS providesthe TerraMax control actions that emulate the ac-tions a human would perform when driving thetruck. The controls provided by the iVMS are steer-

Figure 6. The driving cabin, with the monitor showingthe graphical user interface.

Figure 7. TerraMax iVMS system architecture.



ing, throttle, brake, and transmission control. Steer-ing control is provided through an electronic servoconnected directly to the MTVR steering gearbox.The standard MTVR steering gearbox has dual in-puts so the steering servo for autonomous operationand hand wheel are both connected to the steeringgear allowing the steering control to be switched be-tween manual and autonomous operations withoutchanging mechanical linkages. The steering controlfunction is responsible for providing wheel anglecommands that guide the vehicle to the path definedby the real-time path planner. This is accomplishedby computed deviations from the desired path andconverting the deviations to steer angle commandsthat are sent to the steering servo. The steering con-trol uses capture and track steering control modes.Capture steering control is used for initial coursecapture and track steering control is used duringnormal operation. Capture and track control modesare automatically selected based on currentconditions.

The capture controller uses course error as thecontrol parameter. The controller creates a steerangle command that aligns the ground track of thevehicle with the direct bearing to the active �TO—“next”� waypoint. This type of control is sometimesreferred to as homing control since the path fol-lowed is uncontrolled and the path to the TO way-point may not be a straight line. Capture conditionsoccur during initial course capture so the capturecontroller is only active if certain conditions exist atthe time the autonomous mode is first activated.

The track controller uses linear cross track devia-tion and cross track deviation rate to align the vehi-cle’s path along the ground with the active TO way-point course. Track angle error and steer anglecommand limiters are used to limit the commandedsteer angle to values that are achievable by the ve-hicle. The command limiters incorporate vehicle dy-namic limits with margins built in to ensure the ve-hicle does not get into an unsafe condition. This alsomeans that the vehicle operates at levels below itsmaximum dynamic capability when in autonomousmode. Turn anticipation for waypoint sequences isalso used so the transition onto the new course isaccomplished without overshoots.

The throttle controller interfaces directly to theelectronic engine control unit through a digital PWMinterface. The throttle controller is responsible forcontrolling the vehicle’s speed to the desired speedspecified by the path planner. This is accomplished

primarily through throttle position control but en-gine and service brakes are also used in certain situ-ations to manage the speed.

The throttle position control uses proportionaland integral control. Reset conditions to the throttleposition are provided for transmission up shift anddown shift and to activate the engine brake. Enginebrakes are activated during engine idle so throttleposition overrides are used when engine brakes arerequired. Throttle position faders are used to reacti-vate the throttle position control when the enginebrake is disabled. Engine and service brakes areused primarily to control speed on steep grades andfor speed management during deceleration.

The brake controller provides an analog signalto a pressure actuator connected to the air brake sys-tem �service brakes�. The throttle and behavior con-trol functions provide brake actuation parameters tothe brake controller and the brake controller deter-mines the pressure actuator signal. The brake controlparameter provided by the throttle control functionis speed deviation which is used by the brake con-troller to provide a brake application that is propor-tional to the speed deviation. Behavior control pro-vides brake override signals for emergency stop �e-stop�, e-stop pause, and other special situationsrequiring speed control or position holding. Theemergency stop condition results in a full brakecommand. Brake modulation to limit slipping in fullbrake conditions are provided by the antilock brakesystem that is part of the basic MTVR.

The MTVR has a seven speed automatic trans-mission. The transmission control function providesforward, neutral, and reverse gear control for the au-tomatic transmission. The selection of the transmis-sion gear is through a digital signal to the transmis-sion control unit. The transmission controllerreceives a desired gear signal from the behavior con-trol function and converts the desired gear into thedigital interface to the transmission. Behavior con-trol uses the actual gear position to determine allow-able state transitions and to prevent transmissionfaults due to incorrect gear selection sequences.

4.2. Real-Time Path Planner

The real-time path planner �see Figure 8� is respon-sible for deriving the desired trajectory of the vehicleand providing that trajectory to the vehicle controlfunction. The trajectory includes a desired pathalong the ground as well as the desired speeds and



boundary area. The desired trajectory is derived us-ing the path and speed constraints contained in theDARPA route data definition file �RDDF�, whichcontains a list of waypoints that define a path alongthe ground, a path boundary, and maximum speedfor each leg of the path. The real-time path plannerprovides reactive path corrections to this nominalpath to account for current conditions, such as ve-hicle dynamic limits, obstacles, road edges, terraingrade, etc.

The path planner implements a tree algorithmthat branches from the base at the current TO way-point. Constraints for path boundary and speed areapplied to the tree build function so the tree size isbounded by the constraints. Branches of the tree arecomputed using a model of the steering system andvehicle dynamics to insure that the candidate pathsare drivable. The tree algorithm was derived fromthe rapidly-exploring random tree path planner�Kuffner & LaValle, 2000� where the growth of thetree was limited to a fixed number of branches�levels�.

Once built, the tree represents a series of candi-date paths, one of which is selected as the path to beused by the vehicle control. Selection of the best pathfrom the candidate paths is based on a scoring algo-rithm that considers distance from the route center-line, path curvature, obstacle avoidance, boundary

area constraints, and other factors. Over 2000 candi-date paths are evaluated each planning cycle to de-termine the best path.

The real-time path planner also contains a speedmanagement function that adjusts the speeds as nec-essary to account for path geometry and current con-ditions. The initial desired speed is set to the RDDFspeed constraint for the leg and the speed manage-ment function reduces the speed as necessary.

Figure 8 shows a graphical representation of theRDDF data and the resulting real-time path gener-ated by the path planner. In the main window of theillustration, the central box represents TerraMax ve-hicle, light grey boxes represent the desired path de-fined in the RDDF file, and the black numbers nearthe center of the boxes represent the real-time pathgenerated by the path planner. The width of the greyboxes defines the lateral boundary of the path andthe length of the boxes is defined by the waypointsin the file. The short red lines in the diagram arerepresentations of obstacles that have been detectedby the perception sensors. Dialog boxes along thesides of the main window show data associated withthe path, vehicle state, and control state. As shownin the example diagram, the real-time path is ad-justed to the right of the RDDF path center in orderto avoid the obstacles in the turn.

4.3. Obstacle Detection

LIDAR and vision sensors are used to detect ob-stacles in front of the vehicle. Obstacles detected bythe sensors are registered to the vehicle navigationposition and stored in an obstacle database. The real-time path planner queries the database to determineif obstacle collisions occur on the proposed paths.

Several different types of obstacle clearance in-formation are provided to the path planner to aid inpath selection. Obstacle collision information is re-ported by the database in terms of the closeness ofthe object collision to the proposed path. Buffer re-gions of various sizes are used to determine the col-lision proximity relative to the path.

Bearing and distance to the nearest collision isprovided by the obstacle database that is an indica-tion of the proximity of the obstacles to the proposedpath. Obstacle distance is used primarily in thespeed manager function to lower the speed if an ob-stacle is in close proximity to the vehicle’s plannedpath.

Road and cliff edges are handled as special cases

Figure 8. iVMS graphical user interfaces depicting real-time path planning along a defined route.



by the obstacle database. Since the consequences tothe vehicle of breaching a cliff edge are very severe,additional weight to negative road/cliff edges areused. The database also reports if any negativeroad/cliff edges are in the immediate area that isused by the speed manager to reduce speedsaccordingly.

4.4. Behavior Management

The behavior management module is the central“brain” of the system. Its purpose is to monitor andreact to dynamically changing conditions. This mod-ule receives input from the real-time path planner,obstacle database, navigation sensors, and the ve-hicle interface module.

Several behaviors have been designed into thebehavior module, using a state transition architec-ture. When a specific event or a change from normaloperating conditions is detected, one of the behav-iors is activated to handle the situation at hand. Eachbehavior executes an ordered list of instructions,providing a set of commands to the vehiclecontroller.

Some of the conditions the behavior module willreact to are as follows:

• Transition in e-stop state: When the e-stop isin pause mode, a behavior will command thevehicle to come to a stop. When e-stop tran-sitions to /Run, another behavior is initiatedto begin normal operation.

• No valid path ahead: The behavior initiatedin this condition commands the vehicle tocome to a stop and wait for a valid path. If novalid path is found, it will command the ve-hicle to back up and try again.

• Obstacle detected behind the vehicle whilebacking up: Another behavior will stop thevehicle and command it back into normal op-eration to try to find a valid path ahead.

• A large course change requiring a backupmaneuver: The switchback behavior guidesthe vehicle around a three-point turn.

• Narrow tunnel condition: The tunnel behav-ior will guide the vehicle through a narrowtunnel, using the LIDAR scan data.

• Stuck between obstacles: If the vehicle cannotmake progress along the route because it con-tinues to go back and forth, getting stuck be-

tween obstacles, the stuck behavior will takeover. It will first try to position the vehicle atdifferent angles to search for a valid path. Ifno valid path is found, it then commands thesystem to ignore low confidence obstacles, inan attempt to eliminate false obstacles. Thelast resort is to go forward toward theDARPA route, ignoring all obstacles.

The real-time path planner, behavior manage-ment, and vehicle control functions work together todetermine the actual path the vehicle follows. Nomi-nally, the vehicle follows the path generated by thereal-time path planner but that real-time path can beoverwritten by the behavior manager based on thecurrent conditions. This design approach is similarto the distributed architecture for mobile navigation�DAMN� �Rosenblatt, 1997� developed by CarnegieMellon University where the behavior managementfunctions as the DAMN arbiter and behaviors. Un-like the DAMN architecture, the behavior manageruses rules-based decision logic to determine controlbehavior modes rather than a voting scheme to se-lect the control mode. This approach was chosenover the more complex voting scheme since it is de-terministic and more robust, which were consideredto be important attributes given the nature of thecompetition. This approach lends itself to fleet appli-cations, which was also an important considerationin the iVMS design.

4.5. Navigation

Two Oxford Technical Solutions RT3100’s�www.oxts.co.uk� supply GPS position informationto the iVMS system. The RT3100 is a combined GPS/IMU sensor that provides real-time data even in theabsence of GPS signal. The high 100 Hz update ratehas a very low latency to insure that the system isusing the most accurate position possible. OneRT3100 is configured to use DGPS corrections trans-mitted via RS-232 from an external GPS receiver sub-scribed to the Omnistar correction service. The otherRT3100 is configured to use WAAS corrections.

In the case of loss of GPS signal, such as drivingthrough a tunnel, the IMU portion of the RT3100takes over and begins dead reckoning. In order toaid the INS solution in dead reckoning mode, awheel speed sensor on the vehicle provides input tothe RT3100. Tests have shown that the wheel speed



input helps to keep the IMU solution stable and ex-tends the time the RT3100 is able to dead reckon.

In the case of a failure or short-term loss of theRT3100’s, a second dead reckoner is implementedusing sensed wheel speed and wheel angle. Thisrepresents an independent backup navigation func-tion. Because of the potentially large errors that canbuildup when it is in a dead-reckoning mode, theRDDF boundary area checks in the path planner aredisabled so the vehicle can continue navigation rela-tive to the terrain and terrain obstacles for short pe-riods of time.

The RT3100’s were capable of dead reckoningafter loss of GPS using the IMU but, during fieldtesting, it was found that the error characteristics ofthe wheel speed/wheel angle dead reckoner weremore desirable than the IMU error characteristics.The conditions where the dead reckoner was mostlikely to be used was while traveling through rail-road tunnels and highway overpasses where theGPS satellite signal would be masked. This meantthat the vehicle would be moving in a straight lineand only in the tunnel for a short time �typically lessthan 30 s�. Under these conditions, the wheel speed/wheel angle dead reckoner was able to maintain apredictable accuracy of less than 1 m. The IMU wasalso capable of similar accuracies but the error char-acteristics were less predictable. The predictability ofthe error was important since the obstacles while inthe tunnel �i.e., the tunnel walls� were very close tothe vehicle and small, abrupt changes in position er-ror caused the collision detection function to stopthe vehicle.

The wheel speed/wheel angle dead reckoner re-lied on an analytical model to relate wheel angle andspeed to heading �yaw� rate so the accuracy deterio-rated significantly during turns. The large headingerror build up during turns makes this implementa-tion practical only under a very narrowly defined setof conditions.

5. SENSORS

The sensors were carefully selected to provide the re-quired navigation and perception capability. Figure 9shows the sensor locations on TerraMax. The sensorsselected for the DARPA Challenge 2005 are asfollows:

• Oxford GPS/INS;

• Trimble GPS;• single-plane LIDAR;• multiplane LIDAR; and• forward-looking vision and system.

5.1. Oxford GPS/INS

The OXTS RT3100’s are mounted on the floor of thecab on the approximate centerline of the vehicle. Inorder to obtain a more accurate position solutionand eliminate any errors over time, the position so-lutions from the two RT3100’s were averaged to-gether. In the case of a failure of one of the RT3100’s,the system will switch to using the remainingRT3100 as the sole GPS source.

5.2. Trimble GPS

The Trimble GPS �www.trimble.com/aggps132.shtml� is an agriculture GPS unit used toreceive differential corrections used by the GPS re-ceivers embedded in the Oxford RT3100’s. TheTrimble receiver outputs differential corrections at1 Hz through RS232. In order to output the differen-tial corrections the Trimble receiver is placed in basestation mode and must also have a subscription.

5.3. Single-Plane LIDAR

There are two SICK LMS-291 LIDARs used for posi-tive and negative obstacle detection �see Figures 10and 11�. They are mounted on the outermost edges

Figure 9. The TerraMax sensor suite: the picture showsthe GPS position, the three front looking cameras, the twoSICKs, and the two multiplane laserscanners.



of the front rollbar. They are pointed 10 deg downand 25 deg outward from the truck so that there isgood coverage on extreme turns. The two LIDARsare configured to scan a 100-deg scan area with a1-deg resolution.

The orientation of the SICK LIDARs was chosento gain visibility to positive obstacles near the ve-hicle and detect negative road edges. Positive ob-stacle detection was accomplished be translating therange returns into local level coordinates and com-

paring the relative heights of neighboring scanpoints. A history of scan returns were maintainedthat effectively mapped the surface directly in frontof the vehicle. Detection thresholds were set so ob-stacles below a specific height would not be detectedas an obstacle. The minimum obstacle height was setbased on the capability of the vehicle. A convex hullalgorithm was used to define the outermost edge ofthe obstacle.

Negative road edge detection followed a similarapproach as for the positive obstacle detection. Aspecific search algorithm was used to find any nega-tive height discontinuities. Each discontinuity thatwas detected was further evaluated to determine ifthe true edge and if that discontinuity was a con-tinuation of the previously detected edge.

5.4. Multiplane LIDAR

The IBEO ALASCA LIDAR �Lages, 2004� is a four-plane scanner that is used for positive obstacle de-tection. The LIDAR is mounted level in the frontbumper �see Figure 12� and has two planes that scantoward the ground and two planes that scan towardthe sky. With a range of 80 m and a resolution of0.25 deg it can detect obstacles accurately at longand close range. The 170-deg scan area allows seeingobstacles around upcoming turns.

The LIDAR sends scan data via Ethernet to theLIDAR PC via a TCP connection. An algorithm thentransforms the raw scan data into obstacles by look-ing for large positive slopes in the scan data �seeFigure 13�.

Figure 10. The SICK LIDARs and the cameras are placedon a rigid bar onto the vehicle hood.

Figure 11. iVMS graphical user interface depicting SICKobstacles as green and yellow polygons. In the figure thevehicle is located at waypoint 27 with iVMS calculatedmicro-waypoints extending ahead of the vehicle.

Figure 12. The front of the TerraMax vehicle. Two four-plane laserscanners are visible: the one inside the bumperis the one used during the race, the other �over thebumper� is a backup.



The IBEO obstacle detection algorithm followeda similar approach as the SICK obstacle detectionalgorithms. The scan returns were translated fromthe sensor coordinate frame to a local level coordi-nate frame. The scan returns were then compared toneighboring returns and discontinuities were de-tected. Since the IBEO LIDAR was a multiplanes, theobstacle detection algorithm was able to comparethe scans from each plane to aid in the obstacledetection.

The SICK and IBEO LIDARs were configured sothe data provided by the sensors were complemen-tary. The SICK LIDARs were configured to detectobstacles from 0 to 20 m in front of the vehicle. TheIBEO LIDAR was configured to detect obstaclesfrom 20 to 60 m in front of the truck. Obstacles de-tected from both sensors were put into the obstacledatabase and used for path planning.

Field testing indicated that, although the two LI-DAR sensors provided similar data, the characteris-tics of the data were somewhat different. For ex-ample, the SICK LIDAR data were very consistent

but more susceptible to reflections than the IBEO LI-DAR. The IBEO LIDAR provided more accurate databut would occasionally return spurious data spikes.Putting both sets of data into the database withoutprior filter resulted in multiple copies of some ob-stacles database. A fusion of the LIDAR sensors toeliminate the multiple copies reduced database utili-zation and sped up the execution of the collision de-tection algorithms.

5.5. Trinocular Vision System

The vision system is based on multistereoscopic vi-sion �forward looking trinocular system�. It consistsof three identical cameras mounted on a rigid bar ontop of the hood. The two lateral cameras lay at adistance, which is about 1.5 m, while the central oneis placed asymmetrical at about 0.5 m from the rightone. Thanks to a precise calibration of the cameras—performed on a graduated grid—the three degreesof freedom specifying cameras orientation are fixedto known values, and in particular—in order to easeand speed-up the subsequent processing—the yawand roll angles are fixed to zero for all cameras. Thepitch angle is chosen so that the cameras frame asmall portion over the horizon �to limit direct sun-light� and frames the terrain at about 4 m from thevehicle.

The trinocular system sends three video streamsat 10 Hz �640�480, color with Bayer pattern� to thevision personal computer �PC� via a firewire connec-tion. The PC selects which stereo pair to use depend-ing on the speed of the vehicle. Since the base line ofthe stereo vision system influences the depth ofview, the large base line is used at high vehiclespeeds so that a deeper field of view is obtained, themedium one at medium speeds, and the short baseline is used at low speeds. This is one of the very fewexamples of very large base line stereo systems�1.5 m� used on rough off-road terrain and deliver-ing a robust environmental perception at more than50 m, regardless of terrain slope.

The rationale behind the design is mainly theneed for mechanical robustness: three nonmovingcameras have been preferred with respect to a pan-tilt solution such as the one used by other teams, forexample the Red Team �Whittaker, 2005�. A fewother considerations were the basis for this choice:vision must be able to sense obstacles at large dis-tances �more than 50 m away on rough terrain�,therefore a stereo vision system was the only choice.

Figure 13. iVMS graphical user interface depicting IBEOobstacles as blue lines. In the figure the vehicle is locatedat waypoint 162 with iVMS calculated microwaypoints ex-tending ahead of the vehicle.



Furthermore, the base line �distance between the ste-reo cameras� had to be large enough to guaranteedepth perception at large distances. Systems basedon moving cameras when both stereo and a largebase line are used, are subject to a number of me-chanical problems such as—for example—the non-negligible momentum caused by vehicle vibrationswhich has to be compensated for. As a result, theexperience with multiple cameras providing differ-ent video streams to choose from turned out to be awinning solution, capable of removing most of themechanical problems of a gazing system.

Vision provides sensing for both obstacle detec-tion and path detection �see Figures 14 and 15�.

1. Image disparity is first used to estimate theaverage terrain slope in front of the vehicle�Labayrade, Aubert & Tarel, 2002�. Slope in-formation is then used for both obstacle de-tection and path detection. Any significantdeviation from the average smooth slope de-tected previously is then identified as an ob-stacle. The exact location of obstacles is thenobtained via stereo triangulation between thetwo views of the object. A fairly precise local-ization is obtained, but nonetheless it can befurther refined via sensor fusion with rawdata coming from the multiplane LIDAR. Inthis way it is possible to detect thin verticalposts and fence poles. The system is able todetect even small obstacles �Broggi, Caraffi,Fedriga & Grisleri, 2005�, but—due to boththe size and capabilities of the vehicle and tothe team strategy—it was tuned with veryhigh thresholds, so that the number of falsepositives was reduced to a minimum. Inother words, the capability of detecting smallobstacle was traded for a higher robustnessof the detection. Nevertheless, the systemwas demonstrated to be able to detect smallconstruction cones used during both the testsand the qualification runs. Anyway, since thevehicle is able to negotiate 60 cm steps, ob-stacles smaller than 60 cm needs to be de-tected primarily for speed managementissues.

2. Image disparity is also used to compute thearea in front of the vehicle which features asmooth slope, the so-called free-space. Thefree-space is one of the features that concur toconstruct a representation of the path to be

followed by the vehicle: also similarity in tex-ture, similarity in color, and shape informa-tion are taken into account, fused together,and delivered to the following path planningmodule. Free space is obtained using a stan-dard image warping �Bertozzi, Broggi & Fas-cioli, 1998� in order to localize deviationsfrom a smooth road surface: Figure 15 showsthe right image �a�, the warped left image �b�,and—in green—the matching cluster repre-

Figure 14. Images showing left and right images of dif-ferent situations; on the right images, colors show thepresence of detected obstacles: different colors mean dif-ferent distances. The additional horizontal lines representthe 5 m, 50 m, and horizon position. Posts and thin polesas well as fence posts are correctly detected.



senting free space �c�. Figure 15�d� shows thefinal result of path detection. This algorithmalso signals the presence of a straight seg-ment of road, in order to increase vehiclespeed. When a curved path is present �the reddot at the top right of Figure 15 shows thepresence of a nonstraight road�, vehicle speedis reduced.

Vibrations are automatically filtered out sincethe slope detection algorithm �Broggi et al., 2005�,which is the first to be performed, also extracts in-formation that is used to electronically stabilize theoncoming images. Different light levels are compen-sated for by an automatic gain control scheme,which allows to sense the environment even withdirect sunlight into the camera. Figure 16 showssome examples of the custom gain controlmechanism.

The camera boxes have a sun shade aimed atreducing to a minimum the quantity of direct sun-light hitting the cover glass, in order to avoid oversaturation and reflections due to dirty glass.

The MTVR is a production vehicle with thou-sands of units produced and in service with the US

Marine Corps, US Navy, and other services through-out the world. Performance of the MTVR was notspecifically tracked as part of the TerraMax develop-ment efforts, rather the dynamic capabilities of the

Figure 15. Image showing different steps of path detec-tion: first the free-space is determined then the path is lo-calized. Right image �a�, warped left image �b�, free space�c�, the final result of path detection �d�.

Figure 16. Images captured at sunrise, representing theview with �a� and without �b� the developed camera gaincontrol scheme; �c� and �d� show the result of obstacle de-tection in bad illumination conditions: although a part ofthe images are oversaturated, the terrain is visible and thealgorithm can be run; obstacles can be detected until theyenter the oversaturated area.



vehicle were identified through ADAMS simulationsand used to define the limits within which the realtime path planner developed alternative paths.

6. VEHICLE PERFORMANCE

The vehicle was able to conclude the qualificationruns with excellent results, avoiding obstacles, pass-ing into tunnels, and maneuvering �with backups� inorder to align itself with narrow barriers and gates.Figure 17 shows some pictures taken during thequalification phase.

The iVMS development philosophy was to createan autonomous system that could, in the future, be

utilized in military operations. This allowed for amore rugged implementation of the iVMS for realtime navigation across unknown terrain. As a result,Team TerraMax was one of only five teams to traversethe 132-mile course and the only vehicle to overcomean overnight “pause” of the autonomous system.During the race, TerraMax reached a maximum speedof 42 mph. This is impressive not only due to the sizeand weight of the TerraMax, but due to the fact thattrue obstacle avoidance was achieved at these speeds.Figure 18 shows a few pictures taken during the lastpart of the race.

During the race the TerraMax was paused 13times by DARPA officials to maintain a minimum dis-tance between the competing vehicles or for passing

Figure 17. Some phases of the qualification runs.



stopped vehicles. The TerraMax automaticallystopped and realigned its path approximately 52times during the race. The majority of the path resetsoccurred while traversing beer bottle pass where theroad was very narrow and the turns were very tightcompared to the size of the vehicle �see Figure 18�.

An automatic reversion mechanism was imple-mented to manage redundant sensors. The reversion-ary logic was activated due to sensor abnormalitytwice during the race. The reversionary logic cor-rectly selected the operational sensor and continuedto operate normally after the reversion.

A software application health monitor wasimplemented to monitor the health of the system andstart and stop applications as necessary to keep thesystem executing normally. During the race thehealth monitor function was activated and correctlyreset applications to keep the system operating nor-mally with only minor interruptions in service. A pe-

riodic database integrity check was also performed toprevent fatal errors from corrupting the database andto recover data if a data error was found.

During the race the TerraMax struck the edge ofone of the concrete underpass barriers. The impact ofthe tunnel caused the IBEO LIDAR and passenger vi-sion camera to be severely misaligned. The misalign-ment caused a georegistration error of the detectedobstacles. This caused the path planner to offset thepath to compensate for obstacles, slowing theprogress of the vehicle for the last half of the race. Ter-raMax completed the 132-mile race with an officialtime of 12 h, 51 min and over 28 h of continuousoperation.

7. CONCLUSIONS

This technology demonstrated by TerraMax has thepotential of improving soldier survivability in thebattlefield by removing soldiers from harms way, es-pecially during convoy operations—the ultimate out-come of the Congressional goal. The development offully autonomous systems has allowed Oshkosh andits partners to fully understand the requirements re-lated to both leader-follower and autonomous opera-tion. The opportunity exists to develop and deploythis technology to allow for robotic replacement ofconvoy personnel, allowing the personnel to be refo-cused on more pressing duties, and ultimately reducethe convoy personnel exposure to enemy threats.

On the technical side, the choice of the �i� sensorssuite delivered the sufficient amount of informationfor the successful conclusion of the race and demon-strated to be robust enough to deal with the extremeconditions of a desert environment in summer. Theexperience with multiple cameras providing differentvideo streams to choose from turned out to be a win-ning solution, capable of removing most of the me-chanical problems of a gazing system. The 28 h of un-interrupted service provided by the �ii� processingsystems and software architecture demonstrated thestability and robustness. Finally, the �iii� algorithmicsolutions proved to be fast and reliable, reducing thenumber of wrong detections to a minimum.

The TerraMax partners of Oshkosh Truck, Rock-well Collins, and University of Parma have demon-strated the scalability and portability of the autono-mous technology by installing and operating thesystem on an Oshkosh palletized loading system�PLS�. The PLS is a 10�10 vehicle with a gross vehicle

Figure 18. Two pictures taken during the race �courtesyof DARPA�.



weight of 84 000 lb and is capable of delivering a33 000 lb payload. The vehicle was successfully dem-onstrated at the Yuma Test Center in January 2006, ex-hibiting the same autonomous capabilities as the Ter-raMax. The project was completed in approximately75 days. Figure 19 shows the PLS vehicle during anautonomous run.

REFERENCES

Bertozzi, M., Broggi, A., & Fascioli, A. �1998�. Stereo in-verse perspective mapping: theory and applications.

Image and Vision Computing Journal, 16�8�, 585–590.Braid, D., & Johnson, C.W. �2006, June�. DARPA Grand

Challenge intelligent vehicle management system�iVMS� and the transition to leader/follower. 6th An-nual Intelligent Vehicle Systems Symposium & Exhi-bition, Traverse City, MI �pp. 104–113�.

Broggi, A., Caraffi, C., Fedriga, R.I., & Grisleri, P. �2005,June�. Obstacle detection with stereo vision for off-road vehicle navigation. 2005 IEEE Computer SocietyConference on Computer Vision and Pattern Recogni-tion, San Diego, CA �Vol. 3, pp. 65–65�.

Kuffner, J., & LaValle, S. �2000, April�. RRT-connect: Anefficient approach to single-query path planning. Pro-ceedings 2000 IEEE International Conference on Ro-botica and Automation, San Francisco �Vol. 2, pp. 995–1001�.

Labayrade, R., Aubert, D., & Tarel, J.-P. �2002, June�. Realtime obstacle detection in stereovision on non flatroad geometry through “v-disparity” representation.Intelligent Vehicle Symposium, 2002, IEEE, Paris,France �Vol. 2, pp. 646–651�.

Lages, U. �2004, June�. Laser sensor technologies for pre-vent safety functions, ATA EL 2004, Parma, Italy.

Ozguner, U., Redmill, K.A., & Broggi, A. �2004, June�.Team TerraMax and the DARPA Grand Challenge: Ageneral overview. Intelligent Vehicles Symposium,2004 IEEE, Parma, Italy �pp. 232–237�.

Rosenblatt, J. �1997�. DAMN: A distributed architecture formobile navigation. Doctoral dissertation �Tech. Rep.CMU-RI-TR-97-01�. Pittsburgh, PA: Carnegie MellonUniversity, Robotics Institute.

Whittaker, W. �2005, August�. The Red Team technical pa-per. DARPA for the Grand Challenge 2005.

Figure 19. PLS operating autonomously near Barstow,CA.



the terramax autonomous vehicle - unipr.it · vehicle to negotiate tighter turns without needing...

Documents