a city driving robotic car named “tommy jr.” · a city driving robotic car named “tommy...

A City Driving Robotic Car Named “Tommy Jr.”

Paul J. PerroneCEO – Perrone RoboticsTeam Lead – Team Jefferson

TS-5925

2008 JavaOneSM Conference | java.sun.com/javaone | 2

Learn about a robotic car in a historic city driving race and the Java™ technology that got him there


Agenda

The challengeTommy Jr’s partsTommy Jr’s brainsTommy Jr’s journeyTommy Jr’s advancementConclusions


The DARPA Grand Challenge and Tommy

175+ mile race through desertNo driver onboard or in remote control• Completely Autonomous Ground Vehicles (AGVs)

$2M prizeApplications Video Site Visit NQEs/DGC“Tommy” is born and advances100% Java technology based bot• Java SE platform for navigation & obstacle avoidance• Java ME platform for controls


The DARPA Urban Challenge and Tommy Jr.

60+ mile through a cityAGV city driving• Navigation & obstacle avoidance• Plus all rules involved with city driving

$2M/$1M/$500K prizes$1M grants (“Track A”) & Indies (“Track B”)Applications Video Site Visit NQEs/UCE“Tommy Jr.” is born and advances – Track B100% Java technology based bot• Java SE technology & Java ME technology again• Java technology RTS for real-time nav controls• SunSPOT safety controller


Agenda



Automotive Platform: Affordable & Practical

Scion xB• Used 2004 = “affordable”• Small footprint = “smaller collision surface”• Navigable = “more responsive collision avoidance”• Boxy = “plentiful/easier external mounting”

+ “plentiful/easier internal mounting”Actuation integration & drop-in• Automatic/shifter actuator• Cruise control drop-in• Brake actuator• Steering motor

Tapped• Throttle position• Speedometer• Turn signals


Hardware: COTS & Java technology capableEmbedded x86• All of TJr’s decision making

and sensor fusion• Java SE technology &

Java RTS technology • MAX Standard & Real-Time

Desktop-grade x86• Front-end UI to TJr• Stereo camera processing• Java SE technology;

MAX StandardJStamp micro-controllers• Feedback control of motors• Java ME technology;

MAX MicroSunSPOT• Safety controller• Java ME technology; MAX Micro

Misc: redundant I/O controller, motor drivers, etc…

LAN

RADAR

FeedbackEncoders

VehicleController

Motor Drivers

Motor Motor Motor Motor

Steering Shifting Throttle Brake

INSSensing

GPSPositioning

SideLADARsLADARs

Camera

VehicleState

Sensors

VehicleState

Sensors

Electronics Interface

VehicleController

WatchdogController

E-stop Electronics

IgnitionEbrake

AuxProcessor

ESTOPReceiver

MainProcessor

UserInterface


Sensors: COTS & Precision

GPS – precision positioning• Standard and differential GPS• Hi-precision GPS – 2 to 4 inches

INS – Inertial Navigation System – precision orientation• Heading, pitch, roll• GPS backup

LADAR – laser range sensors – obstacle detection• SICK LMS sensors – one of our strongest capabilities• Detection ahead, on sides, and behind• Virtual “curb feeler”• Low hanging obstacle detection – last minute

Stereo camera – lane detection• Rapid integration• Incomplete usage

RADAR – long range warnings• Integrated, minimal utility

LAN

RADAR

FeedbackEncoders

VehicleController

Motor Drivers

Motor Motor Motor Motor

Steering Shifting Throttle Brake

INSSensing

GPSPositioning

SideLADARsLADARs

Camera

VehicleState

Sensors

VehicleState

Sensors

Electronics Interface

VehicleController

WatchdogController

E-stop Electronics

IgnitionEbrake

AuxProcessor

ESTOPReceiver

MainProcessor

UserInterface


Software: Layered Platform

Java platform, Java platform, Java platform• Java RTS API, Java SE API, Java ME API, Java SunSPOT API, Java

Comm APIMAX Robotics Platform• Real-Time, Standard, and Micro profiles• General purpose rapid integration platform• Many available drivers and libraries

MAX Sensor Drivers• GPS, INS, LADAR, Camera drivers

MAX Actuation Drivers & Frameworks• Motor & feedback controls• Analog/digital I/O

MAX-UGV• UGV framework• Extensible/configurable navigation maneuvers• Extensible/configurable obstacle sensation

Microprocessor/Micro-controller

OS/Runtime

Java

MAX

MAX-Drivers

MAX-UGV

“Tommy Jr.”


Agenda



Main Processing Loops

Route planning (asynch)GPS & INS position & orientation sensing (asynch)LADAR sensing and obstacle detection (asynch)Camera sensing and lane detection (asynch)Navigation maneuvers (synch)Navigation controls (synch)Feedback controls (synch)Safety controls (synch)

GPS

INS

LADAR

Camera

ObstacleDetection

LaneDetection

NavigationControls

NavigationManeuvers

FeedbackControls

SafetyControls

RoutePlanning


Navigation Maneuvers

More driving rules and complexity• More rules for city driving (i.e. more maneuvers)• More complex steps for maneuvers (i.e. more maneuver actions)• Added navigation maneuver framework to MAX-UGV

Detection of maneuvers• Look-ahead for imminence• Activate/de-activate, pause/resume

Maneuver prioritization & exclusion• Maneuver precedence/priority

Maneuver actions• Configurable actions• Sequence of steps

Navigation maneuvers• Intersections, passing, merging, parking, zones, …


Parking Maneuver Sample

public class Parking extends ManeuverGeneric {

/** Detect imminence of maneuver and enter as appropriate **/ public boolean detectIfImminent() { … Course course = super.getMovementPlanner().getCourse(); WaypointProximity nextSpot = course.getWaypoint(NextParkingSpotArrivalPointID); startSpot = nextSpot.getWaypoint();

if(nextSpot.getDistance() < minDistanceForSpotApproach){ super.getMovementPlanner().setSpeed(spotApproachSpeed); if(!parking){ if(nextSpot.getDistance() < minDistanceForStartParking){ arrivalSpot = startSpot; initParking(); // Starts parking maneuver (parking = true) } } … }

Copyright 2001-2008. Perrone Robotics, Inc. All rights reserved.


Parking Maneuver Sample – Cont’d

// Configurable parking actions// Start approach to spotApproachSpot// Start slow down as approach spot (pump brakes, timeout, slow speed)BrakePumpAction// When within x meters of checkpoint, stopApproachCheckpoint // Once speed below certain mph and certain timeout, start applying reverseWaitForCheckpointHalt// Wait for reverse shift, then give a little gas, invert steeringWaitForReverse// Invert steering, backup toward staging point until reachedBackup// Wait, until stop. WaitForBackupHalt // Clear out those spot points that are part of route (i.e. make sure considered visited)// Then proceed out of maneuver (release steering override)WaitForDrive



Parking Maneuver Sample – Cont’d

public class ApproachCheckpoint extends ParkingAction {

public boolean execute() { if(getParking().reachedCheckpoint()){ super.getMovementPlanner().overrideSteering(); super.getMovementPlanner().setSteeringAngle(0.0); super.getMovementPlanner().setSpeed(0.0); reachedCheckpoint = true; return true; }else{ reachedCheckpoint = false; super.getMovementPlanner().setSpeed(pullInApproachSpeed); return false; } }



Obstacle Detection Maneuver

More perspectives, obstacles, and degrees of freedom• More sensor perspectives (front, sides, rear)• Moving and static obstacles• Multiple lanes and routes• Embodied obstacle reactions as MAX-UGV maneuvers

Obstacle detection• Enhancements to LADAR-based detection

Obstacle resolution• Moving obstacle additions• Had size, position, and confidence• Add speed and direction

Obstacle avoidance maneuvers• Lane collision, zone collision, passing, backup, parking…


Passing Another Vehicle Sample

public class Passing extends ManeuverGeneric {… public boolean detectIfActive() { boolean pass = passing || stoppedFollowingDuringPass || (getMovementPlanner().getClosestUnpassableDistance() <= closestDistanceForStopWhenUnpassable) || isReroutingDuringPass(); return pass; }

public boolean detectIfImminent() { boolean immin =

(getMovementPlanner().getClosestUnpassableDistance() <= closestDistanceForImminentPass)|| isReroutingDuringPass();

return immin; }}



Passing Another Vehicle Sample – Cont’d

// Initial passing impasse detection and beginning to slow downInitialImpasseStopAction// Wait for car to slow down (may exit out of maneuver in meantime)WaitForPassingSlowDownAction// Wait for car to come to complete halt (may exit out of maneuver)WaitForPassingStopAction// Wait for an opening, timeout, and begin passingWaitForOpeningAndPass// As pass, continue to check passage way, merging as needed, exit on completionHandleExtendedMerge



Feedback Controls

Navigation controls• Derived steering, speed, and shift commands

Feedback controls• Steering position control• Braking level & position control• Throttle level & position control• Shift position control

Closed-loop PID controls (actuator positioning)Speed controls (combined brake/throttle levels)


Steering Control Sample

public class SteeringControlPlan extends PlanGeneric{

public void init(){ commandSensor = (CommandSensor) super.getSensor(COMMAND_SENSOR_ID); steerControl = (FeedbackControl) super.getComponent(STEER_FEEDBACK_COMPONENT_ID); steeringFeedbackSensor = (Sensor) super.getSensor(STEERING_FEEDBACK_SENSOR_ID); steeringMotor = (Actuator) super.getActuator(STEERING_MOTOR_ID); steerControl.registerSensor(steeringFeedbackSensor); steerControl.registerActuator(steeringMotor); }

public void trigger(){ // Get commanded steering value SensorState state = commandSensor.getSensorState(); double steerValue = state.getDouble(DESIRED_STEERING_VALUE_ID);

// Set desired steering value steerControl.setDesiredPosition(steerValue); }}



Fail-Safety

Safety Risk = high energy + grand challenge + little time + little money

Back Story• Tommy Sr. smarting• E-stop of little use• Safety-critical systems knowledge

Focus on speed controls• Redundant speed sensing (2 speed inputs, 2 GPS speeds)• Redundant throttle position sensing (2 inputs)• Redundant speed calculations• Fail-safe controls

Watchdog safety• Monitor processing• Monitor sensors• Monitor other controllers• Cut throttle, stop car if issue


Agenda



Tommy

Desert-ready robotic dune buggy$60K – 10 man-months software – 1 PC (Dec ’04 to Sept ’05)100% Java technology bot – decisions AND micro-controlMAX for rapid integrationMAX-UGV for navigation & obstacle avoidanceBuilding upon this software platform


Drop-In: Car + Actuators + Hardware + Sensors

Four man-month (one actual month) drop-in (Jan ’07)Power electronics: Alternator, batteries, isolation, convertersActuators: braking, steering, throttle, shifting, E-brakeActuator electronics: Motor drivers, relays, switchesFeedback: signals, encoders, electronicsE-stop electronics: Commercial E-stop, government E-stopSensors: GPS, INS, LADAR, Stereo Camera, RADARRapid after market drop-in approach


Base Software Platform Configuration

One week to fully autonomous vehicle (Feb ’07)Java technology & MAX installsBasic MAX parameter configurationsCalibration & tuningAutonomous navigation & obstacle avoidanceProven generic UGV framework with rapid configuration


Rules Creation and Evolution

Route planning: course maps (RNDF) + mission checkpoints (MDF)Passing vehiclesStopping at intersections & obeying precedence orderQueuing and following behind vehicles3-point U-TurnsProper turn signalingMergingOpen zone/lot navigationParking in parking spotsLane collision avoidanceGround obstacle avoidance (ditches, curbs)Proximity collision avoidanceRoute re-planningGPS outage handlingLane detection & road followingLow hanging obstacle avoidanceEight months of rule creation and testing (Mar ’07 to Oct ’07)


Loose Ends

Dashboard UI: manage & test TJrSimulation: test new rulesCourse maker: plot and load test coursesSafety: fail-safe, don’t rely on E-stop (controlled by DARPA)People: engineering & operationsParts: sensors, car, hardwareFacilities: basic testing, demos, large-scale tests, CA testsTravel: TJr legal to human drive in VA!, trailer to/from CAFinances/Marketing: sponsorship, logos, demos, TV, news


Agenda



Applications, Sponsorships, & Kool-Aid

March 2006: Announcement• $2M/$1M/$500K prizes; $50K to all qualifiers, $100K to all finalists

April 2006: Team Jefferson back• Slam Dunk Plan ($$$): $800K• Winning Chance Plan ($$): $400K• Austerity Plan ($): $90K

Summer 2006: DARPA Kick-Off• No prize money• Vehicles required to have stock chassis – can’t reuse Tommy Sr.

Oct 2006: Track A awards• $1M up front to 11 teams; mostly large teams

Oct 2006: Track B applicants• 89 teams total (11 Track As, 78 Track Bs)• Interest down from 2004 & 2005

Jan 2007: Prize adjustments• Prize money reinstated; qualifier money not reinstated

Jan-Feb 2007: Team Jefferson Accelerates• Sponsorship support• Mechanical/electrical engineering accelerates


Video Submission & Site Visit

March 2007: Video Submission• Demo safety, self-navigation, and passing• 1 week SW configuration + few weeks new passing rules testing• 53 teams advance to site visit (11 Track As, 42 Track Bs)• Tommy Jr. advances• Targeting austerity plan

June 2007: Site Visit Demonstration• Demo safety, self-nav, passing, intersections, route planning,

queuing, U-turns, turn signaling• Demo in front of 3 DARPA officials• 35 teams advance to qualifications (10 Track As, 25 Track Bs)• Tommy Jr. aces & advances – projected among top ranked visits• Austerity plan goals close


Leaving for the Race

October 2007: Leaving Virginia for California• Tommy Jr in trailer, RV, and 3 team members• 3 day – 2 night trip fueled by caffeine & Java technology • Austerity goals met, partial way to winning chance target• More testing required:

• Lane detection, route re-planning, ground obstacles• Low-hanging obstacles?

• Completed:• Self-navigation, route planning, lane collision avoidance, intersections,

merging, queuing, passing, zones, parking, U-turns, signaling, proximity and zone collision avoidance, GPS outages, fail-safety

• Development & test enhancements• Operations & marketing enhancements


The National Qualifiers

Day 1: Teams given course maps on USB stix w/their team nameDay 2: E-stop test• Passed, DARPA presses wrong buttons

Day 3: Circular loop with intersections and re-routing• TJr. navigates intersections and U-turns, re-routes• TJr clips low-hanging railway crossing arm, no E-stop issued• Other teams w/issue, railway crossing arm ruled illegal for course

Day 4: Circular loop with intersections and merging• TJr. navigates intersections and obeys precedence and merging

rules• TJr. goes wide on a sharp turn in course map file• Other teams w/issue, we re-plan for course


The National Qualifiers - Continued

Day 5: Large course representative of city• TJr. navigates well initially• TJr. considers re-routing, course map points clip K-barriers• Tjr. restart, begins re-route, clips barrier, sensor damage, no E-stop

Day 6: Second run of loop w/intersections & re-routing• Team prepared for low-hanging obstacle, but crossing arm removed• Sensor damage causes TJr to detect ground and re-route• Team goes off to prep for Day 7• Gets call back early to run another course, team inquiry• Team gets call from DARPA director that is disqualified, team appeal

Day 7: Second run of loop w/intersections & merging• Team prepared to complete its second runs• DARPA director denies appeal• Urban Challenge Program Manager tries to help, decision is final


The Final Event

11 teams advance to final event (7 Track As, 4 Track Bs)DARPA director selects the teams, claims safety basisSix teams complete the course (5 Track As, 1 Track B)Prize Winners:• 1st Place: Carnegie Mellon – General Motors team (Track A)• 2nd Place: Stanford Univ – Volkswagen team (Track A)• 3rd Place: Virginia Tech – CAT team (Track A)


Tommy Jr. VideoPerrone Robotics, Inc.


Agenda



Reflections on the Journey

What worked well• Extraordinarily large function base generated per unit time• Building on Java technology and a framework definitely way to go

What didn’t work well• Gaps in lane detection & route re-planning – time runs out

Issues with race operations• Rules incomplete and not-specified• Decisions made by director vs committee• E-stop safety not employed

Lessons learned from race• Design for generalized vs specific rules ($$$)• Design for dynamic vs specified rules ($$$)• Design for more fail-safety scenarios paramount ($$)


Reaping the Rewards

Designs for use beyond challenge• High profile race & technology• Have generalized Java technology based robotics & automation

platform• Have generalized Java technology based UGV platform

Awareness generated• TV, news, trade• Hi-profile demonstrations• Sponsors benefit before, during, & after race

Applying to the real-world• Distributed sensor applications for automation• Platform and framework licensing• Commercial UGV applications• Vendor/sponsor technology used and licensed


Tommy Junior’s Sponsors

Sun Microsystems – Java technology, advisors, financialPerrone Robotics – MAX, software/systems/operations, financial

University of Virginia – mechanical/electrical

Northrop Grumman - financialAutoTrader - financial

WowWee Robotics - financial, toy botsBDMetrics – financialSICK - lasers45Fix - financialTDFund – financial


Paul J. PerroneCEO – Perrone Robotics www.perronerobotics.comTeam Lead – Team Jefferson www.teamjefferson.com

TS-5925

a city driving robotic car named “tommy jr.” · a city driving robotic car named “tommy...

Documents