robot-assisted urban search and rescue at the wtc disaster

Robot-Assisted Urban Search and Rescueat the WTC Disaster

Prof. Robin MurphyDirector, Research CRASAR

also Associate Prof., Computer Science & Eng.

University of South Florida [email protected]

John Blitch, Jenn Casper, Mark Micire, Brian Minten www.crasar.org

mailto:[email protected]

http://www.crasar.org/

Feb 19, 2002 Robot-Assisted Search and Rescue at WTC Disaster (NSF) 2

NBC Today Show Sept 20, 2001


Outline

• Why Robots?

• What should robots do?

• What did they do?

• What is needed?


The First Known Robot-Assisted USAR• CRASAR stood up by John Blitch Sept. 2 to transfer military robots

for USAR: deploy, train, evaluate, & research robots for SAR• At WTC, CRASAR robots and personnel worked with FDNY, NYC,

and FEMA teams such as INTF-1, PATF-1, VATF-1, OHTF-1

Duration Tuesday, Sept 11 – Oct. 2

Victims found 10+

CRASAR

Teams/people

Defense Advanced Projects Research Agency (1), US Dept. of Defense (2), Foster-Miller (5), iRobot (8),

US Navy SPAWAR (3), University of South Florida (4)

Robots on site 17, 7 used on rubble pile

Field excursions through 9/22…

11 (8 on rubble pile with depths of 20-45ft)

1 min set up, avg. 7 minutes per run

USF brought 6,2 from NSF

USF team loggedmost hours on

The Pit in 1st 2 weeks(except Blitch)

USF was onlyUSAR certified team


photos courtesy of Justin Reuter


Why Use Robots?

• Things that humans can’t do or can’t do safely– “the human use of

humans” Norbert Wiener– and that applies to dogs

• 135 rescuers died Mexico City, 65 in confined spaces

• Not enough trained people– 1 survivor, entombed: 10

rescuers, 4 hours

– 1 survivor, trapped/crushed: 10 rescuers, 10 hours

photos courtesy of Justin Reuter


FEMA

DoD Liaison

Plan/Ops Safety

Haz Mat

Tech Search

2 CanineSearch Spec.

Search Team

2 Rescue Squads12 Rescue Spec.

Rescue Team

2 MedicalSpecialists

Medical Team

Logs Spec.

Comm. Spec.

PIO

Rigging

StructureSpecialist

Logistics Team

Task Force Leader

Local Jurisdiction

ESF9 Tasks: Search, Assessment, Medical

FEMA Task Force Organizational Chart

photo courtesy of Justin Reuter


Haz Mat

Tech Search

2 CanineSearch Spec.

Search Team Rescue Team Medical Team

StructureSpecialist

Logistics Team

Task Highlights

Rescue Phase9-11 through 9-21

worked with FDNY, FEMAfound 6+ victims

(equivalent to FLTF-2)

Rescue Phase9-11 through 9-21

worked with FDNY, FEMAfound 6+ victims

(equivalent to FLTF-2)

not usednot used

Recovery Phase9-23 through 10-02

NYC DDC engineersfound 3-5 victims

added HazMat sensors

Recovery Phase9-23 through 10-02

NYC DDC engineersfound 3-5 victims

added HazMat sensors

neededneeded


Search Task Priorities

• Rescue first, recover later– first responders (always take care of

your own)– civilians in rubble pile– civilians elsewhere

• Information for triage– where are the survivors?– where are the people likely to be?– where are the survivors likely to

have survived?– which pile of rubble do I remove

first?

• Typical resources– dogs, search cams, acoustics,

sledgehammers

SearchCam:camera on a pole

dogs:injured by sharp metal,smell only 0.3m due torain on 2nd day, lackof circulating air


Robots Compared with SearchCams

• SearchCams ~$10K, Robots ~$12K

• SearchCams reach up to 5 meters

• Robots reach up to 30 meters, averaged between 6 and 13 meters

• Robots can put light on the object, prod it, look at it from different angles

• Robots can go through more twisting tunnels

• SearchCams and small robots take <1.5 minutes to set up and insert

helped to recognize remains of a body

camera with zoom, lights couldn’t do this


History of Robot-Assisted USAR in USA


Timeline of Response(Rescue and Recovery, Inspection not shown)

BlitchFoster-Miller iRobot

JPO SPAWARUSF

11 12 13 14 15 16 17 18 19 20 21 22T W Th F S Su M T W Th F S

MooreCiholas/Alibozek

PrattScheinSmith

BlitchMangoldsHaglundMouruFrost

MooreCiholas

MurphyCasperMicireMinten

EverettBaruchLaird

LevanHudson

FrostPratt

Norman

MangoldsMouruFrostPratt

Norman

MangoldsMouru

MurphyCasperMicireMinten

Inuktuns

SolemU

11 4S DARPAUSF

Foster-Miller

rubble pile(The Pit)

Talon Packbot

UrBot

Foster-MilleriRobot

SPAWAR

collateralbuildings


Where The Robots Were in the Rubble Pile

New York Times 9/23/01

16

7

23

4

5

9/18/01

0-live4-dead

VATF-1

9/16/01

0-live

INTF-1

9/12/01

0-live1-dead

9/12/01

0-live

9/13/01

0-live1-dead

FDNYFDNYFDNY

Not shown:PATF-1OHTF-1NYC


Environment at Ground Zero

Personal Safety:-Thick dust for days - asbestos, glass -Rubble largely stable, but if fall, could be impaled - 1 injury: fell out of chairWeather:-45-70 deg F-rain 2 days, making rubble slick, slippery

Personal Safety:-Thick dust for days - asbestos, glass -Rubble largely stable, but if fall, could be impaled - 1 injury: fell out of chairWeather:-45-70 deg F-rain 2 days, making rubble slick, slippery


Contextual View: WTC 2


Confined Space using Tethered Bots

• 2 types (chemical & sewer, HVAC inspection, TRL 9 but not for USAR)– “polymorphic”: ~60 lbs, 7 hours

battery, fits in 1 backpack, $12K• height of mouse to height of squirrel

– “fixed geometry”: ~70 lbs., 7 hours battery, fits in 1 backpack, $10K

• Tethers: 100-300ft• Terrain & Environment

– voids ~6-12 inch diameter– dirt, rubble (but not mounds of

paper)– inclines: depends

• up steep, go with heavier microTracks

– water resistant– not intrinsically safe, can melt tracks

• 2 types (chemical & sewer, HVAC inspection, TRL 9 but not for USAR)– “polymorphic”: ~60 lbs, 7 hours

battery, fits in 1 backpack, $12K• height of mouse to height of squirrel

– “fixed geometry”: ~70 lbs., 7 hours battery, fits in 1 backpack, $10K

• Tethers: 100-300ft• Terrain & Environment

– voids ~6-12 inch diameter– dirt, rubble (but not mounds of

paper)– inclines: depends

• up steep, go with heavier microTracks

– water resistant– not intrinsically safe, can melt tracks

Inuktun microTracvideo, 2 way audio

Inuktun microVGTV,Inuktun pipe crawlervideo, 2 way audio

www.inuktun.com


Needed Image Processing and Object Recognition Technologies

-tracks of previous robot run -a watch-3, possibly 4 victims (covered in dust, burned) - head, arm socket?, nose, perhaps fingers

WTC 2, 9/18/01


State of Available Information

• what viewpoints have already been explored– example: boot?

• no one rewound tape far enough back to catch earlier viewpoint which disambiguated the object!


Confined Space, Structural Assessment using Wireless Bots

•1 type (military hostage rescue, special order, TRL 9 but not for USAR)– backpackable by 2 people, 1 for bot, 1

for OCU & batteries, 12 hours-7 days (standby)

– fast, can right itself with practice – can add sensors, payloads– ~$30K

•Wireless– depends on material

•Terrain & Environment– voids: mansize– dirt, rubble,inclines: depends– water resistant– not intrinsically safe

•1 type (military hostage rescue, special order, TRL 9 but not for USAR)– backpackable by 2 people, 1 for bot, 1

for OCU & batteries, 12 hours-7 days (standby)

– fast, can right itself with practice – can add sensors, payloads– ~$30K

•Wireless– depends on material

•Terrain & Environment– voids: mansize– dirt, rubble,inclines: depends– water resistant– not intrinsically safe

www.foster-miller.com

FM Solemvideo, striper, audio


Need Sensing of State of the Robot, World

• Need state of robot– pose, size relative to the

environment– health

• some UIs display• fault detection is hard• diagnostics and recovery is slow

(replace or 35 minutes)

• State of World– topological vs. metric– 3D mapping– 3D interpretation

• video overlay• structural reasoning

robot projected a grid to estimatedistance

WTC 4, 9/16/01


Search and Structural Assessment of Collaterally Damaged Buildings

• 3 types (military hostage rescue, special order research, TRL 4-6) ~$30-45K

– backpackable by 2 people, 1 for bot, 1 for OCU & batteries, 12 hours (UrBot), 20min (PackBot)

– fast, stairs, grasping

– self-righting or invertible

– can add sensors, payloads, software

• Wireless

• Terrain & Environment– buildings, large voids, 3 story drops

– dirt, rubble,inclines, stairs

– Packbot is water proof

– not intrinsically safe

• 3 types (military hostage rescue, special order research, TRL 4-6) ~$30-45K

– backpackable by 2 people, 1 for bot, 1 for OCU & batteries, 12 hours (UrBot), 20min (PackBot)

– fast, stairs, grasping

– self-righting or invertible

– can add sensors, payloads, software

• Wireless

• Terrain & Environment– buildings, large voids, 3 story drops

– dirt, rubble,inclines, stairs

– Packbot is water proof

– not intrinsically safe

www.spawar.navy.mil, www.irobot.com

FM Talonvideo, audio, gripper

iRobot PackBotvideo, FLIR, 2 way audio

SPAWAR UrBotvideo, audio

http://www.spawar.navy.mil/

http://www.irobot.com/


General Mobility & Hardware Problems• Lost 1 Solem robot

– lost wireless comms, left in hole, wasn’t there later

• Damage– detracked once (high heat)– speared on rubble– just wear and tear

• Tethers tangle– only twice not immediately recoverable

• 7.75 “stuck assists” per drop (or once a minute)

– but tether handling is significant• 9.25 “gravity assists” per drop

– still have tie a rope around the wireless ‘bots

• Wireless and dropouts– can’t compress and do image processing– too numerous/duration to count– make it hard to do image proxy processing

• Fear of flipping and getting stuck


HRI Issues Overall

• Warning: camera occlusion 12.3% of a drop– teleop doesn’t work well in those cases!

• Operator errors (Norman ‘91)

– Mistakes• 2: wrong robot, had to remove and try another

– microTracks bulldozes in deep dirt, can’t climb

– microVGTV too light to get traction

• 10% of duration of Inuktun runs spent significant time adjusting lights despite auto gain

– need image enhancement

– Slips• 0.25 collisions per drop (oversteer)

• 8.9% of duration, robot wheel slip (high centered, wrong configuration)

• Human-Robot Interfaces scared end-users off


iRobot: PackBot (experimental)

• game joystick plus laptop with video & audio• robot state: battery, comms, orientation, camera, encoders• scared off rescuers: too complicated, too long to train

iRobot PackBotvideo, FLIR, 2 way audio

2 people, 35 kg


9:00PM

11:00PM

1:00AM

3:00AM

5:00AM

7:00AM

65min

~355min

~6.5min

~6.5min

35min

~5min

~105min

~30min

1. Drive to Ground Zero from Javits Center

2. Parts of 9 member group splits off while the rest waits

3. 2 operators, 2 robots and VATF-2 search void; MicroTrac failure

4. 2 operator, 2 robots and VATF-2 search same void; VGTV failure

5. 2 members retrieve spare robot from Javits Center

6. 2 members fix 2 robots using parts from spare robot

7. 9 member group moves to new location to wait

8. Group returns to Javits Center from Ground Zero

12:00AM 2 operators return with robots after VATF-2 retreats

In route

Waiting

Searching

Timeline of Shift w/ VATF-2 (7:00PM 9/18/01 – 7:00AM 9/19/01)

• 13 minutes for entire shift (green)• No drop >7 minutes• <1.5 minute set up time or rescuers walk away• Found 3+ victims


Other problems: Asynergism

• Lack of interoperability / “on the other robot”– Image processing, intelligent assistance

– Software, sensors couldn’t be migrated between platforms (“plug and perceive”)

– Only the “dumbest” robots were well-used

• Known capabilities “lost” or “back in the lab”• robots as wireless repeaters

• self righting

• self reacquistion of comms signal

• general office navigation, obstacle avoidance

Demo, hardware focus, rather than Systems focus


Problems as Point of Departure for Theory• USAR robots can be generic, adapted in the field for the

particular task, thought of a system• Sensors and sensing need to be improved, miniaturized, $$

– State of robot: highly sensate robots, health monitoring– State of world: 3D maps plus understanding structure– State of available information

• Mobility and hardware need to be improved– Polymorphic and elephant trunk sensors masts– Teams of robots

• marsupialism for delivery, relay, proxy processing• physically coupled: tether managers, in-line collaborative teleoperation• distributed: hybrid tethers

• Human-robot Interaction needs to be improved– Perceptual user interfaces not well-defined, mapping & human models of

situational awareness– Training and access is an issue; over the internet


Robot-Assisted USAR as IT

HOSPITAL

Forward StationSection Chief

WARM ZONEBase of OperationsTeam Task Leader

personnel prep

HOT ZONESearch Teams

COLD ZONECommand Post& Staging Area

Incident Commander, Structural Engineers,

Robot Specialists

NEW-TECHLABS

UNIVERSITY

DECIDABILITY

CONNECTIVITY

USABILITY

MOBILITY

ADAPTABILITY

PERCEPT-IBILITY

CRASAR, GOVT LABS


Summary of Performance

• Robots were successfully used at the World Trade Center – Success: quick response (luck)

– effective performance • 12/11 equal to manual technical search for 24/10 days

• user acceptance– Training in Dec. 02 for East Coast firefighters

– Training in Apr. 02 at FDIC in Indianapolis

– ~30 copies of data sets to fire rescue teams to date

• USF as a university provided expertise from field research, managed data collection (and pushing data dissemination)


However, We Could’ve Done More…

• Had more robots, sensors, and people than were utilized due to credentialing, organizational issues– Lost days 3,4

– Never saw a 24/7 deployment cycle

– Only 75% of the available Inuktuns were deployed at any given time, despite effectiveness

– Buildings could have been surveyed, if necessary, stored on videotape

• Capabilities never exploited– Robots could place tubing to transport air/water/meds to survivors

• Dr. Eric Rasmussen, 3rd Fleet Surgeon, sent medical equipment

– Added air quality monitors on larger robots after 9/25

– AI Software on larger robots not interoperable with smaller and newer robots


more information and public video & stills: [email protected]

Robots Can’t Replace the Real Heroes

http://www.crasar.org/

mailto:[email protected]

robot-assisted urban search and rescue at the wtc disaster

Documents

search cams

urban search

wtc disaster nsfthe

wtc disaster nsfrobots

wtc disaster nsfnbc

wtc disaster nsfwhy

sarat wtc

wtc disasterprof