Scaling Human Robot Teams

Prasanna VelagapudiPaul Scerri

Katia SycaraMike Lewis

Robotics InstituteCarnegie Mellon University

Pittsburgh, PA

Large Multiagent Teams

• 1000s of robots, agents, and people

• Must collaborate to complete complex tasks

Large Multiagent Teams

Large Multiagent Teams

• Network Constraints

Large Multiagent Teams

• Human Information Needs

Network Constraints

• Networks affect human interface design– Limited bandwidth– Significant latency– Lossy transmission– Partial/transient connectivity

Network Constraints

• How can we design robust tasks?– Feasible under network constraints– Tolerant of latency– Within bandwidth constraints– Robust to changes in information

Network Constraints

• Humans are a limited resource

Network Constraints

• Humans are a limited resource– Centralized, expensive– Limited attention and workload– Penalties for context switching– Necessary for certain tasks

• Complex visual perception• Meta-knowledge

Network Constraints

• How do we maximize the effectiveness of humans in these systems with respect to network constraints?

MrCSMulti-robot Control System

MrCSMulti-robot Control System

Waypoint Waypoint NavigationNavigation

TeleoperationTeleoperation

Video/ Video/ Image Image ViewerViewer

Status Status WindowWindow

Map Map OverviewOverview

Victims Found in USAR Task

Number of

Victims

[Velagapudi et al, IROS ’08]

Task decomposition[Velagapudi et al, IROS ’08]

Network Constraints

• How we divide tasks between agents may affect performance– What is the best way to factor tasks?– Where should we focus autonomy?

Large Multiagent Teams

• Human Information Needs

Human Information Needs

• Human operators need information to make good decisions

• In small teams, send everyone everything

• This doesn’t work in large systems

Human Information Needs

• Sensor raw datarates – Proprioception

• < 1kbps

– RADAR/LIDAR• 100kbps – 20Mbps

– Video• 300kbps – 80Mbps

Human Information Needs

• Can’t transmit every bit of information– Selectively forward data

• How do agents decide which pieces of information are important?

– Fuse the data• What information are we losing when we fuse

data?

Asynchronous Imagery

• Inspired by planetary robotic solutions– Limited bandwidth– High latency

• Multiple photographs from single location– Maximizes coverage– Can be mapped to virtual pan-tilt-zoom camera

Asynchronous Imagery

• Streaming Mode • Panorama Mode

Panoramas stored for later viewingPanoramas stored for later viewingStreaming live videoStreaming live video

[Velagapudi et al, ACHI ’08]

Victims Found[Velagapudi et al, ACHI ’08]

Average Average # of # of

victims victims foundfound

Accuracy ThresholdAccuracy Threshold

11

22

33

44

55

66

Within Within 0.75m0.75m

Within 1mWithin 1m Within 1.5mWithin 1.5m Within 2mWithin 2m00

PanoramaStreaming

Environmental Factors

• Colocated operators get extra information– Exocentric view of other agents– Ecological cues– Positional and scale cues

Conclusion

• Need to consider the practicalities of large network systems when designing for humans.

• Need to consider human needs when designing algorithms for large network systems.

Our Work

Cognitive modeling

• ACT-R models of user data

• Determine– What pieces of information users are using?– Where are the bottlenecks of the system?

Environmental Factors

• Colocated operators get extra information– Exocentric view of other agents– Ecological cues– Positional and scale cues

Utility-based information sharing

• It is hard to describe user information needs

• Agents often don’t know how useful information will be

• Many effective algorithms use information gain or probabilistic mass

• Can we compute utility for information used by people

MrCSMulti-robot Control System

MrCSMulti-robot Control System

Waypoint Waypoint NavigationNavigation

TeleoperationTeleoperation

Video/ Video/ Image Image ViewerViewer

Status Status WindowWindow

Map Map OverviewOverview

Victims Found

Number of

Victims

Task decomposition

NavigationNavigation

SearchSearch

Task decomposition

Asynchronous Data

• One way to address the latency of networks is to transition to asynchronous methods of perception and control.

• Asynchronous imagery– Decouples users from time constraints in

control

Asynchronous Imagery

• Inspired by planetary robotic solutions– Limited bandwidth– High latency

• Multiple photographs from single location– Maximizes coverage– Can be mapped to virtual pan-tilt-zoom camera

Asynchronous Imagery

• Streaming Mode • Panorama Mode

Panoramas stored for later viewingPanoramas stored for later viewingStreaming live videoStreaming live video

Victims Found

Average Average # of # of

victims victims foundfound

Accuracy ThresholdAccuracy Threshold

11

22

33

44

55

66

Within Within 0.75m0.75m

Within 1mWithin 1m Within 1.5mWithin 1.5m Within 2mWithin 2m00

PanoramaStreaming

Tools

• USARSim/MrCS

• VBS2

• Procerus UAVs

• LANdroids

• ACT-R

USARSim

[http://www.sourceforge.net/projects/usarsim]

• Based on UnrealEngine2

• High-fidelity physics• Realistic rendering

– Camera– Laser scanner

(LIDAR)

MrCSMulti-robot Control System

MrCSMulti-robot Control System

Waypoint Waypoint NavigationNavigation

TeleoperationTeleoperation

Video/ Video/ Image Image ViewerViewer

Status Status WindowWindow

Map Map OverviewOverview

VBS2

[http://www.vbs2.com]

• Based on Armed Assault and Operation Flashpoint

• Large scale agent simulation

• “Realistic” rendering– Cameras– Unit movements

Procerus UAVs

• Unicorn UAV• Developed at BYU• Foam EPP flying wing• Fixed and gimbaled

cameras• Integrated with

Machinetta agent middleware for full autonomy

LANdroids Prototype

• Based on iRobot Create platform

• Integrated 5GHz 802.11a based MANET

• Designed for warfighter networking

• Video capable

ACT-R

• Cognitive modeling framework

• Able to create generative models for testing