human robot teams: concepts, constraints, and experiments

Human Robot Human Robot Teams:Teams:

Concepts, Constraints, and Concepts, Constraints, and ExperimentsExperiments

Michael A. GoodrichMichael A. GoodrichDan R. Olsen Jr.Dan R. Olsen Jr.Brigham Young Brigham Young

UniversityUniversity

Research AgendaResearch Agenda Evaluation TechnologyEvaluation Technology

Neglect ToleranceNeglect Tolerance Behavioral EntropyBehavioral Entropy Fan-OutFan-Out

Interface DesignInterface Design Mixed Reality DisplaysMixed Reality Displays PrinciplesPrinciples HF ExperimentsHF Experiments

Autonomy DesignAutonomy Design Team-Based AutonomyTeam-Based Autonomy UAVsUAVs Perceptual LearningPerceptual Learning

The Presentation AgendaThe Presentation Agenda

The types of questionsThe types of questions Neglect tolerance: Is a team feasible?Neglect tolerance: Is a team feasible? How do we compute neglect tolerances?How do we compute neglect tolerances? Tradeoffs: workload and performanceTradeoffs: workload and performance Is a team optimal?Is a team optimal? The problem with switch costsThe problem with switch costs Some limits, ideas, and proposalsSome limits, ideas, and proposals

A Special Case: A Special Case: The Robotics SpecialistThe Robotics Specialist

One soldierOne soldier Two UAVsTwo UAVs One UGVOne UGV

Can one person Can one person manage all three manage all three assets?assets?

At what level of At what level of performance?performance?

At what level of At what level of engagement?engagement?

A More General Case:A More General Case:Span of ControlSpan of Control

How many “things” can be managed by a How many “things” can be managed by a single human?single human? How many robots?How many robots?

How do we measure Span of Control in How do we measure Span of Control in HRI?HRI? Relationships between NT and ITRelationships between NT and IT

How do we compare possible team How do we compare possible team configurations?configurations? Evaluate performance-workload tradeoffsEvaluate performance-workload tradeoffs Identify performance of feasible Identify performance of feasible

configurationsconfigurations

The Most General Case: The Most General Case: Multiple Robots & Multiple Multiple Robots & Multiple

HumansHumans How many people are responsible How many people are responsible

for a single robot?for a single robot? How many robots can provide How many robots can provide

information to a single human?information to a single human?

VehicleCommander

ICV Driver

ICV

1 CL I UAV System

Platoon Headquarters Organization

PLT LDR

Medic Robotics NCO

ARV-A (L)

1 CL I UAV System

The Presentation AgendaThe Presentation Agenda

The types of questionsThe types of questions Neglect tolerance: Is a team feasible?Neglect tolerance: Is a team feasible? How do we compute neglect tolerances?How do we compute neglect tolerances? Tradeoffs: workload and performanceTradeoffs: workload and performance Is a team optimal?Is a team optimal? The problem with switch costsThe problem with switch costs Some limits, ideas, and proposalsSome limits, ideas, and proposals

Neglect Tolerance:Neglect Tolerance:Neglect Time and Neglect Time and Interaction TimeInteraction Time

How long can the robot “go” without How long can the robot “go” without needing human input?needing human input?

How long does it take for a human to give How long does it take for a human to give guidance to the robot?guidance to the robot?

Neglect Time (NT)

Interaction Time (IT)

Fan-Out (Olsen 2003,2004): Fan-Out (Olsen 2003,2004):

How many homogeneous How many homogeneous robots?robots? How many interaction periods “fit” How many interaction periods “fit”

into one neglect periodinto one neglect period

Two other robots can be handled Two other robots can be handled while robot 1 is neglectedwhile robot 1 is neglected

Fan-out = 3Fan-out = 3

NT

IT IT IT

1

2 3 4

Can a human manage team Can a human manage team T T ? ?

Fan-out and FeasibilityFan-out and Feasibility Fan-out (homoeneous teams)Fan-out (homoeneous teams)

Feasibility (heterogeneous teams)Feasibility (heterogeneous teams)

These are upper boundsThese are upper bounds

The Presentation AgendaThe Presentation Agenda

The types of questionsThe types of questions Neglect tolerance: Is a team feasible?Neglect tolerance: Is a team feasible? How do we compute neglect tolerances?How do we compute neglect tolerances? Tradeoffs: workload and performanceTradeoffs: workload and performance Is a team optimal?Is a team optimal? The problem with switch costsThe problem with switch costs Some limits, ideas, and proposalsSome limits, ideas, and proposals

Neglect Impact CurvesNeglect Impact Curves A task is A task is NeglectedNeglected if if attention is attention is elsewhereelsewhere

Neglect impacts Neglect impacts task performance: task performance: 22ndndary tasksary tasks

T im e-o ff-tas k

R o b o t E ffec tivenes s

Teleoperation

Point-to-Point

Autonom ous

Not Neglect Tolerant Not Neglect Tolerant EnoughEnough

Too Neglect TolerantToo Neglect Tolerant

Old Glory InsuranceOld Glory Insurance

Interface Efficiency Interface Efficiency CurvesCurves

Recovery from “zero” Recovery from “zero” pointpoint

Imprecise switch costsImprecise switch costs

T im e-o n-tas k

R o b o t E ffec tivenes s

Teleoperation Point-to-point

W aypoints

Efficient InterfacesEfficient Interfaces PDA-based UAV control (versus PDA-based UAV control (versus

command line)command line)

Efficient InterfacesEfficient Interfaces Phycon-based UAV control (versus Phycon-based UAV control (versus

command line)command line)

Finding NT and IT from the Finding NT and IT from the curvescurves

ExampleExample Vary minimum performance levelVary minimum performance level MeasureMeasure

Average performanceAverage performance Neglect timeNeglect time Interaction timeInteraction time

Validation of Method: Validation of Method: ComplexityComplexity

As complexity goes As complexity goes up, NT goes down up, NT goes down and IT goes upand IT goes up

Feasibility using Feasibility using NT/IT needs more NT/IT needs more workwork

The Presentation AgendaThe Presentation Agenda

The types of questionsThe types of questions Neglect tolerance: Is a team feasible?Neglect tolerance: Is a team feasible? How do we compute neglect tolerances?How do we compute neglect tolerances? Tradeoffs: workload and performanceTradeoffs: workload and performance Is a team optimal?Is a team optimal? The problem with switch costsThe problem with switch costs Some limits, ideas, and proposalsSome limits, ideas, and proposals

Existing TradeoffsExisting Tradeoffs

Ideal

Increasing Threshold

Types of AutonomyTypes of Autonomy

Using Tradeoffs to Select a Using Tradeoffs to Select a ConfigurationConfiguration

Ideal

Ideal

Ideal

The Presentation AgendaThe Presentation Agenda

The types of questionsThe types of questions Neglect tolerance: Is a team feasible?Neglect tolerance: Is a team feasible? How do we compute neglect tolerances?How do we compute neglect tolerances? Tradeoffs: workload and performanceTradeoffs: workload and performance Is a team optimal?Is a team optimal? The problem with switch costsThe problem with switch costs Some limits, ideas, and proposalsSome limits, ideas, and proposals

Predicting Performance of a Predicting Performance of a Heterogeneous TeamHeterogeneous Team

Each robot may have multiple autonomy Each robot may have multiple autonomy modes and interaction methodsmodes and interaction methods

Each interaction scheme yields NT, IT, and Each interaction scheme yields NT, IT, and average performance valuesaverage performance values

Two interaction schemes Point to point (P) Region of Interest (R)

Three robots

Experiment 23 subjects 148 trials 3 world complexities

Accuracy of Predictions Accuracy of Predictions in a Three-Robot Teamin a Three-Robot Team

The Presentation AgendaThe Presentation Agenda

The types of questionsThe types of questions Neglect tolerance: Is a team feasible?Neglect tolerance: Is a team feasible? How do we compute neglect tolerances?How do we compute neglect tolerances? Tradeoffs: workload and performanceTradeoffs: workload and performance Is a team optimal?Is a team optimal? The problem with switch costsThe problem with switch costs Some limits, ideas, and proposalsSome limits, ideas, and proposals

What are switch costs?What are switch costs? The biggest unknown influence on span of The biggest unknown influence on span of

controlcontrol They come in several flavors:They come in several flavors:

Time to regain situation awarenessTime to regain situation awareness Time to prepare for switchTime to prepare for switch Errors and Change BlindnessErrors and Change Blindness

What really happens here?

Before and AfterBefore and After

Getting a Feel for the Getting a Feel for the ExperimentExperiment

Preliminary ResultsPreliminary Results

0123456789

10

Median Time to Recover

BlankToneTetrisUAV

ToneTone TetrisTetris UAVUAV

BlankBlank p<15p<15%%

p<20p<20%%

p<5%p<5%

ToneTone p>35p>35%%

p<15p<15%%

TetrisTetris p<10p<10%%

6 subjects, none naïve6 subjects, none naïve 207 correct change detections207 correct change detections One-sided T-test, equal variancesOne-sided T-test, equal variances

Important TrendsImportant Trends Differences not just from “time away”Differences not just from “time away”

blank and tetris have same timeblank and tetris have same time UAV and tone have same timeUAV and tone have same time Averages nearly identicalAverages nearly identical

Differences not just from “counting”Differences not just from “counting” UAV and tone both countUAV and tone both count

Differences not just from “motor channel” Differences not just from “motor channel” UAV and tone both selectUAV and tone both select Tetris requires interactionTetris requires interaction

Probably spatial reasoning and changing Probably spatial reasoning and changing perspectivesperspectives

The Presentation AgendaThe Presentation Agenda

The types of questionsThe types of questions Neglect tolerance: Is a team feasible?Neglect tolerance: Is a team feasible? How do we compute neglect tolerances?How do we compute neglect tolerances? Tradeoffs: workload and performanceTradeoffs: workload and performance Is a team optimal?Is a team optimal? The problem with switch costsThe problem with switch costs Some limits, ideas, and proposalsSome limits, ideas, and proposals

How Many Robots?How Many Robots? AssumptionsAssumptions

Goal: Gather battle-related information Goal: Gather battle-related information while minimizing risk while minimizing risk

Media: Mostly camera/video Media: Mostly camera/video informationinformation

PredictionPrediction Interpreting camera information Interpreting camera information

difficultdifficult High robot autonomy won’t help enoughHigh robot autonomy won’t help enough

A Special Case: A Special Case: The Robotics SpecialistThe Robotics Specialist

Can one person Can one person manage multiple manage multiple robot assets?robot assets?

At what level of At what level of performance?performance?

Goal: gather Goal: gather informationinformation

Media: visual Media: visual (camera/video)(camera/video)

Belief: autonomy will Belief: autonomy will help, but not enoughhelp, but not enough

Mixed Reality DisplaysMixed Reality Displays

Eliminate “The world Eliminate “The world through a soda straw”through a soda straw”

Integrate vision with Integrate vision with active sensorsactive sensors

Integrate display with Integrate display with autonomyautonomy

Include sensor Include sensor uncertaintyuncertainty

Control pan-and tiltControl pan-and tilt Study time delay effectsStudy time delay effects

Real World ResultsReal World Results

ObjectiveObjective 51% Faster (p < .01)51% Faster (p < .01) 93% Less Safeguarding (p < .01)93% Less Safeguarding (p < .01) 29% Lower Entropy (p < . 05)29% Lower Entropy (p < . 05) 10% Better on Memory Task (p < .05)10% Better on Memory Task (p < .05)

SubjectiveSubjective 64% Less Workload / Effort (p < .001)64% Less Workload / Effort (p < .001) 70% More Learnable (p < .0001)70% More Learnable (p < .0001) 46% More Confident (p < .05)46% More Confident (p < .05)

Several Thousand WordsSeveral Thousand Words

Experiment ResultsExperiment Results

Mixed Reality Displays Mixed Reality Displays (Pan and Tilt)(Pan and Tilt)

Control the Information Control the Information Source, Source,

Not the RobotNot the Robot Phlashlight ConceptPhlashlight Concept What will UAV see?What will UAV see?

Semantic Maps and Change Semantic Maps and Change HighlightingHighlighting

Video in contextVideo in context Icon-based maps Icon-based maps

w/ semantic w/ semantic labelslabels

““That was then, That was then, this is now this is now comparison” --- comparison” --- change change highlightinghighlighting

Information Information decaydecay

Information in ContextInformation in Context

Prompt prospective memoryPrompt prospective memory Shift in a timely wayShift in a timely way Give time to prepareGive time to prepare

Robot Progress While User is Doing Secondary Task

0

2

4

6

8

10

12

14

Forced Paced w/oPath Planner

Forced Paced w PathPlanner

Self Paced w/o PathPlanner

Self Paced w PathPlanner

Seco

nds

doin

g Se

cond

ary

Task

Stopped

Going

“Neglect Tolerance

”

Support Timely ShiftsSupport Timely Shifts

Human Reaction to Robot Getting Stuck

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

Forced Paced w/opath planner

Forced Paced w/Path Planner

Self Paced w/opath planner

Self Paced w/Path Planner

Sec

onds

for H

uman

to R

espo

nd

w/o Attention Management

w/ Attention Management

“Situation Awareness”

Supporting Task Supporting Task Switching: Etc.Switching: Etc.

History trails. History trails. Knowing recent past helpsKnowing recent past helps Tail on a map-based interfaceTail on a map-based interface Virtual descent into video-based interfaceVirtual descent into video-based interface Change highlighting/morphingChange highlighting/morphing

Plans: Plans: Knowing intention helpsKnowing intention helps Planned path on map-based interfacePlanned path on map-based interface Predicted trajectory on video-based interfacePredicted trajectory on video-based interface Quickened displaysQuickened displays

Task relationships: Task relationships: Knowing relationship Knowing relationship between two tasks helpsbetween two tasks helps Relative spatial location on map-based Relative spatial location on map-based

interfaceinterface Picture-in-picture on video-based interfacePicture-in-picture on video-based interface Progress bar of task X on task Y’s displayProgress bar of task X on task Y’s display

Improve Perception and Improve Perception and Scene Interpretation Scene Interpretation

(Olsen)(Olsen) Use interaction and machine Use interaction and machine

learning to make this robustlearning to make this robust

Future Concept Future Concept (Proposed)(Proposed)

Safe/Unsafe Safe/Unsafe occupancy gridsoccupancy grids Evolutionary image Evolutionary image

classifierclassifier Evolutionary Evolutionary

integration of vision integration of vision and lasersand lasers

Particle-based inverse Particle-based inverse perspective transformperspective transform

Path planningPath planning Uncertainty-based Uncertainty-based

triggers for retrainingtriggers for retraining

Learning interface Learning interface mappings from implicit mappings from implicit user cuesuser cues

ConclusionsConclusions

We can evaluate team feasibilityWe can evaluate team feasibility We can predict team performanceWe can predict team performance We need to understand task switching We need to understand task switching

betterbetter

We need to support realistic task We need to support realistic task switchingswitching Via interfacesVia interfaces Via autonomyVia autonomy

Near-Term Future WorkNear-Term Future Work

Complete validation of task Complete validation of task switching experiment paradigmswitching experiment paradigm

Compare “new and improved” Compare “new and improved” interfaces against baselineinterfaces against baseline

Compare effects of type and size of Compare effects of type and size of interfaceinterface

Answer the questions for the special Answer the questions for the special casecase