plan - rafale malaysiarafalemalaysia.com/wp-content/uploads/rafale-techno-day-control-of... · plan...

94

Upload: others

Post on 27-Mar-2020

2 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief
Page 2: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Plan

• Context• Brief presentation of Télécom Bretagne and Lab-STICC• Brief presentation of bio

• Scientific part• Human supervisory control of complex systems• Human factors and UVs control: a brief state of the art• Playing with swarms of UAVs• New trends and challenges in dialogue management • Perspectives and conclusions

Page 3: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Télécom Bretagne

A graduate Engineering Schooland a Research Centre in the field of

Science and Information Technologies

Page 4: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Institut Mines-Télécom

4

Page 5: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Télécom Bretagne

The Engineering School of the (French) Far West

Page 6: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Télécom Bretagne in few numbers350 Researchers (faculty, masters, PhD)9 Research DepartmentsAffiliations to French National Research Agencies (CNRS)International prices (Marconi, SPIE, IEEE) and National representativeness (French Academy of Sciences)

1200 students

160 Academics

45% International students from 50+ countries

Telecom Bretagne ranked in Top 15 among 200+ Graduate Engineering Schools in France

Page 7: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

International exchanges

• Out: Our student have to spend between 4 and 6 months abroad

• In: Master of Sciences• One semester with English spoken teaching + intensive classes

of French language

• In: Internship• Research / engineering project in one of our labs

• In/Out: Co-tutelle PhD

Page 8: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Lab-STICCCNRS UMR 6285

From Sensors to Knowledge:Communicate and Decide

Page 9: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Télécom Bretagne

UBO

UBS

ENIB

ENSTA-Bretagne

The Research Unit of the (French) Far West

Page 10: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Lab-STICC teams

Page 11: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Short bio

Page 12: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Airborne systems engineering (Thales - 1986/2000)PhD in Computer Sciences - 1999Professor at Telecom Bretagne - 2000 / presentHead of Department (Cognitive Sciences / Usages)Visiting professor MIT - Aero-Astro Department 2006/2007French representative NATO group HF - UAV 2008/2012Deputy director Lab-STICC Research Unit

Page 13: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Plan

• Context• Brief presentation of Télécom Bretagne and Lab-STICC• Brief presentation of bio

• Scientific part• Human supervisory control of complex systems• Human factors and UVs control: a brief state of the art• Playing with swarms of UAVs• New trends and challenges in dialogue management • Perspectives and conclusions

Page 14: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Human supervisory control of complex systems

Page 15: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Human supervisory control

Increasing complexity and autonomy of systems and subsystems have led to • a separation in time and space between command /control and

execution of the task• a mediation through computers between the operator and the

actuators and sensors

Page 16: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Human supervisory control

Operator Situation

Operator’s space

Command

Display

System

Space of task

Informationmanagement

TaskSharing

Coordination

Informationcollection

Command system

• Task sharing and allocation• Human - System InteractionDimension 1

Page 17: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Referring to OODA

• Functional decomposition• Information processing in systemsDimension 2 Functional decomposition

Page 18: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Supervisory control and OODA

Operator Situation

Operator’s space

Command

Display

System

Space of task

Commandsystem

Informationmanagement

TaskSharing

Coordination

Informationcollection

Space of task

Ob

Space of taskSpace of task

A

Operator

Or

Operator’Operator’Operator

D

• Usual functional sharing• ~ Fitts tableDimension 1 + Dimension 2

Page 19: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Plan

• Context• Brief presentation of Télécom Bretagne and Lab-STICC• Brief presentation of bio

• Scientific part• Human supervisory control of complex systems• Human factors and UVs control: a brief state of the art• Playing with swarms of UAVs• New trends and challenges in dialogue management • Perspectives and conclusions

Page 21: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Operator Situation

Operator’s space

Command

Display

System

Space of task

Commandsystem

Informationmanagement

TaskSharing

Coordination

Space of task

Ob

Space of taskSpace of task

A

OperatorOperator

Ors spaceOperator’Operator’Operator s space

D

Supervisory control and OODA

Focusing on Observation

Informationcollection

Page 22: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Remote perception (Observe)

Relative orientation in space

Stability of images

ref. Cooper & al., CERI-UAV 2007

by courtesy of TNO / L. van Breda

North-up Platform up UAV-up

MovingOperatorCentered

MovingUAV

Centered

Page 23: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Remote perception (Observe)

Telepresence UGV (TNO Netherlands / US Army): acuity, contrast sensibility, stereo vision and audio

by courtesy of TNO / L. van Breda

EyeRobot

Page 24: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Operator Situation

Operator’s space

Command

Display

System

Space of task

Commandsystem

Informationmanagement

TaskSharing

Coordination

Informationcollection

Space of task

Ob

Space of taskSpace of task

A

OperatorOperator’OperatorOperator

Ors spaceOperator’Operator’Operator s space

D

Supervisory control and OODA

Focusing on Action

Page 25: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Remote action

ref. Maj. Martin, CERI-UAV 2007

Page 26: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Operator Situation

Operator’s space

Command

Display

System

Space of task

Commandsystem

Informationmanagement

TaskSharing

Coordination

Informationcollection

Space of task

Ob

Space of taskSpace of task

A

Operator

Ors spaceOperator’Operator’Operator s space

D

Supervisory Control and OODA

Focusing on Orientation

Page 27: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Support to situation assessment

Cognitive task analysis• Identification of data structure that are meaningful (or salient) for the

operator

Situation awareness models• Mica Endsley 3 stages process (perception, understanding,

projection)• Gary Klein RPD (Recognition Primed Decision)• and many others ...

Hunn, CERI 2006

Page 28: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Graphical interfaces, multimodality

MUSCIT (US Army / USAF) : monitoring ground scenes from multiple UAVs

• Graphical interfaces, attention allocation, multimodal presentation

Left Control Station displaytactical situation display, payload control

Right Control Station displayaircraft video, sensor management

by courtesy of AFRL / M Patzek

Page 29: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Supervisory Control and OODAGoing ahead: teaming!!!

Operator Situation

Operator’s space

Command

Display

System

Space of task

Commandsystem

Informationmanagement

TaskSharing

Coordination

Informationcollection

Space of task

Ob

Space of taskSpace of task

A

OperatorOperator

Ors spaceOperator’Operator’Operator s space

D

collection

D

collection

OrOb A

Page 30: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Cognitive assistant unitsCognitive Assistant Units (Germany / UBM Münich)From shared knowledge representation, mimicking human operators and pilots to handle selected tasks

by courtesy of Schulte / IFS

Page 31: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Assistant systems

by courtesy of Donath & al. / IFS

Page 32: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Assistant systems

by courtesy of Donath & al. / IFS

Page 33: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Playbook (Miller)

Delegation: one way humans manage supervisory control with heterogeneous, intelligent assets

Require a shared knowledge of domain goals, tasks and actions

Plays reference a defined range of plan/behavior alternatives (that can be further constrained or simulated)

Supervisor calls plays; agents have autonomy within the play’s scope

by courtesy of Shively / AMRDEC

Page 34: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Playbook

Overwatch: sustained surveillance of a fixed target or area

Track target: sustained surveillance of a moving target

Area recon one-pass reconnaissance of an area

Route recon: one pass reconnaissance of a route

Encircle patrol: circle a target

Protect: surveillance of an area while moving

“Plays”

Miller, Goldman, Funk, Wu & Pate 2004

Page 35: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Constraints on Plays

Example of constraints (for overwatch function)• earliest / latest acceptable “time on target”• earliest / latest acceptable “time off target”• stealth• priority• sensors• etc...

ref: Miller, Goldman, Funk, Wu & Pate 2004

AI constraint basedcontrol algorithms

Page 36: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Human supervisory control

• Next step is autonomy handled by pro-active automata (also called agents)

• From “Human-in-the-loop” to “Human--the-loop” to “Human-on-the-loop”• From supervisory control to authority sharing (a.k.a.

mixed initiative)

Page 37: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Supervisory Control and OODAHow to cooperate?

? ? ? ?

Operator Situation

Operator’s space

Command

Display

System

Space of task

Commandsystem

Informationmanagement

TaskSharing

Coordination

Informationcollection

How to cooperate?

Space of task

Ob

Space of taskSpace of task

A

OperatorOperator

Ors spaceOperator’Operator’Operator s space

D

collection

D

collection

OrOb A

Page 38: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Interaction?

Most of the approaches focus on task sharing, in terms of delegation or responsibilityBut very few address the communication and interaction protocol that supports this delegation

Following: a detailed example from the domain of swarm control

Page 39: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Plan

• Context• Brief presentation of Télécom Bretagne and Lab-STICC• Brief presentation of bio

• Scientific part• Human supervisory control of complex systems• Human factors and UVs control: a brief state of the art• Playing with swarms of UAVs• New trends and challenges in dialogue management • Perspectives and conclusions

Page 40: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Playing with swarms

Page 41: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Collective Intelligence & Swarms

• Collective Intelligence paradigm : an intelligent system may be composed of a large number of simple entities interacting together (agents) • Natural collectives systems: social insects, multi-cellular organisms,

communities...• Physical systems: mobile robots, sensors networks, distributed

systems

• Computational systems: multi-agents systems, grid computing, cellular automata

• Properties• Robustness, reactivity, self-organisation, adaptativity

by courtesy of LORIA France

Page 42: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Artificial pheromone

Mobile multi-agent system based on social insect mimickingAgents communicate through the environment where they virtually d r o p a m o u n t s o f a r t i f i c i a l pheromoneA pheromone can have

a repulsive effect so that agents avoid to patrol on the same areaan attractive effect so that agents follow the track opened by others

environment where they virtually a r t i f i c i a l

Page 43: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Principles of swarm

by courtesy of MAIA team LORIA - France

Page 44: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Principles of swarms

by courtesy of MAIA team LORIA - France

Page 45: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

SMAARTSwarms for surveillance and security

Page 46: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Context: intrusion surveillance

Page 47: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Interaction management for swarms

• A swarm is an autonomous complex system• Main characteristics of UAVs system:

• Many vehicles• Intrinsic collective automation, local decision• Complex interactions

• Several concurrent tasks• Manage threads of interaction• Priorities, turn-taking, interruption, change of

context, etc.

Page 48: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

HMI: a global view

Page 49: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

HMI details

Page 50: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Pheromones

2 pheromones in order to ensure patrol and tracking functions• Repulsive pheromone for self organization and covering the

area• Attractive pheromone to concentrate UAVs on tracking (and

possible future identification functions)

Page 51: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

HMI details (2)

Page 52: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Patrolling with swarms

Page 53: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Results

Between +6,5% and -13% on the first 20 minutes of pure

surveillance

Fail:

1. to control system

2. to self evaluate

Page 54: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Interacting with a swarm?

Objects out of usual representations• Interactions through the environment, fields of pheromone, diffusion,

evaporation, random moves• What happens? Why? What is to happen?How to control?How to set systems parameters?

Page 55: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

SUSIESupervising Intelligent Systems based

on Swarms

Page 56: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Interaction management

• Pheromon grids, large amount of subsystems, multiple localized events,”fuzzy” areas ...

• Need to increase the power of expression for the interpretation and for the generation of interaction• New modalities to interact on these

objects• Reflection on an adequate model of

interaction

Page 57: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Control the self-organized systems

Keep human in/on the loopAdapt the algorithm to the new modes of interaction• Speak the language of the

operator• A : Deal with topological +

operational concepts• Control more accurately but

not too much• B : Mix different modes of

control

Page 58: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Changing the point of viewA : Dealing with topological / technical aspects

Mobile Point Open lineClosed

line Area

Watch see followbear regul. a sensor on a given point

bear regul. a sensor on all points / line

bear regul. a sensor on all points /cl. line

bear regul. a sensor on all points / area

Avoid e.g. coll. avoidance

see closed line reduced to a point

do not cross a line

do not cross a given

closed line

get out area if inside

Find find targets systematic search of a

point

find at least one point of

a line

see open line with inside/

outside

see closed line

Follow track N/Amaneuver in

order to align on a line

see open line N.A

Intercept cross mobiletrajectory

N/Amaneuver so as to meet an

open linesee open line N/A

Page 59: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Changing the point of viewB : Mixing modes of control

• Keep pheromone as a basic principle for self organization and surveillance

• Add use of way-points for specific tasks• reach directly the area to patrol• come back to area when shifting because of speed• watch a point (see previous table)

• Add other use of pheromones to extend the expressiveness of commands

• complete exclusion zone (avoid an area - see previous table)• dynamic map of pheromone (work in progress)

Page 60: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Presentation of SUSIE

see video at:

recherche.telecom-bretagne.eu/susieor

deev-interaction.com/project-susie/

Page 61: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Human factors experiment (show)

Page 62: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Human factors experiment (no show)

Page 63: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Experiment configuration

• 47 people

• Non expert but trained

• 15 minutes scenario

• No show (24 people) / Show (23 people)

Same age mean

Page 64: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Experiments conclusions

Show No show

Better time of identification Better results in identification

Less manual transfer of UAVs Less failure in identification

Better usability perceived Lower perceived workload

Lower perceived cognitive load

Lower perceived time pressure

???

!!!

Page 66: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Plan

• Context• Brief presentation of Télécom Bretagne and Lab-STICC• Brief presentation of bio

• Scientific part• Human supervisory control of complex systems• Human factors and UVs control: a brief state of the art• Playing with swarms of UAVs• New trends and challenges in dialogue management • Perspectives and conclusions

Page 67: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

New trends and challenges in dialogue management for human supervisory

control

Page 68: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Did we fix everything?We proposed:

! A user-centered level of language! A way to make algorithms (and machines) compatible! A new intuitive mode of interaction (tabletop)

! but

! what about the cooperation?! what about more complex systems ... needing more complex commands?

Page 69: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Supervisory control is an InteractionSupervisory control is an interaction…

! Between a supervisory and one or more subordinates

! Nature of supervisor! Nature of subordinates! Nature of tasks! Nature/range of subordinate behaviors! Nature of world (operating conditions)

! Reliability! Trust! Criticality/Importance! Social/Emotional issues! Autonomy

from Chris Miller / SIFT 2012

Page 70: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Human-Human Supervisory Control Examples

A Parking ValetA Postal clerk/Fast Food ChefShepherd to sheepdogA young child (~3 yrs)A teenager (~16 yrs)A new secretary/asstA conciergeQuarterback (Am.) to team Project Manager to teamCEO to corporationPresidential AideMovie Director’s GopherRadar O’Reilly

from Chris Miller / SIFT 2012

Page 71: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Describing the Interaction (Simply)

• How frequently do I have to interact?• How explicitly/expressively?• What kinds of things can I ask for?• Two dimensions proposed:

• Intervention Demand = how much time/attention do I have to spend commanding and monitoring for useful work?• How much time must I spend managing the automation vs.

how much time does it perform independently• Scope = what range of functions can the automation provide?

from Chris Miller / SIFT 2012

Page 72: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Scoring Supervisory Control Examples

A Parking ValetA Postal clerk/Fast Food ChefShepherd to sheepdogA young child (~3 yrs)A teenager (~16 yrs)A new secretary/asstA conciergeQuarterback (Am.) to team Project Manager to teamCEO to corporationPresidential AideMovie Director’s GopherRadar O’Reilly

Frequency11610873243331

Scope12326644689910

Useful?+++----++++

++++++

Ratio1.525

1.31.1.75.5.67.38.33.33.1

from Chris Miller / SIFT 2012

Page 73: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

A (Partial) Utility Threshold?

Radar O’ReilyValet

Postal Clerk

Sheepdog

Child

Teenager

New Secretary

Concierge

QB to Team

PM to Team

CEO to Corp

Presidential Aide

Gopher

1

10

110Scope

Frequency from Chris Miller / SIFT 2012

Page 74: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

New kind of Operator Support System

Vehicles Command & ControlSituation AwarenessDSS & Automation

Page 75: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

New kind of Operator Support System (II)

Vehicle Command & ControlSituation Awareness

Automation

Interaction ManagementSemantic Bridge

Automation

Interaction Management+

Situation AwarenessTwo tasks:

1. mission

2. interaction

Page 76: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Need for Semantic Bridge

same principles upstream

Page 77: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

interface manipulation

Need for Interaction Management

• New characteristics of UVSs and their OSSs, e.g.• Several vehicles• More automation, decision support• More complex interactions

• Several concurrent tasks• Manage threads of interaction• Priorities, turn-taking, interruption, change of

context, etc.

interactive

trajectory

planner

inter-UV CoordinationMore automation, decision support

multiple failures or events

Priorities, turn-taking, interruption, change of

multiple payloads + automation

Page 78: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Interaction as a Collaborative Activity

Traditional view [Shannon]: unidirectional, one-m e s s a g e , a d d r e s s e e i s p a s s i v e , n o n -understandings are errors

Collaborative view [Clark]: shared goal, shared effort, interactive refinement, feedbacks

Page 79: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Interaction is a Subordinate Activity

Execute the mission Interact with the OSS

The operator does two things:

>more important

Perfect understanding is not requiredSufficient for the current purpose: mission & context

Shared effort, but unequal contributions

Page 80: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Adjusting the Interaction Load

Non-Understanding

Give up / postpone

“Hold on. Intruder detected at XY”

Ask for recast “Please rephrase”

Request refinement

“Which building?”

Propose refinement

“Do you mean the building North of the airport?”

Understanding

understanding

= positive

feedback

Mor

e co

oper

ativ

e st

rate

gies

Page 81: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Adjusting the Interaction Load

Non-Understanding

Generation & Interpretation

Give up / postpone Keywords

Ask for recast Selfish

Request refinement Cooperative

Propose refinement Mutual awareness

Understanding

understanding

= positive

feedback

only one’s

beliefs

only the addressee’s

beliefs

Mutual awarenesscommon beliefs

Mor

e co

oper

ativ

e st

rate

gies

Page 82: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Why Not Always Be Cooperative?

Maintain the operator’s SAImprove the feeling to be working with the OSSConstruct and improve a common language & representations (e.g. “selfish” at first enables “cooperative” later)Develop more accurate models of each otherUnusual operator’s demand, critical task, etc.

with the OSS

similar to “adverse effects of

automation”

*loss of SA

*complacency

*skill degradation

[Parasuraman et al.]

Page 83: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Summary

• Consider interaction as a task, not only as a an overhead

• Principles similar to human-human interaction/dialogue

• The cooperative attitude of the system can be adjusted

• Interaction load adjustment could be useful to mitigate mission workload

Page 84: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Next step: Trust management

Page 85: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Trust

• A major indirect effect of dialog management• Within a team:

• mutual modeling (task driven)• interaction (speech turn, style, etc. impact trust)

• Constructive / based upon observations

Page 86: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Lee’s Model

Page 87: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Trust and interaction

Page 88: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Trust and interaction

Page 89: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Plan

• Context• Brief presentation of Télécom Bretagne and Lab-STICC• Brief presentation of bio

• Scientific part• Human supervisory control of complex systems• Human factors and UVs control: a brief state of the art• Playing with swarms of UAVs• New trends and challenges in dialogue management • Perspectives and conclusions

Page 90: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Conclusions

Page 91: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Main asserts

• Interaction is a key concept in the control of autonomous systems

• Interaction must be processed as a task by itself• Interaction has to remain intuitive and close to

operators high-level representations as the autonomous system has to interpret these representations in its own OODA space

Page 92: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

(Many) Remaining challenges

Operator multi-tasking / cognitive load• Attention and focus

• Quantitative models

Trust in automation• Extrinsic factors (role of alarms, task environment)

• Intrinsic factors (self-confidence, sharing mental models)

• Situation awareness• Task and mode switching

• Error diagnosis and recovery

Page 93: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

(Many) Remaining challenges

• Understanding the operator(s)

• Physiological and cognitive assessment of the operator’s state

• Assessing team cohesion and performance

• Systems of systems

• UAVs within the complete system

• Insertion of UAVs in civil air traffic / regulation

• Psycho-sociological aspects: UAV operators

• Lack of link between UAV operators and field forces

• Ruptures of contexts

Page 94: Plan - Rafale Malaysiarafalemalaysia.com/wp-content/uploads/Rafale-Techno-Day-Control-of... · Plan • Context • Brief presentation of Télécom Bretagne and Lab-STICC • Brief

Any questions?

[email protected]