Vand

erbi

lt U

nive

rsity

Robotics and Autonomous Systems Lab (RASL)

A novel interface system for seamlessly integrating human-robot

cooperative activities in space

Nilanjan SarkarMechanical Engineering

Electrical Engineering and Computer Science

Craig A. SmithPsychology and Human Development

Vanderbilt UniversityNashville, TN

Vand

erbi

lt U

nive

rsity

Robotics and Autonomous Systems Lab (RASL)

Inspiration“Exploration of space and solar system will be most effective if human capabilities are synergistically combined with those of robots. Such a human-robot system, developed correctly, will reduce exploration risks, improve efficiency, and achieve overall mission goals faster and in a better manner.”[Objective of the ICASE/USRA/LaRC Workshop on Revolutionary Aerospace Systems Concepts for Human/Robotic Exploration of the Solar System, Nov. 2001]

Vand

erbi

lt U

nive

rsity

Robotics and Autonomous Systems Lab (RASL)

Major ObstacleNatural interaction between human and robot

Is implicit communication possible?

- e.g., robot brings the right tool to the astronaut without being explicitly commanded

Can the robot understand the psychological states of the human?

- e.g., robot rushes to help the human if the robot senses “panic”

Vand

erbi

lt U

nive

rsity

Robotics and Autonomous Systems Lab (RASL)

Novel Approachaffect detection and recognition

brainwave monitoring and characterization

design of control architecture for implicit communication

integrates research in signal processing, wearable computing, experimental psychology and control theory

Vand

erbi

lt U

nive

rsity

Robotics and Autonomous Systems Lab (RASL)

HypothesesA robot will:

implicitly understand a task command from an astronaut

sense the psychological state of the astronaut and take necessary actions

Vand

erbi

lt U

nive

rsity

Robotics and Autonomous Systems Lab (RASL)

Rationale

Initially, the research will be:person specificcontext specific

Afterwards, with enough understanding and data, an affect recognizer for a class of people will be attempted

Vand

erbi

lt U

nive

rsity

Robotics and Autonomous Systems Lab (RASL)

Phase I TasksThe following tasks were proposed:

A. Develop Affect Recognizer1. Design human subject experiments to elicit target

affective states (e.g., engagement, anxiety, fatigue, etc.)

2. Assess physiological indices that are important for the target affective states

3. Conduct pilot studies 4. Data analysis and signal processing for affect detection

B. Investigate the currently available brainwave monitoring technologies

Vand

erbi

lt U

nive

rsity

Robotics and Autonomous Systems Lab (RASL)

Phase I Tasks

The following additional tasks were performed:

Developed a functional human-robot system that is affect-sensitive, and

Conducted preliminary brainwave monitoring experiments with EEG

Vand

erbi

lt U

nive

rsity

Robotics and Autonomous Systems Lab (RASL)

Development of

Affect Recognizer

Vand

erbi

lt U

nive

rsity

Robotics and Autonomous Systems Lab (RASL)

Experimental TasksThree Problem-Solving Tasks (Each lasting ~1 hour)

Anagrams

Math Word Problems

Sound Discrimination

Two Sequences for Each Task1. Easy: Fairly Easy --> Trivially Easy

2. Difficult: Fairly Easy --> Virtually Impossible

Vand

erbi

lt U

nive

rsity

Robotics and Autonomous Systems Lab (RASL)

Sample Anagrams

Easy Condition Difficult Condition

AWADR IYTED

DEITYAWARD

Vand

erbi

lt U

nive

rsity

Robotics and Autonomous Systems Lab (RASL)

Easy Math Problem

An astronaut requires 2 pounds of oxygen per day while in space. How many pounds of oxygen are needed for a team of 3 astronauts who are going to spend 5 days in space?

30Answer:

Vand

erbi

lt U

nive

rsity

Robotics and Autonomous Systems Lab (RASL)

Difficult Math Problem

Tammy has $9.70 in nickels, dimes, and quarters. The number of nickels is 4 more than 3 times the number of dimes, and the number of quarters is 5 fewer than 2 times the number of nickels. How many nickels does Tammy have?

19Answer:

Vand

erbi

lt U

nive

rsity

Robotics and Autonomous Systems Lab (RASL)

Sound TaskSequence of three tonesJudge whether first and third tones are the same or differentDifficulty manipulated by varying tone length and frequency difference between tones

Easy Trials Difficult Trials

Vand

erbi

lt U

nive

rsity

Robotics and Autonomous Systems Lab (RASL)

Physiological Data Collection

Vand

erbi

lt U

nive

rsity

Robotics and Autonomous Systems Lab (RASL)

Physiological MeasuresEKG

interbeat interval (IBI)meanvariability

sympathetic power *parasympathetic power *

Finger Pulse Amplitudemean variability

Pulse Transit Time (PTT)Digit Skin Temperature

meanslope of change

Skin ConductanceTonic

meanslope of change

Phasicresponse rate *average amplitude *maximum amplitude

Facial Muscle Activity (EMG)Brow (Corrugator)

meanvariability *

Jaw (Masseter)mean *variability

Vand

erbi

lt U

nive

rsity

Robotics and Autonomous Systems Lab (RASL)

Self Report MeasuresSampled:

Two minutes into each taskEvery seven minutes thereafter

Assessed:Task DifficultyPerceived AbilityAffective States

Key Affective Parameters:

Anxiety IndexAnxietyOverload

Engagement Index

Task ImportanceHopeChallengeInterest(R) Resignation(R) Apathy

Vand

erbi

lt U

nive

rsity

Robotics and Autonomous Systems Lab (RASL)

Correlations of Anxiety with Physiology

-0.5

-0.3

-0.1

0.1

0.3

0.5

Symp Paras SCR-Rate

SCR-Amp

Corr-Var

Mass

Physiological Parameter

Subject 2Subject 4

Vand

erbi

lt U

nive

rsity

Robotics and Autonomous Systems Lab (RASL)

Correlations with Physiology:Anxiety vs. Engagement (Subject 2)

-0.5

-0.3

-0.1

0.1

0.3

0.5

0.7

Symp Paras SCR-Rate

SCR-Amp

Corr-Var

Mass

Physiological Parameter

AnxietyEngagement

Vand

erbi

lt U

nive

rsity

Robotics and Autonomous Systems Lab (RASL)

Schematic of Fuzzy LogicAnalyzer

SympatheticPower

ParasympatheticPower

Skin ConductanceResponse Rate

Skin ConductanceResponse Amplitude

CorrugatorVariability

MasseterAverage

Fuzzy LogicAnalysis

AffectDiagnostic

Output

Vand

erbi

lt U

nive

rsity

Robotics and Autonomous Systems Lab (RASL)

Development of a

Functional Human-Robot

Interactive System

Vand

erbi

lt U

nive

rsity

Robotics and Autonomous Systems Lab (RASL)

Our Robot

Operates in three modes:1. Explore -- Wanders

about environment2. Survival -- Uses sonar to

detect and avoid obstacles in environment

3. Affective -- Responds to affective signals from human operator

If robot senses high anxiety, it uses light detection algorithm to go to operator

Vand

erbi

lt U

nive

rsity

Robotics and Autonomous Systems Lab (RASL)

Robot Control ArchitectureA hybrid subsumption control paradigm

Emergency Stop

Reverse

Affect Layer

Obstacle avoidance

Wall Follow

Wandering

S

S

S

S

S

Sensor Data

Survival Mode

Deliberative Mode

Reactive Mode

Vand

erbi

lt U

nive

rsity

Robotics and Autonomous Systems Lab (RASL)

Human-Robot Interaction Scenario

Human operator works at computer task while the robot explores the environment and monitors

operator’s affective state

Vand

erbi

lt U

nive

rsity

Robotics and Autonomous Systems Lab (RASL)

The Functioning System

QuickTime™ and a DV/DVCPRO - NTSC decompressor are needed to see this picture.

Vand

erbi

lt U

nive

rsity

Robotics and Autonomous Systems Lab (RASL)

Physiological Signals & Fuzzy Logic Output for Moderate Anxiety Trigger

Vand

erbi

lt U

nive

rsity

Robotics and Autonomous Systems Lab (RASL)

Timing Diagram from Robot Experiment

Vand

erbi

lt U

nive

rsity

Robotics and Autonomous Systems Lab (RASL)

Investigation of Brain Wave

Monitoring Technologies

Vand

erbi

lt U

nive

rsity

Robotics and Autonomous Systems Lab (RASL)

Technologies to Assess Brain ActivityEEG - Measures electric currents generated by the brain at the scalp

MEG - Measures magnetic fields generated by the brain

PET - Measures emissions from radioactively labeled chemicals, monitors metabolic rates and blood flow

fMRI - Measures oxygen concentration of blood, correlates to blood flow

Vand

erbi

lt U

nive

rsity

Robotics and Autonomous Systems Lab (RASL)

Modality Comparison

EEG MEG fMRI PET

Spatial Resolution 7-16 mm 3-11 mm 1 mm 5 mm

Temporal Resolution Milliseconds Milliseconds 1 second to

minutes 45 seconds to

minutes

Cost $15-120K

$500k (37 channel)

$300-400k Shielding

Millions Millions

Wearability Yes No No No

Vand

erbi

lt U

nive

rsity

Robotics and Autonomous Systems Lab (RASL)

Example EEG and MEG Systems

http://www.xltek.com/xlwebv2/products/eeg/xlamb.htmhttp://www.vsmmedtech.com/MEG_Technical/MEG_Images_275_151.asphttp://www.biomag.helsinki.fi/meg.htmlhttp://www.elixa.com/mental/MindSet.htmhttp://www.emedicine.com/neuro/topic445.htm

Vand

erbi

lt U

nive

rsity

Robotics and Autonomous Systems Lab (RASL)

EEG Experiment: Cognitive Load

0

5001000

1500200025003000

350040004500

Alp

ha P

ower

Eyes Closed Eyes Open

Vand

erbi

lt U

nive

rsity

Robotics and Autonomous Systems Lab (RASL)

EEG Experiment: Math Problem-Solving

0

100

200

300

400

500

600

700

800

900

Alp

ha P

ower

Eyes Open Easy Moderate Difficult

Problem Difficulty

Vand

erbi

lt U

nive

rsity

Robotics and Autonomous Systems Lab (RASL)

Summaryperformed every proposed task for Phase I

performed two additional tasks beyond what was proposed

were successful in eliciting and detecting the target affective states under controlled situations

developed a functional human-robot system to demonstrate the feasibility of the central concept

Vand

erbi

lt U

nive

rsity

Robotics and Autonomous Systems Lab (RASL)

ConclusionsPhase I work demonstrates that:

implicit human-robot communication is feasible

can detect human affect on-line and in the context of realistic taskscontrol system can be made affect-sensitive

brainwave monitoring will likely supplement peripheral physiology in affect detection and implicit communication

Vand

erbi

lt U

nive

rsity

Robotics and Autonomous Systems Lab (RASL)

Future WorkExpand range of tasks and contexts to which framework can be appliedIncrease the reliability and sophistication of affect recognition

Increase range of affects detected and discriminated beyond anxiety and engagement to include frustration, fatigue, boredom, etc.Advance analytical tools for extracting relevant information from physiological signals

Increase degree to which physiological recording is ambulatoryFurther explore utility of EEG recording to improve affect recognition and implicit communicationFormalization of affect-sensitive control system