Lastin

g R

ela

tion

ship

Cornerstones of Social Robotics in HRI

Cornerstones of Social Robotics in HRI

Team

work

Socia

l Learn

ing

Socia

l Inte

lligen

ceInterdependence

Transparent Communication

Cognitive Compatibility

Perspective Taking

User Studies,Psychology &Social Development

Being a body with a mind in a world of like bodies with like minds yields multi-modal associations

the self and the appearance of other agents

the behaviors (events in time) of the

self and the behaviors of other agents

the internal workings of oneself and other agents

Human Social DevelopmentHuman Social Development

Simulation Theory & Mindreading

Simulation Theory & Mindreading

We use our own cognitive system “off-line” to simulate others (R. Gordon)

Cognitive processes are dual-use Generate own actions from our mental states Infer the mental states responsible other’s

actions by “stepping into their shoes”

We use our own cognitive system “off-line” to simulate others (R. Gordon)

Cognitive processes are dual-use Generate own actions from our mental states Infer the mental states responsible other’s

actions by “stepping into their shoes”

Neural Mechanisms of MindreadingDual Use SELF-TOM

Neural Mechanisms of MindreadingDual Use SELF-TOM

Evidence of overlapping brain regions involved in SELF and TOM

SELF: meta-representational cognitive capacity to apply a “self perspective”

TOM: mindreading capacity to model someone else’s state of mind

Evidence of overlapping brain regions involved in SELF and TOM

SELF: meta-representational cognitive capacity to apply a “self perspective”

TOM: mindreading capacity to model someone else’s state of mind

TOM+SELF+

TOM-SELF+

TOM+SELF-

TOM-SELF-

Vogeley et al, Neuro Image 14, 170-181 (2001)

Simulation Theory & Social Learning

Simulation Theory & Social Learning

Andrew Meltzoff posits that we LEARN to simulate via early infant imitation-based interactions with adults

Social Learning Implications:The experience of others can be mapped to self --- enabling the development of learning by observation, imitation, social referencing, etc.

Andrew Meltzoff posits that we LEARN to simulate via early infant imitation-based interactions with adults

Social Learning Implications:The experience of others can be mapped to self --- enabling the development of learning by observation, imitation, social referencing, etc.

ArchitectureArchitecture

Imitation & Mirror Systems Dual Use Recognition/Production

Imitation & Mirror Systems Dual Use Recognition/Production

Visual-Motor Xform

Motor Out

Motor InVision

In

Interpret Observed Actions wrt Motor Repertoire

Meltzoff&Moore AIM Model Synthesize Action from Motor Repertoire

MotorKnowledge

Breazeal et al, Artificial Life (2005)

Social Structure of Imitation

Social Structure of Imitation

Human Engages robot in imitation game by mimicking the robot’s facial expressions

Social Interaction Affords LearningBody Maps: how robot’s face (body) maps onto

social othersMirror System: Dual use of motor

representations for recognition of action in others and production of own action

Ability to mimic others actions

Human Engages robot in imitation game by mimicking the robot’s facial expressions

Social Interaction Affords LearningBody Maps: how robot’s face (body) maps onto

social othersMirror System: Dual use of motor

representations for recognition of action in others and production of own action

Ability to mimic others actions

Development of Social Referencing

Development of Social Referencing

Social Referencing (~12 mos) Understand meaning of affective

signal from adult (~6 mos) Shared attention to understand

referent of adult (~9 mos) Associate that appraisal with beliefs

and memory (attitudes) toward that referent

Interact with novel object accordingly

Learn how to appraise novel objects (~18 mos)

Social Referencing (~12 mos) Understand meaning of affective

signal from adult (~6 mos) Shared attention to understand

referent of adult (~9 mos) Associate that appraisal with beliefs

and memory (attitudes) toward that referent

Interact with novel object accordingly

Learn how to appraise novel objects (~18 mos)

Social Referencing Challenges

Social Referencing Challenges

#1 Understand the affective meaning of another’s expression

#1 Understand the affective meaning of another’s expression

Evoke Affect from Observed Expression

Evoke Affect from Observed Expression

Exploit Bi-Directional Body-Affect Pathways to learn affective meaning of observed facial expression

Exploit Bi-Directional Body-Affect Pathways to learn affective meaning of observed facial expression

Body-affect loop

Imitate Facialexpression

Empathic MechanismEmpathic Mechanism

Learn association to evoke empathic response in robot Learn association to evoke empathic response in robot

Recognition of Vocal Affective Intent

Recognition of Vocal Affective Intent

Four cross-cultural contours of infant-directed speech A. Fernald

Exaggerated prosody matched to infant’s innate responses

Four cross-cultural contours of infant-directed speech A. Fernald

Exaggerated prosody matched to infant’s innate responses

time (ms)

pitc

h, f

(kH

z)o

approval

That’s a good bo-o-y! No no baby.

time (ms)

pitc

h, f

(kH

z)o

prohibition

Can youget it?

Can youget it?

time (ms)

pitc

h, f

(kH

z)o

attention

time (ms)

pitc

h, f

(kH

z)o

MMMM Oh, honey.

comfort

Evidence for Fernald-like Contours for (Cute) Robot Directed SpeechEvidence for Fernald-like Contours for (Cute) Robot Directed Speech

Valence and Arousal in Feature Space

Valence and Arousal in Feature Space

prohibition &high-energy neutral

attention & approval

soothing & low-energy neutral

Breazeal & Aryananda, Autonomous Robots (2002)

Results, Multiple LanguagesResults, Multiple Languages

Classification ResultTest set Strength Class Test

Size Approval Attention Prohibition Soothing Neutral

%

Correctly

Classified

Approval 84 64 15 0 5 0 76.19

Attention 77 21 55 0 0 1 74.32

Prohibition 80 0 1 78 0 1 97.5

Soothing 68 0 0 0 55 13 80.88

Caregivers

Neutral 62 3 4 0 3 52 83.87

Approval 18 14 4 0 0 0 72.2

Attention 20 10 8 1 0 1 40

Prohibition 23 0 1 20 0 2 86.96

Strong

Soothing 26 0 1 0 16 10 61.54

Approval 20 8 6 0 1 5 40

Attention 24 10 14 0 0 0 58.33

Prohibition 36 0 5 12 0 18 33.33

Medium

Soothing 16 0 0 0 8 8 50

Approval 14 1 3 0 0 10 7.14

Attention 16 7 7 0 0 2 43.75

Prohibition 20 0 4 6 0 10 30

Weak

Soothing 4 0 0 0 0 4 0

Naive

speakers

Neutral 29 0 1 0 4 24 82.76

Objective scorer classifies as strong, medium, weak

Good overall performance for strong instances

Random perf. = 20%

very good for caregivers

good for naive subjects

Acceptable misclassifications

minimal confusion of valence

some confusion of arousal

Responding to Vocal AffectResponding to Vocal Affect

QuickTime™ and aYUV420 codec decompressor

are needed to see this picture.

Social Referencing Challenges

Social Referencing Challenges

#1 Understand the affective meaning of another’s expression

#2 Understand the referent that their emotive reaction is about

#1 Understand the affective meaning of another’s expression

#2 Understand the referent that their emotive reaction is about

Saliency & Visual Attention (Adapted from J. Wolfe VGS 2.0)

Saliency & Visual Attention (Adapted from J. Wolfe VGS 2.0)

Visual attention allows robot to look at salient objects/events around it --- sets focus of robot’s attention

Visual attention allows robot to look at salient objects/events around it --- sets focus of robot’s attention

inh

ibit

rese

t

Frame Grabber

Eye Motor Control

Top down,task-driveninfluences

w w w w

skin tone habituationmotioncolor

attention

“Seek face” –high skin gain, low color saliency gainLooking time 80% face, 20% block

“Seek toy” –low skin gain, high saturated-color gain

Looking time 28% face, 72% block

Looking PreferenceLooking Preference

Internal influences bias how salience is measured The robot is not a slave to its environment Prefers behaviorally relevant stimuli

Directing Attention in Interaction

Directing Attention in Interaction

QuickTime™ and aYUV420 codec decompressor

are needed to see this picture.

Breazeal & Scassellati, IJCAI 1999

Socially Directed AttentionSocially Directed AttentionAdd social cues as stimuli

that explicitly contributes to saliency in addition to environmental “pop outs”

Add social cues as stimuli that explicitly contributes to saliency in addition to environmental “pop outs”

Gaze to Pointing Gaze to Head Pose

Shared Attention: Aboutness

Shared Attention: Aboutness

Extend attention model to distinguish focus of attention (what is salient right now) verses referential focus (what this interaction is about)Keep track of relative-looking-time

of objects in scene that robot and human look at

Hypothesis: object with highest relative looking time is the shared object referent.

Extend attention model to distinguish focus of attention (what is salient right now) verses referential focus (what this interaction is about)Keep track of relative-looking-time

of objects in scene that robot and human look at

Hypothesis: object with highest relative looking time is the shared object referent.

QuickTime™ and a decompressor

are needed to see this picture.

Social Referencing Challenges

Social Referencing Challenges

#1 Understand the affective meaning of another’s expression

#2 Understand the referent that their emotive reaction is about

#3 Use their appraisal to bootstrap its own appraisal of novel objects

#1 Understand the affective meaning of another’s expression

#2 Understand the referent that their emotive reaction is about

#3 Use their appraisal to bootstrap its own appraisal of novel objects

Object BeliefsObject Beliefs

Incoming perceptual features are bound into “beliefs” about objects Object beliefs are tracked perceptual histories over time:

“What this object was like recently...” +“What this object is like right now...” +“What we expect this object to be like soon.”

Incoming perceptual features are bound into “beliefs” about objects Object beliefs are tracked perceptual histories over time:

“What this object was like recently...” +“What this object is like right now...” +“What we expect this object to be like soon.”

Object Beliefs & Working MemoryObject Beliefs & Working Memory

Object templates are long-term prototypical representations of objectsSets expectations of what this thing typically is like…

Learning Associations Bind labels to object templates to learn names of objectsBind affect to object templates to learn attitude toward

objects (i.e., somatic markers)

Object templates are long-term prototypical representations of objectsSets expectations of what this thing typically is like…

Learning Associations Bind labels to object templates to learn names of objectsBind affect to object templates to learn attitude toward

objects (i.e., somatic markers)

Social ReferencingSocial Referencing

Thomaz et al, Ro-Man 2005

QuickTime™ and aYUV420 codec decompressor

are needed to see this picture.

Goal: robots that can learn in the real-world from anyone

Most people don’t have experience with Machine Learning techniques, but they bring a lifetime of experience with social learning interactions

Social-cognitive skills

Social interaction and socio-affective-cognitive skills does “heavy lifting” of framing the learning problem as a collaborative process.

If done correctly, improves learning process and performance for human and robot. Robot learns what is intended in a transparent way.

Summary: HRI meets MLTaking Learning Experience Seriously

Summary: HRI meets MLTaking Learning Experience Seriously

Students

Matt Berlin

Andrew Brooks

Jesse Gray

Guy Hoffman

Our collaborators

Stan Winston Studio

Cory Kidd

Jeff Lieberman

Andrea Thomaz

Dan Stiehl

ContributorsContributors

Funding

TTT & DL Media Lab Consortia

ONR YIP

DARPA MARS, BICA

Toyota

Thank You!

For more infoFor more info www.media.mit.edu/~cynthiab robotic.media.mit.edu Designing Sociable Robots (2002) MIT

Press

www.media.mit.edu/~cynthiab robotic.media.mit.edu Designing Sociable Robots (2002) MIT

Press