socially-aware conversational agents · socially-aware conversational agents justine cassell...
TRANSCRIPT
Justine Cassell
Socially-Aware
Conversational Agents
Justine Cassell
Carnegie Mellon University
July 2017
Justine Cassell
The ineffable quality of Rapport
Children whoreport more rapport with Alex are more likelylearn from the virtual peer
Justine Cassell
Search Engine vs. Socially Aware
Justine: “OK Google, I love Manchester United”
Google:
Justine: “SARA, I love Manchester United”
SARA: “No way! Arsenal wipes the floor with those Red Devils!”
Manchester United Football Club is a professional football clubBased in Old Trafford, Greater Manchester, England, that competesin the Premier League, the top flight of English Football
Justine Cassell
Chatbot vs. Socially Aware
SARA, 00:40,2, well, let’s find another attendee for you to meet who fits your
interests and who is an old timer and can show you around, 00:50,1
XiaoIce SARA
Justine Cassell
Motivation for Socially-Aware Agents
1) 1. People pursue multiple conversational goals in every conversation &expect the same from their interlocutors. To put people at ease, andincrease relationship strength, we must understand the propositional,interactional & interpersonal functions of conversation.
2) 2. People change interaction styles over time. We must manage long-term interactions with people by changing interaction style in a way thatindicates the system’s increased rapport, intimacy, and trust.
Justine Cassell
Goal of Socially Aware Systems
Development of an agent that manages interpersonal rapport (relationship strength) with users over interactions across time, as well as managing propositional and interactional goals, in order to improve task performance.
Automatically recognize rapport-managing conversational strategies from verbal, visual and vocal modalities of speaker and interlocutor, both within the individual and in the dyad.
Automatically generate responses, in verbal, visual and vocal modalities that manage rapport as well as fulfilling system intentions / task goals.
Justine Cassell
Not just chit-chat:
rapport improves task performanceSurveys
– Survey respondents give higher quality answers if they feel rapport with interviewer (Berg (1989)
Health
– Physicians who build rapport during trial interviews enroll more participants (Albrecht et al., 1999).
Sales
– Rapport with sales staff leads to increased likelihood of purchasing goods/service (Brooks, 1989).
– Customers show increased trust and disclosure when rapport is maintained with sales staff (LaBahn, 1996).
Justine Cassell
Theorize & Model
BuildStudy
CollectNatural data
Build formalmodels
Implement system on the basis of model
Design evaluationof use
Realize gaps in understanding
Test
rules
Start here
Methodology
Justine Cassell
Ob
serv
e
Justine Cassell
Analyze
Justine Cassell
With
Ran
Zhao
Data- &Theory-Driven
Model of Rapport Management
Justine Cassell
Agent Model of Rapport must be:
1. Dyadic,
2. Multi-level: differentiate between observable signals & underlying psychological states,
3. Sensitive to effect of time
4. Cross-Modal
Psychological States
ConversationalStrategies
Multimodal Behaviors
Pragmatic / SocialFunctions & Goals
Vocal Visual Verbal
Trust Rapport Intimacy
Multi-level Multi-modal Multi-functional Model of Dyadic Psych States over Time
with L.P. Morency, 2015
Justine Cassell
Temporal association rules: Friends
Example: Friend in high rapport
Tutor: Sweeney you can't do that, that's the whole point {smile} [Violation of Social Norm]
Tutee: I hate you. I'll probably never never do that [Reciprocate Social Norm Violation]
Tutor: Sweeney that's why I'm tutoring you {smile}
Tutee: You're so oh my gosh {smile}.We never did that ever [Violation of Social Norm]
Tutor: {smile} What'd you say?
Justine Cassell
Data- &Theory-Driven process modelRapport building Reasoning Strategies
Rapport
Friends• Violate sociocultural
norms to fit interlocutor’s behavior expectation
• Mark in- and out-group• Learn behavior
expectations
Strangers• Politeness according
to sociocultural norms
• Learn behavior expectations
Shared personal knowledge gap
Self
-dis
clo
sure
Shared personal knowledge
Self-disclosure
Reciprocity
condition
Reasoning Process
Goal
Norm Recip.
Social Norm
Conv. Norm
Relationship Stage (T1/T2)
Attributed Process
outin
GenerateImagined interaction:• Min.
vulnerability• Influence subs.
attr. process
EstimateAppropriate social distance at right time
Rel. def. (soc. ID)
content
Match “likeness” and rel. stage
(T1/T2)
Discloser
Recipient
Trust
Po
sitive Face En
han
cemen
t Sign
al Go
al of R
elation
ship
Develo
pm
en
t In
dex “Tru
e self”
Soci
al V
alid
atio
n
Soci
al C
on
tro
lR
eci
pro
city
of
Self
-dis
clo
sure
smal
l tal
k, g
oss
ip, …
Justine Cassell
Socially-Aware Agent Architecture
Speech RecognitionNatural Language
Understanding
Dialogue
Manager
Content DB(schedule of person,
event)
Social
ReasonerLanguage
Generation
OpenFace
Behavior RecognitionCamera
Microphone
User Model
Recommendation
System
Nonverbal
Behavior
Generation
Virtual Body
Text-to-SpeechGame World
Conversational Strategy
Classifier
Rapport
EstimatorSpeech Analysis
Justine Cassell
Evaluate
Total Interactions: 250+ sessions
Current Dataset: 120 sessions (totally 10+ hours)• Mean duration = approx. 5 min.• SD = 69.00 seconds
Justine Cassell
Some Applications:
Rapport-Aware Peer Tutor (RAPT)
Justine Cassell
ASR BEAT
OpenSmile
Rapport Score Estimator
Camera
Socially-inflected task utterance
Images(640x480, 30fps)
Mic
Acoustic Features
Smart Body
TTSUnity 3D (agent)
Text
Speech Signal
Social Conversational Strategy Classifier
FACS
Step-level problem correctness
OpenFace
Tutoring Move Finite State
Machine
Social Reasoner
WoZ: Task NLU and Task Reasoner
Condition 1:Social behaviors gradually increasing over time
Recorded Video/Audio of User
Screen Recorder
Tutoring and Social utterance database
Problem-Solving Interface
ProblemSelection
Condition 2:Social behavior chosen based on current rapport level, previous social moves, and users’ nonverbal behaviors
Pre- and Post-conditions for Algebra task
Agent’s next tutoring move
Learner Model
Problem and step-level correctness of students’ typed responses
RAPT WoZ System Architecture
Rapport Score
Agent’s selected task utterance
Agent animations
Synthetic Speech Markup Language (SSML)
Behavioral Markup Language (BML)User’s social
utterance label
Justine Cassell
Application: Mobile front-end to apps
Justine Cassell
with Yahoo InMind Team
Justine Cassell
Justine Cassell
SARA: Socially-Aware Robot Assistant at Davos
Justine Cassell
SARA: Socially Aware Robot Assistant
Justine Cassell
