towards affective interactive ecas catherine pelachaud – university of paris 8 isabella poggi –...

Towards Affective Interactive ECAs

Catherine Pelachaud – University of Paris 8

Isabella Poggi – University of Rome 3

WP6 workshop, 10-11 March 2005, Paris

humaine WP6 workshop2

WP6 Emotion in Interaction

Embodied Conversational Agents

WP6 core task: describe an interactive ECA system with capabilities beyond those of present day ECAs

Conceptual framework consists of three key domains: perception, interaction and generation

Advance ECA capabilities in the key domains of perception, interaction and generation



Perception: perception from audio and visual information;

Agent perception of the users / other agents / world.

human perceptual attention

Interaction: role of emotions in interaction; Balanced perception and generation

Interactants model of speaker(s) and listener(s)Agent Interpretation of discourse signals Active interpretation of signals

Social attention for speakers and listeners based on facial expressions during social interaction

Create affective awareness



Generation: design of expressive signs. Produce dynamic expressive visual and auditory behaviours

Achieve coordination of signs of emotion in multiple modalities

Define representation language to encode information and use it to drive animation of ECAs


Current conception

ECAs with more or less sophisticated presentation facilities. ECAs are to a certain extent able:

to compute what to say based on discourse plans

to generate synchronized behaviours

to talk with emotional appropriate intonation

to show some emotional expressions

to perform gesture and body movement


What needs to be done

improve the naturalness of ECAs

multimodal integration and the display of emotional behaviours

improve the believability of ECAs

consistency of an ECA’s behaviour in terms of personality, cultural context and situation.

improve the interactivity of ECAs

take into account the particular user and the particular context

markes up its own model of user and of context


ECA

ECA ought to be able to interact with the User not only through written text, but with all modalities:

words, voice, gesture,

gaze, facial expression,

body movements, body posture

(even through touch?)

conceive, represent and convey all the possible meanings that natural language and multimodal interaction convey in humans


ECA - Capabilities

Capabilities required from an ECA corresponds to steps to move research forward

WP6 exemplar: Definition of Affective Interactive Embodied Conversational Agents with several capabilities covering the 3 domains:

perception

interaction

generation


ECA - Capabilities

Cognitive influences on action Emotion related attention shifts

Adapt politeness behaviours to the user’s emotional state

Creating Affective Awareness Creating Affective Bonds

Imitation

Adaptation

Backchanneling


ECA - Capabilities

Coordination of signs in multiple modalities From multimodal emotional corpora to models of coordination between modalities

Multimodal behaviour

Gesture repositories

Expressivity Behaviour expressivity

Speech Expressivity

Context dependent emotional body gesture


Phases of Research

Empirical researchModelling Implementation Evaluation


Phases of Research

Phase 1 – Empirical Research Find out regularities in mind and behavior of HumansDefine lexicons for all modalities: face, gaze, gesture, … through:

empirical research (questionnaires)analysis of observational data

Cooperation with WP3 (Theories and Models) and WP5 (Data and Database)


Phases of Research

Phases 2 and 3 – Modelling and Implementation

formalisation and representation of the rules found out in the previous phase

Agent made of mind and bodyrepresent signals properties (shape and temporal dynamism)represent mind model

Cooperation with WP4 (Signs of Emotion), WP7 (Emotion in Cognition and Action), WP8 (Communication and Emotion)


Phases of Research

Phase 4 – Evaluationsimilarity of ECA behavior with human behavior

believability of ECA animation

added-value of ECA in interaction system

how ECAs fit user’s need

Cooperation with WP9 (Usability)


Example of ECA

Greta: ECA systemexpressive agent

individuality specification

based on a model of communicative functions by I. Poggi


Model of Discourse and Conversation

Based on Castelfranchi & Parisi and Poggi

A sentence performs an action. It aims at a goal, and possibly at one or more supergoals

A discourse is made of several sentences which aims at the same supergoals

A conversation is a sequence of discourse where interactants “adopt” or “adhere” to each other’s goals

adopt: I1 acts to fulfil I2’s goal even if I2 has not asked I1 to do so

adhere: I1 acts to fulfil I2’s goal that I2 has explicitly asked I1 to do


Multimodal Communication

Taxonomy of Communicative Functions (I. Poggi)

The speaker may provide three broad types of information about:

Information about the world: deictic, iconic (adjectival),…

Information about the speaker’s mind:

belief (certainty, meta cognitive)

Goal (performative, rheme/theme, turn-system, belief relation)

emotion

Information about speaker’s identity (sex, cutlure, age…)


Information about Speaker’s Belief

Speaker has the goal that Addressee believes some specific speaker’s beliefs

Certainty markers: Indicate how reliable the provided information is

Meta-cognitive marker: Meta information about the source or cognitive states of information the speaker is taking about: gaze of thought.

Goal S Bel A (Bel S b)


Information about Speaker’s Goal

Information about intention of single communicative act: performative of a sentence

Information about a whole hierarchy of intentions: planning of a sentence

Information about overall arrangement of conversation: turn-taking

Information about the relation between beliefs

Goal S Bel A Intend S Ag_Act

Ag_Act: agent’s action


Performative Marker

Communicate illocutionary force of speech actI order you to put on your coat

I implore you to put on your coat

I advise you to put on your coat

Communicative intention of speaker that includes:

Her goals (her reason to communicate to the addressee, what she wants the addressee to do)

Her social relationship she wants to establish with addressee


Performative

3 main classes of performative (General Goals):

request, inform, ask

Request: command, implore, advise…

Inform: warn, tell, announce…

Ask: question, interrogate…

Propositional content: set of beliefs speaker is mentioning and what she has to say about them.


Performative

Elements distinguishing performatives within a same General Goal:

In whose interest is the action requested: S or A* Advise vs command, inform vs warn

Degree of certainty of S’s beliefs: certain, uncertain

* Suggest vs assure

Power relationship between S and A:* Command vs implore


Performative: Peremptory Order

Example:In whose interest is the action: S

Interest S aDegree of certainty: unmarkedPower relationship: S has power on A

Bel S (Power-on S A)Performative: S orders A to do action a.

Ex: I order you to come here right awayGoal S (Do A a)

Signal of peremptory order: S looks down on A and frowns

Bel S (if (not Do A a) then (Angry S))


Performative: Implore

Example:In whose interest is the action: S

(interest S a)Degree of certainty: unmarkedPower relationship: A has power on S

Bel S (Power-on A S)

Performative: S implores A to do action aEx: I implore for your help; I can’t do it otherwise

Goal S (Do A a)

Signal of imploration: S bends head aside and raised inner eyebrow

Bel S (if (not Do A a) then (Sad S))


Information about Speaker’s Affective State

Emotion are triggered by an event, action, person

One can feel emotion toward another person (love, scorn)

Emotion triggered by an event and not directed toward someone (fear, surprise)

Signal: facial expressions of emotion

Display of emotion are regulated

Display or not of the emotion based on context

Goal S Bel A (Feel S e)


Language of control of agent’s behaviors

Based on APML: Affective Presentation Markup

Language

XML-based mark-up language

Describe the communicative functions (I. Poggi)

Control the agent’s animation

Provide a mapping between meaning and signal


Expression Meaning

deictic: this, that, here, there

adjectival: small, subtle, difficult,…; or big, long, great,…

certainty: certain, uncertain, …

metacognitive: I'm planning, I'm thinking, …

performative: greet, request (implore, order, suggest...), inform (warn, approve...),

topic comment: this is the topic, this is the comment

Belief relation: contrast,…

turn allocation: take turn, give turn

affective: anger, fear, happy, sorry-for, envy, relief, …


Expression signals

Deictic: gaze direction, head direction

Certainty:Certain: small intensity frown

Uncertain: raised eyebrow

adjectival:Small, tiny…: small eye aperture, small opening of fingers

Belief relation: contrast: raised eyebrow; sequential symmetrical hand palm up

Performative: Suggest: small raised eyebrow, head aside

Emotion:Sorry-for: head aside, inner eyebrow up

…


Example

<performative type="inform"><theme> As far as

<emphasis x-pitchaccent="LplusHstar"> vitamins </emphasis>

are concerned <boundary type="LH"/></theme>

</performative><rheme><emphasis x-pitchaccent="Hstar"> research </emphasis>

has shown <boundary type="LH"/></rheme>

<performative type="inform"><rheme>that eating

<emphasis x-pitchaccent="Hstar">the recommended </emphasis>levels of vitamin

<emphasis x-pitchaccent="Hstar"> A </emphasis>and

<emphasis x-pitchaccent="Hstar">C</emphasis><boundary type="LH"/></rheme>

<rheme affect="sorry-for">can have <emphasis x-pitchaccent="Hstar">beneficial </emphasis><emphasis x-pitchaccent="Hstar">effects </emphasis>for your

<emphasis x-pitchaccent="Hstar">appearance </emphasis> <boundary type="LH"/>

and <emphasis x-pitchaccent="Hstar">health</emphasis><boundary type="LL"/>

</rheme></performative>


Overall Architecture

Björn Hartmann, Myriam Lamolle, Maurizio Mancini


Input

Input text marked with APML Communicative Function tags and their coefficient of communicative strength

Agent DefinitionExpressivity of agent

Spatial, temporal, fluidity, power, repetitivity, overallActivity

Agent-specific hierarchy of modalities (Which modalities the Agent tends to prefer?)

Face, gaze, gesture, posture, …

Predisposition factors (How expressive are each modalities?)Face, gaze, gesture, posture, …

Database of <meaning, signals, relevance>


Video


Conclusion

Creation of an Affective Interactive Embodied Conversational Agent

perception

interaction

generation

Collaboration requires within and across Workpackges

towards affective interactive ecas catherine pelachaud – university of paris 8 isabella poggi –...

Documents