Backchannels Through Backchannels Through Gaze as Indicators of Gaze as Indicators of Persuasive SuccessPersuasive Success

E. Bevacqua, M. Mancini, E. Bevacqua, M. Mancini, C. Peters,C. Peters, C. PelachaudC. Pelachaud

University of Paris 8University of Paris 8

Isabella PoggiIsabella PoggiUniversità Roma TreUniversità Roma Tre

GOALGOAL DefineDefine speaker that alters gaze behaviour during speaker that alters gaze behaviour during

discourse for the purposes of persuasive discourse for the purposes of persuasive interactioninteraction

Speaker monitors amount of interest listener has Speaker monitors amount of interest listener has in what it is sayingin what it is saying

Part of wider framework (WP6): Part of wider framework (WP6): • An agent that can establish, maintain and close a An agent that can establish, maintain and close a

conversation (Sidner, 2004)conversation (Sidner, 2004)

Significance to WP8: Significance to WP8: • consider reactive backchannels through attention and consider reactive backchannels through attention and

interestinterest

• important prerequisites for persuasionimportant prerequisites for persuasion

BackgroundBackground Interaction is by no means a one way Interaction is by no means a one way

communication channel between partiescommunication channel between parties Within an interaction, parties take turns in Within an interaction, parties take turns in

playing the roles of the speaker and of the playing the roles of the speaker and of the listenerlistener

Where persuasion fits in: Where persuasion fits in: • Speaker to monitor listeners attention and interest in Speaker to monitor listeners attention and interest in

what they have to saywhat they have to say Interest maintenance as prerequisite for persuasive Interest maintenance as prerequisite for persuasive

interactioninteraction

• Listener to persuade the speaker that they are paying Listener to persuade the speaker that they are paying attentionattention

Vital for maintaining engagementVital for maintaining engagement

Role of SpeakerRole of Speaker

Speaker:Speaker:• Pointless for a speaker to engage in an act of Pointless for a speaker to engage in an act of

communication if listener does not pay or communication if listener does not pay or intend to pay attentionintend to pay attention

• Important for speaker to assess listener’s Important for speaker to assess listener’s engagement at:engagement at:

when starting an interaction: assess the possibility of when starting an interaction: assess the possibility of engagement in interaction (engagement in interaction (establish phaseestablish phase))

when interaction is going on: check if engagement is when interaction is going on: check if engagement is lasting and sustaining conversation (lasting and sustaining conversation (maintain phasemaintain phase))

Role of ListenerRole of Listener ListenerListener

• attentionattention: pay attention to the signals produced by : pay attention to the signals produced by speaker to perceive, process and memorize themspeaker to perceive, process and memorize them

• perceptionperception: of signals: of signals

• comprehensioncomprehension: understand meaning attached to : understand meaning attached to signalssignals

• internal reactioninternal reaction: the comprehension of the meaning : the comprehension of the meaning may create cognitive and emotional reactionmay create cognitive and emotional reaction

• decisiondecision: decision to communicate the internal : decision to communicate the internal reactionreaction

• generationgeneration: display behaviors: display behaviors

Application to Persuasive ECAsApplication to Persuasive ECAs Persuasive ECAs ought to be able to:Persuasive ECAs ought to be able to:

• establish backchannels with the otherestablish backchannels with the other

• conduct eye contact conduct eye contact properlyproperly Maintain mutual eye contact but…Maintain mutual eye contact but… Don’t stare or threatenDon’t stare or threaten

• interpret looking behaviours of the otherinterpret looking behaviours of the other

Requires engagement capabilitiesRequires engagement capabilities Perception of gazePerception of gaze Establish backchannelsEstablish backchannels Generate gazeGenerate gaze

Key Capabilities in our ModelKey Capabilities in our Model

Listener backchannelListener backchannel Speaker interpreting listener Speaker interpreting listener

behaviourbehaviour Speaker generating or altering its Speaker generating or altering its

own behaviourown behaviour

1. Listener Backchannel1. Listener Backchannel During conversation…During conversation…

• two streams of information exchanged at the two streams of information exchanged at the same time between participants (the speaker and same time between participants (the speaker and listener). listener).

• first one concerned with the topic of the first one concerned with the topic of the conversationconversation

• second provides information on the second provides information on the successfulness of the communicationsuccessfulness of the communication

Listener can show level of engagement in Listener can show level of engagement in conversationconversation• Through linguistic and gestural signals, called Through linguistic and gestural signals, called

backchannel feedbacksbackchannel feedbacks

BackchannelBackchannel

Depending on the type of speech act they Depending on the type of speech act they respond to, a signal will be interpreted as respond to, a signal will be interpreted as a backchannel or not.a backchannel or not.• backchannel: a signal of agreement / backchannel: a signal of agreement /

disagreement that follows the expression of disagreement that follows the expression of opinions, evaluations, planningopinions, evaluations, planning

• not a backchannel: a signal of comprehension / not a backchannel: a signal of comprehension / incomprehension after an explicit question incomprehension after an explicit question « Did you understand? »« Did you understand? »

Listener BackchannelListener Backchannel based on taxonomy of Poggi (2005)based on taxonomy of Poggi (2005)

• attentionattention• comprehensioncomprehension• believabilitybelievability• interestinterest• agreementagreement

positive/negativepositive/negative any combination of the aboveany combination of the above

• pay attention but not understand; understand but not pay attention but not understand; understand but not believebelieve

Our focus: interest and attention on a signal level Our focus: interest and attention on a signal level (not on a cognitive level) (not on a cognitive level) • listener’s feedback is triggered by the speaker’s listener’s feedback is triggered by the speaker’s

behaviour and not by what the speaker is sayingbehaviour and not by what the speaker is saying

Backchannel modellingBackchannel modelling

Reactive modelReactive model• generates an instinctive feedback without generates an instinctive feedback without

reasoningreasoning

Cognitive modelCognitive model• conscious decision to provide backchannel to conscious decision to provide backchannel to

provoke a particular effect on the speaker or to provoke a particular effect on the speaker or to reach a specific goalreach a specific goal

Currently, our model is Currently, our model is reactivereactive in nature in nature• Dependant on perceptionDependant on perception

• Extendable to include limited language Extendable to include limited language reasoning by inclusion of APML tagsreasoning by inclusion of APML tags

Listener BackchannelListener Backchannel Listener’s behaviour produced to Listener’s behaviour produced to

show feedback can be categorized as show feedback can be categorized as follows:follows:• gaze,gaze,

• paraverbals,paraverbals,

• head movements, like nods, jerks, head movements, like nods, jerks, shakes, tilts,shakes, tilts,

• facial expressions,facial expressions,

• other gestures such as hand and body other gestures such as hand and body movements.movements.

Backchannel signals of gazeBackchannel signals of gaze GazeGaze

• shows direction of attentionshows direction of attention• informs on level of engagement or on intention to informs on level of engagement or on intention to

maintain engagementmaintain engagement• indicates degree of intimacy indicates degree of intimacy

but alsobut also• monitors the gaze behavior of others to establish their monitors the gaze behavior of others to establish their

intention to engage or maintain engagedintention to engage or maintain engaged Shared attention Shared attention

• involves mutual gaze at other partner or mutual gaze at involves mutual gaze at other partner or mutual gaze at a same objecta same object

By analysing the listener’s gaze behaviourBy analysing the listener’s gaze behaviour• can compute his level of attention and interest can compute his level of attention and interest

E.g. if the listener starts to frequently avert his eyes from E.g. if the listener starts to frequently avert his eyes from the speaker, it means that he is probably losing interest the speaker, it means that he is probably losing interest and that he is not paying attention anymore.and that he is not paying attention anymore.

2. 2. Speaker interpreting listenerSpeaker interpreting listener

Speaker perceives and interprets Speaker perceives and interprets behaviour of listener behaviour of listener • determine effectiveness and determine effectiveness and

progression of persuasion attemptsprogression of persuasion attempts

• perceives eye, head and body perceives eye, head and body directions of the listenerdirections of the listener

‘‘Snapshot’ perceptual updatesSnapshot’ perceptual updates Weighted sum forms an Weighted sum forms an attention level…attention level… ……the amount of attention that the other is the amount of attention that the other is

paying at paying at thisthis instant of time instant of time

MetricsMetrics Based on virtual visual sensing of the others looking Based on virtual visual sensing of the others looking

behaviour behaviour Used to interpret and generate ECA gaze Used to interpret and generate ECA gaze

behaviours. behaviours. • Attention Level:Attention Level: amount of visual attention one is perceived amount of visual attention one is perceived

to be paying at an instant of time, based on their eye, head to be paying at an instant of time, based on their eye, head and body directions.and body directions.

• Interest Level:Interest Level: amount of attention perceived to be paying amount of attention perceived to be paying over a period of time and is a measurement of the others over a period of time and is a measurement of the others interest in what the speaker has to say. interest in what the speaker has to say.

• Effectiveness: Effectiveness: indication of how well communication is indication of how well communication is progressing – that is, how the listener is perceived to be progressing – that is, how the listener is perceived to be behaving (e.g. attending, looking elsewhere, not interested)behaving (e.g. attending, looking elsewhere, not interested) when the speaker is saying something important.when the speaker is saying something important.

InterpretingInterpreting An agent’s attention at one instant of time can be An agent’s attention at one instant of time can be

misleading…misleading…• … … so a profile of attention over time is consideredso a profile of attention over time is considered• Attention levels over time form an Attention levels over time form an attention profileattention profile

• Integration produces Integration produces interest levelinterest level metric metric• EffectivenessEffectiveness of speakers communication used to alter of speakers communication used to alter

speaker’s behaviour towards the listener speaker’s behaviour towards the listener frames the actions of the listener in the context of what the frames the actions of the listener in the context of what the

speaker is saying.speaker is saying.

3. Speaker generating / altering 3. Speaker generating / altering its behaviourits behaviour

SpeakerSpeaker• Alters its behaviour to try to be more Alters its behaviour to try to be more

persuasive and obtain listener's persuasive and obtain listener's attentionattention

Variation in gaze parametersVariation in gaze parameters

• Abides by timing constraints for gazeAbides by timing constraints for gaze maximum amount of time speaker can look at maximum amount of time speaker can look at

listener without interruptionlistener without interruption maximum amount of time speaker and listener can maximum amount of time speaker and listener can

engage in mutual attentionengage in mutual attention

Speaker generating / altering its Speaker generating / altering its behaviourbehaviour

During behaviour generation:During behaviour generation:• level of effectiveness is constantly level of effectiveness is constantly

calculatedcalculated

• compared to a desired levelcompared to a desired level

• if it is below this threshold, the speaker if it is below this threshold, the speaker increases its tendency to look at the increases its tendency to look at the listenerlistener

• temporal constraints are applied temporal constraints are applied

• final behaviour is decided by a final behaviour is decided by a probabilistic model probabilistic model

AlgorithmAlgorithm

input: APML tagged text, effectiveness input: APML tagged text, effectiveness threshold and maximum duration of gaze threshold and maximum duration of gaze direction for speaker S and listener Ldirection for speaker S and listener L

compute behaviors for S and L based on compute behaviors for S and L based on APML tags specificationAPML tags specification

compute metrics compute metrics • attention profileattention profile

• level of interestlevel of interest

• level of level of effectivenesseffectiveness of conversation of conversation

VideosVideos

Low interestLow interest High interestHigh interest

Future WorkFuture Work Interpretation module based on perceptionInterpretation module based on perception

• listener’s gazelistener’s gaze

Simplified language-based planning Simplified language-based planning schemescheme• use of hard-coded tags in the input APML providing use of hard-coded tags in the input APML providing

semantic information related to that utterance semantic information related to that utterance

Multiple choice selection for the speaker Multiple choice selection for the speaker • at each stage in the dialogue choose between at each stage in the dialogue choose between

predefined utterances associated with slightly different predefined utterances associated with slightly different verbal or behavioural characteristics depending on the verbal or behavioural characteristics depending on the goal of the speaker e.g. increase in volume, increased goal of the speaker e.g. increase in volume, increased expressivity of hand gestures.expressivity of hand gestures.

Future WorkFuture Work Planner uses interpretation of Planner uses interpretation of

• perceived behaviour of the listener during the previous perceived behaviour of the listener during the previous utterance utterance

backchannelling, in particular head nods and facial expressionsbackchannelling, in particular head nods and facial expressions Gaze behaviourGaze behaviour

• comparison with the importance of the utterancecomparison with the importance of the utterance APML tags corresponding to importance of speech segmentsAPML tags corresponding to importance of speech segments effectiveness of conversationeffectiveness of conversation theory of the interaction goal of the listener (e.g. maintain or finish theory of the interaction goal of the listener (e.g. maintain or finish

conversation). conversation).

Enhance listener backchannel modelEnhance listener backchannel model• new backchannel signals (head nods, facial expressions)new backchannel signals (head nods, facial expressions)

• feedbacks triggered also by the semantic of Speaker’s wordsfeedbacks triggered also by the semantic of Speaker’s words

• reactive model -> reactive+cognitive modelreactive model -> reactive+cognitive model