Transcript
Page 1: Complementing Embodied Conversational Agents by Implementing Compliments Measuring the Effect of Compliments in Embodied Conversational Agents Group 23

Complementing Embodied Conversational Agents by Implementing ComplimentsMeasuring the Effect of Compliments in Embodied Conversational Agents

Group 23 Marieke Agterbos | Tim van Bremen | Carel Jansen | Lisa Oud | Justin Post

Page 2: Complementing Embodied Conversational Agents by Implementing Compliments Measuring the Effect of Compliments in Embodied Conversational Agents Group 23

Research Question

To what extent do compliments have an impact on the

user experience of embodied conversational agents?

Page 3: Complementing Embodied Conversational Agents by Implementing Compliments Measuring the Effect of Compliments in Embodied Conversational Agents Group 23

Questions

1. Are the ECA’s compliments recognized as compliments? (Part

I)

2. Experience with vs. without compliments (Part II)

A. Perceived liking of the agent

B. Perceived level of comfort

C. Perceived information quality

Page 4: Complementing Embodied Conversational Agents by Implementing Compliments Measuring the Effect of Compliments in Embodied Conversational Agents Group 23

Background theory

Making machines more human (Picard, 2000)

Minimize gap between real life contact and digital contact

Lifelike ECA (Louwerse et al., 2009)

Is sensitive to moods and sentiments

Politeness and variation in language (Behnam & A. Niloufar, 2011)

Add value and believability

Function of Compliments (Manes en Wolfson, 1981)

Goodwill and solidarity

Three types of personal compliments (Jucker, 2009)

Implicit, explicit or indirect

Behavioral realism might be more important than photorealism (Groom, 2009)

Generating positive response

More productive conversation

Page 5: Complementing Embodied Conversational Agents by Implementing Compliments Measuring the Effect of Compliments in Embodied Conversational Agents Group 23

Pre-test

Are the ECA’s compliments recognized as compliments? Participants read a transcript of an ECA conversation

Questionnaire on different parts of the conversation:

1. Bot: ‘Hello, how are you?’2. User: ‘Great, and you?’3. Bot: ‘Good, so am I.’4. User: ‘Good.’5. Bot: ‘What was the name of the lead guitar player of the band called “A band of Gypsies?”’6. User: ‘Jimi Hendrix.’7. Bot: ‘That is correct.’8. Bot: ‘Someone told me you are very bright.’

Page 6: Complementing Embodied Conversational Agents by Implementing Compliments Measuring the Effect of Compliments in Embodied Conversational Agents Group 23

Pre-test results

‘Yes, this is a significant difference (p<0.01)’

84 participants (53, 31)

Page 7: Complementing Embodied Conversational Agents by Implementing Compliments Measuring the Effect of Compliments in Embodied Conversational Agents Group 23

Pre-test results

 

 Explicit Implicit Indirect

Non-contextual

‘You are such a nice person!’

‘People like you make this world a better

place.’

‘Someone told me you are very bright.’

‘I wish everyone was as nice as you.’

‘It seems like I’m having a conversation with an intellectual.’

‘My colleague told me you have a great

personality.’

Contextual

‘You are smarter than most respondents!’

‘You seem to know your instruments!’

‘According to our system, you score better than most

people!’

‘You answered really fast!’

‘It seems like you know your politics!’

‘According to our database, you know a

lot about music!’

Page 8: Complementing Embodied Conversational Agents by Implementing Compliments Measuring the Effect of Compliments in Embodied Conversational Agents Group 23

Experiment Design

Measured variables Liking

Comfort

Information Quality

Contextual Non-contextual

Based on input data Based on ‘nice-to-hear’statements

Related to conversational content Unrelated to conversational content

‘You seem to know your instruments!’

‘People like you make this world a better place.’

Page 9: Complementing Embodied Conversational Agents by Implementing Compliments Measuring the Effect of Compliments in Embodied Conversational Agents Group 23

Experiment

53 Participants

Conversation with bot Quiz setting

Questionnaire Perceived liking

Perceived level of comfort

Information quality

“How many strings does a guitar have?”“Who is the prime minister of the Netherlands?”

Page 10: Complementing Embodied Conversational Agents by Implementing Compliments Measuring the Effect of Compliments in Embodied Conversational Agents Group 23

Results

Page 11: Complementing Embodied Conversational Agents by Implementing Compliments Measuring the Effect of Compliments in Embodied Conversational Agents Group 23

Effects

Page 12: Complementing Embodied Conversational Agents by Implementing Compliments Measuring the Effect of Compliments in Embodied Conversational Agents Group 23

Results H1. Contextual compliments have a positive effect on the user experience of an ECA.

H1-a. perceived liking of the agent.

H1-b. perceived level of comfort.

H1-c. perceived quality of information.

H2. Non-Contextual compliments have a positive effect on the user experience of an ECA.

H2-a. perceived liking of the agent.

H2-b. perceived level of comfort.

H2-c. perceived quality of information.

H3. Contextual compliments have a more positive effect than non-contextual compliments.

H3-a. perceived liking of the agent.

H3-b. perceived quality of information.

H3-c. perceived level of comfort.

Page 13: Complementing Embodied Conversational Agents by Implementing Compliments Measuring the Effect of Compliments in Embodied Conversational Agents Group 23

Results H1. Contextual compliments have a positive effect on non-contextual

compliments on the user experience of an ECA.

H2. Non-Contextual compliments have a positive effect on the user experience of an ECA.

H3. Contextual compliments have a more positive effect than non-contextual compliments.

H4. Compliments have a positive effect on the user experience of an ECA.

H4-a. perceived liking of the agent.

H4-b. perceived quality of information.

H4-c. perceived level of comfort.

Page 14: Complementing Embodied Conversational Agents by Implementing Compliments Measuring the Effect of Compliments in Embodied Conversational Agents Group 23

Conclusion

Compliments have a positive effect on the user experience of an ECA.

However, not on likingness.

The type of compliment matters

Non-contextual > contextual

Page 15: Complementing Embodied Conversational Agents by Implementing Compliments Measuring the Effect of Compliments in Embodied Conversational Agents Group 23

Discussion

Pros:

Pretest, kind of compliments accounted for.

Pretest in line with experiment outcome.

All experiments under supervision of a test leader.

Improvements:

Sample too small

Language, not native language of all subjects

Variable mount of compliments given

Compliments when correctly answered

Definition of comfort, persuasiveness, likeliness might differ individually

Page 16: Complementing Embodied Conversational Agents by Implementing Compliments Measuring the Effect of Compliments in Embodied Conversational Agents Group 23

Questions?

Page 17: Complementing Embodied Conversational Agents by Implementing Compliments Measuring the Effect of Compliments in Embodied Conversational Agents Group 23

Measuring - Liking

‘I would like to be friends with her’

‘If I was upset, she would be able to cheer me

up.’

‘She is a bad listener’

Rated on Likert scale 1-7

Page 18: Complementing Embodied Conversational Agents by Implementing Compliments Measuring the Effect of Compliments in Embodied Conversational Agents Group 23

Measuring - Level of comfort

During the conversation, I felt:

Comfortable

Involved

Open

Relaxed

Warm

Rated on Likert scale 1-7

Page 19: Complementing Embodied Conversational Agents by Implementing Compliments Measuring the Effect of Compliments in Embodied Conversational Agents Group 23

Measuring - Information quality

The extent to which the information was:

Correct

Helpful

Persuasive

Insightful

Relevant

Rated on Likert scale 1-7

Page 20: Complementing Embodied Conversational Agents by Implementing Compliments Measuring the Effect of Compliments in Embodied Conversational Agents Group 23

Context

Controlled environment

Participants from same population


Top Related