joemon m jose (with ioannis arapakis & ioannis konstas) department of computing science

26
Exploring the role of affective feedback in Interactive IR Joemon M Jose (with Ioannis Arapakis & Ioannis Konstas) Department of Computing Science

Post on 19-Dec-2015

213 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Joemon M Jose (with Ioannis Arapakis & Ioannis Konstas) Department of Computing Science

Exploring the role of affective feedback in Interactive IR

Joemon M Jose(with Ioannis Arapakis & Ioannis Konstas)

Department of Computing Science

Page 2: Joemon M Jose (with Ioannis Arapakis & Ioannis Konstas) Department of Computing Science

Affective Feedback 2

Questions?What is the role of emotions in the

information seeking process?

Do they correspond to any form of relevance feedback?

How can we effectively employ them in information retrieval scenarios?

02/03/2009

Page 3: Joemon M Jose (with Ioannis Arapakis & Ioannis Konstas) Department of Computing Science

Affective Feedback 3

Relevance Feedback• Relevance assessments can contribute in the

disambiguation of the user’s information need

• This is achieved through the application of various feedback techniques

02/03/2009

Page 4: Joemon M Jose (with Ioannis Arapakis & Ioannis Konstas) Department of Computing Science

Affective Feedback 4

Explicit Relevance Feedback• Feedback which is obtained through the explicit

and intended indication of documents as relevant (positive feedback) or irrelevant (negative feedback)

02/03/2009

Page 5: Joemon M Jose (with Ioannis Arapakis & Ioannis Konstas) Department of Computing Science

Affective Feedback 5

Explicit Relevance FeedbackBenefits Drawbacks

Robust method for inferring relevance feedback

Interrupts the flow of the search process

Better query reformulationsIntroduces the cognitive

burden of explicit relevance judgments

Improves considerably the retrieval performance of a

system

Trade-off between the users perusing documents because

the system expects them to do so and because they are

genuinely interested

02/03/2009

Page 6: Joemon M Jose (with Ioannis Arapakis & Ioannis Konstas) Department of Computing Science

Affective Feedback 6

Implicit Relevance Feedback• Implicit Feedback: a passive form of feedback,

which is applied in an intelligent and unobtrusive manner

• Can be used to individualize a system’s responses or develop user models (UM)

02/03/2009

Page 7: Joemon M Jose (with Ioannis Arapakis & Ioannis Konstas) Department of Computing Science

Affective Feedback 7

Implicit Relevance FeedbackBenefits Drawbacks

Disengages users from the cognitive burden of document

rating and relevance judgments

Difficult to interpret

Large amount of data can be obtained very easily

Unreliable (compared to explicit feedback techniques)

Does not account for the individual differences of users

02/03/2009

Page 8: Joemon M Jose (with Ioannis Arapakis & Ioannis Konstas) Department of Computing Science

Affective Feedback 8

Common aspects• Both categories of feedback techniques

determine relevance by considering what occurs on the cognitive and situational level of interaction

• However, they do not account for the affective dimension of the conversational interplay between the user and the system

02/03/2009

Page 9: Joemon M Jose (with Ioannis Arapakis & Ioannis Konstas) Department of Computing Science

Affective Feedback 9

Affective ComputingAffective computing aims in the development of

more natural and flexible systems.Human-machine interactive systems capable of

sensing affect states (stress, inattention, etc) and capable of adapting and responding appropriately to these are likely to be perceived as more natural, efficient and trustworthy (Pantic, Sebe, Cohn, Huang, 2005).

Can we build a multimodal retrieval system that exploits more than one modality?

02/03/2009

Page 10: Joemon M Jose (with Ioannis Arapakis & Ioannis Konstas) Department of Computing Science

Affective Feedback 10

Affective FeedbackCan affective feedback be of any value to IR?

Likely yes, since it is considered a qualitatively rich source of human affect indications, which can be potentially exploited to enhance the information retrieval process.

Affective feedback can be defined asthe sum of all the human affective

expression/indications, which are communicated implicitly to (or identified by) a computer system and can be therefore used to facilitate a more natural, effective and robust interaction.

02/03/2009

Page 11: Joemon M Jose (with Ioannis Arapakis & Ioannis Konstas) Department of Computing Science

Affective Feedback 11

Affective Interaction• Users interact with intentions, motivations and

feelings besides real-life problems and information objects…• Intentions, motivations and emotions are all critical

aspects of cognition and decision-making

02/03/2009

Page 12: Joemon M Jose (with Ioannis Arapakis & Ioannis Konstas) Department of Computing Science

Affective Feedback 12

Affective Interaction• Information systems equipped with the ability to detect

and respond to user emotions could potentially:

1. Improve the naturalness of human-computer interaction

2. Progressively optimize their retrieval strategy

3. Offer a more personalized experience

4. Determine more accurately the relevancy of an information object

02/03/2009

Page 13: Joemon M Jose (with Ioannis Arapakis & Ioannis Konstas) Department of Computing Science

Affective Feedback 13

Affective Interaction• What are the possible reasons of emotion?

1. System?

2. Search strategy & search results?

3. Content design and aesthetics?

4. Other

02/03/2009

Page 14: Joemon M Jose (with Ioannis Arapakis & Ioannis Konstas) Department of Computing Science

Affective Feedback 14

Emotion in IR – Some Conclusions• The co-occurrence of emotions during an

information seeking process, among other physiological, psychological and cognitive processes

• Patterns of emotional variance, which reveal a progressive transition from positive to negative valence as the degree of task difficulty increases

• Depending on their frequency of occurrence the value of the conveyed affective information may potentially vary?

02/03/2009

Page 15: Joemon M Jose (with Ioannis Arapakis & Ioannis Konstas) Department of Computing Science

Affective Feedback 15

Test Collection• For the indexing we used TREC 9 (2000) Web Track

• 1.69 million document subset of the VLC2 collection

• We retained the original content of the TREC topics, but presented them using the structural framework of the simulated information need situations• Introduce a layer of realism, while preserving well-

defined relevance criteria

02/03/2009

Page 16: Joemon M Jose (with Ioannis Arapakis & Ioannis Konstas) Department of Computing Science

Affective Feedback 16

Search Tasks

02/03/2009

Page 17: Joemon M Jose (with Ioannis Arapakis & Ioannis Konstas) Department of Computing Science

Affective Feedback 17

Facial Expression Analysis• Facial expression analysis was applied on the

video recordings of each session

• For each key-frame of the video eMotion calculated the probability of the detected facial expression (assuming there was one) corresponding to any of the seven detectable emotion categories (Neutral, Happiness, Surprise, Anger, Disgust, Fear, Sadness)

02/03/2009

Page 18: Joemon M Jose (with Ioannis Arapakis & Ioannis Konstas) Department of Computing Science

Affective Feedback 18

eMotion• eMotion is an automatic facial expression recognition

system• Developed by Nicu Sebe’s group in Amsterdam/Trento

• It follows a model-based approach, in which a 3-dimensional wireframe model of the face is constructed, once certain facial landmark features are detected

• Head motion of facial deformation can then tracked and measured in terms of motion-units (MU’s), which are eventually classified into one (or more) of the seven detectable emotion categories

02/03/2009

Page 19: Joemon M Jose (with Ioannis Arapakis & Ioannis Konstas) Department of Computing Science

Affective Feedback 19

eMotion

02/03/2009

Page 20: Joemon M Jose (with Ioannis Arapakis & Ioannis Konstas) Department of Computing Science

Affective Feedback 20

Classifier• eMotion has been trained using a generic static

classifier

• The classifier has been developed from a subset of the Cohn-Kanade database

• It performs reasonably well across all individuals, independently of ethnicity-specific features

02/03/2009

Page 21: Joemon M Jose (with Ioannis Arapakis & Ioannis Konstas) Department of Computing Science

Tools & ModalitiesTools:

1) eMotion (Facial Expression Recognition System) + 2d camera2) Pasion (Facial Expression Recognition System) + 3d camera3) Polar RS800 Heart Rate Monitor4) BodyMedia SenseWear Pro3 Armband

• Modalities:Facial Expressions (emotion categories)1

Facial Expressions (motion units)1

HR3

GSR4

Heat Flux4

Skin Temperature4

Acceleration4

Page 22: Joemon M Jose (with Ioannis Arapakis & Ioannis Konstas) Department of Computing Science

Affective Feedback 22

Facial Expressions-

02/03/2009

Page 23: Joemon M Jose (with Ioannis Arapakis & Ioannis Konstas) Department of Computing Science

Affective Feedback 23

Facial Expressions-

02/03/2009

Page 24: Joemon M Jose (with Ioannis Arapakis & Ioannis Konstas) Department of Computing Science

Affective Feedback 24

Biometrics

02/03/2009

Page 25: Joemon M Jose (with Ioannis Arapakis & Ioannis Konstas) Department of Computing Science

Affective Feedback 25

Findingsusers' affective responses will vary across the

relevance of perused information items.

the results also indicate that prediction of topical relevance is possible and

to a certain extent models can benefit from taking into account user affective behaviour.

02/03/2009

Page 26: Joemon M Jose (with Ioannis Arapakis & Ioannis Konstas) Department of Computing Science

Affective Feedback 26

Open Questions

How to select different modalities?Large-scale body movements; Hand-gesture

recognition; Gaze-detection; Speech/voice analysis

How to integrate multiple modalities?Modelling challenge?

How to develop a practical system that respond to users emotional behaviour?

02/03/2009