Avar Pentel PhD student, Tallinn University, School of Digital TechnologiesSupervisor: Tobias Ley

• Area of reasearch:– User profiling– Detecting users motor behaviour via

standard input devices such as keyboard and mouse

– and connecting it to users demographic data, emotions, etc

Title of Presentation

Employing Think-Aloud Protocol to Connect User Emotions and Mouse

Movements*

* Based on the paper (2015) with the same titleavailable at IEEE Xplore digital library: http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=7387970

Goal

To predict user’s emotional state by analyzing mouse movements logs

Outline of the Presentation

• Related work• Experimental setup

– Data collection procedure– Associating tasks to emotional states– Features– Machine learning

• Results• Conclusion

Related work

• Special equipment• Small samples• Specific tasks, no general link between

emotion and mouse movement is studied

Data Collection Procedure

Idea from Christmas Calendar

Data Collection Procedure

Data collection

• The game has collected data about:–Each mouse click.–Time when button was clicked.–All mouse movements with

timestamps

Data Collection

Example of mouse log (x,y, timestamp)

70,34,1365354712662,74,34,1365354713453,78,36,1365354713488,81,38,1365354713517,85,44,1365354713537,87,50,1365354713560,89,53,1365354713573,90,58,1365354713598,91,63,1365354713622,92,59,1365354713903,95,53,1365354713927,97,49,1365354713942,100,45,1365354713954,103,40,1365354713976,106,36,1365354714001,110,34,1365354714049,105,33,1365354714390,100,30,1365354714414,96,27,1365354714439,93,25,1365354714475,93,21,1365354714561,95,18,1365354714598,98,15,1365354714622

Data

• 916 game sessions played by 282 individual users. Participants were between 12 and 52 years old.

• As each game session consisted of 24 searching tasks, we had all together 21984 comparable (standardized session-wise)records, each of them presenting mouse movement logs between two button clicks.

Connecting Emotions with Tasks

Old and New approach

1) Retrospective feedback2) Concurrent Think-Aloud protocol

Old Approach – retrospective feedbackFirst pilot:

There was no room of variety of emotions

Self-Reports on Russel’s Model

Self-Reports on Likert Scale• Interviews with selected particiapants

(N=44)• Right after game session• Still image of the game session was shown

Content Confused

Binary mapping: (1-3) content, (5-7) confused, (4) neutral

Self-Reports on Likert Scale• Emotion data about 44*24=1056 tasks

• All target finging times standardized session-wise

• Pearson correlation between self reports and standardized finding time was found (r = 0.86)

Content Confused

Self-Reports on Likert Scale

• Tasks reported as confused had finding speed 0.5 standard deviation below mean

• Tasks reported as content had finding speed 0.5 above mean

Content Confused

Binary mapping: (1-3) content, (5-7) confused, (4) left out a neutral

Separation of Classes

Standardized item finding speed

Second half ofeach of these

logs counted as characterizing non confusion

First half of each of these logs counted

as characterizing

confused state

Using Think-Aloud Protocol

Think-Aloud Protocol• Users reported five kinds of emotions -

confusion, frustration, shame, content and flow. Strongest emotions were confusion and frustration. Here is an example how users were expressing themselves during states of confusion and frustration:

• “Where is number x, where is number x, it is not there, it is impossible, I looked everywhere, it is missing. “

• “It can’t be, you hide a button, it is not there.”• “I saw it before, but now it is not there any

more.”

Russel’s Model

Think-Aloud protocol

Using Think-Aloud Protocol

• 400 sessions (20 users * 20)• 400*24 = 9600 comparable tasks with

emotion data

Separation of Classes

Final Datasets

• Confused class with 3170 examplesand all the rest with 18814 examples.

• In the case with separation gap between classes, the second class had 12381 examples.

• Before applying classification algorithms, we balanced our datasets.

Features

Distance (curvature)

1

23

4

1

2 3

12

3

45

Ratio between the 3-6 movements length and shortest path between the beginning and end point

Speed (σ of the speed)

s4s5

s1

s2 s3

s6s7

s8

s9s10

Speed is measured for each 10px movement separately

N

S

W O

Direction

Angle based features• Sum of consecutive turns greater than an angle A (A counted by

45-degree step), normalized by number of movements.

• 18 features representing turns from 0 to 180 degrees’ by 10-degreestep. Counted results were normalized by the number of movements.

• Sum of all angles divided by number of movements – 1.

• σ of angles.

A

Bα

Feature selection

10

Machine learning

• Logistic Regression• Support Vector Machine• Random Forest• C4.5

– Motivation based on literature. – Java implementations of data analysis

package Weka.– 10-fold cross validation

Results (with separation gap)

Results (without separation gap)

Conclusion

• Mouse movements reveal users emotsional states

• But is the confusion and frustration in current study comparable with confusion and frustration caused by solving mathematical equation or some other cognitively more demanding task?

Conclusion

However, if we relay on 2D Circumplex Model ofEmotion, then all kind of confusion and frustration is located in the sameplace.

Thank You!

Q&A