Techniques for Recording and Analyzing Posture and Gesture as

a Means of Inferring Students’ Emotional States

1

Learning & Emotion Lab •  Focuses on uncovering the relationships between students;

learning and their emotional (i.e., affective states. The research goals include refining psychological theory and developing educational applications, such as emotionally adaptive learning environments. •  Dr. Roger S. Taylor •  Zachary Bradley •  Matthew Doyle

2

3

Measuring Affect: Through Facial Expression - Boredom

AU 27 + 64 – Mouth Stretch + Eyes Down

AU 43 + 64 – Eye Closure + Eyes Down

Neutral AU 27 AU 64 AU 43 AU 55

4

Examples of Facial Expression

Boredom

Happiness/Excitement Confusion

Calmness/Neutral

5

Affect Sensors: New Setup

•  Facial Displays •  Webcam •  Human & Computer Coding

•  Posture •  Replace Pressure Sensitive Chair with Kinect

•  Affect Map •  Java Implementation •  Mobile Devices (hopefully coming soon)

Current Research Projects Posture Analysis •  Determining students’ emotional states through

their postures.

Facial Analysis •  Determining students’ emotional states through

their facial expressions.

Self-Report Assessment •  Determining students’ emotional states through

the Affect Map self-report instrument.

6

Utilizing the Kinect Motion Sensing Input Device to Record

Students Affective States

7

Assessment Techniques

•  Internal - Observations Reference Frame – Self-reports – Affect Map

•  External - Observations Reference Frame – Behavioral Traces – Kinect Device

8

9

Affect: Theoretical Background •  Core Affect Theory (Russell, 1980, 2003; Russell & Barrett, 1999

• Dimension 1: Activation Represented vertically, with higher levels of energy toward the top and lower levels of energy toward the bottom

• Dimension 2: Valence Represented horizontally, with “positive” feelings (pleasant) toward the right and “negative” feelings (unpleasant) toward the left

10

•  Measurement about every minute

•  Activation: Level of Energy (Vertical)

•  Valence: Level of Pleasure (Horizontal)

Affect Map

High Activation (+)

Low Activation

Low

Val

ence

Hig

h V

alen

ce (+

)

Kinect Device •  Motion sensing input device •  Records participants postures and gestures

and exports joint positional coordinates X, Y & Z to a .CSV

11

Kinect

12

Methods

•  Participants: 37 SUNY Oswego undergraduates, 2 males, 35 females

•  Materials: Algebraic Problems; levels of difficulty = Easy, Medium & Hard

•  Procedure: Participants attempted to answer as many algebraic equations as possible in one half hour.

13

Open Sesame

14

Data Visualization

•  Students’ emotional states over time •  Distance from screen (meters) •  Emotive State (Valence | Activation) •  Problem difficulty (Easy | Medium | Hard)

15

Deducer Output

16

Thank you! If you have any questions, please contact:

Dr. Roger S. Taylor| roger.taylor@oswego.edu

Matthew C. Doyle| mdoyle@oswego.edu

Or visit: www.learningandemotionlab.org

17