detection of and response to online users' emotion

Beverly Park Woolf College of Information and Computer Sciences,

University of Massachusetts-Amherst bev@cs.umass.edu

Detection of and Response to Online Users’ Emotion

Department of Quantitative Health SciencesUniversity of Massachusetts Medical School

March 24, 2017

Bounce between two systems . . .

Mathematics Tutor: Exists and contains most of the described features.

Patient Care Tutor: Proposed and can be built with these features.

Introduction and Care for the Patient

Model the Patient

Model Emotion (Sensors & Computer Vision)

Model the Medical Domain

Provide Interventions

Assess Patient Learning

Agenda

Caring: A Definition

Caring implies actions based on another person’s wants and desires (Noddings, 1986).

Entities that care consider the other person’s point of view, her needs and what she expects of us (Cooper, 2003).

Caring includes the ability to empathize with others and take responsibility for their needs.

Caring Impacts Learning

Findings from neuroscience suggest that all learning is affective in nature (Damasio, 1994).Students learn most effectively in a climate in which people care about them (Cooper, 2003).A major weakness of traditional psychology is to separate the intellect and affect (Vygotsky, 1986). Every person’s idea contains an affective attitude.

When Caring is Present . .

Positive emotions and interactions create the ambience for learning and enable a student’s brain to remain curious and open (Cooper, 2002; 2003).

A continuing, positive sense of self produces a constant positive feeling throughout the body, which leads to greater openness and willingness to engage in interactions (Winkley, 1996). Babies brains grow when they feel cared for.

When Caring is NOT Present . . .

Students flounder in internal confusion (Cooper, 2003).

Negative affect tends to produce a shutting down of one’s self, a withdrawal, stimulating protection and defense.

Research Questions about Emotion

How are users’ emotions evidenced and measured?

How do emotions predict learning?

How accurate are emotion models (e.g., Markov or Bayesian Models) at predicting future emotions from student behaviors?

How effective are interventions at responding to emotion??

Introduction and Care for the Patient

Model the Patient

Model Emotion (Sensors & Computer Vision)

Model the Medical Domain

Provide Interventions

Propose Health Care Tutor

Agenda

Model the Patient

Model the Disease

Personalize Tutoring

Assess Learning

Intelligent TutoringSystems

Adele explains the importance of palpating the patient’s abdomen.

Background Research: Shaw, Johnson , & Ganeshan (1999). In Proceedings of the third annual conference on Autonomous Agents (pp. 283-290). ACM.

Adele advising in a critical care scenario on the World Wide Web. Pedagogical Agents on the Web

Erin Shaw, W. Lewis Johnson, and Rajaram Ganeshan

Adele instructs a student to answer a quiz when the student selects a urine dipstick test. Pedagogical Agents on the Web

Erin Shaw, W. Lewis Johnson, and Rajaram Ganeshan

Background Research: Taking the Time to Care: Empowering Low Health Literacy Hospital Patients

with Virtual Nurse AgentsTimothy W. Bickmore, Laura M. Pfeifer

College of Computer & Information Science Northeastern University {bickmore,laurap}@ccs.neu.edu

• Ninety million Americans have inadequate health literacy, resulting in a reduced ability to read and follow directions in the healthcare environment.

• Animated, empathic virtual nurse interface for educating and counseling hospital patients with inadequate health literacy in their hospital beds at the time of discharge.

• Results indicate that hospital patients with low health literacy found the system easy to use, reported high levels of satisfaction, and most said they preferred receiving the discharge information from the agent over their doctor or nurse.

Patient Interacting with Virtual Nurse

Taking the Time to Care Timothy W. Bickmore, Laura M. Pfeifer

Chi 2009

Sample dialogue - relational aspects highlighted Taking the Time to Care Timothy W. Bickmore, Laura M. Pfeifer

Chi 2009

Taking the Time to Care Timothy W. Bickmore, Laura M. Pfeifer

Chi 2009

Self-report Ratings of the Virtual Nurse (mean (SD))

Taking the Time to Care Timothy W. Bickmore, Laura M. Pfeifer

Chi 2009

Introduction and Care for the Patient

Model the Patient

Model Emotion (Sensors & Computer Vision)

Model the Medical Domain

Provide Interventions

Propose Health Care Tutor

Agenda

MathSpring

An intelligent tutor for mathematics;

Contains hundreds of math problem, grades 5-10;

Used by tens of thousands of students;

Aligned with Common Core standards;

Detects and responds to student emotion;

Improves student passing of state standardized tests;

Uses multimedia characters to support student emotion;

Built by Woolf & Arroyo at UMass-Amherst.

23

The MathSpring System

Pedagogical AgentsAdult Users Require Adult Companions

Help to develop a positive emotional environment; Agents offers advice and encouragement.

Empathize with patientAgents express full sentences of cognitive, meta-cognitive and emotional feedback.

Build patients’ self-esteem and self-worthAgents are gendered and multicultural: White, Hispanic, African-American

Mass Statewide Standardized Tests

Passing grades for experimental (dark grey) and control group (light grey); same grade, same school and same teacher.

Students represented by the yellow/green polygon used the Math Tutor; those represented by the blue polygon did not. The distribution of students using the tutor is shifted towards the right, towards more proficiency and above proficiency. Students were matched in terms of teacher; all were 7th grade students.

Introduction and Care for the Patient

Model the Patient

Model Emotion (Sensors & Computer Vision)

Model the Medical Domain

Provide Interventions

Propose Health Care Tutor

Agenda

Sensors to Detect Student EmotionDavid Cooper, Ph.D. Dissertation

Sensors to measureStudent Emotion

(Clockwise): mental state camera, skin conductance bracelet, pressure sensitive mouse, pressure sensitive chair.

31

Use Sensors

32

The Students

Rural-Area High School in MA (35 students)Geometry and Algebra classes

UMASS 114 (29 students)Math for Elementary School Teachers

33

Students Self-Report EmotionsFour bipolar emotional axes

How are you feeling? Please rate your level of interest in this

Using Sensors to Measure Emotion:A Linear Regression Model

R represents the fit of the modelN is the number of cases available, for each emotion self-report and each sensor

Significantresults

Student self-report

Cooper et al., UMAP 2009;Arroyo et al., AIED 2009

David Cooper, Ph.D. Dissertation

36

Models of Emotions with SensorsFrom Tutor-Context Variables and Sensors

Linear Models to Predict Emotions Variables Entered in Stepwise Regression

Confidence InterestFrustration Excitement

SitForwardStdev

“Concentrating”Max. Probability

Camera Facial Detection Software

SitForwardMean

Seat Sensor

# Hints Seen

Solved? 1st Attempt

# Incorrectattempts

Gender Ped. Agent

Seconds to1st Attempt

Time in Tutor

Seconds To Solve

Tutor Context Variables (for the last problem)Tutor Only All Sensors+Tutor

R=0.53 R=0.43 R=0.37R=0.49

R=0.72 R=0.70 R=NAR=0.82

Sensor Variables (Mean, Min, Max, Stdev for the last problem)

Results of Emotion Prediction Studies

Sensors predict self-report, accuracy 75-80%. Sensors improve student emotion self-reports. Self-reports predicts post-tutor math performance, attitudes and perceptions of tutoring software.

Introduction and Care for the Patient

Model the Patient

Model Emotion (Sensors & Computer Vision)

Model the Medical Domain

Provide Interventions

Propose Health Care Tutor

Agenda

Computer Vision to Monitor & Measure Grit

• Several written Instruments are used to measure non-cognitive skills, primarily self-report surveys.– Self-regulation– Motivation strategies (MSLQ)

• Students with grit & persistence have good self-regulation skills:– Goal setting, self-monitoring, and self-instruction; – They manage their emotions and waning motivation

Head Position

The right mouth corner is tracked during student-computer interactions with significant changes of head position and orientation

Low Level Features

Extraction of low-level features from face, (a) furrows, (b) iris, and (c) mouth, for FACS-based facial expression analysis, yielding the interpretation “student exhibits anger.”

Trace Student Attention

The Mathspring GUI overlaid with a “heat map” of student attention, as measured

by computer-vision. In red, screen regions the student particularly focused

on

Engaged while reading and solving the problem

Head tilts

Starting to get annoyed with the learning companion

• Student was actually fine with the learning companion at first but seems to get annoyed later on

• Seem to be offended when the learning companion states “See, I told you that hints are really helpful”. He never took any hints and did everything correctly on every first try

• Finally closes the learning companion as soon as it pops up after doing six problems

Looking to the right side of the screen means he is looking at learning companion

Closing learning companion

Companion removed by

student.

Summary for GritCamera6

• Student is very engaged• Student takes time to answer every question

– Made no mistake– Took no hint– Got it right in every first try

• Takes time to answer survey– Only two survey shows up

Introduction and Care for the Patient

Model the Patient

Model Emotion (Sensors & Computer Vision)

Model the Medical Domain

Provide Interventions

Propose Health Care Tutor

Agenda

Cardiac Tutor, Built at UMass-Amherst

The Cardiac Tutor teaches advanced cardiac support techniques to medical personnel. It presents cardiac problems and, using a variety of steps, students select various interventions. It provides clues, verbal advice, and feedback to personalize and optimize the learning..

Simulated patient in the Cardiac Tutor. The intravenous line has been installed

( “IV in ”); chest compressions are in progress, ventilation has not yet begun,

and the electronic shock system is discharged. Eliot, Williams & Woolf, 1996, An intelligent learning

environment for advanced cardiac life support.

Rashi Case DescriptionBuilt at UMass-Amherst

Dragon et al., 2006 Coaching Within a Domain Independent Inquiry Environment

Rashi Interview Tool

Dragon et al., 2006 Coaching Within a Domain Independent Inquiry Environment

Rashi Exam Tool

Dragon et al., 2006 Coaching Within a Domain Independent Inquiry Environment

Rashi Lab Test Tool

Rashi Inquiry Notebook

Dragon et al., 2006 Coaching Within a Domain Independent Inquiry Environment

Rashi Argument Editor

Dragon et al., 2006 Coaching Within a Domain Independent Inquiry Environment

Background Research: Medical Knowledge

Bioworld patient scenario and evidence palette.Susanne Lajoie 2001

Constructing knowledge in the context of BioWorld

Model Medical Knowledge

(Left) Conducting Diagnostic Tests; (Right) constructing the closing argument Lajoie 2001; Constructing knowledge in the context of BioWorld

Lajoie & Greer, 2003 Establishing an Argumentation Environment to Foster Scientific Reasoning With Bio-World

Introduction and Care for the Patient

Model the Patient

Model Emotion (Sensors & Computer Vision)

Model the Medical Domain

Provide Interventions

Propose Health Care Tutor

Agenda

Research Questions

• What to do in the moment when users are frustrated, bored, etc.?– Increase Challenge? Decrease Challenge?– Provide extra scaffolds?– Provide “affective” scaffolds? What are those?– Encourage users to stop and think about what is going

on?• How to measure changes in student affect, or to

capture micro-changes in student affective states?

Interventions have a strong impact on Learning

• Average effect size of metacognitive instruction across 20 studies was .72, a very large effect.

– Teach a skill; track the student, measure the skill before and after use of the software.

– Measure a student’s aptitude for learners trained in a skill against students who are not trained in that skill.

Many people: • relate to computers in the same way they relate to

humans (Nass, 2010);• continue to engage in frustrating tasks significantly

longer after an empathic digital response (Picard); • have lowered stress levels after receiving an empathetic message from a

digital character (Arroyo et al., 2009); • recalled more information when interacting with an

artist agent compared to scientist agent; • report reduced frustration and more general interest

when working with gender-matched characters.

People and Agents

Affirmation Theory Messages

Affirmation Theory: Propose that users’ motivation is rooted in their belief about why they succeed or fail. Students can be taught to understand that failure is the result of a lack of effort instead of a lack of ability.

Example Messages “People have myths about math, like, that only some people are good in

math. The truth is that we can all be successful in math if we give it a try”

“We will learn new skills only if we are persistent. If we are very stuck,

let's call the teacher, or ask for a hint!”

“When we realize we don't know why the answer was wrong, it helps us

understand better what we need to practice.”

Effort Interaction Messages

Effort Interactions: Acknowledge effort and incorrect answers. The goal is to make students realize that praise is not always appropriate and that effort is the primary goal.

Example Messages“That was too easy for you. Let's hope

the next one is more challenging so that we can learn something.”!”

“Good job! See how taking your time to work through these questions can

make you get the right answer?”

Incorrect ResponseStudent effort shown/correct response

Student effort shown /incorrect response

Agent Emotion

Frustrated students are supported by helpful companions.

Arroyo et al., AIED2009

• Our current Gold standard/Truth: self-reports• During Experiment:

– Ask students about their “interest” level and their “excitement” level every 7 minutes, on average.

Gathering Student Affect “During”

Low

HighNeutral / Middle

Large Data Sets

EventLog Table of a Math Tutoring System. 571,776 rows, just in a year time.

The Student Progress Page

73

Dovan Rai, PhD, WPI

Three Experimental Conditions

74

• No access to the Student Progress Page

• “My Progress” button was present (student choice)

• Prompt invitation to see “my progress” upon bored (disinterested) or unexcited

• Force student to see my progress when student bored (disinterested) or unexcited

How to analyze changes in student affect, from moment to moment?

MARKOV CHAIN MODELS

Student InterestTen thousand Data Points N~230.

SPPAbsent

SPPPresent

SPP Prompted

SPP Forced

How to compare Markov Chain Models, quantitatively?

• Probability of student following a specific path• What is the probability that a student will end

up excited, after 3 transitions?• Given that they started in a specific state?

Resulting Data

Introduction and Care for the Patient

Model the Patient

Model Emotion (Sensors & Computer Vision)

Model the Medical Domain

Provide Interventions

Propose Health Care Tutor

Agenda

Health Care Research Questions

Which interventions improve patients’ affect and learning?

Which interventions are best for which patients?

What is the impact of deeper companion characteristics?

Do affective characters make users “feel” better?

How does the gender (ethnicity) of companions impact patients’ attitudes/emotions/learning?

Detect patient emotion, using Computer Vision or sensors (camera, wrist band, pressure mouse, seat sensors).

Dynamically collect data streams of students’ physiological activity and self-reports of emotions.

Apply interventions once emotion is detected, e.g., animated embodied agents

Integrate computer vision techniques to improve detection of emotion.

Train researchers and educators in techniques to recognize and respond to patients’ emotion and make predictions over large data sets.

Proposed Health Care System

Study Design

Patients

Control

ExperimentalIntervention

Control

ExperimentalIntervention

Visit # 1 Visit # 2 Visit # 3 Visit # 4

Randomized

• Survey• Audio tape with physician

• Survey• Audio tape with physician

• Survey• Audio tape with physician

• Survey

• Survey • Survey• Survey

Baseline data

Greenfield et al., 1986; Patients’ participation in Medical Care General Internal Medicine

Emotion is vital to learning and using technology to recognize users’ emotion has led to powerful performance results.

Detecting and responding to users’ emotion while they work with online environments improves learning.

Summary

Thank You !Any Questions?

Detection of and Response to Online Users’ Emotion

Department of Quantitative Health SciencesUniversity of Massachusetts Medical School

March 24, 2017