lecture 5: physiological and brain computingict.usc.edu/~gratch/csci534/lecture2015-11.pdf ·...
Post on 10-Jun-2018
221 Views
Preview:
TRANSCRIPT
Outline
Review of ACII
Discuss EMA and Battleship
Brain and Body Computing
– Previously learned about physiological measures
Guess Lecture by Sarah Townsend, Business School
This one type of body measurement
– Talk more generally about brain/body measures
– Talk about different ways computers can induce and use these signals
– Focus on EEG
Embodied Cognition (how body effects mind)
Workshops
– 4th Workshop on Affective Brain Computer Interaction (aBCI 2015) development of human-computer interfaces able to react and adapt to users' emotions and related
cognitive states as measured from neurophysiological signals
– Affective Social Multimedia Computing analysis of affective signals in social multimedia (e.g., twitter, wechat, weibo, youtube, facebook, etc)
– Workshop on ENgagement in HumAN Computer IntEraction focuses on engagement modeling and recognition in human-human and human-machine interaction
– Workshop on Automatic Sentiment Analysis in the Wild analysis of human sentiment, and empathic and social behaviour observed in multi-party games,
user-centric healthcare and online services, automatic market research analysis, etc
– Workshop on Affective Touch Touch is important in social interactions, such as hugging, and can elicit strong affective responses.
Think for example of using touch for social communication with virtual agents or social robots,
Conference Themes
Special sessions– Laughter
– Affect in Games
Main tracks– Affect in human-machine
interaction
– Modeling emotion and cognition
– Affect in Psychophysiology
– Affect in Speech
– Affect in Social Communication
– Affect in Health
– Multimodal Perception
– Machine learning
– Affect synthesis
– Video perception
Best paper
Zakia Hammal, Jeffrey Cohn, Carrie Heike and Matthew Speltz. What
Can Head and Facial Movements Convey about Positive and
Negative Affect?
Best paper
Zakia Hammal, Jeffrey Cohn, Carrie Heike and Matthew Speltz. What
Can Head and Facial Movements Convey about Positive and
Negative Affect?
Best paper
Zakia Hammal, Jeffrey Cohn, Carrie Heike and Matthew Speltz. What
Can Head and Facial Movements Convey about Positive and
Negative Affect?
The Appraisal Equivalence Hypothesis
The projects or efforts depicted were or are sponsored by the U.S. Army Research,
Development, and Engineering Command (RDECOM). The content or information
presented does not necessarily reflect the position or the policy of the Government, and no
official endorsement should be inferred.
Jonathan Gratch, Stacy Marsella and Lin Chen
University of Southern California and Northeastern University
Is it really possible to make domain-independent
computational models of emotion?
11
Good Old-fashioned AI
Classical AI rests on the foundation that general intelligence
is symbol manipulation
e.g. Alan Newell’s (‘80) Physical Symbol System Hypothesis
– Necessary and sufficient condition for a system to exhibit general
intelligence is that it be a symbol system
And this applies to human intelligence as well
– “symbol systems are the appropriate class within which to seek the
phenomena of mind” (Newell 1980)
And this a powerful and useful concept
– The foundation of domain independence
– AI systems involve general reasoning methods that operate over
symbols, regardless of what those symbols denote
Thinking = Rule-governed manipulation
of symbolic representations
In humans, symbols
are instantiated in the
brain
The same symbols
can be instantiated in
a computer
12
But does this apply to emotion?
This hypothesis has come under sustained attack (cf Nilsson 2007)
– Embodiment: Intelligence symbols to be connected to the world via a
physical body that senses, acts and experiences (Niedenthal, Damasio)
– Non-symbolic: Intelligence involves analog signal processing
– Nonconscious: Intelligent behavior is really mindless, reflexive,
chemical activity
These are the hallmarks of emotion
Intelligence requires more than a brain in a vat
13
System 1 System 2Integration
Intelligence
Counter claim: dual-process theories of minde.g., Kahneman
Symbol SystemAffect System Symbol System
• slow
• conscious
• reflective
• forward-looking
• self-regulatory
• effortful
• exhaustible
Affect System
• fast
• unconscious
• reflexive
• myopic
• effortless
14
Emotions are not symbol systems?
Emotion arises from non-propositional content (e.g., vividness)(Nisbett and Ross 1980)
Jack sustained fatal injuries in a car crash
Jack was killed by a semi trailer that rolled over on his car and
crushed his skull
Niedenthal et al
– Just as people appear to know that CARS possess the features
engines and tires, they know that ANGER involves the experience of a
thwarted goal, and a willingness to strike
– But this is not what we mean by emotion
15
What does this say for the enterprise of creating
computational models of emotion?
Modeling?
Depends on Newell’s Physical Symbol System Hypothesis
16
Appraisal
Desirability
Expectedness
Controllability
Causal Attribution
Bodily Response
ExpressionAction
Tendencies
PhysiologicalResponse
Environment Mental State(beliefs, goals)
Theoretical Perspective: Appraisal Theory
Scherer, Klaus R., Angela Ed Schorr, and Tom Ed Johnstone. Appraisal processes in emotion: Theory, methods, research. Oxford University Press, 2001.
Emotion reflects the “person-environment relationship”
17
Appraisal
Desirability
Expectedness
Controllability
Causal Attribution
Bodily Response
ExpressionAction
Tendencies
PhysiologicalResponse
Environment Mental State(beliefs, goals)
Theoretical framework: Appraisal Theory
Coping
Problem-focused
coping
Emotion-focused
coping
18
RetakeCause: self
Intend: yes
Prob.: 50%
Get PhDUtility: 50Prob: 50%Intend: True
Past Present Future
Get PhD
Utility: 50 Prob.: 100%Belief: FalseFail Exam
Cause: Stupid
Teacher
Intend: yes
Prob: 100%
FacilitatesInhibits
Appraisal Theory in EMAUses good old-fashioned AI to derive appraisals
Working memory of beliefs, desires, intentions
HOPE (25)Desirability: 50
Likelihood: 50%
Causal Attribution: self
Coping Potential: Moderate
ANGER (50)Desirability: -50
Likelihood: 100%
Causal Attribution: Other
Coping Potential: moderate
Appraisal = Symbolic inference
19
Dimensional
Appraisal
ARElliott
ARElliott
EMNeal Reilly
EMNeal Reilly
FLAME
El Nasr
FLAME
El Nasr
EMILEGratch
EMILEGratch
CBIMarsella
CBIMarsella
EMAGratch/Marsella
EMAGratch/Marsella
FearNot!Dias
FearNot!Dias
PEACTIDMMarinier
PEACTIDMMarinier
CATHEXISVelásquez
CATHEXISVelásquez
ArmonyArmony
Scheultz&
Sloman’01
Scheultz&
Sloman’01
ALMA
Gebhard
ALMA
Gebhard
WASABIBecker-Asano
WASABIBecker-Asano
ACRES
Swagerman
ACRES
Swagerman
WILLMoffat
WILLMoffat
Anatomical
RationalNML1Beaudoin
NML1Beaudoin
MINDER 1Wright
MINDER 1Wright
Alvila-Garcia,
Canamero
Alvila-Garcia,
Canamero
FrijdaFrijda
OCCOCC
LazarusLazarus
SchererScherer
MehrabianMehrabian
DamasioDamasio
LeDouxLeDoux
SlomanSloman
ParleEBui
ParleEBui
THESPIANSi et al.
THESPIANSi et al.
Gmytrasiewicz
Lisetti ‘00
Gmytrasiewicz
Lisetti ‘00
TABASCOStaller&Petta
TABASCOStaller&Petta
ActAffActRank
ActAffActRank
Most models rely on appraisal theory
20
EMA as a symbol system
EMA is domain independent
– A domain is represented as propositions
– EMA leverages general reasoning methods that operate over
symbols, regardless of what those symbols denote
Emotion Symbol System Hypothesis:
(Appraisal Equivalence Hypothesis)
Emotion should be determined by the deep symbolic structure (goals,
actions, causal threats..)
Competing hypothesis (Embodiment hypothesis)
Emotion should be determined by surface characteristics (e.g., vividness)
21
Test of the Symbol System approach to Emotion
Create two domains that share identical deep
propositional structure but differ considerably in terms
of surface characteristics
Implement the deep structure in EMA
See if EMA predicts emotional responses in these two
domains
28
Failure!
Success!
State
Being Ahead
Being Behind
Being Even
Win
nin
g
Losing
Automatically measure smiles (Mousewars only)
Self-reported
• Appraisal
• Emotion
• Coping
Measures of Emotion
29
EMA Predictions
Emotions influence by probability of goal attainment
Emotions influenced by Utility of goal attainment
Should hold independent of surface form
Winning
Losing
30
Results
Emotion intensity
reflects probability
of goal attainment
– Up as win approaches
– Down as win recedes
Intensity reflects
goal importance
– Strongest changes for
those a priori most
invested in winning
Identical pattern for
Mousewars
Ho
pe
Jo
y
Ho
pe
Jo
y
31
Results
Control influenced
by probability and
initial utility
Subjective
probability reflects
objective change but
also initial utility
Same pattern for
mousewars
Battleship Mousewars
33
Limitations
Supportive but not definitive
– Many ways to manipulate surface form:
Vividness
Incidental emotion
– Many ways to manipulate appraisals
Trajectory
Control
Loss vs. Gain
What if we had failed?
– Wouldn’t necessarily rule out symbol-system hypothesis
– Surface changes could have evoked different goals, actions
34
Coin-flip game
Probably less sense of control
– In battleship, can place ships (sense of control)
But not obviously reflected in mousewars/battship distinction
– In mousewars/battleship, perceived control whereas coin-flip is
perceived random
Physiological and Brain Computing
Psychological constructse.g., frustration
Physiological measures
e.g., skin conductance
Physiological and Brain Computing
Physiological measures
e.g., skin conductance
Psychological constructs
e.g., frustration
1 23
4
fMRI – functional magnetic
resonance imaging
EEG – electro encephalogram
fNIRS - functional near-infrared
spectroscopy
Measures
fMRI – functional magnetic
resonance imaging
EEG – electro encephalogram
fNIRS - functional near-infared
spectroscopy
Hormone levels in blood, saliva or
urine
Measures
fMRI – functional magnetic
resonance imaging
EEG – electro encephalogram
fNIRS - functional near-infared
spectroscopy
Hormone levels in blood, saliva or
urine
EMG - electromyography
Measures
fMRI – functional magnetic
resonance imaging
EEG – electro encephalogram
fNIRS - functional near-infared
spectroscopy
Hormone levels in blood, saliva or
urine
EMG - electromyography
GSR/EDA – skin conductance
HRV – heart-rate variability
BP – blood pressure
TPR – total peripheral resistance
Measures
fMRI – functional magnetic
resonance imaging
EEG – electro encephalogram
fNIRS - functional near-infared
spectroscopy
Hormone levels in blood, saliva or
urine
EMG - electromyography
GSR/EDA – skin conductance
HRV – heart-rate variability
BP – blood pressure
TPR – total peripheral resistance
Disease history
White blood cell counts
Measures
Properties of measures: resolution
Resolution– Spatial: how specific is the part of the body/brain measured
fMRI has high spatial resolution
EEG / fNIRS have low spatial resolution
– Temporal: what is the “frame rate” EEG has high temporal resolution
fMRI has low temporal resolution (depends on metabolic processes)
“effective” resolution– Also have to consider properties of system being measured
Immunological changes: days or weeks
Endocrine changes: minutes
Visceral changes: seconds
Neural changes: milliseconds
– Slower systems tend to be less localized (GSR same at hands and feet)
Measurement resolution
Temporal Resolution
Spatial R
esolu
tion
High
Low
Low HighWhite
blood cellsCortisol
Impedance
Cardio.
fMRI
(metabolic)
EEG
(electrical)EMG
(electrical)
How do we use these measures
e.g., EMG tells us state information– Brows are furrowed
Affect tends to be short term response to stimuli– Emotion stimuli specific and momentary
– Mood more generalized and tends to last minutes
Measures most useful to index response– i.e., state change
Properties of measures: probing or monitoring
Evoked (Probing / Event-related)– System produces stimulus and measures immediate response
e.g, Flash an angry face
– Analysis: often average over repeated presentations to control noise e.g., IAPS: present multiple positive and negative images
Induced (Monitoring / Endogenous)– System monitors changes in user state
– Changes considered “endogenously” produced – i.e. Reflects some
mental processing
– Could include indirect responses to computer stimuli: e.g., frustration
in response to computer delivered exercise
– Analysis: apply frequency transformation and look for oscillations
One example: Gene expression (Steven Cole UCLA)
Affect impacts your genes
– Genes determined by heredity
– Gene expression determined by affect and environment?
– DNA analysis identified 209 genes that were differentially expressed in
high- versus low-lonely individuals
up-regulation of genes involved in immune activation, and cell proliferation
down-regulation of genes supporting lymphocyte and interferon response
Correlation or causation?
Mindfulness-based stress reduction
training reduces loneliness and pro-
inflammatory gene expression in older
adults: a small randomized controlled trial
8 week training program
EEG Measurement approaches
Evoked: Time-domain correlates (ERP)– Derived by averaging multiple traces following stimulation events of
same condition
– Example of “evoked” (probing) approach
– Can look for potentials in different parts of head
EEG Measurement approaches
Evoked: Time-domain correlates (ERP)– Derived by averaging multiple traces following stimulation events of
same condition
– Example of “evoked” (probing) approach
– Can look for potentials in different parts of head
EEG Measurement approaches
P300
Time-domain correlates (ERP)– Early ERPs: reflect automatic evaluation
P1/N1: reflect initial perception and automatic evaluation of stimuli
– Late ERPs reflect higher-level processes.
P200: sensation-seeking behavior
P300 unexpected stimuli
N400 processing difficulty (e.g., difficulty in
understanding meaning of a word)
P600: follows syntax violation in language
Problem: requires multiple probes
Not practical in many apps.
Example: P300
Present series of stimuli– Beep, Beep, Beep, Beep, BOOP, Beep, Beep,….
– P300 bigger when stimuli important to subject (increased utility) e.g., loose $5 every time BOOP occurs
– P300 bigger when unexpected stimuli less common (lower probability)
– Thus, related to two common appraisal variables
Evokes P300
EEG Measurement approaches
Alternative: brain rhythms in frequency domainApply frequency transformation and look for oscillations
FERRARI, Rosana; ARCE, Aldo Ivan Cespedes; MELO, Mariza Pires de and COSTA, Ernane Jose Xavier. Noninvasive method to assess the electrical brain activity
from rats. Cienc. Rural [online]. 2013, vol.43, n.10
EEG Measurement approaches
Alternative: brain rhythms in frequency domainApply frequency transformation and look for oscillations
– Delta rhythm: associated w/ hunger and drug craving (reflects workings of
brain reward system). Also correlates w/ P300 ERP. May be associated with
detection of emotionally salient stimuli
– Theta rhythm: indexes working memory / memory demands;
Frontal-medial theta associated w/ positive valence
– Alpha rhythm: associated with sensory processing (e.g., music, films)
Frontal alpha symmetries vary as function of valence.
Rightward lateralization associated w/ positive or approach-related emotions
(contrasting w/ negative withdrawal-related emotions)
– Beta rhythm: associated w/ sensory-motor system: increased activity
associated w/ positive emotions
– Gamma rhythm: associated w/ attention, memory and consciousness.
Correlated w/ positive valence.
Posterior increases associated with highly arousing visual stimuli.
Gamma over somatosensory cortex associated w/t pain
EEG Measurement approaches
Alternative: brain rhythms in frequency domainApply frequency transformation and look for oscillations
– Delta rhythm: associated w/ hunger and drug craving (reflects workings of
brain reward system). Also correlates w/ P300 ERP. May be associated with
detection of emotionally salient stimuli
– Theta rhythm: indexes working memory / memory demands;
Frontal-medial theta associated w/ positive valence
– Alpha rhythm: associated with sensory processing (e.g., music, films)
Frontal alpha symmetries vary as function of valence.
Rightward lateralization associated w/ positive or approach-related emotions
(contrasting w/ negative withdrawal-related emotions)
– Beta rhythm: associated w/ sensory-motor system: increased activity
associated w/ positive emotions
– Gamma rhythm: associated w/ attention, memory and consciousness.
Correlated w/ positive valence.
Posterior increases associated with highly arousing visual stimuli.
Gamma over somatosensory cortex associated w/t pain
Physiological and Brain Computing
Physiological measures
e.g., skin conductance
Psychological constructs
e.g., frustration
2
Psychological constructs
Any affective brain state that can be inferred with reliability
– Emotion, mood, appraisals
Can distinguish by temporal characteristics
– Tonic state: a longer-term steady state (e.g., mood) E.g. average skin conductance over a 5minute speech
– Phasic state: a short term state or transition (e.g., emotion) E.g., skin conductance 5 seconds following presentation of disgusting picture
Psychological constructs
Any affective brain state that can be inferred with reliability
– Emotion, mood, appraisals
Can distinguish by temporal characteristics
– Tonic state: a longer-term steady state (e.g., mood) E.g. average skin conductance over a 5minute speech
– Phasic state: a short term state or transition (e.g., emotion) E.g., skin conductance 5 seconds following presentation of disgusting picture
Can distinguish if “deliberate” or not– Active (explicit): states “created” intentionally by user
Remember a stressful event
Try to relax
– Passive (implicit): states occurring w/o conscious control Subliminal presentation of angry face
Physiological and Brain Computing
Physiological measures
e.g., skin conductance
Psychological constructs
e.g., frustration3
Mappings
Predictors of psychological state
Physiological
measures
Psychological constructs
e.g., skin conductance e.g., frustration
Properties of mappings
Specificity: characteristic of f(measure) = construct
– One-to-one: amygdala activity represents appraisal of relevance
– Many-to-one: cortical activity, BP and heart rate together predict effort
– One-to-many: skin conductance indicates positive or negative emotions
– Many-to-many
Context-dependence: invariance of measure across situations
– Context-independent mapping would hold across lab and “real world” and
across social and task contexts
– Context-dependent would hold only in certain situations (e.g., smiles might
predict true feelings in presence of friends by not strangers)
Sensitivity: correlation between measure and construct
Some applications
Physiological measures
e.g., skin conductance
Psychological constructs
e.g., frustration
(Response to stimuli)(Endogenous)
(Intention to control -- e.g. imagined arm movements)
(No intentional to control)
Affect
monitoring
Affect
Regulation
Affect
probing
Affect
Communication
(Response to stimuli)(Endogenous)
(Intention to control -- e.g. imagined arm movements)
(No intentional to control)
Affect
monitoring
Affect
Regulation
Affect
probing
Affect
Communication
Examples
Neuro-feedback Systems
(e.g., depression treatment)
• Teach positive emotion regulation
techniques
• Use EEG to measure success
• Analyze EEG in frequency
domain
• Look for alpha rhythm
asymmetries
• Ring bell when user achieves
desired brain state
• Proved clinically effective
Hammond, D. Corydon. "Neurofeedback
treatment of depression and anxiety."Journal of
Adult Development 12.2-3 (2005): 131-137.
(Response to stimuli)(Endogenous)
(No intentional to control)
Affect
monitoring
Affect
Regulation
Affect
probing
Affect
Communication
Examples
Sensing user boredom or frustration
• Could use to adjust difficulty in
computer game to maintain user
satisfaction
• Could use as feedback in
tutoring system to adjust learning
level or pedagogical feedback
(Response to stimuli)
(No intentional to control)
Affect
monitoring
Affect
Regulation
Affect
probing
Affect
Communication
Examples
Assess affective responses across multiple “probes” (e.g., music or video clips
• Monitor the user response to media
• Tag media with affect it produced
• Selectively offer or automatically play
back media items known to induce a
certain affective state in the user
• (e.g., Koelstra et al., 2012; Soleymani
et al., 2011).
Use audio probe and measure if
attending to it as measure of
workload
(Response to stimuli)
(Intention to control -- e.g. imagined arm movements)
(No intentional to control)
Affect
monitoring
Affect
Regulation
Affect
probing
Affect
Communication
Examples
Standard approach for non-affective
BCI
• e.g., P300 speller: relies on P300
response to letter “probes”
Summarize
Emotions shape human decision-making– Can try to account for this in a general sense in system design
Emotions reflected in physiological signals– Can try to make use of this to recognize and react to moment-to-
moment emotions
Many approaches proposed– Measuring different physiological systems
– Exploiting different measurement strategies (active probing vs. passive
monitoring)
– Examining different characteristics of signals (tonic, phasic, …)
Work in progress
top related