joschabach implementing emotion and motivation in ai
TRANSCRIPT
Joscha Bach
ImplementingEmotionandMotivationinAIArchitectures
TutorialAAAI2018
FromComputationtoEmotion
Howcanacomputationalsystemexperienceemotion?
Whatisemotion?Whatisexperience?Relationshipbetweenexperiencerandexperience?
AAAI 2018 Emotion and Motivation Tutorial 2
Emotionalstatesascognitiveconfigurations
AAAI 2018 Emotion and Motivation Tutorial 3
Emotionsascognitiveconfigurations
AAAI 2018 Emotion and Motivation Tutorial 4
Emotionsascognitiveconfigurations
AAAI 2018 Emotion and Motivation Tutorial 5
Overview
• AGIperspectiveonminds• Basicarchitecturalcomponents• Modelingmotivation(MicroPsi model)• Modelsofemotion• EmotioninthePsitheory• Modelingpersonality• Emotion,selfandprosociality
AAAI 2018 Emotion and Motivation Tutorial 6
BasicperspectiveongeneralAI
• Mindasmachine
• Machine=computationalsystem• computation=regularstatechange
• universalcomputation:setofcomputablefunctionsthatcancomputeallcomputablefunctions(whengivenunboundedresources)
AAAI 2018 Emotion and Motivation Tutorial 7
BasicperspectiveongeneralAI
• Accesstouniverseviadiscernibledifferences® Information
• Meaningofinformation:relationshiptochangesinotherinformation
• Intelligence:abilitytomodel• Modelingisfunctionapproximation• Purposeofmodelingisregulation,tomaximizerewards
AAAI 2018 Emotion and Motivation Tutorial 8
BasicperspectiveongeneralAI
• ClassicalAI:directmodelingofcognitivefunctionality® “firstorderAI”
• DeepLearning:systemsthatmodelfunctionalitythemselves;compositionalfunctionapproximation® “secondorderAI”
• MetaLearning:systemsthatlearnhowtobuildlearningsystems® “thirdorderAI”
AAAI 2018 Emotion and Motivation Tutorial 9
BasicperspectiveongeneralAI:openquestions
• Arehumansmetalearningsystems?• Isevolutiona(slowandineffective)searchformetalearners?
• Isthereaclassofuniversalfunctionapproximatorsthatcanapproximateanyfunctionthatcanbeapproximatedbyacomputer(whengivenunboundedresources),anddoesitcontainitselfandus?
AAAI 2018 Emotion and Motivation Tutorial 10
AIperspectiveonthemind
• Constructedarchitectures(Minsky,Simon,Newell)— classicalcognitivearchitectures
• Generatedarchitectures(Schmidhuber,Hutter,…):generalrecursivefunctionapproximation+rewardsystem
• Hybridperspective:mostlygenerated,butwithcomplexprerequisitesandbiases
AAAI 2018 Emotion and Motivation Tutorial 11
LayersofCognition
Reflective
Deliberative
Reactive
AAAI 2018 Emotion and Motivation Tutorial 12
ColumnsofCognition
Perception CognitiveProcessing Action
AAAI 2018 Emotion and Motivation Tutorial 13
CognitiveGrid
ReflexivePerception Meta-
ManagementManagement
Action
DeliberativePerception
Planning,Reasoning
DeliberativeAction
ReactivePerception Reflexes Reflexive
Action
AAAI 2018 Emotion and Motivation Tutorial 14
ConceptualAnalysis:HCogAff(Sloman2001)
Emotion and Motivation TutorialAAAI 2018 15
Artificial GeneralIntelligence
Methodsshouldfocusoncomponentsandperformancesnecessaryforintelligence:
• Whole,testable architectures• UniversalRepresentations:
Dynamicmodel of environment,possible worlds,and agent• (Semi-)UniversalProblemSolving:
Learning,Planning,Reasoning,Analogies,ActionControl,Reflection ...
• UniversalMotivation:Polythematic,adaptivegoal identification
• Emotionand affect
AAAI 2018 Emotion and Motivation Tutorial 16
• Universalmentalrepresentations(compositional +distributedà neurosymbolic)
Componentsfor Cognitive AI
AAAI 2018 Emotion and Motivation Tutorial 17
Componentsfor Cognitive AI
• (Semi-)Generalproblem solving:Operations over theserepresentations
(neural learning,categorization,planning,reflection,consolidation,...)
AAAI 2018 Emotion and Motivation Tutorial 18
Op3 Op4Op2Op1 Opn...
Componentsfor Cognitive AI
• Perceptual grounding
AAAI 2018 Emotion and Motivation Tutorial 19
Op3 Op4Op2Op1 Opn...
Perception
Componentsfor Cognitive AI
• Perceptual grounding and action
AAAI 2018 Emotion and Motivation Tutorial 20
Perception Action
Op3 Op4Op2Op1 Opn...
Componentsfor Cognitive AI
• Perceptual grounding and action
AAAI 2018 Emotion and Motivation Tutorial 21
Perception Action
Op3 Op4Op2Op1 Opn...
• Modelof current situation,and protocol of past situations
Componentsfor Cognitive AI
AAAI 2018 Emotion and Motivation Tutorial 22
Perception Action
Situationmodel
Protocol memory
Op3 Op4Op2Op1 Opn...
• Modelof self
Componentsfor Cognitive AI
AAAI 2018 Emotion and Motivation Tutorial 23
Op3 Op4Op2Op1 Opn...
Perception Action
Situationmodel
Protocol memory
Self model
• Abstractions of objects,episodes and types
Componentsfor Cognitive AI
AAAI 2018 Emotion and Motivation Tutorial 24
Op3 Op4Op2Op1 Opn...
Perception Action
Self model
Declarative memory
Procedural memory Frame
• Anticipation of future developments
Componentsfor Cognitive AI
AAAI 2018 Emotion and Motivation Tutorial 25
Op3 Op4Op2Op1 Opn...
Perception Action
Self model
Declarative memory
Procedural memory Frame Expect.
Plans
• Actionselection and executive control
Componentsfor Cognitive AI
AAAI 2018 Emotion and Motivation Tutorial 26
Perception Action
Longterm memory Worldmodel
Mentalstage
• Actionselection and executive control
Componentsfor Cognitive AI
AAAI 2018 Emotion and Motivation Tutorial 27
Perception Action
Longterm memory Worldmodel
Mentalstage
Action selection
• Universalmotivation:autonomous identification of goals
Componentsfor Cognitive AI
AAAI 2018 Emotion and Motivation Tutorial 28
Perception Action
Longterm memory Worldmodel
Mentalstage
Action selection
Motivational system:Motive Selection
and Decision making
• Emotionalmodulation and affect
Componentsfor Cognitive AI
AAAI 2018 Emotion and Motivation Tutorial 29
Perception Action
Memory
Action selection
Motivational system:Motive Selection
and Decision making
Urges/drives
Resolution
Arousal
Selectionthreshold
Securing rate
AAAI 2018 Emotion and Motivation Tutorial 30
CognitiveArchitectures
SoarACT-R
“ClassicalCognitiveArchitectures” tendtofocusoncognitionasanisolatedproblemsolvingcapability.
Componentsfor Cognitive AI
AAAI 2018 Emotion and Motivation Tutorial
Perception Action
Memory
Action selection
Motivational system:Motive Selection
and Decision making
Urges/drives
Resolution
Arousal
Selectionthreshold
Securing rate
31
MicroPsi architecture
AAAI 2018 Emotion and Motivation Tutorial 32
PSItheoryPrinciples of Synthetic Intelligence
(Dörner1999;Bach2003,2009)
AAAI 2018 Emotion and Motivation Tutorial 33
MicroPsiArchitecture–simplified
Short TermMemory/
Local PerceptualSpace
Long Term Memory(LTM)
Body Parameters
MicroPSI Agent
UrgeSensor PerceptSensorSensors/
Modulators
MemoryMaintenance
Behaviour Script Space /Execution Space
Motivation Execution
Internal Behaviours
Perception
Meta-Management
ExternalBehaviors
Action
AAAI 2018 Emotion and Motivation Tutorial 34
AgentFunctionality
• Episodiclearning• Goal-directedbehavior,motivationalsystem• Emotionalmodulation• Hypothesisbasedperception• Simpleplanning• Executionofhierarchicalplans
AAAI 2018 35
Implementation:MicroPsi (Bach03,05,04,06)
Node Net Editor
Net Simulator/Agent
Execution
World Editor
Monitoring ConsoleApplication
World Simulator
3D DisplayServer
3D DisplayClient
Eclipse Environment
Emotion and Motivation Tutorial
AAAI 2018 36
Node Net Editor
Net Simulator/Agent
Execution
World Editor
Monitoring ConsoleApplication
World Simulator
3D DisplayServer
3D DisplayClient
Eclipse Environment
Implementation:MicroPsi(Bach03,05,04,06)
Low-levelperception
Emotion and Motivation Tutorial
AAAI 2018 37
Node Net Editor
Net Simulator/Agent
Execution
World Editor
Monitoring ConsoleApplication
World Simulator
3D DisplayServer
3D DisplayClient
Eclipse Environment
Implementation:MicroPsi(Bach03,05,04,06)
Low-levelperception
ControlandsimulationEmotion and Motivation Tutorial
AAAI 2018 38
Node Net Editor
Net Simulator/Agent
Execution
World Editor
Monitoring ConsoleApplication
World Simulator
3D DisplayServer
3D DisplayClient
Eclipse Environment
Implementation:MicroPsi(Bach03,05,04,06)
Low-levelperception
ControlandsimulationMulti-agentinteractionEmotion and Motivation Tutorial
AAAI 2018 Emotion and Motivation Tutorial 39
Node Net Editor
Net Simulator/Agent
Execution
World Editor
Monitoring ConsoleApplication
World Simulator
3D DisplayServer
3D DisplayClient
Eclipse Environment
Implementation:MicroPsi(Bach03,04,05,06)
Low-levelperception
ControlandsimulationMulti-agentinteraction
Robotcontrol
AAAI 2018 Emotion and Motivation Tutorial 40
Differentsimulationenvironments
AAAI 2018 41
Implementation:MicroPsi 2(Bach,Welland,Vuine,Herger12,14,…)
Emotion and Motivation Tutorial
Cognitive Artificial Intelligence
Methodsshouldfocusoncomponentsandperformancesnecessaryforintelligence:
• UniversalRepresentations:Dynamicmodel of environment,possible worlds,and agent
• (Semi-)UniversalProblemSolving:Learning,Planning,Reasoning,Analogies,ActionControl,Reflection ...
• UniversalMotivation:Polythematic,adaptivegoal identification
• Emotionand affect• Whole,testable architectures
AAAI 2018 Emotion and Motivation Tutorial 42
ModelingMotivation
AAAI 2018 Emotion and Motivation Tutorial 43
AAAI 2018 Emotion and Motivation Tutorial 44
ModelingMotivationinaCognitiveArchitecture
• GeneralintelligenceneedsGeneralMotivation• Motivationalsystemstructurescognition• Motivationaldynamics:physiological,socialandcognitivedrives
• Intentionselectionandactioncontrol• Motivationvs.affect
Motivationvs.emotion
• Motivation:– reflectsneeds– givesrisetogoalsanddirectedbehavior– doesnothavetobeassociatedwithemotions
• Emotion:– modulatesperception,cognition,action– receivesvalencefrommotivation– receivesobjectsfromcognition– leadstoaffectiveexpression
AAAI 2018 Emotion and Motivation Tutorial 45
MotivationalSystem
• Allgoals attempt to satisfy a(hard-wired)demandà flexiblegoals,but(evolutionary)suitable behavior
AAAI 2018 Emotion and Motivation Tutorial 46
Intention
Motive selection
Urges/drives
Food
Wat
er
Inte
grity
Affi
liatio
n
Legi
timac
y
Com
pete
nce
Cer
tain
ty
Aes
thet
ics
MotivationalSystem
• Drivescorrespond to set of demands of the agent
AAAI 2018 Emotion and Motivation Tutorial 47
Physiological
Food
Wat
er
Inte
grity
Affi
liatio
n
Inte
rnal
Le
gitim
acy
Gen
eral
and
Spec
ific
Com
pete
nce
Unc
erta
inty
Red
uctio
n
Evol
utio
nary
and
Abs
tract
Aes
thet
ics
Social Cognitive
PhysiologicalDrives
• if autonomous regulation of body processes failsà actively managephysiology (seek food,water,healing,shelter,rest,warmth,...)à escape perilious situationsà implicitly seek physical survival
AAAI 2018 Emotion and Motivation Tutorial 48
Social Drives
• Affiliation:structuresocialinteractionbeyondrationalutility
• increasedby‘legitimacysignals’,decreasedby‘antilegitimacysignals’ (andadaptivelyovertime);allowsfornon-materialrewardandpunishment
• external legitimacy:groupacceptance• internal legitimacy:“honor”,conformancetointernalizedsocialnorms
AAAI 2018 Emotion and Motivation Tutorial 49
Cognitive Drives
• Competence– epistemic (problem specific)– general (ability to satisfy demands)– effect oriented
• Uncertainty reduction– novelty seeking
• Aesthetics– evolutionary preferences (stimulus oriented)– abstract (representation oriented)
AAAI 2018 Emotion and Motivation Tutorial 50
MotivationalSystem
Allpossible goals correspond to (at leastone)demand
AAAI 2018 Emotion and Motivation Tutorial 51
Physiological
Food
Wat
er
Inte
grity
Affi
liatio
n
Inte
rnal
Le
gitim
acy
Gen
eral
and
Spec
ific
Com
pete
nce
Unc
erta
inty
Red
uctio
n
Evol
utio
nary
and
Abs
tract
Aes
thet
ics
Social Cognitive
FromNeedstoBehavior
NeedsUrge
Signals
Memory
Perception
Action
Priming + Modulation
Learning
Decision Making
AAAI 2018 Emotion and Motivation Tutorial 52
AAAI 2018 Emotion and Motivation Tutorial 53
Rewardsignals
Pleasureanddistress:– Change ofademandisreflectedinpleasureor distresssignal
– Strengthisproportional toamountofchange
– Pleasureanddistresssignalsdeliverreinforcementvaluesforbehavioralproceduresandepisodicsequencesanddefineappetitive andaversive goals.
• drive =demand +urge indicator
MotivationalSystem
AAAI 2018 Emotion and Motivation Tutorial 54
Wat
ertarget
currentlevel urge indicator
swater
• motive =urge +goal situation
MotivationalLearning
AAAI 2018 Emotion and Motivation Tutorial 55
Wat
ertarget
currentlevel urge indicator
swater
goal
• motive =urge +goal situation
MotivationalLearning
AAAI 2018 Emotion and Motivation Tutorial 56
Wat
ertarget
currentlevel urge indicator
swater
goal situation
aversive situation
+
–
Goals
• Goal:situationoractionthataffordstosatisfyaneed
• Aversivegoal:situationoractionthatfrustrateaneed
• Allbehaviorisdirectedonsatisfyinganappetitivegoaloravoidinganaversivegoal
• Needsarepredefined,goalsarelearned
AAAI 2018 Emotion and Motivation Tutorial 57
Physiologicalneeds
• Thirst• Hunger• Rest• Warmth• Libido• …
à Survivalasemergentproperty
AAAI 2018 Emotion and Motivation Tutorial 58
Socialneeds
• Affiliation(Attentionfromothers,externallegitimacy)
• Internallegitimacy• Nurturing(caringforothers)• Affection• Dominance
AAAI 2018 Emotion and Motivation Tutorial 59
Cognitiveneeds
• Competence:– Skillacquisition(epistemiccompetence)– Coping/controlability(generalcompetence)– Effectgeneration
• Uncertaintyreduction:– Exploration
• Aesthetics:– Stimulusoriented– Structureoriented(abstractaesthetics)
AAAI 2018 Emotion and Motivation Tutorial 60
Needsandurges
AAAI 2018 Emotion and Motivation Tutorial 61
• association by learning:
MotivationalLearning
AAAI 2018 Emotion and Motivation Tutorial 62
urgeindicator
si goalsituation
aversivesituation
+
–
w1
w2
changeindicator
Δsi
V+
V-
A+
A-
valence associator
demand
• retrogradient reinforcement
MotivationalLearning
AAAI 2018 Emotion and Motivation Tutorial 63
urgeindicator
siw1
changeindicator
Δsi
V+ A+
valence associator
demand
protocol chaingoal
+
Motivator:
MotivationalLearning
AAAI 2018 Emotion and Motivation Tutorial 64
urge
siw1
Δsi
V+ A+
valence associator
autonomousregulation
goal
situations leading up to goal =plan
+
Intention:
MotivationalLearning
AAAI 2018 Emotion and Motivation Tutorial 65
urge
si
autonomousregulation
goal 1
goal 2
goal 3
goal n
...
Motivationallearning
AAAI 2018 Emotion and Motivation Tutorial 66
Motiveselection
AAAI 2018 Emotion and Motivation Tutorial 67
Needparameters
• Strength:relativeimportance• Decay:rateofreplenishment• Gain:effectofsatisfaction• Loss:effectoffrustration
• differentconfigurationofneedparameters=differentpersonalitytraits
AAAI 2018 Emotion and Motivation Tutorial 68
Anticipatedneeds
• Actualrewarddeterminesreinforcement(striatum,basalganglia)
• Anticipatedrewarddeterminesaction(prefrontaldopamine)
• Valenced reactionsarenotonlycausedbypresentrewards,butalsobyimagined/anticipatedrewards(expectations,memories)
AAAI 2018 Emotion and Motivation Tutorial 69
Implementationofmotivation
𝑁𝑒𝑒𝑑 ≔
𝑡𝑦𝑝𝑒 ∈ 𝑝ℎ𝑦𝑠𝑖𝑜𝑙𝑜𝑔𝑖𝑐𝑎𝑙, 𝑠𝑜𝑐𝑖𝑎𝑙, 𝑐𝑜𝑔𝑛𝑖𝑡𝑖𝑣𝑒 ,𝑣𝑎𝑙𝑢𝑒𝑡 ∈ 0, 1 ,𝑣𝑎𝑙𝑢𝑒𝑡0 ∈ 0, 1 ,𝑢𝑟𝑔𝑒𝑡 ∈ 0, 1 ,
𝑢𝑟𝑔𝑒𝑛𝑐𝑦𝑡 ∈ 0, 1 ,𝑝𝑙𝑒𝑎𝑠𝑢𝑟𝑒𝑡 ∈ 0, 1 ,𝑝𝑎𝑖𝑛𝑡 ∈ 0, 1 ,𝑤𝑒𝑖𝑔ℎ𝑡 ∈ ℝ+,
𝑑𝑒𝑐𝑎𝑦𝑡𝑖𝑚𝑒 ∈ −1, 1 ,𝑔𝑎𝑖𝑛 ∈ 0, 1 ,𝑙𝑜𝑠𝑠 ∈ 0, 1 ,
𝑠𝑎𝑡𝑖𝑠𝑓𝑎𝑐𝑡𝑖𝑜𝑛𝑖𝑚𝑔 ∈ 0, 1 ,𝑓𝑟𝑢𝑠𝑡𝑟𝑎𝑡𝑖𝑜𝑛𝑖𝑚𝑔 ∈ 0, 1 ,
𝑝𝑙𝑒𝑎𝑠𝑢𝑟𝑒𝑠𝑒𝑛𝑠𝑖𝑡𝑖𝑣𝑖𝑡𝑦 ∈ ℝ+,𝑝𝑎𝑖𝑛𝑠𝑒𝑛𝑠𝑖𝑡𝑖𝑣𝑖𝑡𝑦 ∈ ℝ+,
𝑝𝑙𝑒𝑎𝑠𝑢𝑟𝑒𝑑𝑒𝑐𝑎𝑦𝑡𝑖𝑚𝑒 ∈ ℝ+,𝑝𝑎𝑖𝑛𝑑𝑒𝑐𝑎𝑦𝑡𝑖𝑚𝑒 ∈ ℝ+,
𝑝𝑙𝑒𝑎𝑠𝑢𝑟𝑒𝑠𝑒𝑛𝑠𝑖𝑡𝑖𝑣𝑖𝑡𝑦𝑖𝑚𝑔 ∈ 0, 1 ,𝑝𝑎𝑖𝑛𝑠𝑒𝑛𝑠𝑖𝑡𝑖𝑣𝑖𝑡𝑦𝑖𝑚𝑔 ∈ 0, 1
AAAI 2018 Emotion and Motivation Tutorial 70
Urgestrengthandurgency
• 𝑢𝑟𝑔𝑒𝑡 = 𝑤𝑒𝑖𝑔ℎ𝑡 1 − 𝑣𝑎𝑙𝑢𝑒𝑡−1 012
• 𝑢𝑟𝑔𝑒𝑛𝑐𝑦𝑡 = 𝑤𝑒𝑖𝑔ℎ𝑡 𝑘−𝑟𝑒𝑚𝑎𝑖𝑛𝑖𝑛𝑔𝑡𝑖𝑚𝑒𝑡𝑘 0
12
AAAI 2018 Emotion and Motivation Tutorial 71
Valueanddecayofaneed
• 𝑣𝑎𝑙𝑢𝑒𝑡 =
decay 𝑣𝑎𝑙𝑢𝑒𝑡−1+𝑔𝑎𝑖𝑛×𝛿𝑡𝑐𝑜𝑛𝑠𝑢𝑚𝑒
+𝑙𝑜𝑠𝑠×𝛿𝑡𝑎𝑣𝑒𝑟𝑠𝑖𝑣𝑒
+𝑔𝑎𝑖𝑛×𝑠𝑎𝑡𝑖𝑠𝑓𝑎𝑐𝑡𝑖𝑜𝑛𝑖𝑚𝑔𝛿𝑡𝑖𝑚𝑔𝑐𝑜𝑛𝑠𝑢𝑚𝑒
+𝑙𝑜𝑠𝑠×𝑓𝑟𝑢𝑠𝑡𝑟𝑎𝑡𝑖𝑜𝑛𝑖𝑚𝑔𝛿𝑡𝑖𝑚𝑔𝑎𝑣𝑒𝑟𝑠𝑖𝑣𝑒
0
1
• decay 𝑣𝑡−1 ≔ 𝜎 𝜎−1 𝑣𝑡−1 + duration 𝑡,𝑡−1𝑑𝑒𝑐𝑎𝑦𝑡𝑖𝑚𝑒
AAAI 2018 Emotion and Motivation Tutorial 72
Pleasureandpainassociatedwithaneed
• 𝑝𝑙𝑒𝑎𝑠𝑢𝑟𝑒𝑡 =decay 𝑝𝑙𝑒𝑎𝑠𝑢𝑟𝑒𝑡−1, 𝑝𝑙𝑒𝑎𝑠𝑢𝑟𝑒𝑑𝑒𝑐𝑎𝑦𝑡𝑖𝑚𝑒+𝑔𝑎𝑖𝑛×𝑝𝑙𝑒𝑎𝑠𝑢𝑟𝑒𝑠𝑒𝑛𝑠𝑖𝑡𝑖𝑣𝑖𝑡𝑦×𝛿𝑡𝑐𝑜𝑛𝑠𝑢𝑚𝑒
+𝑔𝑎𝑖𝑛×𝑝𝑙𝑒𝑎𝑠𝑢𝑟𝑒𝑠𝑒𝑛𝑠.𝑖𝑚𝑔×𝛿𝑡𝑖𝑚𝑔𝑐𝑜𝑛𝑠𝑢𝑚𝑒
0
1
• 𝑝𝑎𝑖𝑛𝑡 =
decay 𝑝𝑎𝑖𝑛𝑡−1, 𝑝𝑎𝑖𝑛𝑑𝑒𝑐𝑎𝑦𝑡𝑖𝑚𝑒+𝑙𝑜𝑠𝑠×𝑝𝑎𝑖𝑛𝑠𝑒𝑛𝑠𝑖𝑡𝑖𝑣𝑖𝑡𝑦×𝛿𝑡𝑎𝑣𝑒𝑟𝑠𝑖𝑣𝑒
+𝑙𝑜𝑠𝑠×𝑝𝑎𝑖𝑛𝑠𝑒𝑛𝑠𝑖𝑡𝑖𝑣𝑖𝑡𝑦𝑖𝑚𝑔×𝛿𝑡𝑖𝑚𝑔𝑎𝑣𝑒𝑟𝑠𝑖𝑣𝑒
+painfromdepletion(value𝑡−1) 0
1
• painfromdepletion 𝑣𝑎𝑙𝑢𝑒 ≔ 1 − 𝑣𝑎𝑙𝑢𝑒𝜃 0
12
AAAI 2018 Emotion and Motivation Tutorial 73
Eventsandconsumptions
• 𝐸𝑣𝑒𝑛𝑡 ≔
𝑐𝑜𝑛𝑠𝑢𝑚𝑝𝑡𝑖𝑜𝑛 ∈ 𝐶𝑜𝑛𝑠𝑢𝑚𝑝𝑡𝑖𝑜𝑛𝑠,𝑒𝑥𝑝𝑒𝑐𝑡𝑒𝑑𝑟𝑒𝑤𝑎𝑟𝑑 ∈ −1, 1 ,
𝑐𝑒𝑟𝑡𝑎𝑖𝑛𝑡𝑦 ∈ 0, 1 ,𝑠𝑘𝑖𝑙𝑙 ∈ 0, 1 ,
𝑟𝑒𝑚𝑎𝑖𝑛𝑖𝑛𝑔𝑡𝑖𝑚𝑒𝑡 ∈ ℝ+
• 𝐶𝑜𝑛𝑠𝑢𝑚𝑝𝑡𝑖𝑜𝑛 ≔
𝑛𝑒𝑒𝑑 ∈ 𝑁𝑒𝑒𝑑𝑠,𝑟𝑒𝑤𝑎𝑟𝑑𝑡 ∈ ℝ,
𝑡𝑜𝑡𝑎𝑙𝑟𝑒𝑤𝑎𝑟𝑑 ∈ ℝ,𝑟𝑒𝑤𝑎𝑟𝑑𝑑𝑢𝑟𝑎𝑡𝑖𝑜𝑛 ∈ ℝ+,
𝑚𝑎𝑥𝑟𝑒𝑤𝑎𝑟𝑑 ∈ ℝ,𝑑𝑖𝑠𝑐𝑜𝑢𝑛𝑡 ∈ 0, 1
AAAI 2018 Emotion and Motivation Tutorial 74
Rewardsignalandrewardsummation
• 𝑠𝑖𝑔𝑛𝑎𝑙 𝑡 : = 𝑡𝑒−12𝑡2
• 𝑡1 = 𝑡 − 𝑡𝑜𝑛𝑠𝑒𝑡 ]
𝑟𝑒𝑤𝑎𝑟𝑑𝑑𝑢𝑟𝑎𝑡𝑖𝑜𝑛duration(t, t − 1)
• 𝑡1 = 𝑡 − 𝑡𝑜𝑛𝑠𝑒𝑡 ]
𝑟𝑒𝑤𝑎𝑟𝑑𝑑𝑢𝑟𝑎𝑡𝑖𝑜𝑛duration(t, t − 1)
• 𝑟𝑒𝑤𝑎𝑟𝑑𝑡 =]×𝑡𝑜𝑡𝑎𝑙𝑟𝑒𝑤𝑎𝑟𝑑𝑟𝑒𝑤𝑎𝑟𝑑𝑑𝑢𝑟𝑎𝑡𝑖𝑜𝑛 ∫ 𝑡𝑒−
12𝑡2𝑡2
𝑡1 −𝑚𝑎𝑥𝑟𝑒𝑤𝑎𝑟𝑑
𝑚𝑎𝑥𝑟𝑒𝑤𝑎𝑟𝑑
= ]×𝑡𝑜𝑡𝑎𝑙𝑟𝑒𝑤𝑎𝑟𝑑𝑟𝑒𝑤𝑎𝑟𝑑𝑑𝑢𝑟𝑎𝑡𝑖𝑜𝑛
𝑒−12𝑡12− 𝑒−
12𝑡22
−𝑚𝑎𝑥𝑟𝑒𝑤𝑎𝑟𝑑
𝑚𝑎𝑥𝑟𝑒𝑤𝑎𝑟𝑑
AAAI 2018 Emotion and Motivation Tutorial 75
Actualizedrewardschangevaluesofneeds
• 𝛿𝑡𝑐𝑜𝑛𝑠𝑢𝑚𝑒 = 𝑟𝑒𝑤𝑎𝑟𝑑𝑡𝑐𝑜𝑛𝑠𝑢𝑚𝑝𝑡𝑖𝑜𝑛
0
∞
• 𝛿𝑡𝑎𝑣𝑒𝑟𝑠𝑖𝑣𝑒 = 𝑟𝑒𝑤𝑎𝑟𝑑𝑡𝑐𝑜𝑛𝑠𝑢𝑚𝑝𝑡𝑖𝑜𝑛
−∞
0
AAAI 2018 Emotion and Motivation Tutorial 76
Dealingwithanticipatedrewards
• 𝛿𝑡𝑖𝑚𝑔𝑐𝑜𝑛𝑠𝑢𝑚𝑒 = 𝑐𝑒𝑟𝑡𝑎𝑖𝑛𝑡𝑦×𝑠𝑘𝑖𝑙𝑙× 𝑟𝑒𝑤𝑎𝑟𝑑𝑡
𝑐𝑜𝑛𝑠𝑢𝑚𝑝𝑡𝑖𝑜𝑛
1+𝑑×𝑟𝑒𝑚𝑎𝑖𝑛.𝑡𝑖𝑚𝑒𝑡 0
∞
• 𝛿𝑡𝑖𝑚𝑔𝑎𝑣𝑒𝑟𝑠𝑖𝑣𝑒 = 𝑐𝑒𝑟𝑡.×(1 − 𝑠𝑘𝑖𝑙𝑙)× 𝑟𝑒𝑤𝑎𝑟𝑑𝑡
𝑐𝑜𝑛𝑠𝑢𝑚𝑝𝑡𝑖𝑜𝑛
1+𝑑×𝑟𝑒𝑚𝑎𝑖𝑛.𝑡𝑖𝑚𝑒𝑡 −∞
0
(d isdiscount;hyperbolicdiscounting)
AAAI 2018 Emotion and Motivation Tutorial 77
Cognitive Artificial Intelligence
Methodsshouldfocusoncomponentsandperformancesnecessaryforintelligence:
• UniversalRepresentations:Dynamicmodel of environment,possible worlds,and agent
• (Semi-)UniversalProblemSolving:Learning,Planning,Reasoning,Analogies,ActionControl,Reflection ...
• UniversalMotivation:Polythematic,adaptivegoal identification
• Emotionand affect• Whole,testable architectures
AAAI 2018 Emotion and Motivation Tutorial 78
ModelingEmotion
AAAI 2018 Emotion and Motivation Tutorial 79
Emotionalexpression
• PaulEkman:FacialActionCoding
AAAI 2018 Emotion and Motivation Tutorial 80
Emotionalexpression
• PaulEkman:FacialActionCoding
AAAI 2018 Emotion and Motivation Tutorial 81
Emotionalexpression
• PaulEkman:FacialActionCoding
AAAI 2018 Emotion and Motivation Tutorial 82
Affectivecomputing
Recognize,process,simulate,expresshumanaffects
AAAI 2018 Emotion and Motivation Tutorial 83
Appraisalmodels
• MagdaArnold,RichardLazarus:– Emotionascognitiveappraisalsofrelations,motivation,cognition
• KlausScherer:– Stimulusevaluationchecks– innate(sensorymotor)® learned(schemas)® deliberate– relevance® implication® copingpotential® normativesignific.
AAAI 2018 Emotion and Motivation Tutorial 84
AAAI 2018 Emotion and Motivation Tutorial 85
Conceptualanalysis:Ortony,Clore,Collins1988:
AAAI 2018 Emotion and Motivation Tutorial 86
AffectivedimensionsinthePSItheory(Dörner1999)
Valence(appentence/aversion)
Arousal(unspecificsympathicussyndrome)
Selectionthreshold(motivedominance)
AAAI 2018 Emotion and Motivation Tutorial 87
Compare:Affectivedimensions(Wundt1910)
AAAI 2018 Emotion and Motivation Tutorial 88
AffectivedimensionsinthePSItheory
Valence(appentence/aversion)
Arousal(unspecificsympathicussyndrome)
Selectionthreshold(motivedominance)
Resolutionlevel(focus)
Securingrate
Goaldirectedness
AAAI 2018 Emotion and Motivation Tutorial 89
ThePsitheoryaboutemotion
• Affectisseenasaconfiguration ofacognitivesystem
• Modulatorsofcognition:– arousal,selectionthreshold,securingthreshold,resolutionlevel
– estimateofcompetenceandcertainty– pleasure/distresssignals® valence
• Affectivestateisemergentpropertyofmodulation
• Directedaffects(higher-levelemotions)emergebyassociationofdemandwithappetive oraversiveobjects/situations
AAAI 2018 Emotion and Motivation Tutorial 90
Purposeofemotionalmodulation
• Controlwidth,depthandbiasofoperationsonmentalrepresentationsoftheagent®modifyperception,memory,planningandactionselection
• Reducecomplexityofcognitiveprocesses
AAAI 2018 Emotion and Motivation Tutorial 91
ModulationinPSI/MicroPsi
AAAI 2018 Emotion and Motivation Tutorial 92
Primarymodulators
Attentional modulators
AAAI 2018 Emotion and Motivation Tutorial 93
Modulatordynamics
AAAI 2018 Emotion and Motivation Tutorial 94
Modulatorparameters
• Baseline• Range• Volatility• Duration
• Differentmodulatorparameterconfigurations=differenttemperaments
AAAI 2018 Emotion and Motivation Tutorial 95
Emotions as directed affect +Modulation
Examples:
Fear:anticipation of aversive events (à neg.valence)+arousalAnxiety:uncertainty (à neg.valence)+low competence +arousal,highsecuring behavior (frequent background checks)
AAAI 2018 Emotion and Motivation Tutorial 96
Emotions as directed affect +Modulation
Examples:
Anger:Perceived obstacle (usually agent)manifestly prevented reachingof anactive,motivationally relevantgoal (àneg.valence),sanctioningbehavior tendency (à goal relevance is re-directed to sanctioning ofobstacle),arousal,low resolution level,highaction readyness,highselection threshold
Sadness:Manifestprevention from all conceived ways of reachingactive,relevantgoal,without relevantobstacle (à neg.valence),support-seeking behavior (by increased demand for affiliation),low arousal,inhibition of active goalà decreased action readyness
AAAI 2018 Emotion and Motivation Tutorial 97
Emotions as directed affect +Modulation
Examples:
• Pride:highcompetence (à low securing rate),highinternal legitimacy,likely coincidence with highexternallegitimacy
• Joy:higharousal +highperceived reward signal fromsatisfying ademand
• Bliss:low arousal +highperceived reward signal fromsatisfying ademand (since physiological demands ofteninvolve higharousal,mostly related to cognitive demands,suchas aesthetics)
AAAI 2018 Emotion and Motivation Tutorial 98
Implementationofaffectivemodulation
• 𝑀𝑜𝑑𝑢𝑙𝑎𝑡𝑜𝑟 ≔
𝑚𝑖𝑛 ∈ ℝ𝑚𝑎𝑥 ∈ ℝ,
𝑙𝑒𝑣𝑒𝑙𝑡 ∈ [𝑚𝑖𝑛,𝑚𝑎𝑥],𝑏𝑎𝑠𝑒𝑙𝑖𝑛𝑒 ∈ [𝑚𝑖𝑛,𝑚𝑎𝑥],
𝑣𝑜𝑙𝑎𝑡𝑖𝑙𝑖𝑡𝑦 ∈ ℝ+,𝑑𝑒𝑐𝑎𝑦𝑡𝑖𝑚𝑒 ∈ ℝ+
• 𝑖𝑛𝑡𝑒𝑟𝑣𝑎𝑙 = d𝑚𝑎𝑥 − 𝑏𝑎𝑠𝑒𝑙𝑖𝑛𝑒, if𝑙𝑒𝑣𝑒𝑙𝑡−1 > 𝑏𝑎𝑠𝑒𝑙𝑖𝑛𝑒𝑏𝑎𝑠𝑒𝑙𝑖𝑛𝑒 − 𝑚𝑖𝑛, else
• 𝛿𝑡 = 𝑡𝑎𝑟𝑔𝑒𝑡×𝑖𝑛𝑡𝑒𝑟𝑣𝑎𝑙 + 𝑏𝑎𝑠𝑒𝑙𝑖𝑛𝑒 − 𝑙𝑒𝑣𝑒𝑙𝑡−1 𝑣𝑜𝑙𝑎𝑡𝑖𝑙𝑖𝑡𝑦
AAAI 2018 Emotion and Motivation Tutorial 99
Implementationofvalence
• marginalsum 𝑉, 𝑙𝑖𝑚𝑖𝑡 : = ∑ 𝑆𝑛|𝑉|𝑛=0 l 𝑆𝑛 ≔
𝑙𝑖𝑚𝑖𝑡−𝑆𝑛−1𝑙𝑖𝑚𝑖𝑡
𝑣𝑛• 𝑙𝑖𝑚𝑖𝑡 = max 𝜔|𝜔 ∈ 𝑤𝑒𝑖𝑔ℎ𝑡𝑛𝑒𝑒𝑑𝑠
• 𝑐𝑜𝑚𝑏𝑖𝑛𝑒𝑑𝑝𝑎𝑖𝑛 = marginalsum weightneed×𝑝𝑎𝑖𝑛𝑛𝑒𝑒𝑑
• 𝑐𝑜𝑚𝑏𝑖𝑛𝑒𝑑𝑝𝑙𝑒𝑎𝑠𝑢𝑟𝑒 = marginalsum weightneed×𝑝𝑙𝑒𝑎𝑠𝑢𝑟𝑒𝑛𝑒𝑒𝑑
• 𝑡𝑎𝑟𝑔𝑒𝑡𝑣𝑎𝑙𝑒𝑛𝑐𝑒 = 𝑐𝑜𝑚𝑏𝑖𝑛𝑒𝑑𝑝𝑙𝑒𝑎𝑠𝑢𝑟𝑒−𝑐𝑜𝑚𝑏𝑖𝑛𝑒𝑑𝑝𝑎𝑖𝑛𝑙𝑖𝑚𝑖𝑡
AAAI 2018 Emotion and Motivation Tutorial 100
Implementationofarousal
• 𝑐𝑜𝑚𝑏𝑖𝑛𝑒𝑑𝑢𝑟𝑔𝑒 = marginalsum weightneed×𝑢𝑟𝑔𝑒𝑛𝑒𝑒𝑑𝑙𝑖𝑚𝑖𝑡
• 𝑐𝑜𝑚𝑏𝑖𝑛𝑒𝑑𝑢𝑟𝑔𝑒𝑛𝑐𝑦 = marginalsum weightneed×𝑢𝑟𝑔𝑒𝑛𝑐𝑦𝑛𝑒𝑒𝑑𝑙𝑖𝑚𝑖𝑡
• 𝑡𝑎𝑟𝑔𝑒𝑡𝑎𝑟𝑜𝑢𝑠𝑎𝑙 = 𝑐𝑜𝑚𝑏𝑖𝑛𝑒𝑑𝑢𝑟𝑔𝑒 + 𝑐𝑜𝑚𝑏𝑖𝑛𝑒𝑑𝑢𝑟𝑔𝑒𝑛𝑐𝑦 − 1
AAAI 2018 Emotion and Motivation Tutorial 101
Implementationofaggression/regression
• 𝑒𝑝𝑖𝑠𝑡𝑒𝑚𝑖𝑐𝑐𝑜𝑚𝑝𝑒𝑡𝑒𝑛𝑐𝑒 = 𝑠𝑘𝑖𝑙𝑙𝑐𝑢𝑟𝑟𝑒𝑛𝑡𝑔𝑜𝑎𝑙𝑒𝑣𝑒𝑛𝑡
• 𝑔𝑒𝑛𝑒𝑟𝑎𝑙𝑐𝑜𝑚𝑝𝑒𝑡𝑒𝑛𝑐𝑒 = valuecompetence×𝑒𝑝𝑖𝑠𝑡𝑒𝑚𝑖𝑐𝑐𝑜𝑚𝑝.�
• 𝑡𝑎𝑟𝑔𝑒𝑡𝑎𝑔𝑔𝑟𝑒𝑠𝑠𝑖𝑜𝑛 = 𝑔𝑒𝑛𝑒𝑟𝑎𝑙𝑐𝑜𝑚𝑝.+𝑒𝑝𝑖𝑠𝑡𝑒𝑚𝑖𝑐𝑐𝑜𝑚𝑝.−1
AAAI 2018 Emotion and Motivation Tutorial 102
Emotionviewer
AAAI 2018 Emotion and Motivation Tutorial 103
Personalitymodeling
• Motivationparameters:Personalityproperties• Modulationparameters:Temperament
AAAI 2018 Emotion and Motivation Tutorial 104
IndividualVariations by Parameterizing
Possible grounding of personality properties (FFM):• Openness:appreciation of art and new ideas,curiousity
• Conscientiousness:rulefollowing vs.chaotic• Extraversion:tendency to seek stimulation byenvironment and others
• Agreeableness:tendency for cooperativeness andcompassion
• Neuroticism:emotionalstability,effect of failureto self-confidence
AAAI 2018 Emotion and Motivation Tutorial 105
NeedsandBigFive
AAAI 2018 Emotion and Motivation Tutorial 106
Example:Mappingto FFM(BigFive)
Demanddynamics:
AAAI 2018 Emotion and Motivation Tutorial 107
Physiological
Food Water Integrity Affiliation Internal Legitimacy
Gen./Epis.Competence
UncertaintyReduction Aesthetics
Social Cognitive
Example:Mappingto FFM(BigFive)
Demanddynamics:
AAAI 2018 Emotion and Motivation Tutorial 108
Physiological
Affiliation Internal Legitimacy
Gen./Epis.Competence
UncertaintyReduction Aesthetics
Social Cognitive
Food Water Integrity
Example:Mappingto FFM(BigFive)
Demanddynamics:
AAAI 2018 Emotion and Motivation Tutorial 109
Affiliation Internal Legitimacy
Gen./Epis.Competence
UncertaintyReduction
Aesthetics
Example:Mappingto FFM(BigFive)
Valence:Pleasure/Pain signals
AAAI 2018 Emotion and Motivation Tutorial 110
Affiliation Internal Legitimacy
Gen./Epis.Competence
UncertaintyReduction
V+
V-
Aesthetics
Example:Mappingto FFM(BigFive)
AAAI 2018 Emotion and Motivation Tutorial 111
Affiliation Internal Legitimacy
Gen./Epis.Competence
UncertaintyReduction
V+
Aesthetics
V-
Neuroticism:strongerexperienceofnegativeemotions,loweremotionalstability(strongnegativereward/strongerlossofcompetence,certainty;strongerdecay->morefrequentneedforreplenishment,possiblylowergain),pronenesstoanxietyduetolossofcertainty
Example:Mappingto FFM(BigFive)
Extraversion:surgency,activity insocial relations,expressivity(stronggain for affiliation and competence,highdecay of affiliation)
AAAI 2018 Emotion and Motivation Tutorial 112
Affiliation Internal Legitimacy
Gen./Epis.Competence
UncertaintyReduction
V+
V-
Aesthetics
Example:Mappingto FFM(BigFive)
Openness:desire for novelty,intellectual independence,non-conservatism,appreciation for art and new ideas(stronggain for uncertainty reduction,highepistemic competence foruncertainty reduction,probably strongrewards for alldemands)
AAAI 2018 Emotion and Motivation Tutorial 113
Affiliation Internal Legitimacy
Gen./Epis.Competence
UncertaintyReduction
V-
AestheticsV+
Example:Mappingto FFM(BigFive)
Agreeableness:(strongpositiveand negativereward for affiliation,lowergains for competence)
AAAI 2018 Emotion and Motivation Tutorial 114
Affiliation Internal Legitimacy
Gen./Epis.Competence
UncertaintyReduction
V+
V-
Aesthetics
Example:Mappingto FFM(BigFive)
Conscientousness,Rigidity:(highloss incompetence,highselection threshold,possibly low effect of demands onarousal and resolutionlevel)
AAAI 2018 Emotion and Motivation Tutorial 115
Affiliation Internal Legitimacy
Gen./Epis.Competence
UncertaintyReduction
V+
V-
Aesthetics
Example:Mappingto FFM(BigFive)
• Why notone free variableperFFMdimension?
AAAI 2018 Emotion and Motivation Tutorial 116
FFMdoes nottell the complete story
Shyness !=Introversion(highloss of affiliation,low competence)
AAAI 2018 Emotion and Motivation Tutorial 117
Affiliation Internal Legitimacy
Gen./Epis.Competence
UncertaintyReduction
V+
V-
Aesthetics
Howcanweevaluateamodelofmotivation?
• Games!
Emotion and Motivation TutorialAAAI 2018 118
SpaceInvaders(1978,TomohiroNishikado)
AAAI 2018 Emotion and Motivation Tutorial 119
WorldofWarcraft (RobPardo,JeffKaplanetal.2004)
AAAI 2018 Emotion and Motivation Tutorial 120
Playerpersonalitytypes
RichardBartle(1996):“Hearts,Clubs,Diamonds,Spades:PlayersWhosuitMUDs”
AAAI 2018 Emotion and Motivation Tutorial 121
Motivationandpersonality
• Personalitypropertiescanbemodeledasmotivationalvariability
Affiliation
CompetenceUncertaintyreduction
AAAI 2018 Emotion and Motivation Tutorial 122
Needsandplayertypes(withS.Tekovsky)
AAAI 2018 Emotion and Motivation Tutorial 123
EmotionandtheSelf
AAAI 2018 Emotion and Motivation Tutorial 124
Neocortexasamodelingsystem
• corticalmodelingisapredictivemodelofsensorypatterns,asadynamicsimulationofworldandagent
AAAI 2018 Emotion and Motivation Tutorial 125
Neocortexasamodelingsystem
AAAI 2018 Emotion and Motivation Tutorial 126
Neocortexasamodelingsystem
AAAI 2018 Emotion and Motivation Tutorial 127
Neocortexasamodelingsystem
• Primarymodel:environmentandagent• Secondarymodel:interactionofagentwithenvironment
• Tertiarymodel:functioningofsecondarymodelingàSelf
• Consciousnessasamodelofattention
AAAI 2018 Emotion and Motivation Tutorial 128
Integrationofemotionandmotivation
• Controlandselfregulationdependsonlearnedfunctionalrepresentationsincorticalstructure
• Attentionalbiases
• Attentionismechanismfordirectedlearning• Experientialaccessviaattentionalprotocol
• Structureofselfdeterminesexperienceofemotion
AAAI 2018 Emotion and Motivation Tutorial 129
Socialemotions
• Objectofemotionrequiresrepresentationandmotivationalrelevance
• Socialurges:affiliation,romanticaffect,libido,dominancearetransactional
• Love:non-transactionalemotion
• Loverequiressharedpurpose,vialegitimacy
AAAI 2018 Emotion and Motivation Tutorial 130
Acknowledgements
WorkonMicroPsi2iscollaborativeeffort:- RonnieVuine,DominikWelland,PriskaHerger,JonasKemperarecontributorstothecurrentversion
- Architecture/conceptshavebeeninspiredbyDietrichDörner,AaronSloman,MarvinMinsky,StanFranklinandmanyothers
- SupportfromHumboldtUniversityofBerlin,UniversityofOsnabrück(InstituteforCognitiveScience),BerlinSchoolofMindandBrain,HarvardProgramofEvolutionaryDynamics,MITMediaLab
AAAI 2018 Emotion and Motivation Tutorial 131
Thank you!
Interesting questions:
• Is recursive function approximation plusmotivationalsystem sufficient for generalintelligence?
• Could we functionally recreate human-likemindswith our model?
• How does amotivated/emotionalsystem evolvewhen it can modify itself?
AAAI 2018 Emotion and Motivation Tutorial 132