Path-finding Using Reinforcement Learning and Affective States

Johannes Feldmaier1 and Klaus Diepold1

Abstract— During decision making and acting in the en-vironment humans appraise decisions and observations withfeelings and emotions. In this paper we propose a frameworkto incorporate an emotional model into the decision makingprocess of a machine learning agent. We use a hierarchicalstructure to combine reinforcement learning with a dimen-sional emotional model. The dimensional model calculatestwo dimensions representing the actual affective state of theautonomous agent. For the evaluation of this combination, weuse a reinforcement learning experiment (called Dyna Maze) inwhich, the agent has to find an optimal path through a maze.Our first results show that the agent is able to appraise thesituation in terms of emotions and react according to them.

I. INTRODUCTION

The integration of affective states and emotions into nextgeneration artificial intelligence (AI) could be the nextconsequent step in the development of robotic systems. Inmodern psychology emotions are an important componentin decision making. Humans judge situations with emotionsand the reactions depends on them [1], [2]. Emotions can alsobe seen as a way to simplify actual decisions, enabling oneto compare different events and actions against each other[3]. Thereby, emotions can be used as a utility function foraction selection and for event classification. Using objectiveutility functions for decision making is a common basisin AI systems. In the future, it would be desirable touse affective utility functions instead. This could improvethe believability and behaviour of AI systems. At a firstglance, emotions could be seen as a direct result of thestate and previous actions and hence there is no differencecompared to objective utility functions. But generally emo-tions incorporate more than objective features. They dependon personality (especially the mood), the attachment tothings and previous experiences. Especially knowledge andexperience are main components of emotions. In psychologythere is a phenomenon called mental time travel describinga situation in which the current action promises an imme-diate positive emotion but also simultaneously an expectednegative consequence in the future (e.g. stealing something).Remembering consequences requires a memory process andthe ability to learn. Particularly reinforcement learning (RL)combines rewards with previous actions and is a commonlyused strategy in artificial intelligence. However, not manyexisting systems combine reinforcement learning with re-wards calculated using artificial emotions. One approachintegrating emotional attachment and decision making into

1J. Feldmaier and K. Diepold are with Department of Elec-trical Engineering, Institute for Data Processing, Technische Uni-versitat Munchen, Germany johannes.feldmaier@tum.de,klaus.diepold@tum.de

an agent is described in [4]. This approach shows somesimilarities to our system, but focuses mainly on the human-machine interaction part. However, our approach focuses onthe affective appraisal of decisions made by the agent itself.Therefore, we integrate affective states into reinforcementlearning. Furthermore, we describe why such a combinationis a contribution to social robotics. Social robotics means inthis context that a robotic system should act in a way that canbe understood by humans. Today, many robots act like bigblack boxes with some actuators and thereby elicit feelingsof fear and anxiety in their human operators. On the onehand, this is a result of their often cryptic textual outputsand signals. On the other hand, there exists only few outputdevices displaying the current emotional state of the systemusing profound integrative and functional frameworks of thedisplayed emotion and mood. A display showing the user theactual emotional state (e.g. the feelings and mood) enablesthe AI agent more transparent to non experts and thereforemore attractive and less frightening.

The primary goal of this work is the combination andintegration of affective states and models with reinforcementlearning. We use a reinforcement learning framework (DynaMaze) simulating a small maze, which is bounded with wallsand contains some obstacles. In Section II we point outsome basic principles of reinforcement learning. Section IIIdescribes the main points of our dimensional emotion modelwhich is then, in Section IV, used in combination with thereinforcement learning framework to appraise the learningprogress of the agent. The last two Sections discuss theresults and conclude the work.

II. REINFORCEMENT LEARNING

For the evaluation of our emotional model and the com-bination with reinforcement learning we use the well knowngridworld example Dyna Maze [5]. This example representsa rectangular maze with some walls and obstacles. The agenthas in each state four possible actions, up, down, right, andleft (cf. Fig. 1). Selecting one of the actions takes the agentto the corresponding neighbouring state. In cases when themovement is blocked by a wall or the boundaries of themaze, the agent remains in its current position. The agentbegins in the start state S and has to reach the goal stateG. We define the state space S = s1, s2, ..., sn and theavailable set of actions as A(s). Hence, st ∈ S denotes theactual state of the agent at time t and at ∈ A(s) denotesthe action executed at time t. Executing action at causes atransition change from state st to state st+1. Additionally,we can assume a deterministic world so that each transitionor action is executed properly.

The 23rd IEEE International Symposium on Robot andHuman Interactive CommunicationAugust 25-29, 2014. Edinburgh, Scotland, UK,

978-1-4799-6765-0/14/$31.00 ©2014 IEEE 543

S

G

actions at

left right

up

down

st st+1

Fig. 1. Gridworld example and the four possible actions at. Each field ofthe grid corresponds to a possible state st of the agent.

The experiment is conducted in episodes. In each episodethe agent has to find its way from the starting point tothe goal. The maze always remains unchanged. In the firstepisode the agent is placed at the start state S without anyknowledge about the structure of the maze. The agent firstselects an action (randomly) and performs the movementwithin this new environment. After performing each actionthe agent receives a reward rt from the environment. Thefirst steps or episodes can be seen as an exploration process.During the exploration the reward is either negative orrelatively low. If the agent reaches the goal state G, it receivesa high positive reward. After some episodes, the cumulativereward at the end of each episodes increases because theagent gains knowledge about the maze.

In our experiment we use the Dyna-Q algorithm [5] forlearning and selecting the actions in each state. Dyna-Qincludes planning, acting, model-learning and direct rein-forcement learning. Each process occurs continually duringthe exploration and path-finding.

First, the agent selects an action at and performs thetransition st, at → st+1, rt+1 to the next state st+1. Afterthe state transition the agent gets the reward which dependsonly on state st+1. With the reward and the performedaction the model records in its table entry for the previousstate and action pair st, at the prediction for the rewardin the following state st+1 taking action at+1. Thus, themodel is able to predict the next reward using the last-observed next state and reward. The model update step isfollowed by the planning step. While planning the Dyna-Q algorithm randomly samples previously observed state-action pairs improving the action-value (Q-) function. Theaction-value function Q is learned or updated after eachstate transition. The Q-function estimates the future expectedreward taking an action in a specific state. This enables theagent to determine how good it is to be in a given state andperform a given action [5]. We use Q-learning to update theaction-value function. The update of the Q-function

Q(st, at)← Q(st, at)+α[rt+1 + γmaxaQ(st+1, a)−Q(st, at)](1)

is independent of the policy being followed (off-policy) [6]and depends on the step-size α and discount-rate γ (cf. TableI). After planning, updating the Q-function and the model,the agent uses the action-value function to select the nextaction. For the action selection we used the ε-greedy policy.

This policy selects most of the time the greedy action (theone with the highest Q value), but once in a while with asmall probability ε it selects a random action.

We use the described Dyna-Q learning algorithm for thepath-finding task in a small maze environment because thisexperiment provides a manageable complexity, sufficientevents for the stimulation of our emotional model andadditionally, the structure can be integrated in the affectdesign structure of Norman et al. [7].

III. EMOTIONAL MODEL

There is no unique definition of emotions but ratherthere are several different models of emotional systems. Onetheory describes the emotional system of humans with so-called basic emotions. This theory was mainly influenced byPaul Ekman [8] and Jaak Panksepp [9]. The basic emotiontheory describes emotions as discrete states disregardingsome intermediate states and the integration of the theory intoan overarching framework. Such a complete and continuousstructure of emotions is offered by so called dimensionaltheories [10], [11], [12]. Commonly, dimensional modelsrepresent emotions as a point in a two- or three-dimensionalspace. The different dimensions were determined in em-pirical studies in which subjects had to appraise emotionalscenes.

There is an ongoing discussion [13], [14] between thedimensional and the basic emotion views of emotion, butit is not our objective to support one model explicitly.Furthermore, we consider the different models regardingtheir usability in technical systems. Therefore, we decidedto implement and evaluate an emotional model based on thedimensional theory of emotions.

VA-Space Model

In dimensional models the affective state of an organismis represented in a space of two- or three-dimensions. Thisspace is called core affective state space. Most computationalmodels build on the three-dimensional PAD model of Mehra-bian and Russell [10]. In the PAD model the three dimensionsof the core affective state are denoted as pleasure P, arousalA, and dominance D. Dropping dominance D is a commonsimplification of the model. The dimension of dominance isnot significant in the present scenario involving only a singleagent. Pleasure denotes a measure of valence and indicateswhether an emotion, an event, or an experience is good orbad for you (the general perception of positivity or negativityof a situation). In psychology, the terms pleasure and valenceare interchangeable. In the following, we will use valenceV. The second fundamental dimension is called arousal Aand indicates the level of affective activation of an agent.Arousal can be defined as the mental alertness and physicalactivity [15]. The core affective state space spanned by thetwo dimensions of valence and arousal is further described bythe discrete emotions which subjects have reported for eachquadrant (Q1 – Q4 in Figure 2) [16]. They serve as additionaldiscrete labels for each core affective state. Placing a pointin this two-dimensional space and pushing it around by a

544

continuous time-varying process appraising eliciting eventsis a commonly used implementation for the core affect of anagent [17], [18]. Besides direct influences of eliciting events,the core affect could additionally be modified by incorpo-rating the impact of dispositional tendencies such as moodstate and personality. There is some disagreement withinpsychology to use dimensional models as emotion elicitingprocess [13], [14], [18]. However, in technical systems therepresentation of events within the two-dimensional affectivestate space is a convenient and economical way to appraiseevents in terms of emotions [18], [19]. For this reason, weuse a two-dimensional emotion model for calculating theaffective state during a reinforcement learning experiment.

arousalhigh

low

valencepositivenegative

excited

happy

relaxed

calm

sad

depressed

anxious

fearful

Q1

Q2Q3

Q4

Fig. 2. Core affect represented in two-dimensional space – valence andarousal. Possible reported affective states (emotions) are stated in eachquadrant (words in italics). The left half-plane of the VA-space is related tonegative (red) experiences and the right half-plane to positive (green) ones(adapted from [16]).

IV. INTEGRATION OF THE EMOTIONAL MODEL INTODYNA MAZE

Norman et al. published a paper about affect and machinedesign describing a model of affect and cognition. Theypropose a three-level theory of human behaviour basicallyapplicable to the architecture of affective computer systems[7].

The three levels are the reaction level, the routine level,and the reflection level. These levels enable a processing ofthe surrounding world in terms of affective evaluation andcognitive interpretation of the environment.

Figure 3 depicts the three levels including their assignedcomponents of the Dyna Maze example and the emotionalmodel. Lower levels perform fast computations and processesand higher levels involve more information resulting inincreased effort.

The lowest level – the reaction level – processes low levelinformation, performs rapid reactions to the current state(such as reflexes), and controls motor actions. Therefore, wehave used the reaction level to perform the state transitions

Routine Level

Reaction Level

Check DirectionGoal in Sight

Calculate Valence and Arousal

Update ModelPerform Planning

Discretize State

ε - Greedy SelectionUpdate Q-Learning

Do Action

Wall Detection

Get Reward

Fig. 3. Integration of reinforcement learning and Dyna Maze into the threelevel model of affect and cognition.

of the agent within the maze, detect collisions with walls andobstacles and receive the reward from the environment.

In the routine level, routinized actions are performed. Inhumans, the routine level is responsible for most motor skills,language generation, and other skilled and well-learned be-haviours [7]. It has access to working memory and morepermanent memory to guide decisions and update planningmechanisms. Therefore, it is the perfect level to performtwo important steps of reinforcement learning: updating theaction-value function (Update Q-learning) and selecting thefollowing action according to a routinized policy (e.g. ε-Greedy Action Selection). In the present maze scenario,the discretization of the state is not a complex step, butin alternative scenarios with continuous states and a non-deterministic world, the Discretize State step would be achallenging and computationally expensive step and thus hasto be performed in the routine level.

The highest level – the reflection level – calculates themost computationally intensive steps. Reflection is a meta-process involving all available data to build own internalrepresentations about the surrounding environments. It rea-sons about the environment and interprets the pre-processedsensory inputs of lower levels. The main tasks of thislevel are planning, problem solving, and reasoning aboutfacts. This suggests to implement the update and planningfunctions of reinforcement learning into the reflection level.Updating the model means to use the current and previousstate, the actual performed action, and the earned reward toupdate the previously saved model with this new information.Similarly, the planning step of reinforcement learning usesthe model (the long-term memory of the agent) and selectsrandom previously observed states and performs virtually arandom action at this state. The model outputs a rewardfor this random action which in turn is used to updatethe model itself. This kind of planning internally simulatesthe environment and produces simulated experience [5].

545

Besides these high-level functions of reinforcement learningthe reflection level accommodates the calculation of the coreaffect of the agent.

Calculation of Arousal and Valence

The calculation of the arousal and valence value requiresinformation of lower levels and, additionally, reasoning stepsand thus takes place in the reflection level. The reactionlevel monitors changes in direction, collisions with walls,and obstacles. This triggers short term memory processes inthe reflection level. Each memory process is modeled by afrequency variable. The frequency is high, if the event occursfrequently and decays to zero, if the event does not occuranymore.

If a collision occurs (c = 1) this is reported to thereflection level. The reflection level uses this information andcomputes a frequency of collisions given by

fcol(t) =

{fcol(t− 1) + (1− fcol(t− 1)) · ν if c = 1fcol(t− 1) + (−1− fcol(t− 1)) · ν otherwise,

(2)using a discrete dynamical system where ν describes theamount of residue that is added or subtracted from thefrequency value. The system responds to collision in anexponential way. The more collisions occur the faster (ex-ponential) the frequency value converges to one. If there areno collisions, the frequency value decays exponentially andconverges to minus one.

A similar approach was chosen to model the frequency ofchanges in direction fdir(t). If the agent changes its directionin every step, a value should go up indicating these frequentchanges. For this the reaction level reports the actual action tothe reflection level. A comparison with the previous actionresults in setting a variable d = 1 indicating the directionchange. A second frequency value is calculated by

fdir(t) =

{fdir(t− 1) + (1− fdir(t− 1)) · θ1 if d = 1fdir(t− 1) + (−1− fdir(t− 1)) · θ1 otherwise,

(3)

with θ1 determining the amount of fdir(t − 1) added to orsubtracted from the previous frequency value. The responseto changes in direction is also exponential and increases ordecreases at the beginning very fast and slowly convergesthen to its maximal value of minus or plus one. We ad-ditionally modified θ1 if the agent follows the policy andselects the optimal selection or if the agent performs randomexploration steps.

After these calculations, the agent checks if the goal stateis in sight. Therefore, we used the Bresenham line algorithm[20] to calculate the states between the actual state and thegoal state and compare the resulting states with the positionsof walls and obstacles. If there is no obstacle in the lineof sight, the agent is able to see the goal state (g = 1).This information is used to calculate a third frequency valuefgoal(t) which is high if the goal is regularly in sight anddecaying if the goal gets out of sight. For this calculation weuse the same type of equation as in Equation 2 and 3 withdifferent variables fgoal(t), θ2, and g.

Finally we use these three frequencies to calculate thearousal value A(t) of the agent as a weighted average givenby

A(t) =

{A(t) = fdir(t)+fcol(t)

2 if g = 0

A(t) =fdir(t)+fcol(t)+fgoal(t)

3 if g = 1.(4)

The valence value V (t) is calculated using fgoal(t) andthe information if the agent is moving away from the startingpoint. This information is derived by calculating the distancebetween starting point and actual position. If this distanceincreases (i = 1) the agent assumes to be on the right way,otherwise, it assumes something is going wrong (i = −1).The variable i is used in a discrete dynamical system tocalculate a frequency equivalent fright(t) (i is the limit thefrequency value converges to and κ the factor determiningthe slope) as

fright(t) = fright(t− 1) + (i− fright(t− 1)) · κ. (5)

Using these two frequencies, fgoal(t) and fright(t), thevalence component V (t) for the actual core affect state iscalculated as

V (t) =fgoal(t) + fright(t)

2. (6)

All these calculations are performed in the reflectionlevel at each time step during the reinforcement learningexperiment. The resulting arousal and valence values areused to affectively appraise the learning process.

V. SIMULATION AND RESULTS

In the following we show some results of this appraisaland explain them according to the elicited emotions.

In our experiments the agent is placed at the beginningof each episode in the start state S and has to find thegoal state G. Each episode consists of a maximum of 2000steps. Within this period the agent has to find the goal state,otherwise the episode is terminated. At the beginning of anexperiment the model and Q-table is reset to the initial valueof zero and the agent is placed at its starting position. Allother variables and temporary tables are set to their initialvalues (Table I). Those initial values are depended on theepisode length and the desired behavior of the agent. Theslope factors ν, θ1, and θ2 determine how fast the agentreacts on events and how long they have an effect. Aftereach episode, the agent’s position is reset to the start state,but the already learned model and Q-tables are reused inthe following episode(s). One experiment consists of severalepisodes, in which the agent advances its performance andreduces the steps needed to find the goal.

There are two variants of this experiment. In the first, theagent’s performance is only appraised with the described coreaffect. In the second variant of the experiment the actual coreaffect is used to bias the reward function of the reinforcementlearning framework. At the beginning of both experimentsthe reward function is set to rt = −1 in each state exceptthe goal state (at the goal the agent receives a reward ofrt = 10). In the second variant of the experiment, in which

546

TABLE IPARAMETERS USED IN THE EXPERIMENTS

Parameter Definition Value

n number of episodes 20maxsteps maximum number of steps per episode 2000p steps number of planning steps 50α step-size 0.01γ discount-rate 0.95ε probability for random action in ε-greedy policy 0.1ν factor determining the slope of fcol(t) 0.2

θ1, θ2 factor determining the slope 0.2of fdir(t) and fgoal(t)

κ factor determining the slope of fright(t) 0.15

the reward function is biased by the actual affective stateof the agent, the reward in each state (except the goal state)depends additionally on the core affect. In those states wherethe core affect is valenced positively (quadrant 1 and 2), theagent receives (or perceives) a reward rt = 0 for the actualstate. This should influence the learning behaviour in a waythat the agent actively searches for states with a positivelyvalenced core affect.

In Figure 4 we have plotted the average affective statesof the agent during 20 episodes of the first variant of thereinforcement learning experiment. It should be noted thatthe reinforcement learning experiment is non-deterministic,therefore we had to average the affective state of eachepisode over several experiments. For each point, we haveused the affective appraisal of 50 independent iterations ofthe experiment.

Valence

Aro

usa

l

Q1

Q2Q3

Q4

−0.4 −0.3 −0.2 −0.1 0 0.1 0.2 0.3−0.25

−0.2

−0.15

−0.1

−0.05

0

0.05

0.1

12

3

4

5 6

7

8

910 111213 14

151617

18

1920

Fig. 4. Average affective states during 20 episodes of the reinforcementlearning experiment.

Next to each point the episode number is plotted. Thebrighter the point the older the episode. There is a cleartemporal movement of the core affect form quadrant fourto quadrant two. During the first three episodes the agentgains knowledge about the maze. This requires a lot ofsteps and changes in direction resulting in a highly negativevalence value and a positive arousal value, corresponding

to an affective state of anxiety and fear. After about fourepisodes the reinforcement learning algorithm has gatheredenough model knowledge about the maze enabling the agentto find a short path to the goal state. This requires less stepsand directional changes and therefore the arousal is reducedand the overall feeling (valence) is positive. In the followingepisodes (7 to 20) the path is further optimized and the agentsselects frequently the optimal action. This further reduces thearousal and the valence value increases. The result shows thatthe agent is able to appraise the learning progress in termsof an affective state.

Figure 5 shows the results of the second variant of theexperiment which incorporates the affective state into thereinforcement learning process. Compared to the first variantof the experiment, the agent experiences a wider range and amore uniform temporal movement of affective states, whichis accompanied by a slower learning progress.

−0.3 −0.2 −0.1 0 0.1 0.2 0.3 0.4−0.3

−0.25

−0.2

−0.15

−0.1

−0.05

0

0.051

2

3 4

5

67

8 9

1011

1214

1516

17

18

1920

Q1

Q2Q3

Q4

Valence

Aro

usa

l

13

Fig. 5. Results of the second experiment which incorporates the core affectinto the learning process.

The slower learning progress of the second experimentis characterized by a slower decrease of needed steps perepisode as compared to the first experiment (Figure 6). Onaverage, the learning performance of both experiments issimilar.

2 4 6 8 10 12 14 16 18 200

200

400

600

800

Experiment 1Experiment 2

1Episodes

Ste

ps

per

epis

ode

Fig. 6. Average learning curves for both experiments. Experiment 1 usesunmodified Dyna-Q-Learning and Experiment 2 incorporates the calculatedaffective state of the agent.

This slower learning progress is the result of the in-corporation of the affective state of the agent. The agentsimultaneously tries in each episode to improve the policyand the perceived valence of the situation. As a result,

547

the agent learns the optimal policy more slowly, but isable to perceive (and display) its current learning progressthrough its core affect. Additionally, the more diverse anduniform states enable a more believable communication ofthe affective states towards human observers [21].

In the first experiment the agent is only able to displaysudden changes of its core affect. Whereas in the secondexperiment, the agent can communicate its learning progresswith a more uniform transition of its affective state. A humancould perceive the transition of the agent’s core affect asfollows: First, the agent displays fear and anxiety due to thenew situation it is encountered. After these first episodes, theagent is depressed because of its bad performance. However,the improvement of the policy results in core affects of calmand satisfaction.

Overall, the results indicate that our approach for integrat-ing affective states into reinforcement learning can be usedin autonomous affective systems. The generated core affectscan be used as an alternative way for communicating theactual system state towards a human observer.

VI. CONCLUSIONS AND FUTURE WORK

In this paper, we present an architecture for integrating adimensional emotional model into a virtual agent. We believethat agents or robots using affective models to guide theirdecisions or express their current state are highly beneficialin holding a better social interaction with humans. To provethis belief, we described a reinforcement learning frameworksimulating a virtual agent making decisions to find theshortest path to the goal in a maze. Further we showed howthe reinforcement learning framework could be integratedinto a model of affect and cognition. Finally, we extendedthis framework with an dimensional emotional model tocalculate the core affect of the agent during the learningprocess. The affective appraisals of the learning progressshow that the emotional model is successfully integrated intothe machine learning algorithm and is able to evaluate thecurrent state of the agent in terms of an affective state. Asecond experiment, which uses these affective states to guideor bias the decision behaviour of the agent was conducted.The results show that the core affect can be used to bias thereward function to guide the decisions of an agent.

However, an extensive study is necessary for a statisticalprove of the system and an evaluation of its performancein different scenarios. Furthermore, in a future study we aregoing to use the affective state for displaying the learningprogress in a more interactive way to the user and implementthe whole system to a small two-wheeled robot performingthe same path-finding task in a real environment.

In future implementations we will integrate free-floatingmoods into the system. Moods are long-term influences ontothe core affective state of the system. This means that theaffective state of the agent is additionally influenced throughthe personality of the system and its long term reactions onevents.

ACKNOWLEDGMENT

This work has been supported by the German ResearchFoundation (DFG) funded Cluster of Excellence ”CoTeSys”- Cognition for Technical Systems.

REFERENCES

[1] H.-R. Pfister and G. Bohm, “The multiplicity of emotions: A frame-work of emotional functions in decision making,” Judgment andDecision Making, vol. 3, no. 1, pp. 5–17, 2008.

[2] G. Loewenstein and J. Lerner, “The role of affect in decision making,”in Handbook of Affective Science, R. Davidson, H. Goldsmith, andK. Scherer, Eds. Oxford, NY, USA: Oxford University Press, 2003,pp. 619–642.

[3] E. Peters, “The functions of affect in the construction of preferences,”in The construction of preference, S. Lichtenstein and P. Slovic, Eds.New York, NY, US: Cambridge University Press, 2006, pp. 454–463.

[4] M. B. Moussa and N. Magnenat-Thalmann, “Toward socially responsi-ble agents: integrating attachment and learning in emotional decision-making,” Computer Animation and Virtual Worlds, vol. 24, no. 3–4,pp. 327–334, 2013.

[5] R. Sutton and A. Barto, Reinforcement Learning: An Introduction,1st ed. Cambridge, MA, USA: MIT Press, 1998.

[6] C. J. Watkins, “Learning from delayed rewards,” PhD thesis, Cam-bridge University, Cambridge, England, 1989.

[7] D. A. Norman, A. Ortony, and D. M. Russell, “Affect and machinedesign: Lessons for the development of autonomous machines,” IBMSystems Journal, vol. 42, no. 1, pp. 38–44, 2003.

[8] P. Ekman, “Are there basic emotions?” Psychological Review, vol. 99,no. 3, pp. 550–553, 1992.

[9] J. Panksepp, Affective neuroscience: The foundations of human andanimal emotions. New York, NY, USA: Oxford University Press,1998.

[10] A. Mehrabian and J. A. Russell, An approach to environmentalpsychology. Cambridge, MA, USA: The MIT Press, 1974.

[11] J. A. Russell, M. Lewicka, and T. Niit, “A cross-cultural study ofa circumplex model of affect,” Journal of Personality and SocialPsychology, vol. 57, no. 5, pp. 848–856, 1989.

[12] D. Watson, D. Wiese, J. Vaidya, and A. Tellegen, “The two generalactivation systems of affect: Structural findings, evolutionary consid-erations, and psychobiological evidence.” Journal of Personality andSocial Psychology, vol. 76, no. 5, pp. 820–838, 1999.

[13] C. E. Izard, “Basic emotions, natural kinds, emotion schemas, and anew paradigm,” Perspectives on Psychological Science, vol. 2, no. 3,pp. 260–280, 2007.

[14] J. Panksepp, “Neurologizing the psychology of affects: How appraisal-based constructivism and basic emotion theory can coexist,” Perspec-tives on Psychological Science, vol. 2, no. 3, pp. 281–296, 2007.

[15] A. Mehrabian, “Pleasure-arousal-dominance: A general frameworkfor describing and measuring individual differences in temperament,”Current Psychology, vol. 14, no. 4, pp. 261–292, 1996.

[16] J. A. Russell and L. F. Barrett, “Core affect, prototypical emotionalepisodes, and other things called emotion: dissecting the elephant.”Journal of Personality and Social Psychology, vol. 76, no. 5, pp. 805–819, 1999.

[17] P. Gebhard, “Alma: a layered model of affect,” in Proceedings ofthe Fourth International Joint Conference on Autonomous Agents andMultiagent Systems. ACM, 2005, pp. 29–36.

[18] M. Mendl, O. H. P. Burman, and E. S. Paul, “An integrative andfunctional framework for the study of animal emotion and mood,”Proceedings of the Royal Society B: Biological Sciences, vol. 277, no.1696, pp. 2895–2904, 2010.

[19] K. R. Scherer, “Emotion and emotional competence: conceptual andtheoretical issues for modelling agents,” in Blueprint for AffectiveComputing: A Sourcebook, K. R. Scherer, T. Banziger, and R. Etienne,Eds. New York, USA: Oxford University Press, 2010, pp. 3–20.

[20] J. E. Bresenham, “Algorithm for computer control of a digital plotter,”IBM Systems Journal, vol. 4, no. 1, pp. 25–30, 1965.

[21] F. D. Schonbrodt and J. B. Asendorpf, “The challenge of constructingpsychologically believable agents,” Journal of Media Psychology:Theories, Methods, and Applications, vol. 23, no. 2, pp. 100–107,2011.

548