the role of moral utility in decision making: an

Copyright 2008 Psychonomic Society, Inc. 390

Imagine that you are the democratically elected presi-dent of a country and have just been informed that a pas-senger airplane has been hijacked. The hijackers appear to be headed for a city with well-populated skyscrapers. Your air force is in place and ready to shoot the passenger pplane down at your command. You can take many differ-ent courses of action: shoot or not, negotiate or not. Howshould you decide? Would it make a difference whether 10 or 500 passengers are on board the airplane, or whether the passengers have voted for you? Would it make a dif-ference if the air force is not in place and the only way to stop the hijackers in time is to ask the pilot of a nearby sec-ond passenger airplane to crash into the first? Would you try to calculate how likely it is that the hijackers actuallywill crash the airplane into a skyscraper and how many innocent people would die if the hijackers were to do so?Assuming that the probability of the hijackers’ headingfor a skyscraper appears high, should you have the planecrashed by the other plane with its x additional passengersbeing killed with certainty, in order to probabilisticallybeing killed with certainty, in order to probabilistically

save y people on the ground ( y x)? Or do not only hi-jackers, but also presidents, have an unconditional duty not to kill innocent people?

The questions above concern moral decision making, particularly what normative claims should guide us whenwe make moral decisions and how we actually make such moral decisions. In the present review, we attempt to pro-vide a road map for the empirical investigation of moraldecision making. Our starting point will be the important philosophical distinction between is and ought. This dis-

tinction and the need for normative claims sometimes getsforgotten by politicians and members of the public when they make statements about what should be done, given

d the facts. Then we will briefly introduce utilitarian anddeontological moral theories and will review psychologi-cal and neuroscientific findings regarding moral decision making. This research will suggest that people may usemore than one moral theory, as well as moral heuristics, and that the prefrontal cortex may be particularly impor-tant for moral decision making. Finally, we will propose

INTERSECTIONS AMONG PG HILOSOPHY,PSYCHOLOGY, AND NEUROSCIENCE

The role of moral utility in decision making: An interdisciplinary framework

PHILIPPE N. TOBLERUniversity of Cambridge, Cambridge, England

ANNEMARIRR E KAKK LISUtrecht University, Utrecht, The Netherlands

AND

TOBIAS KAKK LENSCHERUniversity of Amsterdam, Amsterdam, The Netherlands

What decisions should we make? Moral values, rules, and virtues provide standards for morally acceptabledecisions, without prescribing how we should reach them. However, moral theories do assume that we are, at least in principle, capable of making the right decisions. Consequently, an empirical investigation of the methods and resources we use for making moral decisions becomes relevant. We consider theoretical parallels of economic decision theory and moral utilitarianism and suggest that moral decision making may tap into mechanisms and pro-cesses that have originally evolved for nonmoral decision making. For example, the computation of reward value occurs through the combination of probability and magnitude; similar computation might also be used for deter-mining utilitarian moral value. Both nonmoral and moral decisions may resort to intuitions and heuristics. Learning mechanisms implicated in the assignment of reward value to stimuli, actions, and outcomes may also enable us to determine moral value and assign it to stimuli, actions, and outcomes. In conclusion, we suggest that moral capa-bilities can employ and benefit from a variety of nonmoral decision-making and learning mechanisms.bilities can employ and benefit from a variety of nonmoral decision-making and learning mechanisms.

Cognitive, Affective, & Behavioral Neuroscience2008, 8 (4), 390-401doi:10.3758/CABN.8.4.390

P. N. Tobler, [email protected]

MORALORAL DECISIONSECISIONS 391391

ment (Brandt, 1979), and any other valuable consequencesof actions and rules. For utilitarian theories, something ismorally valuable because it promotes nonmoral valuessuch as happiness, pleasure, or other kinds of utility.

Deontological theories argue that certain things aremorally valuable in themselves. Such moral value does notderive from the fact that it promotes happiness, pleasure, or any other utility (Rawls, 1971). The moral quality of actions arises from the fact that the action is done to pro-tect such moral values. The consequences and outcomes of actions are secondary. Impermissible actions remain soirrespective of how good their consequences are. Some deontological theories claim that all moral values can be traced to one ultimate moral principle. One example of such an ultimate principle is autonomy, conceived as the ability of a person to generate self-binding laws (Kant, 1797). Another example is the ability to act (Gewirth, 1998). Other positions claim that there are several moralvalues, which have equal standing. Ross (1930), for ex-ample, proposes seven basic values, which correspond toseven moral duties: fidelity, reparation, justice, benefi-cence, gratitude, self-improvement, and noninjury. Kant’s (1797) work on morals is the paradigmatic example of a deontological theory. The core rule in Kant’s system is thecategorical imperative, which requires that we act accord-ing to maxims that could become universal laws. Rulesoften arise from the rights of those affected by the agent’sactions (Kamm, 2007). In Kantian terminology, this is ex-pressed by the claim that one ought to treat others as endsin themselves, and never only as means to an end.

Virtue ethics focuses on the motives and character traits of actors. The bearers of moral quality are not ac-tions but life, people, and their capabilities (Nussbaum, 2000; Sen, 1985). Virtues are goods in themselves, and their continued practice contributes to the good life. Ex-amples of virtuous dispositions are wisdom, courage,moderation, justice, kindness, generosity, self-respect, compassion, altruism, forgiveness, and sincerity. Virtues may fall in between more extreme character dispositions.Having too little or too much of the disposition results in vices (e.g., cowardice or rashness for courage; Aristotle, Nicomachean Ethics, 2002 ed.). Virtues should be care-fully cultivated through moral education by role models. The development of habits helps to make the expressionof virtuous dispositions more stable and reliable, but vir-tuous behavior should nevertheless remain goal directed and purposeful. Applying and reconciling virtues requires practical wisdom, which helps to discern what is importantfor a good life. Virtues are associated with an appropriate feeling of the right emotions. Aristotle and, more recently,Anscombe (1958), MacIntyre (1985), and Foot (2001) are some of the prime proponents of virtue ethics.

Moral Decisions in Theory and PracticeAn implication of the is–ought gap is that whereas moral

theories provide standards for how we should act, they do not describe how moral judgments and decisions are achieved in practice. Accordingly, moral theories cannot be falsified by empirical findings that we often do not behave in ways that would be prescribed by the theories. In fact, even if

that moral decisions may rely on mental processes and neuronal circuitry that originally evolved to serve non-moral functions, such as reward and punishment process-ing. We will do so by discussing parallels between moral and microeconomic utility theories and by suggesting thatclassical learning theory, with its principle of error-drivenadjustment of reward predictions, can give clues as tohow people acquire moral behavior. Neuronally, we will propose that reward regions, such as the striatum and theprefrontal cortex, may contribute to the acquisition and expression of moral behavior. Our proposal implies thatthe reward system may have beneficial functions not onlyfor individual survival by computing reward value, butalso for society and culture by contributing to the compu-tation of the moral value of decisions and actions.

The Is–Ought GapWhen we ponder the morally right course of action, we

always face a problem. Philosophers since Hume (1739) have argued that the factual characteristics of the situationwill themselves never tell us how we ought to act. Knowingwho the hijackers are, what they are going to do, and whatthe consequences of their actions will be is not enough to de-duce what the president ought to do. To use Hume’s words,“you cannot derive an ought from an is” (Hume, 1739; for a related argument, see Moore, 1903; for a comparison, seeBruening, 1971; for a dissenting view, see Searle, 1964). In order to find out what the president ought to do, we need not only information about the facts of the current situation,but also some normative claims about what is valuable. Wewould commit a logical fallacy if we were to deduce moral conclusions just from descriptive facts.

In the airplane example, one possible normative claimmight be that we are morally obliged to provide happinessfor the largest possible number of people. This would be a utilitarian normative claim (e.g., Mill, 1861). Another possibility would be to presuppose the moral rule thatone ought not to kill. This would be a deontological nor-mative claim (e.g., Kant, 1797). Descriptive information about the situation must be combined with such norma-tive claims in order to reach a conclusion about what oneought to do. For example, if one embraces the normativeclaim that we are morally obliged to save as many peopleas possible, we could determine what to do by combiningthis claim with factual information about how different possible actions would affect the number of people saved.Thus, normative claims and descriptive information takentogether allow us to make moral decisions.

Moral TheoriesWe derive normative claims from moral theories. The

main classes of moral theories are the utilitarian (conse-quentialist), deontological, and virtue theories. Utilitarian-ism states that the moral quality of actions is determined by their consequences (utilitas Latin for [public] good,usefulness). This means that we have the duty to maximize the sum or mean utility of all individuals in the society. Traditionally, utility is interpreted hedonistically as bal-ance of pleasure and pain, but it could also include beauty and truth (Moore, 1903), (informed) preference fulfill-

392392 TOBLEROBLER, K, KALISALISKKK ,, ANAND KALENSCHERALENSCHERKK

Empirical Findings for Moral Judgment and Decision Making

Research has focused on moral dilemmas that oppose deontological and utilitarian theories. In the dilemma of the runaway trolley (Thomson, 1985), a trolley will kill five workers unless diverted to a sidetrack where it will kill one worker. The majority of people choose to switch the trolley to the sidetrack. The outcome of this choice isin agreement with the utilitarian argument that one oughtto maximize overall utility. In an alternative version of thedilemma (the footbridge dilemma), the runaway trolley cannot be switched off the main track. The trolley can be prevented from reaching the five workers only by push-ing a fat man from a bridge onto the track, thus killing him. The majority of people refuse to push and kill the fat man in order to save the five workers. One interpreta-tion of these findings (Greene, Nystrom, Engell, Darley, & Cohen, 2004) suggests that the deontological consid-eration not to use a person purely as a means to an end (Kant, 1797) overrides considerations of utility maximi-zation. This would mean that in real life, people are, at least at first glance, capable of approaching moral issuesfrom both utilitarian and deontological points of view.Moreover, they do not commit themselves to one moraltheory but switch between different theories as the cir-cumstances change. People may thus eclectically combineconsiderations from different moral theories. However, this point is somewhat speculative, since we do not knowhow participants have actually come to their decision. The mere fact that the outcome of the decision maximizesutility does not make it a utilitarian decision; instead, thedecision might be made on deontological grounds or onno grounds at all.

An alternative interpretation of the difference between the two dilemmas suggests that the emotional involvement is higher for the footbridge than for the trolley problem because the act of pushing a person constitutes a more personal and harmful action than does changing a switch (Greene et al., 2004; Greene, Sommerville, Nystrom, Dar-ley, & Cohen, 2001; but see Nichols & Mallon, 2006). Given that at least rough dissociations can be made withrespect to the functions of different brain regions (e.g., Fuster, 1997), the emotion-based interpretation of the footbridge problem may be endorsed by results fromfunctional neuroimaging and lesion studies. For example,within the prefrontal cortex, ventromedial regions aremore strongly involved in the processing of emotions, whereas dorsolateral regions fulfill more cognitive func-tions (Duncan & Owen, 2000; Fuster, 1997). An emotion-based account of the footbridge problem would therefore predict activation of ventromedial prefrontal regions while participants solve footbridge-like problems. This is indeed what was found (Greene et al., 2001).

Conversely, making decisions in trolley-like problemsmay draw on more cognitive capacities, in that it involves weighting up costs (e.g., of one man dying) and benefits(e.g., of five men being saved). One may therefore expect a stronger involvement of the dorsolateral prefrontal cor-tex during trolley-like than during footbridge-like prob-lems, and this is also what was found (Greene et al., 2001).

all of our actions and dispositions were to contradict whatmoral theories require from us, what is morally prescribed would still remain the same. Thus, in further illustration of the is–ought gap, moral theories are about how we ought to behave, not about how we do behave.

Similarly, to answer the question of what we should do does not automatically answer the question of how moraldecisions should be achieved (e.g., Bales, 1971). It is, in fact, open to discussion whether moral theories do or do not suggest specific methods for moral decision making. For example, from a Kantian perspective, it is unclear how weshould determine whether our maxims could become uni-versal laws. Some deontological theories suggest that weshould follow the ( juridical) law in order to behave morally(Wikström, 2007). Virtue theories explicitly require delib-erate and purposeful application of the positive disposi-tions. Utilitarian theories would seem to suggest that weshould make moral decisions on the basis of a process of calculating utilities. But in fact, several utilitarians actually recommend that we do not calculate moral utility every time before we make a decision (e.g., Mill, 1861; Sidg-wick, 1874)—for example, because the required calcula-tions are too difficult and, therefore, error prone or take toomuch time. Conversely, following our intuitions will oftenlead to utility-maximizing decisions (Hare, 1981).

The Relevance of Decision-Making Research for Moral Theories

As we have seen above, moral theories are prescriptive,not descriptive. Nevertheless, moral theories cannot pre-scribe things we cannot possibly live up to. This is oftenexpressed as the ought-implies-can principle, which isascribed to Kant (1788). The idea is that the concept of having a duty already logically entails that the addressee is able to fulfill it. To say that you ought to do A already presupposes that you are able to do A. The notion of canis often interpreted to refer both to ability and to opportu-nity (Copp, 2008). For example, I can have a duty to savea drowning child only if I am actually capable of savingthe child (if I can’t swim and would drown myself beforereaching the child, I do not have a duty to save it) and if Ihave the opportunity to do so (if I am in China at the mo-ment, I do not have a duty to save a child drowning in apond in Manchester). The ought-implies-can principle isalso used to defend the claim that there would be no roomfor moral responsibility if the universe were deterministic(Copp, 1997; Widerker, 1991).

This ought-implies-can principle provides us with a con-nection with empirical research on decision making. Moraltheories presuppose that we can have a duty to perform actsonly insofar as we are able to perform them, and to performthe morally right action we must first reach the morally right decision. And that is where the mechanisms of moraldecision making enter the picture. Insofar as moral theories suggest certain preferred methods of moral decision mak-ing (and we have seen that they sometimes appear to do that—e.g., by advocating intuitive decision making), theymust presuppose that these methods enable us, at least insome cases, to make the right decision. Empirical researchmight tell us whether that is a valid presupposition.


cognition and decision making, computed in a dedicated class of neurons and structures (for such a view, see Har-man, 2000; Hauser, 2007; Mikhail, 2007; for a critique,see Dupoux & Jacob, 2007)? Due to space restrictions,we will focus in the following on utilitarian moral theory. On the basis of (1) parallels of utilitarian moral theory and economic decision theories, (2) findings of neuronal cor-relates of terms used by economic decision theories, and (3) potential relations with learning theory, we will pro-pose that at least some moral cognition may not be special but, rather, may make use of evolutionary old and proven reward decision mechanisms.

Utilitarians argue that in order to make the morally right decision, we need to determine which action or rule maxi-mizes utility. In order to determine whether a particular action or rule maximizes overall utility, we need to con-sciously calculate or intuitively estimate utility and, thus, predict and weigh the consequences of an action. We also have to predict how the utility of the people affected will change, which requires that we recognize others as hav-ing different preferences and taste. By empathizing and identifying with others, we can determine what it would feel like to be in their situation. We can extrapolate howone specific action or rule implementation, rather than another, will maximize overall utility by stepping intomany different shoes and comparing the utilitarian con-sequences of the action or rule (Harsanyi, 1955). Thus,moral utilitarianism implicates several cognitive and emo-tional processes and, consequently, provides leads for in-vestigating moral decision making.

In the following section, we will explore the parallelsbetween moral and nonmoral utility-based decisions. Aswas mentioned in the introduction, there is some disagree-ment between utilitarians who argue that the right moral decision requires conscious calculation of utility and thosewho argue that other (more intuitive) methods are just asgood or better. This discussion bears remarkable similari-ties to issues discussed in economic theory. It is possible that we can learn something about moral decision making by looking at parallels with other theories and at findingsthat support the theories. By extension, we could then in-vestigate whether moral decision making taps into theseprocesses, which have not originally evolved to subserve moral reasoning as such.

Parallels of Moral UtilitarianismWith Economic Decision Theory

Utility has a long-standing tradition in economics. Ber-noulli (1738) argued that each additional unit of a good, such as money, becomes subjectively less valuable (di-minishing marginal utility). An additional $1,000 yieldsmore marginal utility in our poor student times than whenwe have reached some financial security and wealth. In the moral domain, diminishing marginal utility provides an argument for wealth redistribution and supporting the poor, because it maximizes overall utility (Mill, 1861). Assuming similar utility functions for the poor and rich, the utility loss incurred by the rich from giving to the poor is much smaller than the utility gain of the poor. Moderneconomic decision theory stresses the notion that utility

The dorsolateral prefrontal cortex is also more active withutilitarian judgments in footbridge-like problems when participants accept, rather than reject, personal norm vio-lations if these lead to a substantial increase in happiness(Greene et al., 2004). In summary, these imaging resultssuggest a dissociation of the ventromedial and dorsolateralprefrontal cortex during the solving of moral problems, with ventromedial regions processing the emotionally sa-lient aspects of footbridge-like problems and dorsolateralregions analyzing costs and benefits.

The functional dissociation of ventromedial and dorso-lateral prefrontal activations concurs well with the find-ing that lesions of the ventromedial prefrontal cortex re-sult in a higher proportion of utilitarian moral judgmentsin footbridge-like problems, but not in less emotional, trolley-like problems (Koenigs et al., 2007; see also Moll,Zahn, de Oliveira-Souza, Krueger, & Grafman, 2005).Without emotional input from the ventromedial prefrontalcortex, the utilitarian reasoning of the dorsolateral pre-frontal cortex may dominate behavior in footbridge-like problems. Taken together, these results may suggest thathealthy participants weigh up ventromedially processed emotional factors with dorsolaterally processed utility-related factors to determine the morality of a judgment.

The interpretation of the ventromedial lesion results issomewhat complicated by studies of the ultimatum game.In the ultimatum game, a proposer offers a way in whichto split a monetary amount, and a responder either acceptsor rejects the offer. In the case of acceptance, the amount gets split according to the offer; in the case of rejection, neither player receives anything. As with footbridge-likeproblems, unfair ultimatum offers (people typically re-ject offers of about 20% or less of the monetary amount) may elicit a combination of emotional and utility-related processes. Fairness motives, inequality aversion, disgust,and anger may motivate rejection of unfair offers, whereasutility-related factors, such as self-interest and the mini-mization of personal cost, may motivate their acceptance.Ventromedial patients are more likely to reject unfair of-fers (Koenigs & Tranel, 2007; for reviews, see Greene,2007, and Moll & de Oliveira-Souza, 2007; see also Knoch, Pascual-Leone, Meyer, Treyer, & Fehr, 2006; San-fey, Rilling, Aronson, Nystrom, & Cohen, 2003). Thus,with unfair offers in the ultimatum game, ventromedialpatients appear to weigh emotional and fairness factors more strongly than utilitarian self-interest, whereas theopposite appears to be the case with footbridge-like prob-lems. Differences in the types of emotion (empathy vs. anger) and utility (welfare of others vs. monetary self-interest) involved in footbridge-like problems and the ul-timatum game may explain the apparent discrepancies inthe behavior of ventromedial patients.

What Cognitive Processes a Utilitarian Moral Theory May Implicate

From a philosophical perspective, the moral domain isspecial in that it is concerned with what we ought to do,rather than with what we actually do. Is the moral domain also special from a psychological and neuroscientific per-spective, in that there are dedicated mechanisms for moral


a position to compute adaptive behavioral signals with integrated reward value (Lau & Glimcher, 2008; Padoa-Schioppa & Assad, 2006; Sakagami & Watanabe, 2007; Samejima, Ueda, Doya, & Kimura, 2005). In humans, ac-tivations scaling with reward magnitude and probability are prominent in the striatum and prefrontal cortex (Abler, Walter, Erk, Kammerer, & Spitzer, 2006; d’Acremont & Bossaerts, 2008; Knutson, Taylor, Kaufman, Peterson, &Glover, 2005; Tobler, O’Doherty, Dolan, & Schultz, 2007; Yacubian et al., 2006). Moreover, risk-related activations in the prefrontal cortex vary with risk attitude, with morelateral prefrontal regions showing stronger risk signals with increasing risk aversion and more medial regions showing them with increasing risk proneness (Tobler et al., 2007). As was explained above, risk attitude relates to the shape of the utility function, with concave functions reflecting risk aversion and convex ones risk proneness. Taken together with the finding that lateral prefrontal re-gions may weigh up costs and benefits in moral problems and medial prefrontal regions may process the emotionalaspects of moral problems (Greene et al., 2004; Knochet al., 2006; Koenigs et al., 2007), these results suggestthat a variety of reward regions contribute to the calcula-tion of an economic utility signal and that some of theseregions overlap with those implicated in the processing of moral utility.

The neuronal data on footbridge- and trolley-like prob-lems suggest involvement of at least two interacting sys-tems, one deliberative and one emotional (Greene et al.,2004; Greene et al., 2001). This suggestion is reflected in the neuroeconomic domain, with research proposingone set of systems involved in high-level deliberative pro-cesses, such as problem solving, planning, and trading off costs with benefits, and another set of systems engaged bythe emotional aspects of the decision, such as impatience, distress of deferring immediate gratification, and fear of loss. Multiple-system proponents in neuroeconomics con-cur with their moral neuroscience colleagues in locating the more deliberative aspects of decision making within the dorsolateral prefrontal (and parietal) cortex. Con-versely, the more affective aspects of decision making are thought to be located within the ventromedial prefrontal cortex (as well as within the insula and subcortical struc-tures, such as the amygdala and striatum; Berns, Laib-son, & Loewenstein, 2007; McClure, Ericson, Laibson, Loewenstein, & Cohen, 2007; McClure, Laibson, Loe-wenstein, & Cohen, 2004; Miller & Cohen, 2001; San-fey, Loewenstein, McClure, & Cohen, 2006). However, alternative views suggest that the distinction into different neuronal systems and mental processes may be more a matter of gradation than of category. In this view, the brain acts as a single information-processing system (Glimcher, Kable, & Louie, 2007; Kable & Glimcher, 2007; Kalen-scher & Pennartz, 2008; Tom, Fox, Trepel, & Poldrack, 2007). The single- versus multiple-system question is thefocus of ongoing research (Rustichini, 2008). In the moraldomain, avenues for further research may be suggested, in that single-system views presently appear to be moreprominent than multiple-system views in theoretical work,whereas the opposite may hold in empirical work.

is content free; thus, the connotation of utility as pleasureand happiness no longer holds. People reveal their prefer-ences in overt choice behavior between two options, and utility theory captures these revealed preferences with a utility function. Preferences differ between individuals,and there are no normative prescriptions of what peopleought to prefer.

As long as preferences fulfill some axioms, such asbeing well ordered, transitive, complete, and indepen-dent, a utility function can be defined (Marschak, 1950; Von Neumann & Morgenstern, 1944). We can then obtainthe expected utility of a choice option by multiplying theutility (u) of all possible outcome magnitudes (m) of theoption with their probabilities ( p) and integrating acrossproducts: EUT [ p * u(m)]. In order to maximize ex-pected utility, we should choose the option with the largestsum. Diminishing marginal utility results in concave (Ber-noulli proposed logarithmic; alternatives are other power or quadratic and exponential) utility functions. Concaveutility functions imply risk aversion (Bernoulli, 1738). Arisk-averse decision maker prefers $500 for sure over the toss of a fair coin for $1,000 or $0, even though the twooptions have the same expected value. A concave utilityfunction implies that the first $500 yields more marginal utility than does the second $500. Therefore, the sure op-tion will be preferred over the average utility of $1,000and $0. The more concave the utility function, the more risk averse the agent. Conversely, the more convex theutility function, the more risk seeking the agent, and themore he is willing to accept a risky option with probabi-listic high-magnitude outcomes.

The neuronal systems processing economic value partlyoverlap with those processing moral value. Although a neuronal correlate of a formal economic expected util-ity signal has not yet been identified, correlates of com-ponents such as magnitude and probability are known tobe processed by the reward system (for more details onthe reward system and its functions, see, e.g., Berridge & Robinson, 2003; Rushworth & Behrens, 2008; Schultz,2006). Dopamine neurons in the midbrain show phasic activations with short latencies of about 100 msec to un-predicted rewards and to reward-predicting stimuli. Theseactivations scale with the magnitude and probability of predicted reward and combine the two parameters (To-bler, Fiorillo, & Schultz, 2005). It currently remains to be tested whether dopamine neurons code expected value(sum of probability-weighted reward magnitudes), ex-pected utility (sum of probability-weighted utilities of re-ward magnitudes), prospect (wealth changes, weighted bya distorted probability function), or another combinationof magnitude and probability.

Dopamine neurons send widespread projections tomany subcortical and cortical regions, particularly thestriatum and prefrontal cortex (Gaspar, Stepniewska, &Kaas, 1992). Both striatal and prefrontal neurons codereward magnitude and probability (Cromwell & Schultz,2003; Kobayashi, Lauwereyns, Koizumi, Sakagami, &Hikosaka, 2002; Wallis & Miller, 2003). Furthermore,neurons in these regions combine reward information withmore detailed sensory and motor information and are in


2004; for a review, see Frith, 2007). We suggest that a de-rived, utilitarian moral value signal could be computed by reward-processing regions in a manner similar to that for a standard reward value signal. It would be interesting totest where and how such a signal is combined with other-regarding processes. One prediction could be that the basic components of the utility signal will be combined in the prefrontal cortex and striatum similarly for economicand moral utility but that moral utility, in addition, maydraw on other-regarding structures, such as the insula and cingulate cortex.

Parallels of Moral UtilitarianismWith Behavioral Economics

Just like ethical theories, expected utility theory has anormative flavor entailed in the axioms and computations that it requires for optimal behavior. As a consequence, one may want to consider what decision makers can possiblydo and what they actually do when discussing normative models in both ethics and economics. Whereas moral theo-ries define optimal behavior as morally good behavior, for expected utility theory it is defined in terms of rational-ity. Expected utility theory thus prescribes what behavior would be most rational. However, empirically, our prefer-ences sometimes violate the axioms posited by expected utility theory (Allais, 1953; Ellsberg, 1961; Kahneman & Tversky, 1979; Loomes, Starmer, & Sugden, 1991). Wemay prefer option y over y , but after adding z to both of zthese options, we may prefer y z over z y z. Or we may prefer y over y when the options are described as gains but may prefer y over y when they are described as losses. Recall the example of airplane hijackers from the begin-ning. Imagine that they are expected to kill 600 people. We have to choose between two options to combat them.When framed in terms of gains, either 200 people will besaved for certain (safe, risk-free option), or there is a 1/3chance that 600 people will be saved and a 2/3 chance thatno one will be saved (risk-seeking option). When framed in terms of losses, 400 people will die for certain (risk-free option), or there is a 1/3 chance that no one will die and a2/3 chance that 600 people will die (risk-seeking option).As a variety of different studies have shown (reviewed inKuhberger, 1998), participants tend to choose the safe op-tion (200 saved for certain) when the problem is framed interms of gains. Conversely, they choose the risk-seekingoption (1/3 chance that no one will die and a 2/3 chance that 600 will die) when the problem is framed in terms of losses. However, such behavior can not be accommodated by expected utility theory, because the two versions of the problem differ only in how the outcomes are described; the actual numbers of dead or alive people are the same.

Functional neuroimaging suggests that amygdala ac-tivity is susceptible to economic versions of the framing effect, whereas stronger activation of the medial and or-bital prefrontal cortex correlates with stronger resistanceto framing (De Martino, Kumaran, Seymour, & Dolan, 2006). On the basis of these findings, an obvious predic-tion would be that patients with lesions in medial and or-bital prefrontal regions would be more susceptible, and amygdala patients less susceptible, to framing in both the

Expected utility theory can be extended into the socialand moral domain with similar formalisms (Harsanyi, 1955). Economic utility functions represent our egoistic preferences with respect to risky economic outcomes; so-cial or moral utility functions represent impersonal pref-erences over income distributions or social states (Har-sanyi, 1953). Thus, social preferences are preferences thatwe hold irrespective of our own situation and interests.An example would be our view of what a just tax systemshould look like, irrespective of how much we actuallyearn. The social utility functions can vary between indi-viduals, just as with personal utility functions. Within anindividual, the social and the personal utility functions can be in opposition—for example, with respect to an income distribution that favors the individual. Such a distributionmay have low social but high personal utility. To maxi-mize social utility, we should use the same principles and formalism as those proposed by expected utility theorydescribed above (Harsanyi, 1955). By adding up the in-dividual social utilities and maximizing the probability-weighted sum, we determine the income distribution or social state that maximizes the expected social utility of all members of the society.

Personal and social outcomes of decisions could be combined in single utility functions. As a proposer in theultimatum game, we might offer a relatively high amount to a responder partly because we prefer the responder to getan amount similar to our own. More specifically, a person (i) with inequality aversion might discount their personalgains (xi) by both self-interested disadvantageous and other-regarding advantageous inequality. In the two-player case with just one other player ( j), the utility of player icorresponds to UiUU (x(( ) xi (x(( jx xi) i(x(( i xjx ) (Fehr & Schmidt, 1999). Disadvantageous inequality results in the utility loss (x(( jx xi), advantageous inequality in theutility loss i(x(( i xjx ). The two weighting parameters,and could be interpreted as individual propensity for envy (that you get more than I) and compassion (I feel withyou because I get more than you), respectively. In mostpeople, is larger than . Assuming a utility function thatincorporates other-regarding inequality aversion can alsoexplain why we may prefer to cooperate, rather than defect,in the prisoner’s dilemma and punish defecting players at a cost to ourselves (Fehr & Camerer, 2007).

We know that reward-processing regions encode notonly the value of primary outcomes, such as food and drink, but also that of higher order rewards, such as money, abstract points, cooperation, and fairness (e.g., Delgado,Frank, & Phelps, 2005; Delgado, Labouliere, & Phelps,2006; Huettel, Stowe, Gordon, Warner, & Platt, 2006;Knutson et al., 2005; Rilling et al., 2002; Singer, Kiebel, Winston, Dolan, & Frith, 2004; Tabibnia & Lieberman, 2007; Yacubian et al., 2006). For example, activation in the ventral striatum increases with the ratio of what oneparticipant receives, as compared with another (Fliessbachet al., 2007). By extension, this social reward comparisonoutput from the ventral striatum could form the basis for a formal inequality aversion signal. Other-regarding mo-tives such as empathy are processed by the anterior insulaand the anterior cingulate cortex (Singer, Seymour, et al.,


organized criteria and suffice—that is, stop the decision process—once a criterion has been met. For example, wemay search only for the option yielding the fastest out-come, while overlooking other, slower options that would potentially yield higher outcomes. Thereby, we satisfy the minimize waiting time rule and suffice once the optionwith the shortest waiting time has been found. Satisficinghas also been introduced in utilitarian moral theory (Slote, 1984). The notion deals with the objection that utility maximization cannot be a norm for daily moral behavior.Instead, we should satisfice: aim to increase utility onlyto a certain level. So, more generally, we may use similar heuristics for making moral and economic decisions.

It is currently a point of contention whether heuristicsare near-optimal tools that evolved for optimal decisions, given cognitive and environmental constraints, or empiri-cal deviations from normative economic and moral theo-ries (Gigerenzer & Goldstein, 1996; Sunstein, 2005). Like economic heuristics, moral heuristics may be sufficientmost of the time but can fail or result in inconsistent be-havior with difficult decisions, such as the trolley and thefootbridge problems. Examples of specifically moral heu-ristics are do not knowingly cause a human death, peopleshould not be permitted to engage in moral wrongdoing for a fee, punish and do not reward betrayals of trust, penalties should be proportional to the outrageousnessof the act, do not play God or tamper with nature, and harmful acts are worse than harmful omissions. The rule-of-thumb-like nature of these heuristics might give them a deontological flavor, but it is possible to remain withina strictly utilitarian framework: Rule-utilitarianism has argued that we ought to follow utility-maximizing rules (see, e.g., Brandt, 1967; Hooker, 2000). In such a view, the moral quality of actions depends on whether they accord with a rule that would maximize the sum or mean utility if everybody were to follow it.

Although people often make decisions consistent with utilitarianism, they sometimes maximize utility only of members of a certain group (e.g., fellow citizens, co-workers, etc.), but not of members of another group (e.g., citizens of another country, workers from another factory,etc.). Such parochialism (Baron, in press) refers to the tendency of a decision maker to accept personal costs tomaximize the benefit of other in-group members, whileneglecting or even disproportionally increasing the nega-tive effects on outsiders (for experimental evidence, see Bornstein & Ben-Yossef, 1994). As a consequence, the net overall utility (when everyone is taken into account) isreduced. Parochialism seems to be a moral heuristic in it-self ( protect the interests of my fellow in-group members),which would suggest that we make some utilitarian deci-sions only with respect to the groups we belong to.

If it is unclear why, when, and whether ethical intu-itions maximize utility, one may ask why we all seem touse them and why they have developed in the first place.Moral heuristics, including group loyalty and parochial-ism, may have provided an evolutionary advantage to our ancestors and could have evolved from simple social and emotional responses (Greene, 2003; Greene et al., 2004). For example, it is certainly beneficial for the prosper-

moral and the economic domains. Interestingly, people with unconditional moral values as proposed by deonto-logical theories are less susceptible to framing effects in the moral domain (Tanner & Medin, 2004). These people appear to pay more attention to the question of whether their actions are compatible with their moral principlesthan to the consequences of their actions (to the amountof utility it generates). However, it should be noted thatmost people are willing to soften even their most protected unconditional values if the consequences of rigidly stick-ing to them would be too outrageous (Baron & Ritov, inpress). In any case, amygdala activity appears to contrib-ute to flexibly adjusting behavior with economic frames,and it might be interesting to investigate whether this holds also with moral framing.

The framing effect contradicts standard economic the-ory. In response to such contradictions, which usually boildown to axiom violations, one can either relax some of theaxioms (e.g., Fishburn, 1982; Machina, 1982) or pursue anonaxiomatic approach (Kahneman & Tversky, 1979; cf.Kalenscher & Pennartz, 2008). Nonaxiomatic approaches describe actual, rather than deduce optimal, choice behav-ior. For example, Kahneman and Tversky empirically in-vestigated human choice behavior and found several con-sistent and systematic deviations from the normative idealdictated by expected utility theory. These observations arebundled in prospect theory, which, contrary to expected utility theory, submits that we overweigh small and under-weigh large probabilities, are usually risk-seeking and notrisk averse for losses, and are more sensitive to losses thanto gains. However, the computational core of prospect the-ory and expected utility theory is very similar: Accordingto both theories, value is computed as the sum of utilitiesof all possible outcomes, weighted by distorted or linear probability. People assign these values to each available alternative and choose whichever alternative yields the highest value. Thus, some basic mechanism of combiningprobability and magnitude and its neuronal correlates mayalso be relevant for non- or less axiomatic theories, in both the economic and the moral domains.

Economic and Moral HeuristicsExpected utility theory and moral utilitarianism have

often been criticized as requiring excessive processingand calculation (multiplying probabilities and magnitudesand summing the products). Current theories of decisionmaking therefore often propose that humans use simpleand straightforward decision rules (heuristics) to make economic or moral decisions. These heuristics are usuallyeffective and put little demand on cognitive processing butcan occasionally lead to systematic biases, misjudgments, and deviations from the normative standard (Tversky &Kahneman, 1974). On the other hand, heuristics may havehigh descriptive validity while performing equally well,or better, in complex environments, as compared with in-tricate normative models. Examples in the economic do-main include the satisficing (Simon, 1955, 1956) and thegtake-the-best algorithms (Gigerenzer & Goldstein, 1996;tGoldstein & Gigerenzer, 2002). We often satisfy a simplechoice criterion that is part of a sequence of hierarchically


principles that achieve justice, which involves impartiallytaking the perspective of all parties and respecting them equally. According to the theory, we move from one stage to the next by thinking about moral problems, perhaps after our viewpoints have been challenged by others.

Views from learning theory. Kohlberg’s theory has been criticized for putting deontological over other moraltheories (e.g., Campbell & Christopher, 1996). Moreover,it is concerned with abstract thinking about moral situa-tions, which might differ from action. Are there alternativemechanisms that could explain the acquisition of moralbehavior and real-life decision making? One possibility may come from learning theory. Cultural differences inethical values suggest that moral reasoning may indeed belearned. Learning theory aims to explain how rewards and punishments modify behavior, and how learning agentsform associations between such reinforcing stimuli and actions or conditioned stimuli. Rewards and punishmentscan be survival related, as is the case with drink, food, sex,and pain, or of a derived, higher order nature, as is the case with monetary gains and losses, social recognition,and contempt. It is conceivable that we learn to decide morally by associating actions and stimuli with social and nonsocial rewards and punishments. Through higher order conditioning, we may have learned that sometimes it takesa very long time until a reward occurs. For example, we may learn to punish individuals who violate social norms(de Quervain et al., 2004; Fehr & Gächter, 2002) or to trustothers (King-Casas et al., 2005) because this will lead tomore reward for us in the long run. On the other hand,deciding morally and following a moral principle might be rewarding in itself. As has been mentioned, to increasethe utility of others may increase our own utility (Fehr & Schmidt, 1999). In summary, not only learning about rein-forcement value, but also learning about moral value mayfollow principles from learning theory. In the next section,we therefore will briefly introduce a prominent learning theory and explore its applications to the moral domain.

Modern theories of conditioning. Modern learningtheories (for a review, see Dickinson, 1980) describe as-sociation formation as a function of the degree to which participants process the stimuli and their reinforcing out-comes. According to Rescorla and Wagner (1972; see also Sutton & Barto, 1981, 1990), learning of an association de-pends on the extent to which the outcomes are surprising or elicit a prediction error. Learning is captured as changes in associative strength between the stimuli and the outcomes:

V ((( V ), where V denotes the change in asV -sociative strength of a stimulus in a trial; and denotethe salience or intensity of the stimulus and the outcome, respectively; corresponds to the asymptotic processing of the outcome when it is completely unpredicted; and V deV -notes the sum of the associative strengths of all the stimuli present for the outcome. V corresponds to the error Vin prediction of the outcome and can assume positive or negative values in a bidirectional fashion.

Actions and stimuli do not always lead to the same re-inforcing outcomes. Outcome distributions can be char-acterized by their mean (first moment), by their variance(second moment), and by higher moments. Traditionally,

ity and evolutionary stability of a society if many of itsmembers follow certain moral rules of thumb, such asdon’t kill or l respect the personal property and integrity of in-group members. Such social instincts presumablyevolved in situations that primarily involved direct per-sonal contact (small, closed societies), but not in environ-ments in which spatially and temporally distant strang-ers helped each other. This can explain why we usually feel more obliged to help individuals that are presented to us in flesh and blood than to help remote, abstract, and temporally and spatially distant individuals. This evolved social rule obliges us not to kill the fat man with our ownhands by throwing him onto the train tracks in the foot-bridge dilemma. But because this heuristic does not applyif we are not face to face with the fat man, it does not sufficiently discriminate between the choice alternatives in the scenario in which we manipulate the track switch from a remote control room. Thus, we rely on a second rule of thumb: Try to save as many lives as possible by minimizing the number of deaths. A satisficing algorithm can thus provide an alternative explanation for the ethical eclecticism we described earlier: Perhaps people are notsometimes utilitarians and sometimes deontologists but use rules of thumb to make decisions that are essentiallyabout utility. This does not mean that people never makeuse of deontological moral principles; it means only thatthe use of rules does not necessarily imply such deonto-logical principles. As was outlined above, the hallmark of deontology is its reference to values that are themselvesmoral in nature. Whether or not a rule is a deontological rule thus depends on the nature of the values the rule re-fers to. Someone who employs the don’t kill rule can jus-ltify that rule on a utilitarian basis (such a rule brings about the most happiness), but also on a deontological basis (lifehas moral value and thus must be protected). In the firstcase, the rule is justified by referring to values that are themselves not moral (such as happiness); in the second case, the underlying values are themselves moral.

Can We Learn to Decide More Morally?From both a personal and a societal point of view, we

might be interested in how we acquire morality and whether it is possible to make better moral decisions. Building on Piaget (1932), Kohlberg proposed that moral development consists of the stagewise acquisition of a deontologicalmoral theory (for a review, see Colby & Kohlberg, 1987). Kohlberg ascribed moral theories to children on the basisof the justifications they gave for their moral decisions.He identified six stages of moral development. At the first stage, the child is oriented to obedience and punishment.The concern is with what authorities and the law permit and punish. At the second stage, the child is concerned with self-interest and fair exchange. The third stage is reached at the beginning of the teenage years and is concerned with the motives of actions, interpersonal relations, and emo-tions such as empathy and trust. The fourth stage is aboutduty to keep social order, respect authorities, and obey thelaw because of its role for society. At the fifth stage, peo-ple reason about social contracts, the democratic process,and human rights. The sixth stage concerns the universal


ing elicit amygdala activation, similar to self-experienced pain conditioning (reviewed in Olsson & Phelps, 2007). In addition, regions coding empathy, such as the anterior insula and the anterior cingulate cortex, and language re-gions contribute to observational and instructed learning,respectively.

In the moral domain, both observational and laboratoryevidence suggests that significant others pass moral ruleson to children, or at least provide instances for children to extract moral rules. The children subsequently incor-porate the rules into their moral decisions (reviewed inDarley & Shultz, 1990). Observational learning without having to directly experience punishment can occur in the classroom, by watching television, by taking part inpretend play episodes, and by observing parents interactwith siblings. Moreover, knowledge about the moral status of others modulates the occurrence of prediction errors inthe trust game (Delgado et al., 2005). In the trust game,a proposer decides how much of an initial endowment they want to pass on to a responder. The amount passed to the responder will be multiplied (e.g., tripled) by the experimenter before the responder decides how much to pass back to the proposer. In the experiment of Delgadoet al. (2005), participants assumed the role of proposer and played with three different fictional responders: one por-trayed as morally praiseworthy and good, one as neutral, and one as bad. Irrespective of moral status, the respond-ers passed back nothing or half of the tripled amount, so that the proposers earned the same amount of money with all three responders. The proposers experienced a negative prediction error when the responders passed back noth-ing or a positive prediction error when the responderspassed back half of the money. One would therefore expect prediction-error-related activation in the striatum, as de-scribed previously (McClure et al., 2003; O’Doherty et al.,2003). Activation in the dorsal striatum indeed discrimi-nated strongly between positive and negative predictionerrors with the neutral responder. However, the discrimina-tion was much weaker or even absent with the bad and thegood responders. Throughout the experiment, the propos-ers continued to invest more money when they played with the good rather than with the bad responder. These data suggest that moral perceptions can alter behavioral and neuronal reward-learning mechanisms. Thereby, the find-ings further reinforce the main notion of the present article, that moral decision making may rely on, and interact with, reward- (and punishment-)processing circuitry.

ConclusionsThe reward system and its neuronal basis have com-

monly been portrayed as amenable to hijacking by suchdisastrous substances and practices as drugs of abuse, pathological gambling, and irrational risk taking (for areview, see, e.g., Kauer & Malenka, 2007). Here, we pro-pose that at least in the domain of utilitarian moral value,the reward system could be put to more beneficial use, in that it may help compute moral value. As exemplified in dopamine neurons, the striatum, and the lateral prefrontal cortex, the reward system may combine the probabilityand magnitude of valuable consequences associated with

prediction errors are computed for the mean reward (firstmoment). However, the mechanism can, in principle, alsobe used to learn about higher moments. The activation pro-files of single dopamine neurons and of the striatal BOLD response correspond to a prediction error signal for meanreward expectation, possibly normalized by the standard deviation (square root of variance) of the prediction (Mc-Clure, Berns, & Montague, 2003; O’Doherty, Dayan, Fris-ton, Critchley, & Dolan, 2003; Tobler et al., 2005; Waelti,Dickinson, & Schultz, 2001). Formal variance prediction signals may occur in structures such as the insula (Knutson& Bossaerts, 2007), the cingulate (Brown & Braver, 2007),the posterior parietal cortex (Huettel et al., 2006), and theprefrontal cortex (Hsu, Bhatt, Adolphs, Tranel, & Cam-erer, 2005; Tobler et al., 2007). Errors in the prediction of mean and variance of behavior could possibly be used todetermine deviations of actual behavior from behavior asit would be expected if it were to follow a normative moralstandard (Montague & Lohrenz, 2007). Accordingly, suchprediction errors may motivate punishing actions, which,in turn, may be reinforced by observing increased norm conformity of the punished. Errors in the prediction of distress and empathy experienced by others or oneself may elicit moral learning and reduce actions with conse-quences harmful to others (Blair, 2007). Thus, the samemechanisms as those used for learning about reward and punishment may serve for learning about morally relevant emotions. In return, self-evaluating moral emotions, such as shame, guilt, embarrassment, and moral pride, them-selves generate punishment and reinforcement and, thus, an occasion to learn. After learning, we can predict what emotions we are likely to experience for each alternativecourse of action and can decide accordingly.

When a neutral stimulus predicts a previously condi-tioned stimulus, the neutral stimulus will be conditioned as well (Rescorla, 1980). Importantly, such second-order and even higher order associations can occur without ever directly pairing the to-be-learned stimuli with reinforce-ment. Higher order conditioning can also be explained with prediction errors, as is suggested by a real-time ex-tension of the Rescorla–Wagner model (Sutton & Barto,1990). Activation in the ventral striatum and the anterior insula reflects second-order conditioning of pain (Sey-mour et al., 2004). It is possible that higher order condi-tioning plays a role in moral decision making because itallows us to bridge extended periods of time without re-ward. In seemingly other-regarding behavior such as trustand inequity aversion, motives of self-interest may play alarger role than has previously been assumed, as long as these behaviors eventually lead to more reward.

We learn not only through our own experience, but also from others. For example, by observing others undergo-ing pain conditioning and following others’ instructions,we are in a position to learn about pain-predicting stimuli without having to experience the aversive outcome our-selves (Berber, 1962; Mineka & Cook, 1993; Phelps et al.,2001). The behavior and facial expressions of others and the meaning of the instructions can elicit prediction errors, and standard learning mechanisms may apply (Lanzetta &Orr, 1980). Both observed and instructed pain condition-


choice—toward an integrative framework. Trends in Cognitive Sci-ences, 11, 482-488.

Berridge, K. C., & Robinson, T. E. (2003). Parsing reward. Trends inNeurosciences, 26, 507-513.

Blair, R. J. (2007). The amygdala and ventromedial prefrontal cortex inmorality and psychopathy. Trends in Cognitive Sciences, 11, 387-392.

Bornstein, G., & Ben-Yossef, M. (1994). Cooperation in intergroupand single-group social dilemmas. Journal of Experimental Social Psychology, 30, 52-67.

Brandt, R. B. (1967). Some merits of one form of rule-utilitarianism. University of Colorado Studies in Philosophy, 3, 39-65.

Brandt, R. B. (1979). A theory of the good and the right. New York: Oxford University Press.

Brown, J. W., & Braver, T. S. (2007). Risk prediction and aversion by anterior cingulate cortex. Cognitive, Affective, & Behavioral Neuro-science, 7, 266-277.

Bruening, W. H. (1971). Moore and “is–ought.” Ethics, 81, 143-149.Campbell, R. L., & Christopher, J. C. (1996). Moral development

theory: A critique of its Kantian presuppositions. Developmental Re-view, 16, 1-47.

Colby, A., & Kohlberg, L. (1987). The measurement of moral judg-ment: Vol. 1. Theoretical foundations and research validation. New York: Cambridge University Press.

Copp, D. (1997). Defending the principle of alternative possibilities:Blameworthiness and moral responsibility. Nous, 31, 441-456.

Copp, D. (2008). “Ought” implies “can” and the derivation of the prin-ciple of alternative possibilities. Analysis, 68, 67-75.

Cromwell, H. C., & Schultz, W. (2003). Effects of expectations for different reward magnitudes on neuronal activity in primate striatum. Journal of Neurophysiology, 89, 2823-2838.

d’Acremont, M., & Bossaerts, P. (2008). Neurobiological studies of risk assessment: A comparison of expected utility and mean–variance approaches. Cognitive, Affective, & Behavioral Neuroscience, 8, 363-374.

Darley, J. M., & Shultz, T. R. (1990). Moral rules: Their content and acquisition. Annual Review of Psychology, 41, 525-556.

Delgado, M. R., Frank, R. H., & Phelps, E. A. (2005). Perceptions of moral character modulate the neural systems of reward during the trust game. Nature Neuroscience, 8, 1611-1618.

Delgado, M. R., Labouliere, C. D., & Phelps, E. A. (2006). Fear of losing money? Aversive conditioning with secondary reinforcers.Social Cognitive & Affective Neuroscience, 1, 250-259.

De Martino, B., Kumaran, D., Seymour, B., & Dolan, R. J. (2006). Frames, biases, and rational decision making in the human brain. Sci-ence, 313, 684-687.

de Quervain, D. J.-F., Fischbacher, U., Treyer, V., Schellham-mer, M., Schnyder, U., Buck, A., et al. (2004). The neural basis of altruistic punishment. Science, 305, 1254-1258.

Dickinson, A. (1980). Contemporary animal learning theory. Cam-bridge: Cambridge University Press.

Duncan, J., & Owen, A. M. (2000). Common regions of the humanfrontal lobe recruited by diverse cognitive demands. Trends in Neuro-sciences, 23, 475-483.

Dupoux, E., & Jacob, P. (2007). Universal moral grammar: A criticalappraisal. Trends in Cognitive Sciences, 11, 373-378.

Ellsberg, D. (1961). Risk, ambiguity, and the Savage axioms. Quar-rrterly Journal of Economics, 75, 643-669.

Fehr, E., & Camerer, C. F. (2007). Social neuroeconomics: The neu-ral circuitry of social preferences. Trends in Cognitive Sciences, 11, 419-427.

Fehr, E., & Gächter, S. (2002). Altruistic punishment in humans. Na-ture, 415, 137-140.

Fehr, E., & Schmidt, K. M. (1999). A theory of fairness, competition and cooperation. Quarterly Journal of Economics, 114, 817-868.

Fishburn, P. C. (1982). Nontransitive measurable utility. Journal of Mathematical Psychology, 26, 31-67.

Fliessbach, K., Weber, B., Trautner, P., Dohmen, T., Sunde, U., Elger, C. E., & Falk, A. (2007). Social comparison affects reward-related brain activity in the human ventral striatum. Science, 318, 1305-1308.

Foot, P. (2001). Natural goodness. Oxford: Oxford University Press, Clarendon Press.

action options. Formal learning theory suggests mecha-nisms by which utilitarian moral value may come to guide behavior.

But what about deontological moral theories? These theories presuppose that we value certain moral prin-ciples or acts or things in themselves, regardless of theamount of utility they produce. How should we under-stand these kinds of value judgments? Insofar as they takeother things than consequences into account, they can-not be related to reward, because reward (at least at face value) is a consequence of actions or preceding stimuli. Sothe reward system is unlikely to underlie such judgments.It seems hard to conceive any other system that could,without introducing a separate moral motivation system.If it turned out that it is impossible for human beings tovalue things regardless of utility-related characteristics (if human beings can value things only by relating them toreward), this might present deontological moral theorieswith a problem. After all, the ought-implies-can principlestates that human beings should, at least in principle, have the capacity to make the morally right decision. If humanbeings were unable to ascribe purely deontological valueto actions without any regard for utility, this would be a strong objection against deontological theories.

AUTHORAA NR OTE

This work was supported by the Wellcome Trust, the Swiss NationalScience Foundation, the Roche Research Foundation, and Senter Novem.The collaboration between the authors was made possible by a grant from the Volkswagen Foundation, and the article has been written as part of theEuropean Platform for Life Sciences, Mind Sciences, and the Humanitiesinitiative of the Volkswagen Foundation. Correspondence concerning this article should be addressed to P. N. Tobler, Department of Physiology, De-velopment and Neuroscience, University of Cambridge, Downing Street,Cambridge CB2 3DY, England (e-mail: [email protected]).

REFERENCERR S

Abler, B., Walter, H., Erk, S., Kammerer, H., & Spitzer, M. (2006).Prediction error as a linear function of reward probability is coded inhuman nucleus accumbens. NeuroImage, 31, 790-795.

Allais, M. (1953). Le comportement de l’homme rationnel devant le risque: Critique des postulats et axiomes de l’école Américaine.Econometrica, 21, 503-546.

Anscombe, G. E. M. (1958). Modern moral philosophy. Philosophy,33, 1-19.

Aristotle (2002). Nicomachean ethics (S. Broadie, Ed., & C. Rowe,Trans.). Oxford: Oxford University Press.

Bales, R. E. (1971). Act-utilitarianism: Account of right-making char-acteristics or decision-making procedures? American Philosophical Quarterly, 8, 257-265.

Baron, J. (in press). Parochialism as a result of cognitive biases. In R. Goodman, D. Jinks, & A. K. Woods (Eds.), Understanding so-cial action, promoting human rights. New York: Oxford UniversityPress.

Baron, J., & Ritov, I. (in press). Protected values and omission biasas deontological judgments. In D. M. Bartels, C. W. Bauman, L. J.Skitka, & D. L. Medin (Eds.), The psychology of learning and moti-vation: Moral judgment and decision making (Vol. 50). San Diego: Academic Press.

Berber, S. M. (1962). Conditioning through vicarious instigation. Psy-chological Review, 69, 450-466.

Bernoulli, D. (1738). Specimen theoriae novae de mensura sortis.Commentarii Academiae Scientiarum Imperialis Petropolitanae, 5,175-192.

Berns, G. S., Laibson, D., & Loewenstein, G. (2007). Intertemporal


Knoch, D., Pascual-Leone, A., Meyer, K., Treyer, V., & Fehr, E.(2006). Diminishing reciprocal fairness by disrupting the right pre-frontal cortex. Science, 314, 829-832.

Knutson, B., & Bossaerts, P. (2007). Neural antecedents of financial decisions. Journal of Neuroscience, 27, 8174-8177.

Knutson, B., Taylor, J., Kaufman, M., Peterson, R., & Glover, G. (2005). Distributed neural representation of expected value. Journal of Neuroscience, 25, 4806-4812.

Kobayashi, S., Lauwereyns, J., Koizumi, M., Sakagami, M., & Hikosaka, O. (2002). Influence of reward expectation on visuospa-tial processing in macaque lateral prefrontal cortex. Journal of Neu-rophysiology, 87, 1488-1498.

Koenigs, M., & Tranel, D. (2007). Irrational economic decision mak-ing after ventromedial prefrontal damage: Evidence from the ultima-tum game. Journal of Neuroscience, 27, 951-956.

Koenigs, M., Young, L., Adolphs, R., Tranel, D., Cushman, F., Hauser, M., & Damasio, A. (2007). Damage to the prefrontal cortexincreases utilitarian moral judgments. Nature, 446, 908-911.

Kuhberger, A. (1998). The influence of framing on risky decision:A meta-analysis. Organizational Behavior & Human Decision Pro-cesses, 75, 23-55.

Lanzetta, J. T., & Orr, S. P. (1980). Influence of facial expressionson the classical conditioning of fear. Journal of Personality & Social Psychology, 39, 1081-1087.

Lau, B., & Glimcher, P. W. (2008). Value representations in the primatestriatum during matching behavior. Neuron, 58, 451-463.

Loomes, G., Starmer, C., & Sugden, R. (1991). Observing violations of transitivity by experimental methods. Econometrica, 59, 425-439.

Machina, M. J. (1982). “Expected utility” analysis without the indepen-dence axiom. Econometrica, 50, 277-323.

MacIntyre, A. (1985). After virtue. London: Duckworth.Marschak, J. (1950). Rational behavior, uncertain prospects, and mea-

surable utility. Econometrica, 18, 111-141.McClure, S. M., Berns, G. S., & Montague, P. R. (2003). Temporal

prediction errors in a passive learning task activate human striatum.Neuron, 38, 339-346.

McClure, S. M., Ericson, K. M., Laibson, D. I., Loewenstein, G., & Cohen, J. D. (2007). Time discounting for primary rewards. Journal of Neuroscience, 27, 5796-5804.

McClure, S. M., Laibson, D. I., Loewenstein, G., & Cohen, J. D.(2004). Separate neural systems value immediate and delayed mon-etary rewards. Science, 306, 503-507.

Mikhail, J. (2007). Universal moral grammar: Theory, evidence and thefuture. Trends in Cognitive Sciences, 11, 143-152.

Mill, J. S. (1861). Utilitarianism. Fraser’s Magazine for Town & Coun-try, 64, 391-406.

Miller, E. K., & Cohen, J. D. (2001). An integrative theory of prefron-tal cortex function. Annual Reviews in Neuroscience, 24, 167-202.

Mineka, S., & Cook, M. (1993). Mechanisms involved in the obser-vational conditioning of fear. Journal of Experimental Psychology:General, 122, 23-38.

Moll, J., & de Oliveira-Souza, R. (2007). Moral judgments, emotionsand the utilitarian brain. Trends in Cognitive Sciences, 11, 319-321.

Moll, J., Zahn, R., de Oliveira-Souza, R., Krueger, F., & Graf-man, J. (2005). Opinion: The neural basis of human moral cognition.Nature Reviews Neuroscience, 6, 799-809.

Montague, P. R., & Lohrenz, T. (2007). To detect and correct: Normviolations and their enforcement. Neuron, 56, 14-18.

Moore, G. E. (1903). Principia ethica. Cambridge: Cambridge Uni-versity Press.

Nichols, S., & Mallon, R. (2006). Moral dilemmas and moral rules. Cognition, 100, 530-542.

Nussbaum, M. C. (2000). Women and human development: The capa-bilities approach. New York: Cambridge University Press.

O’Doherty, J. P., Dayan, P., Friston, K., Critchley, H., & Dolan,R. J. (2003). Temporal difference models and reward-related learning in the human brain. Neuron, 38, 329-337.

Olsson, A., & Phelps, E. A. (2007). Social learning of fear. NatureNeuroscience, 10, 1095-1102.

Padoa-Schioppa, C., & Assad, J. A. (2006). Neurons in the orbitofron-tal cortex encode economic value. Nature, 441, 223-226.

Frith, C. D. (2007). The social brain? Philosophical Transactions of theRoyal Society B, 362, 671-678.

Fuster, J. M. (1997). The prefrontal cortex. Anatomy, physiology, and neuropsychology of the frontal lobe. Philadelphia: Lippincott-Raven.

Gaspar, P., Stepniewska, I., & Kaas, J. H. (1992). Topography and collateralization of the dopaminergic projections to motor and lateralprefrontal cortex in owl monkeys. Journal of Comparative Neurology,325, 1-21.

Gewirth, A. (1998). Self-fulfillment. Princeton, NJ: Princeton Univer-sity Press.

Gigerenzer, G., & Goldstein, D. G. (1996). Reasoning the fast and frugal way: Models of bounded rationality. Psychological Review,103, 650-669.

Glimcher, P. W., Kable, J., & Louie, K. (2007). Neuroeconomic stud-ies of impulsivity: Now or just as soon as possible? American Eco-nomic Review, 97, 142-147.

Goldstein, D. G., & Gigerenzer, G. (2002). Models of ecological ratio-nality: The recognition heuristic. Psychological Review, 109, 75-90.

Greene, J. D. (2003). From neural “is” to moral “ought”: What are themoral implications of neuroscientific moral psychology? Nature Re-views Neuroscience, 4, 846-849.

Greene, J. D. (2007). Why are VMPFC patients more utilitarian? A dual-process theory of moral judgment explains. Trends in CognitiveSciences, 11, 322-323.

Greene, J. D., Nystrom, L. E., Engell, A. D., Darley, J. M., &Cohen, J. D. (2004). The neural bases of cognitive conflict and con-trol in moral judgment. Neuron, 44, 389-400.

Greene, J. D., Sommerville, R. B., Nystrom, L. E., Darley, J. M., &Cohen, J. D. (2001). An fMRI investigation of emotional engagement in moral judgment. Science, 293, 2105-2108.

Hare, R. M. (1981). Moral thinking. Oxford: Oxford University Press,Clarendon Press.

Harman, G. (2000). Explaining value and other essays in moral phi-losophy. Oxford: Oxford University Press.

Harsanyi, J. (1953). Cardinal utility in welfare economics and in thetheory of risk-taking. Journal of Political Economy, 61, 434-435.

Harsanyi, J. (1955). Cardinal welfare, individualistic ethics, and theinterpersonal comparison of utility. Journal of Political Economy, 63,309-321.

Hauser, M. D. (2007). What’s fair? The unconscious calculus of our moral faculty. Novartis Foundation Symposium, 278, 41-50.

Hooker, B. (2000). Ideal code, real world: A rule-consequentialist theory of morality. Oxford: Oxford University Press.

Hsu, M., Bhatt, M., Adolphs, R., Tranel, D., & Camerer, C. F.(2005). Neural systems responding to degrees of uncertainty in human decision making. Science, 310, 1680-1683.

Huettel, S. A., Stowe, C. J., Gordon, E. M., Warner, B. T., & Platt,M. L. (2006). Neural signatures of economic preferences for risk and ambiguity. Neuron, 49, 765-775.

Hume, D. (1739). A treatise of human nature: Being an attempt to in-troduce the experimental method of reasoning into moral subjects.London: Noon.

Kable, J. W., & Glimcher, P. W. (2007). The neural correlates of sub-jective value during intertemporal choice. Nature Neuroscience, 10,1625-1633.

Kahneman, D., & Tversky, A. (1979). Prospect theory: An analysis of decision under risk. Econometrica, 47, 263-291.

Kalenscher, T., & Pennartz, C. M. A. (2008). Is a bird in the hand worth two in the future? The neuroeconomics of intertemporal deci-sion making. Progress in Neurobiology, 84, 284-315.

Kamm, F. M. (2007). Intricate ethics: Rights, responsibilities, and per-rrmissible harms. Oxford: Oxford University Press.

Kant, I. (1788). Critik der practischen Vernunft. Riga: Johann Friedrich Hartknoch.

Kant, I. (1797). Die Metaphysik der Sitten in 2 Theilen. Königsberg: Nicolovius.

Kauer, J. A., & Malenka, R. C. (2007). Synaptic plasticity and addic-tion. Nature Reviews Neuroscience, 8, 844-858.

King-Casas, B., Tomlin, D., Anen, C., Camerer, C. F., Quartz, S. R.,& Montague, P. R. (2005). Getting to know you: Reputation and trust in a two-person economic exchange. Science, 308, 78-83.


Singer, T., Seymour, B., O’Doherty, J., Kaube, H., Dolan, R. J., & Frith, C. D. (2004). Empathy for pain involves the affective but not sensory components of pain. Science, 303, 1157-1162.

Slote, M. (1984). Satisficing consequentialism. Proceedings of the Ar-rristotelian Society, 58, 139-163.

Sunstein, C. R. (2005). Moral heuristics. Behavioral & Brain Sciences,28, 531-542.

Sutton, R. S., & Barto, A. G. (1981). Toward a modern theory of adaptive networks: Expectation and prediction. Psychological Review,88, 135-170.

Sutton, R. S., & Barto, A. G. (1990). Time-derivative models of Pavlovian reinforcement. In M. Gabriel & J. Moore (Eds.), Learning and computational neuroscience: Foundations of adaptive networks(pp. 497-537). Cambridge, MA: MIT Press.

Tabibnia, G., & Lieberman, M. D. (2007). Fairness and cooperation are rewarding: Evidence from social cognitive neuroscience. In C. Senior & M. J. R. Butler (Eds.), The social cognitive neuroscience of orga-nizations (Annals of the New York Academy of Sciences, Vol. 1118, pp. 90-101). New York: New York Academy of Sciences.

Tanner, C., & Medin, D. L. (2004). Protected values: No omission biasand no framing effects. Psychonomic Bulletin & Review, 11, 185-191.

Thomson, J. J. (1985). The trolley problem. Yale Law Journal, 94,1395-1415.

Tobler, P. N., Fiorillo, C. D., & Schultz, W. (2005). Adaptive coding of reward value by dopamine neurons. Science, 307, 1642-1645.

Tobler, P. N., O’Doherty, J. P., Dolan, R. J., & Schultz, W. (2007).Reward value coding distinct from risk attitude-related uncertainty coding in human reward systems. Journal of Neurophysiology, 97,1621-1632.

Tom, S. M., Fox, C. R., Trepel, C., & Poldrack, R. A. (2007). The neural basis of loss aversion in decision making under risk. Science,315, 515-518.

Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty:Heuristics and biases. Science, 185, 1124-1131.

Von Neumann, J., & Morgenstern, O. (1944). Theory of games and economic behavior. Princeton, NJ: Princeton University Press.

Waelti, P., Dickinson, A., & Schultz, W. (2001). Dopamine re-sponses comply with basic assumptions of formal learning theory.Nature, 412, 43-48.

Wallis, J. D., & Miller, E. K. (2003). Neuronal activity in primate dor-solateral and orbital prefrontal cortex during performance of a reward preference task. European Journal of Neuroscience, 18, 2069-2081.

Widerker, D. (1991). Frankfurt on “ought implies can” and alternativepossibilities. Analysis, 51, 222-224.

Wikström, P.-O. (2007). In search of causes and explanations of crime.In R. King & E. Wincup (Eds.), Doing research on crime and justice(pp. 117-139). Oxford: Oxford University Press.

Yacubian, J., Gläscher, J., Schroeder, K., Sommer, T., Braus, D. F., & Büchel, C. (2006). Dissociable systems for gain- and loss-related value predictions and errors of prediction in the human brain. Journal of Neuroscience, 26, 9530-9537.

(Manuscript received March 8, 2008;revision accepted for publication July 7, 2008.)

Phelps, E. A., O’Connor, K. J., Gatenby, J. C., Gore, J. C., Gri-llon, C., & Davis, M. (2001). Activation of the left amygdala to a cognitive representation of fear. Nature Neuroscience, 4, 437-441.

Piaget, J. (1932). The moral judgment of the child. London: Routledge & Kegan Paul.

Rawls, J. (1971). A theory of justice. Cambridge, MA: Harvard Uni-versity Press.

Rescorla, R. A. (1980). Pavlovian second-order conditioning: Studies in associative learning. Hillsdale, NJ: Erlbaum.

Rescorla, R. A., & Wagner, A. R. (1972). A theory of Pavlovian conditioning: Variations in the effectiveness of reinforcement and nonreinforcement. In A. H. Black & W. F. Prokasy (Eds.), Classical conditioning II: Current research and theory (pp. 64-99). New York: Appleton-Century-Crofts.

Rilling, J., Gutman, D., Zeh, T., Pagnoni, G., Berns, G., & Kilts, C.(2002). A neural basis for social cooperation. Neuron, 35, 395-405.

Ross, W. D. (1930). The right and the good. Oxford: Oxford University Press, Clarendon Press.

Rushworth, M. F., & Behrens, T. E. (2008). Choice, uncertainty and value in prefrontal and cingulate cortex. Nature Neuroscience, 11, 389-397.

Rustichini, A. (2008). Dual or unitary system? Two alternative models of decision making. Cognitive, Affective, & Behavioral Neuroscience, 8, 355-362.

Sakagami, M., & Watanabe, M. (2007). Integration of cognitive and motivational information in the primate lateral prefrontal cortex. InB. W. Balleine, K. Doya, J. O’Doherty, & M. Sakagami (Eds.), Reward and decision making in corticobasal ganglia networks (Annals of the New York Academy of Sciences, Vol. 1104, pp. 89-107). New York: New York Academy of Sciences.

Samejima, K., Ueda, Y., Doya, K., & Kimura, M. (2005). Representa-tion of action-specific reward values in the striatum. Science, 310,1337-1340.

Sanfey, A. G., Loewenstein, G., McClure, S. M., & Cohen, J. D. (2006). Neuroeconomics: Cross-currents in research on decision mak-ing. Trends in Cognitive Sciences, 10, 108-116.

Sanfey, A. G., Rilling, J. K., Aronson, J. A., Nystrom, L. E., &Cohen, J. D. (2003). The neural basis of economic decision making in the ultimatum game. Science, 300, 1755-1758.

Schultz, W. (2006). Behavioral theories and the neurophysiology of reward. Annual Review of Psychology, 57, 87-115.

Searle, J. (1964). How to derive “ought” from “is.” Philosophical Re-view, 73, 43-58.

Sen, A. (1985). Well-being, agency and freedom. Journal of Philosophy,82, 169-221.

Seymour, B., O’Doherty, J. P., Dayan, P., Koltzenburg, M.,Jones, A. K., Dolan, R. J., et al. (2004). Temporal difference mod-els describe higher-order learning in humans. Nature, 429, 664-667.

Sidgwick, H. (1874). The methods of ethics. London: Macmillan.Simon, H. A. (1955). A behavioral model of rational choice. Quarterly

Journal of Economics, 69, 99-118.Simon, H. A. (1956). Rational choice and the structure of the environ-

ment. Psychological Review, 63, 129-138.Singer, T., Kiebel, S. J., Winston, J. S., Dolan, R. J., & Frith, C. D.

(2004). Brain responses to the acquired moral status of faces. Neuron,41, 653-662.

the role of moral utility in decision making: an

Documents