[ieee 2012 ieee conference on computational intelligence and games (cig) - granada, spain...

Update Rules, Reciprocity and Weak Selection in EvolutionarySpatial Games

Garrison W. GreenwoodDepartment of Electrical & Computer Engineering

Portland State UniversityPortland, OR 97207–0751 USAEmail: [email protected]

Phillipa AveryDepartment of Computer Science & Engineering

University of NevadaReno, NV 89557–0208 USA

Email: [email protected]

Abstract—Cooperation in nature is a complex topic and itsstudy has left scientists with many open questions. Over the pasttwo decades research has been undertaken into how cooperationworks in an evolutionary context and how we can emulateit for social analysis. Numerous computer models have beendeveloped and analyzed, with many models formulated as spatialor network games. These games use various update rules to evolvecooperative strategies. Despite two decades of effort, arguablylittle progress has been made.

This paper exposes some of the problems with these spatialand network games and shows why they are ill-suited to get anyreal answers. Recommendations on future research directionsthat might provide some insight are presented.

I. INTRODUCTION

Cooperation gives a benefit b to the recipient while exactinga cost c from the actor (with b > c). The actor may ormay not derive some personal benefit in the future fromthis cooperation. Why cooperate then? One of the largestopen questions in the biological and social sciences is howcooperation began and why is it so pervasive.

Over the past two decades many theories have beenoffered—and an equal number of computer models have beencreated—to try and get some answers. Yet, little headway hasbeen made. In a recent paper Greenwood and Ashlock [1]discussed why answers have been so allusive despite all ofthis research effort. They identified two primary reasons• Invalidated modelsAll models are constructed presumably to emulate dynamics

observed in nature. Computer simulations are run and theresults undergo statistical analysis. But rarely (if ever) doresearchers compare their simulation results with field data tosee how well they match. Field data is available, but modelsare not validated. Yet this lack of validation has not stoppedresearchers from making claims about conditions that promotecooperation.• Unrealistic ModelsMany models incorporate dynamics that have no functional

equivalence in nature. In other words, cooperation evolves inthese models by methods that either don’t appear in any formin nature or have not been observed in human experiments.Nevertheless, claims are still made on the influence of theseresults.

Cooperation exists throughout the animal world, the plantworld and, of course, in human populations. Researcherspropose models but rarely (if ever) restrict them to just one ofthese domains. The models are “general” in the sense that, bynot specifying any particular domain, they imply the modelapplies to any domain. However, the cooperative dynamicsbetween animal/plant populations and human populations arequite different. As pointed out in [2], in systems of interestfitness is not measured by the number of offspring, but by apotential to reproduce. In human experiments the evolution iscultural or social, not genetic, so reproduction refers to theproliferation of behaviors or strategies. On the other handkin selection is believed by many biologists to be a crucialcomponent of population viability (e.g., in social insects). Yetit wasn’t shown until very recently that kin selection andevolutionary game theory have something in common [3].

How and why cooperation develops when social dilemmasare present is of particular interest with human populations. Asurvey of the literature shows three social dilemma scenariosappear most often: the prisoner’s dilemma, the public goodsgame and the tragedy of the commons. In each of thesescenarios the entire population benefits when everyone coop-erates but free-riding is expected because it provides higherindividual payoffs. Yet cooperation must develop otherwisesocial order would disintegrate. So, under what conditionsdoes cooperation develop in human populations?

The tragedy of the commons (and most public goods games)are well-mixed games because in each iteration a playercould play against any other player. This situation doesn’texist in many real-world situations, particularly with humans.People tend to have more restricted, localized interactionsresulting from, for example, cultural or ethnic reasons. Thissituation has given rise to spatial models where individualscan only interact with nearest neighbors. Interestingly enough,computer simulations showed cooperation can persist in spatialenvironments because repeated interactions with the samepartners creates clusters of cooperators. Unconditional imita-tion of better neighbor strategies was also required [4]. Butrecently when spatial prisoner’s dilemma games with humanswere conducted the results differed significantly from what thecomputer simulations predicted [5]. For instance, they foundvirtually no evidence of network reciprocity.

978-1-4673-1194-6/12/$31.00 ©2012 IEEE 9

In this paper we will explain why the results from humanspatial experiments differed from those observed throughcomputer simulation. Specifically we discuss the flaws incommonly used update rules and, as a result, why theydon’t mimic human decision-making processes very well.Furthermore, many (most?) models incorporate features thathave nothing to do with evolving strategies and were notobserved in human experiments but are added merely forcomputational convenience. A particularly flawed tactic isresearchers invariably analyze solely their computer modelsrather than analyze cooperation studies involving humans. Assuch, few if any models that have appeared in the literatureover the past two decades are suitable for gaining insight intohuman cooperation.

We present a number of recommendations on directionsthat should focus cooperation research in more productivedirections. Two of our more important recommendations areremoving weak selection from the computer models anddropping fixation probability as a meaningful metric.

The paper is organized as follows. Section II provides abrief review of key definitions and concepts. Readers familiarwith this material may skip to the following section. SectionIII discusses the problems with most computer models andwhy they aren’t likely to provide any insight into humancooperation. The discussion focuses primarily on spatial gamesbut many ideas apply to network games as well. Finally, inSection IV We recommend what direction future researchefforts should take.

II. BACKGROUND

This section begins with a description of the types of gamestructures used in computer emulation for cooperation scenar-ios. This is followed by a discussion the different methods ofevaluating individuals in a population or the selection require-ments. The main update rules used to evolve the populationare described and the different reciprocity methods are listed.

A. Game Structures

1) Network Games: In network games players are assignedto nodes of an undirected graph. Two players interact if andonly if they share a graph edge. Three common networkstructures are regular networks, small world networks andscale free networks. Regular networks have a motif so allnodes have the exact same number of neighbors. For example,a 2-D torus gives each player a N, S, E and W neighbor. Smallworld networks are large diameter graphs with a small numberof randomly moved edges to reduce the graph diameter [6].Scale free networks distribute edges according to a power law.That is, the probability that a node has m neighbors is α−m

for some positive constant α. See Figure 1 for an example.2) Well-Mixed Games: Players interact with any other

player with some probability p, where p depends on the gametype. For example, p = 1 in public goods games whereasp = 1/N in games where only one interaction takes place perround. (N is the population size.)

Fig. 1. A portion of a scale free network. The vast majority of nodes havefew interconnections (white nodes) but there are a small number of nodes,called ‘hub nodes’, that have a large number of interactions (red nodes).

(a)

(b)

Fig. 2. (a) Von Neumann neighborhood and (b) Moore neighborhood. Thefocal individual is shown in red and its neighborhood in purple. These arespatial games not network games because there are no edges connecting nodes;here Euclidean distance defines the neighborhoods.

2012 IEEE Conference on Computational Intelligence and Games (CIG’12) 10

A well-mixed game is equivalent to a network game witha completely connected graph. The graph edges have weightscorresponding to the interaction probability p.

3) Spatial Games: Spatial games place each player on aunique node in a regular lattice. Players can only interactwith their nearest neighbors based on Euclidean distance. Twocommon neighborhoods are shown in Figure 2.

Spatial games are equivalent to network games on regulargraphs. Consider, for example, a network game on a 2Dgrid. Removing the edges and forcing players to interact withnearest neighbors (subject to wrap-around at the boundaries)yields the identical neighborhoods that exist in the 2D grid.

B. Selection

1) Frequency Dependent Selection: An individual’s ex-pected fitness depends on the relative abundance (or fre-quency) of its strategy in the current population. Let xi denotethe frequency of strategy i. Then the fitness of any individualwith strategy i is fi =

∑nj=1 xjaij where

∑nj=1 xj = 1 and

aij is the payoff for strategy i playing against strategy j.Initially this type of selection was restricted to infinitely

large, well mixed populations, but it has been recently appliedto finite populations as well [7], [8].

2) Weak, Strong & Neutral Selection: The selection inten-sity is specified by a real parameter β ∈ [0, 1]. It determineshow much a game’s payoff contributes to an individual’sfitness. Let fA(i) be the fitness of the i-th individual playingstrategy A. Then with a game payoff of πA(i), the fitness ofthe i-th individual is

fA(i) = 1.0− β + βπA(i) (1)

where β is small enough to make sure fA(i) > 0. The firstterm is a background fitness. If β � 1 the selection is ‘weak’and player fitness depends very little on a game payoff. (Therationale behind weak selection is discussed in Section III.)In ‘strong’ selection β = 1 so the game payoff is the player’sfitness. With β = 0 the selection is ‘neutral’ so game payoffsdon’t affect fitness and any population changes come solelyfrom random drift forces.

3) Kin Selection: This selection method is based on thebelief people who are related to each other are more likelyto cooperate with each other. Hamilton’s rule [9] says naturalselection will favor cooperation when the relatedness r be-tween two individuals exceeds the cost-to-benefit ratio—i.e.,r > c/b.

Kin selection is often called ‘inclusive fitness’ [10]. It is animportant concept that answered a long standing question Dar-win couldn’t answer: why doesn’t natural selection eliminateworker ants because they don’t produce offspring. Supposefitness is defined simply as a measure of the degree to whichan individual’s genes are passed on to the next generation.Two related individuals share genetic material so either onehaving offspring would pass genes. Hamilton believed the totalfitness of one individual was actually composed of two distinctthings: direct fitness and indirect fitness. The former comes

from an individual having offspring whereas the latter comesfrom a relative having offspring. If individual i cooperatedwith a related individual j, and j produced offspring as aresult of that cooperation, then the indirect fitness of i shouldincrease (which results in a total fitness increase). The indirectfitness contribution, however, was weighted by the degree ofrelatedness.

C. Update Rules

Update rules determine how strategies proliferate or declinein a population. There are four update rules that frequentlyappear in the literature.

1) Birth-Death (BD): Two individuals are randomly se-lected from the current population. The first individual isselected proportional to fitness whereas the second individualis chosen with equal probability. The offspring of the firstindividual replaces the second individual.

2) Death-Birth (DB): A randomly selected individual fromthe current population dies and is replaced by one of hisneighbors. The neighbors compete to provide the replacement(e.g., via roulette wheel selection).

3) Imitation (IM): An individual is randomly selected fromthe current population. This individual competes against itsneighbors (proportional to fitness) and replaces his currentstrategy with the winner’s strategy. The individual may keephis current strategy only if he wins the competition.

4) Replicator Dynamics: In well mixed populations thereplicator dynamics show how the population changes overtime. These dynamics are represented as a differential equationof the strategy frequencies. With n strategies the populationstate in any round is ~x = (x1, . . . , xn). Then the replicatorequation is

xi = xi (fi − 〈x〉) (2)

where 〈x〉 =∑n

i=1 xifi is the average population fitness.Mutation is not allowed so only frequency dependent selectiondetermines any changes.

Ohtsuki and Nowak [7] studied replicator dynamics ongraphs. They defined an n × n matrix B = [bij ] that was afunction of the payoff matrix A = [aij ]. This function varieddepending on whether DB, BD or IM updates were used. Then,with gi =

∑nj=1 xjbij the replicator equation on a graph is

xi = xi (fi + gi − 〈x〉) (3)

D. Reciprocity Forms

Reciprocity uses past interactions with opponents to influ-ence current decisions on whether to cooperate or defect.

a) Direct Reciprocity: This form is used whenever re-peated interactions occur between the same two individuals.The main idea is if my opponent has cooperated with me inthe past, then I’m more likely to cooperate with him in thecurrent round. The quintessential example of direct reciprocityis Axelrod’s tit-for-tat strategy.


b) Indirect Reciprocity: This reciprocity form does notrequire any previous interaction between two specific players.It relies instead on reputation. A player who has cooperatedwith others in the past is more likely to cooperate in the future.Players who cooperate often get good reputation and a goodreputation increases the likelihood that future opponents willcooperate with you.

c) Network Reciprocity: This form of reciprocity is usedin network games. Players only play against their neighbors,which are defined by adjacency in the underlying undirectedgraph. These interactions are fixed and therefore repeat withevery new round. This reciprocity form is often combined withdirect reciprocity.

III. DISCUSSION

It is often not possible to get large amounts of observeddata from physical systems. One solution to this problem isto construct a mathematical model that accurately capturesthe dynamics of the physical system. These dynamics areexpressed in either differential equations or difference equa-tions. Researchers can then synthesize additional data under avariety of environmental conditions—assuming the underlyingmathematics of the model are accurate.

All models make simplifying assumptions. Sometimes cer-tain dynamics, such as non-linearities, can be neglected with-out producing any noticeable effects. In other cases assump-tions must be made because certain dynamics are just notknown and therefore cannot be mathematically expressed.Virtually all of the models that have appeared within the lastdecade or so rely on variations of a DB or BD update, whichbasically implement a Moran process [11] in graphs or spatialenvironments. Part of the appeal for these updates rules is theyare trivial to program and can be justified because the Moranprocess is widely accepted in the evolutionary biology field.

Nevertheless, validation is still absolutely essential and noclearer evidence is needed than the results from some recenthuman experiments conducted by Traulsen et. al [5]. Co-operators always eventually disappear in well-mixed iteratedprisoner’s dilemma (IPD) games but previous studies withcomputer models indicated in spatial games cooperators wouldpersist in small clusters. Traulsen wanted to see if that holds inhuman experiments so they constructed a human experimentto mimic the computer models. Humans were placed in avirtual lattice, which restricted interactions to the four nearestneighbors. Their results differed the theoretical models inthree significant ways. First, the human experiment showedreassigning players in the neighborhoods each round producedno significant changes in cooperation levels, which suggestsnetwork reciprocity may actually contribute little or nothingto cooperation levels. Second, they found humans randomlychange strategies at much higher probabilities than assumedby the theoretical models. Finally, the selection intensity β(described below) was quite high, which led the authorsto conclude any analytical results based on weak selectionassumptions don’t apply.

Unfortunately, researchers don’t try to validate but insteadspend their time conducting detailed mathematical analysis oftheir models and then positing conditions under which cooper-ation prevails. But unvalidated predictions cannot and shouldnot be accepted unequivocally because they are unproven.

Many researchers have adopted the philosophy that a modelisn’t credible unless it has gone through a rigorous mathemat-ical analysis. Virtually every proposed mathematical modelof cooperation has been analyzed to determine its fixationprobability. The fixation probability ρA is defined as theprobability a mutant strategy A will eventually take overa population of B strategies. If natural selection favors nostrategy than the fixation probability of any neutral mutant is1/N (in a population size of N ). This analysis is meticulouslyconducted resulting in rather elaborate equations for ρ [1].

Authors usually avoid giving actual values for ρ but oneexception is Ohtsuki et al. [12] who came up with a simple rulefor when natural selection would favor cooperation in graphs.They ran a series of simulations with lattices, random regulargraphs, random graphs and scale-free networks for values ofk between 2 and 10 with DB updating. If ρC > 1/N thenthe probability of cooperative fixation exceeds that achievedvia neutral mutation. If, in addition, ρC > ρD, then naturalselection favors cooperation over defection. Using an IPDpayoff matrix they found ρC > 1/N > ρD if b/c > kwhere k is the average number of neighbors1. This result holdsunder weak selection and if N is much larger than k. Theyran numerous simulations and obtained good agreement withtheir simple rule. ρC was determined by counting the numberof runs, out of 106, where cooperators reached fixation underweak selection (β = 0.01). Population sizes of N = 100 andN = 500 were studied. Some deviation from the rule wasobserved for random and scale-free graphs, but that was tobe expected because the simple rule was actually derived forregular graphs.

The problem is the size of ρC . While it was true ρC > 1/Nif b/c > k in the graphs, ρC was not all that much biggerthan 1/N . For N = 100 the fixation probability of neutralevolution is 1%. For a regular lattice with k = 4 and b/cabove 5 it turns out ρC < 1.2%. Put another way, thereis greater than a 98.8% probability cooperation will notfixate in a spatial game with Von Neumann neighborhoods!The difference between ρC and neutral mutation gets evensmaller as N grows. An objective observer would conclude theprobability of cooperative fixation in Von Neumann neighborsunder DB updating is not much better than what is achievedwith neutral mutation.

Fitness can mean many things. In the context of evolutionarygames high fitness could mean several things: increased lifes-pan of the individual; larger accumulated wealth; increasedfrequency in a population; or greater reproductive success.A fairly large number of papers use weak selection becauseit greatly simplifies the mathematical analysis of the model.Some authors claim the analysis obtained from weak selection

1equivalently the average node degree in the graph.


extends to strong selection as well. Nevertheless, ease inmathematical analysis is not a compelling argument if itdistorts the evolutionary process.

Random drift and selection are opposing evolutionaryforces. Researchers control this interplay by incorporating aselection intensity parameter β into their models. Eq. 1 showshow β influences fitness. The game payoff has almost no effectunder weak selection. Another common method uses β toamplify any game payoff differences. For example, consider apopulation evolving under a BD process. Let ∆π = πA − πB

be the fitness difference between two individuals A and B. Theprobability p that A replaces B is determined by the Fermifunction [13] from statistical physics

p =1

1 + e−β∆π(4)

.Strong selection has a large β while a small value representsweak selection. Under weak selection ∆π has very little effecton p. Specifically, β → 0 ⇒ p → 1/2, which is consideredneutral selection.

Consider the IPD payoff matrix

( C D

C b− c −cD b 0

)(5)

Assume the benefit b = 1 and choose c � 1, which meansthere is little or no cost for cooperating. This situation makesp roughly equivalent to neutral selection because πC ≈ πD.Essentially this eliminates the social dilemma because the onlything that matters is the opponent’s choice. Weak selectionproduces the same effect because it also makes p → 1/2.Weak selection can actually alter the dynamics of the game.

This gives rise to a philosophical argument against weakselection. If weak selection reduces or eliminates any gain,what is the point of cooperating? Yet weak selection isubiquitous in computer models. Why?

The recent paper by Fu et al. [14] is representative ofwhere some of the confusion surrounding weak selection iscoming from. There are three problem areas that surface fromstatements made in that paper.

1) inclusive fitness requires weak selection

Fu et al. [14] say the following:“Indeed, inclusive fitness analysis exclusively relieson the assumption of weak selection”

and they cited the Taylor et al. [15] work as evidence.This claim of exclusivity is an overstatement, which becomesobvious by looking at what Taylor and his colleagues actuallydid. All evolution was done via a BD or DB process, whichhas not been observed in either the animal world or humanexperiments. All graphs also had to be transitive. Some ex-ample graphs given in [15] include ring graphs, graphs withinterconnected pentagonal cycles and island structures withdemes of size 4—hardly structures depicting likely interactions

found in real-world situations. Finally, the payoff matrix formwas

( C D

C a11 a12

D a21 a22

)(6)

with the restriction that a11 + a22 = a12 + a21. (Eq. 5 showsa payoff matrix like this.) Weak selection was achieved bymaking b and c small. But that choice blurs the distinctionbetween cooperation and defection, which eliminates anyadvantage of cooperating.

2) lack of model validation

Fu, et al. state“For weak selection, population configurations arenearly independent of the game but remain affectedby the underlying graph topology.”

This points out another major flaw in the widespread useof weak selection: model validation. If weak selection makesgraph topology the primary influence on strategy proliferation,then it becomes imperative that real-world examples of groupinteractions with connectivity like these graphs are identified.Otherwise any analytical results corresponding to these graphgames have no relevance to the real world. This importantstep has been neglected in current research and needs to beaddressed in any future work on the subject.

3) mixed messages

This particular problem area is perhaps the most devastatingone related to weak selection: mixed messages on when itshould be used. Recently researchers have tried to be on bothsides of the fence on this issue. For instance, Fu et al. [14]claim

“For weak selection (w � 1), the payoff obtainedfrom game interactions makes a small contributionto the overall fitness of an individual. This situationcan be justified in two ways: (i) the results derivedfrom weak selection often remain as valid for largerselection strength... (ii) the weak selection limit hasa long tradition in theoretical biology...”

A year earlier Langer et al. [16] published a paper on thespatial invasion of cooperation. That investigation requiredrunning a large number of simulations. They stated

“ ... under weak selection random drift dominates,which makes it much harder to extract characteristicfeatures of the evolutionary process. In contrast, ourapproach based on strong selection facilitates clear-cut conclusions.”

Obviously these two papers express completely oppositeviewpoints on the efficacy of weak selection. What’s inter-esting is two of the co-authors of [14] were also co-authorsof [16] and are widely published in the field. You can’t have itboth ways; either weak selection does provide some tangiblebenefit in evolutionary game theory studies or it doesn’t.


Some researchers claim weak selection is okay to use be-cause any results apply as well to stronger selection intensities(e.g., see quote from Fu et al. earlier in this section.) But Wuet al. [17] claim just the opposite is true.

“ ... under the assumption of weak selection, someinteresting results can be obtained analytically. It hasto be pointed out that these results do, in general,not carry over to stronger selection.”

In the Traulsen et al. [5] human experiments in spatialenvironments data fitting yielded a value of β > 1. The authorsthus concluded

“... β is also so high that analytical results obtainedunder weak selection may not always apply.”

Researchers appear to be split on the effectiveness of weakselection. However, the onus is on the proponents of weakselection to provide evidence it is a viable (and realistic)selection method. With field data existing to do just this, thecommon use of weak selection should be re-evaluated beforefurther conclusions are made.

The update rules in common use are unrealistic for a numberof reasons particularly in the context of human populations.The problem is both DB and BD updates are just too simplisticto mimic the way humans make decisions—particularly whenthe payoffs involve money. For example, in the Traulsen etal. human experiments payoffs were in euros and players couldkeep whatever they won at he end of the experiment [4]. Thisprovided players with the incentive not to contribute, whichcan skew DB and BD payoffs. In these simplistic update rulesonly one out of N players is updated each round; the otherN − 1 players are forced to make the same choice they madein the previous round. And even when updates do occur it isstrictly by chance because the player chosen to die is pickedrandomly. Humans don’t make decisions that way.

Self-interested players don’t have their strategy equal theiraction. Each action has an underlying reasoning behind it.Humans take a number of things into consideration beforemaking a final decision. For example, a human player mightconsider the amount of money already accrued; the risk ofcontributing in the next round; how their opponents haveplayed in recent rounds or anticipation of how opponentsmight play in future rounds. These factors are consideredfrequently, possibly in every round. DB and BD updates don’t(can’t) take any of these factors into consideration.

Several models evolve the population by replacing an in-dividual’s strategy with a randomly chosen neighbor (e.g.,[18]–[20]). This kind of update rule makes almost no sense.Consider an iterated spatial game involving humans withpairwise interactions. A focal player who is thinking aboutadopting a different strategy either outright knows or can, ata minimum, deduce what his neighbors’ strategies are. Theplayer has been interacting with the neighbors for severalrounds and payoffs are awarded via a payoff matrix after eachround. He therefore can deduce future payoffs by adoptingeach neighbor’s strategy and choosing the one that wouldwork best. There is absolutely no need to randomly chooses a

neighbor and then imitate that neighbor’s strategy with someprobability. Switching strategies via some random process—which is precisely what these update rules do—ignores thecognitive ability human players possess. That is, by assumingevery player uses the same method to decide when to switchstrategies assumes all players have the same ability. Clearlythat is not the case because some players are just more skillfuland thus win more often.

Computer simulations showed clusters of cooperators couldform in spatial prisoner’s dilemma games, which suggestsnetwork reciprocity contributes to cooperation. But in thesesimulations used unconditional IM updating. Humans don’talways unconditionally imitate neighbors regardless of howhigh the payoff differences are. The Traulsen et al. [5] exper-iments substantiated this. Still IM updating is appealing andcomputationally easy to do so some models adopt the Fermifunction in Eq. (4) to cast imitation with some probability.

The problem with the Fermi function is the probabilityof imitation is still tied to payoff differences. Presumablythe higher the payoff difference the higher the probabilityimitation occurs. But this ignores the fact humans have cog-nitive abilities and sometimes it doesn’t make sense to imitatea neighbor’s strategy—even if that neighbor has accrued asignificantly higher payoff. An example will help fix ideas.Consider a moderate size population of N = 25 individualsin a square lattice with Von Neumann neighborhoods. Payoffsare awarded according to a snowdrift payoff matrix

( C D

C b− c/2 b− cD b 0

)(7)

Suppose a focal player is surrounded by defectors butthe east neighbor has all C neighbors (including the focalplayer). Over the course of many rounds this east neighborwould accrue a maximal payoff. However, the focal playercan see imitating his east neighbor’s strategy (defect) wouldyield lower future payoffs. Therefore he would not imitate hisneighbor’s strategy but continue to cooperate.

Nowak et al. [21] suggested social learning as a secondinterpretation of DB updating. The idea is a random individualdecides to change his strategy and so replaces it with aneighbor’s strategy chosen proportional to fitness. But eventhat interpretation is problematic because, as stated above,there is no reason to choose neighbors with some probability.

Hauert and Doebeli claim that population structure inhibitscooperation in snowdrift games [18]. They conjecture thereason is in snowdrift games the payoffs are such that in wellmixed games it is best to pick a different strategy than youropponent. However, this strategy actually prevents formingclusters of cooperators in spatial games. There is anothermore fundamental reason why cooperation is difficult to formin human spatial experiments and it ties directly back tothe update rules. Referring back to Figure 2(a), consider thefocal player and his neighbor to the west. Suppose this westneighbor has a considerably higher accumulated payoff, which


might make the focal player consider adopting that neighbor’sstrategy with a high probability. The problem is this neighbor’saccumulated payoff was largely acquired by interacting withhis neighborhood—and the focal player is only one of hisfour neighbors. The strategy the west neighbor uses may begreat in his neighborhood but it may be awful in the focalplayer’s neighborhood where different interactions take place.No rational player would think about imitating someone else’sstrategy without first checking to see how it might perform inhis own neighborhood. Yet, the update rule mentioned abovetake none of that into consideration.

Another problem with spatial games is DB and BD updatescannot be implemented without abandoning the defined popu-lation structure. These update rules can only be implemented ifglobal communication is allowed, which effectively transformsa spatial game into a well-mixed game and a network gameinto completely connected graph.

a

b

d

cx

Fig. 3. Example of a DB update. Player x is selected to die. The VonNeumann neighborhood of x consists of the nodes marked {a, b, c, d}, whichcompete according to fitness to replace x.

Figure 3 shows a DB update. Suppose individual x, shownwith the dashed border, is selected to die. His neighbors thencompete to pick the replacement. But exactly how does thiscompetition take place? A line of communication (LOC) onlyexists between a focal player and his nearest neighbors. Thereis no LOC between say a and d because they aren’t nearestneighbors of each other. Without any LOC between themthere is no communication path to conduct the competition.This constraint holds for both Moore neighborhoods and VonNeumann neighborhoods. Competition can occur only if theneighborhood of x temporarily becomes well-mixed. It isreasonable to ask why this mixture isn’t permitted duringnormal rounds of play. That would allow neighborhoods tocollaborate and possibly agree to form a local cooperativecluster. It seems unfair to create neighborhood LOCs onlywhen updating strategies.

BD update rules must ignore the population structure aswell. Suppose individual x is selected to die and an individual

y—residing on the other side of the lattice—provides thereplacement strategy. How does a clone of y’s strategy getpassed to x if only local interactions are enforced? Here theentire population must be well-mixed at least long enough topass the strategy.

x y

Fig. 4. Example of indirect reciprocity in a Von Neumann neighborhood.The neighborhood of x consists of y and the red nodes. The neighborhood ofy is x and the blue nodes. Notice the payoffs x and y receive each round areonly partially determined by their pairwise interaction with each other. Forinstance, most of the payoff x receives comes from interaction with his redneighbors, which are not accessible to y.

IV. SUMMARY & RECOMMENDATIONS

The recent work by Traulsen et al. [5] is singularly impor-tant. The research community now has field data to validatespatial computer models. But perhaps even more important isit exposes flaws in the design of current computer models andthe possible misdirection of research effort.

First is this obsession with obtaining analytical resultsfor computer models. Models aren’t deemed credible unlessthey have been mathematically analyzed to obtain equationsfor long-term evolutionary behavior such as fixation times.Weak selection is assumed—i.e., β � 1. Although there isa genetic basis for weak selection, the primary reason theanalysis depends on it is because weak selection simplifies thecalculations. But the human experiment results found β wasslightly larger than 1.0, which is 2-3 orders of magnitude largerthan weak selection values frequently reported. This value isalso large enough to show humans do not unconditionallyswitch to a strategy that outperform their current strategy asrequired by IM updating. β is also large enough to cast seriousdoubt on the continued use of weak selection.

Weak selection was not observed anywhere in these humanexperiments. The advocates of weak selection claim theiranalytical results apply at larger selection intensities but, aspointed out in the previous section, there is clearly no consen-sus. The weak selection advocates now have an opportunity toend this debate. They can use the selection intensities reportedin [5] in their models and see if those models predict the resultsfrom the human experiments. In other words, with field datanow available the models analyzed under weak selection canundergo validation. If the predictions are reliable, then that


will prove weak selection analytical results apply at higherselection intensities2.

The IM, DB and BD updates are largely too simplisticto explain cooperation in human populations. Humans don’tsimply unconditionally accept better behaving strategies (IMupdate) nor do they blindly accept being forced to switchstrategies (DB and BD updates). Again the Traulsen et al.experiments provide some interesting perspectives. Althoughthey observed a player’s probability of cooperating increasedas more of his neighbors cooperated, this probability was lessthan 50%—even when all of the neighbors cooperated. Theyconcluded players tend to imitate what is common but alsothat humans make decisions in more complex ways. The IM,DB and BD updates can’t provide any insight into what thesemore complex ways might be. In particular, they all ignore thehuman cognitive ability to assess the effects of switching toanother strategy and making a final decision accordingly.

Section III discussed why spatial environments under DB orBD updating requires periodic structural changes—i.e., addingnew LOCs—or they won’t work. The spatial structure is finefor studying direct or indirect reciprocity, but some have begunto question if network reciprocity plays any role in socialsystems [4]. Traulsen et al. [5] did not find any significanteffects on cooperation levels. This may be due in part to thespatial structure itself. As illustrated in Figure 4, if player y isa neighbor of player x, the total payoff of y is mostly acquiredby interacting with players x does not interact with. Thus theneighborhood structure inhibits network reciprocity effects.

It is not obvious why fixation probability remains so im-portant in computer models. Yes, fixation is inevitable in aMoran process, but the update rules that produce that behaviorhave no relevance in human populations. Fixation is scarce inspatial games. Traulsen et al. did not see it in their humanexperiments. Hauert and Doebeli [18] showed spatial structureinhibits cooperation in Snowdrift games. Natural selection mayfavor cooperation when b/c > k, but it apparently is notsuccessful very often. Indeed, Ohtsuki et al. [12] found ρC

to be only slightly higher than the fixation probability withneutral evolution. Further studies in this area may not be veryproductive if these are typical fixation probability value sizes.

Finally, West et al. [22] recently published a paper on mythsassociated with cooperation research. People in the biologicalsciences and the social sciences were studying cooperationand they weren’t necessarily using the exact same definitionsfor terms like cooperation, altruism and so forth. West etal. believe this inconsistency was causing confusion. But itisn’t clear how widespread this belief is because there doesn’tappear to be any resulting negative consequences; cooperationresearch is booming. Rather than debating semantics, it mightbe more productive to work on designing more realistic modelsand validation efforts.

So what should models of human populations look like?The model currently under development has a moderate size

2As a side note, there are no analytic results for scale-free networks. Infact, this is true for all heterogeneous graphs except for star graphs [21].

population (16-20). The population is partitioned into small,well-mixed groups. Players have tags and are forced to interactwith all players with the same tag each round. These tagswill enforce local interactions. Each player has a short termmemory to allow reciprocity studies. Players may switchstrategies but this would be an individual decision and not aforced switch. Periodically a player may self-mutate his tag tomove to a new group to acquire higher profits. This new groupwould be permitted to accept new members on a probationarybasis. Any expelled player would be required to return to hisoriginal group. This type of model would give players somecognitive ability to choose their actions accordingly.

REFERENCES

[1] G. Greenwood and D. Ashlock. Evolutionary games and the studyof cooperation: why has so little progress been made? In Proc. 2012Congress on Evol. Comput., pages 680–687, 2012.

[2] A. Traulsen, M. Nowak, and J. Pacheco. Stochastic dynamics of invasionand fixation. Phys. Rev. E, 74:011909–011914, 2006.

[3] H. Ohtsuki. Evolutionary games in wright’s island model: kin selectionmeets evolutionary game theory. Evol., 64(12):3344–3353, 2010.

[4] D. Helbing and W. Yu. The future of social experimenting.Proc. Nat’l. Acad. Sci., 107(12):5265–5266, 2010.

[5] A. Traulsen, D. Semmann, R. Sommerfeld, H. Krambeck, andM. Milinski. Human strategy updating in evolutionary games.Proc. Nat’l. Acad. Sci., 107(7):2962–2966, 2010.

[6] D. Watts and S. Strogatz. Collective dynamics of ’small-world’ net-works. Nature, 393(6684):440–440, 1998.

[7] H. Ohtsuki and M. Nowak. The replicator equation on graphs.J. Theo. Biol., 243(1):86–97, 2006.

[8] G. Greenwood and S. Chopra. A numerical analysis of the evolutionaryiterated snowdrift game. In Proc. 2011 Congress on Evol. Comput.,pages 2010–2016, 2011.

[9] W. Hamilton. The genetical evolution of social behaviour. I.J. Theo. Biol., 7:1–16, 1964.

[10] M. Nowak. Five rules for the evolution of cooperation. Science,314(5805):1560–1563, 2006.

[11] P. Moran. Random processes in genetics. Math. Proc. CambridgePhil. Soc., 54(01):60–71, 1958.

[12] H. Ohtsuki, C. Hauert, E. Lieberman, and M. Nowak. A simple rulefor the evolution of cooperation on graphs and social networks. Nature,441:502–505, 2006.

[13] A. Traulsen, J. Pacheco, and M. Nowak. Pairwise comparison andselection temperature in evolutionary game dynamics. J. Theo. Biol.,246:522–529, 2007.

[14] F. Fu, L. Wang, M. Nowak, and C. Hauert. Evolutionary dynamics ongraphs: efficient method for weak selection. Phys. Rev. E, 79:046707–1–046707–9, 2009.

[15] P. Taylor, T. Day, and G. Wild. Evolution of cooperation in a finitehomogeneous graph. Nature, 447:469–472, 2007.

[16] P. Langer, M. Nowak, and C. Hauert. Spatial invasion of cooperation.J. Theo. Biol., 250:634–641, 2008.

[17] B. Wu, P. Altrock, L. Wang, and A. Traulsen. Universality of weakselection. Phys. Rev. E, 82, 2010.

[18] C. Hauert and M. Doebeli. Spatial structure often inhibits the evolutionof cooperation in the snowdrift game. Nature, 428:643–648, 2004.

[19] C. Roca, J. Cuesta, and A. Sanchez. Effect of spatial structure on theevolution of cooperation. Phys. Rev. E, 80:046106, 2009.

[20] G. Szabo, A. Szolnoki, M. Varga, and L. Hanusovszky. Orderingin spatial evolutionary games for pairwise collective strategy updates.Phys. Rev. E, 82(2):026110, 2010.

[21] M. Nowak, C. Tarnita, and T. Antal. Evolutionary dynamics in structuredpopulations. Phil. Trans. R. Soc. B, 365:19–30, 2010.

[22] S. West, C. Mouden, and A. Gardner. Sixteen common misconceptionsabout the evolution of cooperation in humans. Evol. and Human Behav.,32:231–262, 2011.


[ieee 2012 ieee conference on computational intelligence and games (cig) - granada, spain...

Documents