the evolution of cooperation in in–nitely repeated games...

The Evolution of Cooperation in In�nitely RepeatedGames: Experimental Evidence

Pedro Dal Bó Guillaume R. Fréchette

Brown University and New York University

Dal Bó and Fréchette (Brown and NYU) The Evolution of Cooperation 1 / 46

IntroductionMotivation I

A central issue in the social sciences is the tension between personalincentives and the �common good�.

Example: the prisoner�s dilemma.

How can we support e¢ cient outcomes?

Our focus will be on in�nitely repeated games.


IntroductionMotivation II

Realistic Model: in�nitely repeated games

We may meet again.Credible reward and punishments.Used to explain:

Trench Warfare in WWI (Axelrod, 1984).Tacit Collusion (Friedman, 1971).Informal Contracts (Klein and Le er, 1981).Theory of the Firm (Baker et al, 2002).Macroeconomics (Rotemberg and Saloner, 1986).


IntroductionMotivation III

Usual criticism of the theory of in�nitely repeated games: multiplicityof equilibria.

Fudenberg and Maskin (1993): �The theory of repeated games hasbeen somewhat disappointing. . . . the theory does not make sharppredictions.�


Theoretical LiteratureEquilibrium Selection in In�nitely Repeated Games

Evolutionary Stable Strategies (ESS): Axelrod (1981), Boyd andLorberbaum (1987), Boyd (1989) and Kim (1994).

Finite Automata: Rubinstein (1986) and Abreu and Rubinstein(1988).

Finite Automata plus Evolutionary Stable Strategies: Binmore andSamuelson (1992), Cooper (1996) and Volij (2002). Also Fudenbergand Maskin (1990, 1993).

Finite Automata plus Stochastic Stability: Volij (2002).

Random matching games and Stochastic Stability (Johnson, Levineand Pesendorfer 2001 and Levine and Pesendorfer 2007).

Summary of results: E¢ ciency is selected, or not.


Objective of this paper

When will cooperation arise in in�nitely repeated games?

Vary two theoretically relevant parameters.

Under what conditions will cooperation increase with experience?

Use a design which will allow subjects to gain experience.


Main Results

Cooperation being an equilibrium action is necessary but not su¢ cientfor subjects to learn to cooperate.

A more stringent condition (risk-dominance) is not su¢ cient either.

Subjects do learn to cooperate under very favorable conditions.


Previous Experimental LiteratureFirst Generation

Roth and Murnighan (1978, 1983) �PD.

Holt (1985) �Cournot duopoly.

Feinberg and Husted (1993) �PD.

Palfrey and Rosenthal (1994) �VCM.


Previous Experimental LiteratureResults from these early experiments

Cooperation % in the �rst roundδ

0 0.105 .5 .895 .9Roth & Murnighan (1978): 19 30 36Murnighan & Roth (1983): 18 37 29Palfrey & Rosenthal (1994): 28 41

�So the results remain equivocal.�Roth (HEE 1995).

�This contrast between our one-shot and repeated play results is notencouraging news for those who might wish to interpret as gospel theoft-spoken suggestion that repeated play with discount rates close toone leads to more cooperative behavior. True enough it does-but notby much.�Palfrey and Rosenthal (1994).


Previous Experimental LiteratureRecent Work

Dal Bó (2005) �PD: �nitely versus in�nitely repeated games.

Du¤y and Ochs (2004) �PD: cooperation with random matching.

Aoyagi and Fréchette (2004) �PD: cooperation with imperfectmonitoring.

Engle-Warnick and Slonim (2004, 2006) �Trust Game: identifystrategies.

Stahl (2008) �PD: reputation mechanism with random matching.

Camera and Casari (2008) �PD: reputation mechanism with randommatching.

Blonski, Ockenfels, and Spagnolo (2007) �PD: determinants ofcooperation.


Previous Experimental LiteratureResults from Recent Work:

Increasing the continuation probability increases cooperation. Forinstance Aoyagi and Frechette

Other comparative static predictions of the theory �nd support.

Increasing the noise in a public signal decreases average payo¤s (AF).Random termination games result in higher cooperation rates than�nitely repeated games (Dal Bo).

However, repeated games with random rematching are not assuccessful (Du¤y and Ochs).





Increasing the noise in a public signal decreases average payo¤s (AF).

Random termination games result in higher cooperation rates than�nitely repeated games (Dal Bo).






Increasing the noise in a public signal decreases average payo¤s (AF).Random termination games result in higher cooperation rates than�nitely repeated games (Dal Bo).



Previous Experimental LiteratureWhat�s Di¤erent Between the Early Experiments and the Recent Ones?

The early literature focussed on 1 play of the repeated game.

The more recent experiments play more than 1 repeated games.

Dal Bo: 6 to 10 per treatment.Aoyagi and Frechette: 6 to 10 per session.

Palfrey and Rosenthal: �The single play results are open to thecriticism that subjects lacked su¢ cient task experience. In fact, taskinexperience has been demonstrated to be an important factor inexplaining cooperative play.�


Our New Experimental DesignTreatment Variables

Simple PD Games:C D

C R , R 12 , 50D 50 , 12 25 , 25

Three payo¤s from cooperation: R = 32, 40 and 48.

Two continuation probabilities: δ = 1/2 and 3/4.


Theory Background

Can cooperation be supported in equilibrium?

R = 32 R = 40 R = 48δ = 1/2 NO YES YESδ = 3/4 YES YES YES

QUESTION 1: Do subjects learn to defect when it is the onlyequilibrium action?

QUESTION 2: Do subjects learn to cooperate when it is anequilibrium action?


Theory Background

Lessons from coordination games: subjects may fail to play thee¢ cient equilibrium.

Example:A B

A 1000 , 1000 0 , 800B 800 , 0 800 , 800

Both (A,A) and (B,B) are NE.

(A,A) is the e¢ cient one but subjects tend to play (B,B).

Cooper et al 1992: 160 (B,B) action pairs vs 5 (A,B) or (B,A) actionpairs.Similar results in Cooper et al 1990 and Van Huyck et al 1990.

Risk-dominance (RD) selects (B,B)

Which strategy is optimal against a 50-50 strategy of the opponent?


Theory Background

Risk-Dominance with in�nite number of strategies is complicated.

Focus on the �ultimate� cooperative and non-cooperative strategies:

Grim (G): Start cooperating and then cooperate unless there has beena defection in the past.Always Defect (AD).


Theory Background

Can cooperation be supported in a RD equilibrium?

R = 32 R = 40 R = 48δ = 1/2 NO NO YESδ = 3/4 NO YES YES

QUESTION 3: Do subjects learn to cooperate when it isrisk-dominant?


Sessions�Characteristics

We conducted 18 sessions with NYU undergraduates: 3 sessions pertreatment.

Total of 266 subjects, average of 15 subjects per session.

Earnings: min=$16.29, max=$42.93, average=$25.95.

Number of repeated games: min=23, max=77, average=51.


Experimental Details

One treatment per session (between subject design).

Supergames (referred to as matches) are randomly terminated butthey all last the same number of rounds (stage game) for every one ina given session.

Random rematching between matches.

First match after 50 minutes of play is the last.

Forced pause between rounds.

No context / framing (labels 1 and 2).

Everyone chooses as a row player.


Experimental ResultsGeneral Description

Cooperation in �rst round of the �rst repeated game:

δnR 32 40 481/2 34.09 <* 54.00 < 56.52

= _ ^3/4 34.09 < 36.84 <* 56.82

Note: * signi�cance at 10%, ** at 5% and *** at 1%.

In �rst repeated game:

Cooperation does not necessarily increase with δ or R.Cooperation is not signi�cantly larger when it can be supported inequilibrium.


Experimental ResultsGeneral Description

Cooperation in �rst round of all repeated games:

δnR 32 40 481/2 9.81 <*** 18.72 <*** 38.97

^*** ^*** ^***3/4 25.61 <*** 61.10 <*** 85.07

Note: * signi�cance at 10%, ** at 5% and *** at 1%.

In all repeated games:

Cooperation does increase with δ and R.Cooperation is signi�cantly larger when it can be supported inequilibrium.


Experimental Results

QUESTION 1: Do subjects learn to defect when it is the only equilibriumaction?

Evolution of Cooperation (�rst rounds) when it is not an equilibriumoutcome (δ = 1/2 and R = 32)

0.1

.2.3

.4.5

.6.7

.8.9

1C

oope

ratio

n R

ate

0 1 2 3 4 5 6 7 8 9 10 11 12 13Interaction Group

Answer: YES!Dal Bó and Fréchette (Brown and NYU) The Evolution of Cooperation 22 / 46


QUESTION 2: Do subjects learn to cooperate when it is anequilibrium action?Evolution of Cooperation (�rst rounds) when it is an equilibriumoutcome (all but δ = 1/2 and R = 32)

0.1

.2.3

.4.5

.6.7

.8.9

1C

oope

ratio

n R

ate

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14Interaction Group

Looking at aggregate data: YES, but far away from full cooperation.



QUESTION 2: Do subjects learn to cooperate when it is anequilibrium action?

Evolution of Cooperation (�rst rounds) by Treatment: Graph .

Answer: Not necessarily.



QUESTION 3: Do subjects learn to cooperate when it isrisk-dominant?Evolution of Cooperation (�rst rounds) when it is risk-dominant

0.1

.2.3

.4.5

.6.7

.8.9

1C

oope

ratio

n R

ate

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14Interaction Group

RD

Not RD

Looking at aggregate data: YES, but far away from full cooperation.



QUESTION 3: Do subjects learn to cooperate when it isrisk-dominant?

Evolution of Cooperation (�rst rounds) by Treatment: Graph .

Answer: Not necessarily.

When cooperation is risk-dominant subjects may reach fullcooperation!


Summary of Experimental Results

When cooperation is not an equilibrium action, subjects learn todefect reaching one-shot levels.

Subjects do not necessarily learn to cooperate when it is anequilibrium action or even when it is risk-dominant.

Subjects may learn to cooperate and reach full cooperation when it isa risk-dominant equilibrium.

Out of time


Additional Observations

Beyond the treatment e¤ects, it is interesting to understand whatsubjects were doing.

4 observations:

Basins of attraction Graph .Changes over time.

Beliefs about others.Length of matches.

Strategies used.


The importance of Basins of Attraction

Basins of attraction matter in determining �nal outcome.

Probit: Cooperation in Round 1

of Last Repeated Game

Coef. Clustered Coef. Clustered

Est. Std Error Est. Std Error

Size of Basin of AD -7.276*** 2.432 -013.297*** 4.076

Size of Basin Square 3.153 2.061 7.074** 3.202

SGPE 0.840 0.919

RD -1.126 0.947

Extra Length of Rep. G. 0.942*** 0.213 0.753*** 0.276

Constant 2.553*** 0.628 4.238* 2.416

Observations 266 266

Out of time!


Adaptation

Subjects who are paired with someone who starts by defecting(cooperating) are more likely to start by defecting (cooperating) inthe next repeated game.

Subjects are more likely to start by defecting following a short match.

Out of time!


Correlated Random E¤ects Probit of Round 1 Cooperation

δ = 0.5 δ = 0.75R = 32 R = 40 R = 48 R = 32 R = 40 R = 48

O. C 0.425*** 0.355*** 0.593*** 0.393*** 0.944*** 0.857***R1 PM (0.117) (0.070) (0.065) (0.105) (0.118) (0.130)No. of 0.018 0.051** 0.098*** 0.007 0.032* 0.007R PM (0.025) (0.021) (0.021) (0.012) (0.017) (0.016)C in 1.149*** 0.718*** 1.770*** 0.498 1.730*** 0.494R1M1 (0.218) (0.246) (0.471) (0.375) (0.494) (0.347)Const. -2.204*** -1.787*** -2.113*** -1.309*** -0.801** 0.541**

(0.152) (0.191) (0.364) (0.239) (0.314) (0.269)ρ 0.271*** 0.399*** 0.702*** 0.553*** 0.655*** 0.502***

(0.063) (0.058) (0.055) (0.069) (0.067) (0.079)Obs. 2840 3534 3300 1268 1304 1376Subj. 44 50 46 44 38 44Out of time!


Impact of Supergame Length on Round 1 Cooperation

7 4 7 3 15

3 5 13

7 7 5 1110 1 3 3 2 1 6 1 8 1 2

18

2

2 7

14 9

2 24 2 3 3 7 1 1 9 2 2 10 4 1 2 6 1 8 2 1 6 6 9 5 1

1 6 22 1 2

2 9

1 52 3 2

2 35 1 11 2 1

21 8 2 1 10

31 2 1 1 6 4 4 1 3

1 1 10 1 2 4 4 1 1 2 8

0.5

10

.51

0.5

1

0 50

delta=.75 r=40

delta=.75 r=40

delta=.75 r=40

Coo

pera

tion

SupergameGraphs by date

In Blocks of 5 , Out of time!


Categories of Behavior

By simply focussing on two types of strategy, much of the data canbe organized.

Randomδ R Always D Grim Total Baseline Out of

32 82% 6% 88% 33% 28841/2 40 67% 11% 78% 34% 3584

48 50% 29% 78% 34% 334632 58% 11% 69% 15% 1312

3/4 40 22% 46% 68% 22% 134248 8% 71% 80% 18% 1420

Average 51% 27% 78% 27% 13,888

Out of time!


Strategies Over Time

R = 32 R = 40 R = 48

AD G Other AD G Other AD G Other

δ = 12 AD 0.93 0.04 0.04 AD 0.87 0.06 0.07 AD 0.88 0.06 0.06

G 0.45 0.36 0.19 G 0.33 0.50 0.17 G 0.12 0.75 0.13

Other 0.56 0.30 0.14 Other 0.46 0.29 0.24 Other 0.29 0.50 0.21


δ = 34 AD 0.82 0.04 0.14 AD 0.71 0.08 0.21 AD 0.49 0.20 0.31

G 0.26 0.43 0.31 G 0.03 0.78 0.19 G 0.02 0.89 0.09

Other 0.31 0.25 0.44 Other 0.24 0.42 0.34 Other 0.14 0.46 0.40


Limit Distributions

R = 32 R = 40 R = 48


δ = 12 AD 0.87 0.07 0.06 0.75 0.15 0.10 AD 0.57 0.33 0.10

δ = 34 AD 0.61 0.15 0.24 0.24 0.53 0.23 AD 0.07 0.78 0.15


Conclusion

Long lasting criticism of the theory of in�nitely repeated games:multiplicity of equilibria.

Many theories, we provide experimental evidence:

Cooperation being an equilibrium action is necessary but not su¢ cientfor subjects to learn to cooperate.Risk-dominance is not su¢ cient either.Subject do learn to cooperate under very favorable conditions.

Implication for equilibrium selection:

Assuming that agents will cooperate when possible is not appropriate.Similarly for selection theories that yield defection even for patientagents.First step towards a new theory where initial behavior, basins ofattraction and noise matter.


Next Step

Elicitation of strategies.

Limit to two stage machines: if then statements.Give them experience without machines.

Can determine:

If two stage machines are enough.If they are, what strategies subjects use.If elicitation method a¤ects how they play.


Next Step


Limit to two stage machines: if then statements.

Give them experience without machines.

Can determine:



Next Step



Can determine:



Next Step



Can determine:

If two stage machines are enough.

If they are, what strategies subjects use.If elicitation method a¤ects how they play.


Next Step



Can determine:

If two stage machines are enough.If they are, what strategies subjects use.

If elicitation method a¤ects how they play.


Next Step



Can determine:



What we Have So Far

2 pilots (R = 48, δ = 34 )

Similar overall evolution in the part prior to the introduction of thestrategy elicitation.

In the �rst match, 75% of period 1 decisions are cooperateBy the end of phase I, cooperation has increased to 94% in round 1.

Specifying their strategy does not seem to have an impact on theirchoices.T

The rates of round 1 cooperation remain comparable at the end ofphase II of the new sessions and after a similar length of experience inthe original data (98% vs 100%).

It seems like a two states machine is su¢ cient to express theirstrategies in this environment.

In 90% of all 638 matches in phase II, decisions corresponds to thosethe machine would have taken, that is 90% of 2,222 decisions.This would be 27% for random machines.


What we Have So Far

2 pilots (R = 48, δ = 34 )


In the �rst match, 75% of period 1 decisions are cooperate

By the end of phase I, cooperation has increased to 94% in round 1.






What we Have So Far

2 pilots (R = 48, δ = 34 )








What we Have So Far

2 pilots (R = 48, δ = 34 )






In 90% of all 638 matches in phase II, decisions corresponds to thosethe machine would have taken, that is 90% of 2,222 decisions.

This would be 27% for random machines.


What we Have So Far

2 pilots (R = 48, δ = 34 )








The Machines

Always AlwaysCooperate Defect TFT Grim

Match 1 0.045 0.023 0.364 0.250Last Match 0.045 0.023 0.364 0.341


Previous Experimental LiteratureResults from Recent Work: Aoyagi and Frechette

.2.4

.6.8

0 2 4 6 8 10Cycle

beta = 0 one-shotLinear Fit Linear Fit

Cooperation Rate

Go back


0.1

.2.3

.4.5

.6.7

.8.9

1C

oope

ratio

n

0 20 40 60 80Repeated Game

Neither SGPE nor RDdelta=.5 r=32

0.1

.2.3

.4.5

.6.7

.8.9

1C

oope

ratio

n


SGPEdelta=.5 r=40

0.1

.2.3

.4.5

.6.7

.8.9

1C

oope

ratio

n


SGPE & RDdelta=.5 r=48

0.1

.2.3

.4.5

.6.7

.8.9

1C

oope

ratio

n


SGPEdelta=.75 r=32

0.1

.2.3

.4.5

.6.7

.8.9

1C

oope

ratio

n

0 10 20 30 40 50Repeated Game


0.1

.2.3

.4.5

.6.7

.8.9

1C

oope

ratio

n



Go back to question 2 , Go back to question 3 , Go back to Observations


Impact of Supergame Length on Round 1 Cooperation

7.2

76 2.6 3.666667

4

3.63.8 3 4.6 3.6 5.25

2.4 3.8 2.4 4 2.8 3.4 3.2 3.2 2.4 5

0.5

10

.51

0.5

1

0 5 10

delta=.75 r=40

delta=.75 r=40

delta=.75 r=40

Coo

pera

tion

Blocks of 5 Repeated GamesGraphs by date

Go Back


A Learning Model

Beliefs: Pit (ajt = G ) =βGit

βGit+βADit,

where βai1 � 0, βait+1 = θi βait + 1(at = a) and θi 2 [0, 1].

Decisions: P(ait = G ) = e1

λitEUit (G )

e1

λitEUit (G )+e

1λit

EUit (AD ),

where λit = λFi + φt�1i λVi and φi 2 [0, 1].

Many alternative models: main results are robust.

Go back


0.1

.2.3

.4.5

.6.7

.8.9

1C

oope

ratio

n



0.1

.2.3

.4.5

.6.7

.8.9

1C

oope

ratio

n

0 20 40 60 80Rep eated Game

SGPEdelta=.5 r=40

0.1

.2.3

.4.5

.6.7

.8.9

1C

oope

ratio

n



0.1

.2.3

.4.5

.6.7

.8.9

1C

oope

ratio

n


SGPEdelta=.75 r=32

0.1

.2.3

.4.5

.6.7

.8.9

1C

oope

ratio

n

0 20 40 60 80Rep eated Game


0.1

.2.3

.4.5

.6.7

.8.9

1C

oope

ratio

n



Go back


0.1

.2.3

.4.5

.6.7

.8.9

1C

oope

ratio

n

0 200 400 600 800 1000Repeated Game


0.1

.2.3

.4.5

.6.7

.8.9

1C

oope

ratio

n

0 200 400 600 800 1000Rep eated Game

SGPEdelta=.5 r=40

0.1

.2.3

.4.5

.6.7

.8.9

1C

oope

ratio

n

0 200 400 600 800 100 0Repeated Game


0.1

.2.3

.4.5

.6.7

.8.9

1C

oope

ratio

n

0 200 400 600 800 1000Repeated Game

SGPEdelta=.75 r=32

0.1

.2.3

.4.5

.6.7

.8.9

1C

oope

ratio

n

0 200 400 600 800 1000Rep eated Game


0.1

.2.3

.4.5

.6.7

.8.9

1C

oope

ratio

n

0 200 400 600 800 100 0Repeated Game


Go back


0.1

.2.3

.4.5

.6.7

.8.9

1P

ropo

rtion

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14Number of cooperative actions


0.1

.2.3

.4.5

.6.7

.8.9

1P

ropo

rtion


SGPEdelta=.5 r=40

0.1

.2.3

.4.5

.6.7

.8.9

1P

ropo

rtion



0.1

.2.3

.4.5

.6.7

.8.9

1P

ropo

rtion


SGPEdelta=.75 r=32

0.1

.2.3

.4.5

.6.7

.8.9

1P

ropo

rtion



0.1

.2.3

.4.5

.6.7

.8.9

1P

ropo

rtion



Go back


the evolution of cooperation in in–nitely repeated games...

Documents