6/4/03Genetic Algorithm

The Genetic Algorithm

The Research of Robert Axelrod

The Algorithm of John Holland

Reviewed by Eyal Allweil and Ami Blonder

6/4/03Genetic Algorithm

The Prisoner’s Dilemma

Player 1

Player 2

DefectCooperate

Defect:1

P(unishment)

1

0S(ucker)

5

Cooperate:5

T(emptation)

0

3R(eward)

3

6/4/03Genetic Algorithm

The Iterated Prisoner’s Dilemma (IPD)

• Two tournaments held by Robert Axelrod

• The highest average score was Anatol Rapoport’s Tit-For-Tat (TFT)

• Eight representatives explain 98% of the variance in the tournament results.

• We will also hear criticism of these claims. ( Ken Binmore)

6/4/03Genetic Algorithm

Introduction: Axelrod’s Motivation

• Axelrod wanted to prove a point: The success of Tit-For-Tat in his Iterated Prisoner’s Dilemma was not based on the preconceptions of those who submitted entries.

• The hammer and the nail.

6/4/03Genetic Algorithm

More practical motivations

• We have a group aim, and we want agents that can fulfill it. More accurately, we want our agents to be as efficient as possible in achieving this goal.

• Example: The Mars element-gathering, subsumption architecture experiment

6/4/03Genetic Algorithm

The Genetic Algorithm

1. Construct an initial population

2. Test and score population

3. Calculate number of offspring

4. Reproduction: Choose parental pairs

5. Reproduction: Crossover / Mutation

6/4/03Genetic Algorithm

Encoding a Strategy (1)

0123456 ……….. 69

0101010 . 1 . 0 . 1 ……… 0 . 1

•Axelrod encoded strategies as sequences of 70 bits. How does this work?

•Note: 0 = C, 1 = D

6/4/03Genetic Algorithm

Encoding a Strategy (2)

Move

-3 / him

Move

-3 / me

Move

-2 / him

Move

-2 / me

Move

-1 / him

Move

-1 / me

What do I do now?

0000000 = C

0010010 = C

…………………

1111111 = D

6/4/03Genetic Algorithm

1. Constructing an Initial Population

• Two possibilities for creating an initial population:

1. Initial strategies can be assembled (as in the first two tournaments) or -

2. They can be randomly created

6/4/03Genetic Algorithm

Constructing an Initial Population (cont’d)

• Axelrod used normalized population of 20

• A rule of thumb is that the product of the number in the population and the number of generations should exceed 100,000

• In addition, the number of individuals in the population must considerably exceed the number of genes in each individual's chromosome.

6/4/03Genetic Algorithm

2. Testing the Population

• In the Iterated Prisoner’s Dilemma, the outcome of each interaction is added to produce a player’s score for that generation.

• In general, an ordering function is needed.

• This function must be efficient!

6/4/03Genetic Algorithm

3. The Number of Offspring (in Axelrod’s simulation)

• Strategies that were one standard deviation above the average score produced two matings.

• Strategies that were one standard deviation below the average were barren- no offspring.

• Other strategies produced a single mating

• Each generation is disjoint from its predecessor

6/4/03Genetic Algorithm

The Number of Offspring (cont’d)

• There are other possibilities available when calculating offspring:

• Normalization / growth of population size

• Preservation of arbitrary amount of previous generation’s strategies (not done in Axelrod model)

6/4/03Genetic Algorithm

4. Choosing Parents

• In Axelrod’s simulations, pairs were chosen randomly to mate and produce, each, two offspring.

• Other possibilities exist:

• Mating by excellence (short term exploitation)

• Mating by geographic proximity

6/4/03Genetic Algorithm

5. Reproduction: Crossover

• “Crossover selects one or more places to break the parents’ chromosomes in order to construct two offspring each of whom has some genetic material from both parents.”

• Other forms of crossover are possible.

• Syntactic integrity must be preserved!

• What are the advantages of high/low crossover? (more on this later)

6/4/03Genetic Algorithm

5. Reproduction: Mutations

• In every offspring born, there is a small chance of bit reversal- a change in strategy.

• Don’t forget syntactic integrity!

• What are the advantages of high/low mutation rates?

6/4/03Genetic Algorithm

Technical Details

• Population size: 20

• Round length: 151 meetings

• Each population member met one of 8 “representatives” (not each other)

• Therefore 24000 meetings per generation

• Number of generations: 50

• 40 2-parent (sexual) experiments, 40 asexual ones

6/4/03Genetic Algorithm

Conclusions (1) : Effectiveness• “the problem for evolution can be conceptualized as a

search for relatively high points in a multidimensional field of gene combinations, where height corresponds to fitness.”

• Axelrod: TFT-like strategies are the big winner!

• The genetic algorithm produces results better than the second Axelrod tournament: 450 weighted score vs. 428 for TFT

• But these results were in 11 (out of 40) experiments which resulted in a strategy which tried to exploit “sucker” strategies! (more later)

6/4/03Genetic Algorithm

Successful Alleles: TFT-like behavior

1. Don’t rock the boat ( C after RRR )

2. Be provocable ( D after RRS )

3. Accept apologies: ( C after TSR )

4. Forget: ( C after SRR )

5. Accept a rut: ( D after PPP )

6/4/03Genetic Algorithm

Conclusions (2) : Sexual Reproduction

• In biology, sexual reproduction carries a stiff price – useless males

• Computationally, sexual reproduction is cheap.

• Asexual runs of IPD resulted in lower average scores (5 out of 11 had higher median scores than TFT)

• Parasite theory. (only if we have time)

6/4/03Genetic Algorithm

Conclusions (3) : Arbitrariness

• Hitch-hiking genes- typically present, not typically employed. (TFT example – PPR)

• Premises and their results ( TFT example – the original six bit premise )

• Two basin theory ( Ken Binmore ) : By choosing the correct premises, we can produce different (stable) evolutionary end-products

6/4/03Genetic Algorithm

Conclusions (4) : Tradeoffs

• The trade off that exists is between flexibility and specialization.

• This can be translated into short and long term gains (exploitation vs. exploration)

• Varying mutation based on how dynamic the environment is.

• Invasion

6/4/03Genetic Algorithm

Conclusions (5) : Irreversibility

• The possibility exists of getting stuck in a local maximum.

• This results from adaptation to a set of premises, which strengthens them in the future.

• There are those who claim that TFT is such a local maximum.

6/4/03Genetic Algorithm

Conclusions and Criticism

• Kristain Lindgren (1991) – found cyclical history of stability / instability

• Lombard (1996) : more extensive simulations, copy-and-innovate instead of standard genetic algorithm

• Lombard: Noise is a crucial factor

6/4/03Genetic Algorithm

More Criticism and Conclusions

• Binmore - TAT-FOR-TIT is a better strategy?

• Were more experiments called for? Probst (1996) found exploitive machines thriving after running longer experiments!

• His criticism is against the simulation results, not against the use of the algorithm in general.

• But this is a lesson that applies only in pairwise interactions. In multi-person interactions, it need not be the injured party who punishes a cheater.

6/4/03Genetic Algorithm

References

• Axelrod, Robert. The Complexity of Cooperation

• Online: Review of The Complexity of Cooperation (Ken Binmore)

• Online: Simulation for the Social Scientist (Nigel Gilbert and Klaus G. Troitzsch)