Evolving Evil: Optimizing Flocking Strategies through Genetic Algorithms for the Ghost Team
in the Game of Ms. Pac-Man
F. Liberatore (University Rey Juan Carlos)A. Mora (University of Granada)P. Castillo (University of Granada)J.J. Merelo (University of Granada)
Outline
- The Game of Ms. Pac-Man.- State of the Art.- Flocking.- Flocking Strategies for the Ghost Team.- A Genetic Algorithm for Flocking Strategies.- Experiments.- Conclusions.
What is this about?
In this work we propose:- A Controller for the Ghost Team in the game of Ms. Pac-Man …- … based on Flocking Strategies (more about this in the following) …- … optimized by a Genetic Algorithm.
The game of Ms. Pac-Man [1/2]
The game of Ms. Pac-Man:- Released on 1981.- Chosen for the Ms. Pac-Man vs Ghosts competition.- Participants can submit controllers for Ms.Pac-Man and the Ghosts.- The objective of Ms. Pac-Man is to maximize the final score.- The objective of the Ghosts is to minimize it.
The game of Ms. Pac-Man [2/2]
Movement restrictions for the Ghosts:- A ghost can never stop.- A ghost can choose its direction only at a junction.- A ghost cannot turn back on itself.- At every tick of the game all the ghosts obligatorily reverse their direction according to a small random probability.
State of the Art [1/2]Most contributions proposed controllers for Ms. Pac-Man:- Genetic Programming (Alhejali and Lucas 2010, 2011, Brandstetter and Ahmadi 2012),- Genetic Algorithms for parameters optimization (Thawonmas 2010),- Evolved Neural Networks (Burrow and Lucas 2009, Keunhyun and Sung-Bae 2010),- Ant Colonies (Martin et al. 2010),- Monte Carlo Tree Search (Samothrakis et al. 2011, Ikehata and Ito 2011, Alhejali and Lucas 2013),- Reinforcement Learning (Bom et al. 2013).
State of the Art [2/2]
Controllers for the Ghosts:- Monte Carlo Tree Search (Nguyen and Thawonmas 2011, 2013),- Influence Maps (Svensson and Johansson 2012),- Neural Networks (Jia-Yue et al. 2011),- Genetic Algorithm + Rules (Gagne and Congdon 2012),
- Competitive Co-Evolution of Ms. Pac-Man and Ghosts controllers (Cardona et al. 2013).
Flocking [1/2]
Swarm Intelligence technique:- Coordinated intelligence that arises from the collective behavior of decentralized, self-organized systems.
Flocking:- Developed to mimic lifelike behaviors of groups of beings.- Flocking systems consist of a population of simple agents (or boids) interacting locally with one another depending on the distance between them.
Flocking [2/2]- The agents follow very simple steering behaviors:
· Separation: makes the agent steer away from close flock mates.· Alignment: makes the agent steer toward the average heading of
the flock.· Cohesion: makes the agent steer toward the average position of
distant flock mates.- The interactions between such agents lead to the emergence of "intelligent" global behavior
Flocking Strategies [1/4]
- Generalization of the concept of Flocking.- The direction taken by a ghost at a junction depends on:
· State:- Normal,- Vulnerable,- Flashing.
· Position WRT that of the other actors:- Ms. Pac-Man,- Powerpills,- Other Ghosts (depending on their state).
Flocking Strategies [2/4]
- Each Ghost is surrounded by concentric rings called Neighborhoods.- The neighborhoods can have different radius and size.- Each neighborhood defines the strength of the interaction between the ghost and the other actors:
· Negative value → Separation· Positive value → Cohesion
Flocking Strategies [3/4]Formally, a Flocking Strategy is represented by a tuple (N, δ, Α):
- N, number of neighborhoods, indexed by n=1,…,N.- δ∈ℝN
>0, vector of N real positive numbers. Each element δ
n is
associated to one neighborhood and represents its maximum radius.- Α∈Μ, three-dimensional matrix with real entries. Each element α
s,a,n is the magnitude of the steering force on the current ghost in
state s, resulting from the interaction with the actor a, falling into neighborhood n.
Flocking Strategies [4/4]
- First, all the interactions with the other actors are calculated as vectors.- Then, all the vectors are summed to find the total interaction vector.- Finally, the direction taken by the ghost (UP, DOWN, LEFT, RIGHT) depends on the component of the total interaction vector having maximum absolute value.
A Genetic Algorithm for Flocking Strategies [1/3]
Flocking Strategies could be designed manually by an expert.We propose a standard Genetic Algorithm
A Genetic Algorithm for Flocking Strategies [2/3]
The Genetic Algorithm has been applied for the automatic definition of effective Ghosts agents:- Each individual is a Flocking Strategy, represented as a tuple (N, δ, Α). - Random initial population.- Roulette wheel selection.- Cross-over: two parents generate one child by random recombination of their chromosomes.
A Genetic Algorithm for Flocking Strategies [3/3]
- Fitness function:· The objective is to create effective Ghosts that perform well against
different Ms. Pac-Man strategies. · Each individual is tested against two Ms. Pac-Man agents:
StarterPacMan (SPM) and NearestPillPacMan (NPPM).· To ensure robustness in the face of random events, the game is
simulated 30 times for each Ms. Pac-Man agent and 95% confidence intervals are calculated.
· Fitness is given by:FITNESS = 1/CI+
SPM + 1/CI+
NPPM
Where CI+ represents the upper limit of the 95% confidence interval.
Experiments [1/2]
Comparison of the performance of the Ghosts controllers obtained with different values of the parameter N (number of Neighborhoods)
N=1 N=2 N=3 N=4 N=5
Best FITNESS-1
783.38 726.84 815.66 766.96 720.20
Avg. FITNESS-1
871.30 ±
55.45861.57 ±
70.13876.31 ±
65.06905.86 ±
62.66863.17 ±
72.15
Avg. CPU time (s)
1373 ±
150.661484.3 ±
122.011561 ±
193.941562.60 ±
109.901473.00 ±
74.02
Experiments [2/2]
Performances of the controllers included in the competition framework.
Controller Aggressive Ghosts
Legacy Legacy 2 The Reckoning
Random Ghosts
Starter Ghosts
FITNESS-1 1893.13 2210.9 1429.20 4200.70 1603.49
Conclusions [1/2]
- Sample controller (link)- Flocking Strategies: sets of behavior rules that determine the next move of an agent as a force resulting from the interaction of the agents in the game.- Genetic Algorithm presented to design optimized strategies.- The fitness function evaluates each individual by pitting it against two Ms. Pac-Man controllers 30 times, so as to avoid noise in the function.
Conclusions [2/2]
- Future lines of research:· Include in the fitness function the best Ms. Pac-Man controllers
that took part to the competition.· Compare the Flocking Strategy-based controller with the best
Ghosts controllers that took part to the competition.· Explore optimization methods other than Genetic Algorithms.· Change the fitness function structure to reduce the
computation requirements.