Game Trees and Heuristics

15-211: Fundamental DataStructures and Algorithms

Margaret Reid-Miller

7 April 20052

Announcements

! HW5 Due Monday 11:59 pm

Games

X O X

O X O

X O O

4

! !!!!Two players

! Deterministic

! no randomness, e.g., dice

! Perfect information

! No hidden cards or hidden Chess pieces…

! Finite

! Game must end in finite number of moves

! Zero sum

! Total winnings of all players

Game Properties

5

MiniMax Algorithm

! Player A (us) maximize goodness.

! Player B (opponent) minimizes goodness

Player A maximize (draw)

Player B minimize (lose, draw)

Player A maximize (lose, win)

! At a leaf (a terminal position) A wins, loses, or draws.

! Assign a score: 1 for win; 0 for draw; -1 for lose.

! At max layers, take node score as maximum of thechild node scores

! At min layers, take nodes score as minimum of thechild node scores

0

-1

-1 1

0

6

Heuristics

! A heuristic is an approximation that istypically fast and used to aid inoptimization problems.

! In this context, heuristics are used to“rate” board positions based on localinformation.

! For example, in Chess I can “rate” aposition by examining who has morepieces. The difference in black’s andwhite’s pieces would be the evaluation ofthe position.

7

Heuristics and Minimax

! When dealing with game trees, theheuristic function is generallyreferred to as the evaluationfunction, or the static evaluator.

! The static evaluation takes in aboard position, and gives it a score.

! The higher the score, the better it isfor you, the lower, the better for theopponent.

8

Heuristics and Minimax

! Each layer of the tree is called a ply.

! We cut off the game tree at acertain maximum depth, d. Called ad-ply search.

! At the bottom nodes of the tree, weapply the heuristic function to thosepositions.

! Now instead of just Win, Loss, Tie,we have a score.

9

Evaluation Function

! Guesses the outcome of an incompletesearch

! Eval(Position) = !i wi * fi(Position)

! Weights wi may depend on the phase ofthe game

! Features for chess fi:

! #of Pawns (material terms)

! Centrality

! Square control

! Mobility

! Pawn structure10

Minimax in action

10 2 12 16 2 7 -5 -80

10 16 7 -5

10 -5

10Max (Player A)

Max

(Player A)

Min (Player B)

Evaluation function applied to the leaves!

11

How fast?

! Minimax is pretty slow even for amodest depth.

! It is basically a brute force search.

! What is the running time?

! Each level of the tree has someaverage b moves per level. Wehave d levels. So the running timeis O(bd).

12

Alpha Beta Pruning

! Idea: Track “window” of expectations. Two variables:

" – Best score so far at a max node: increases.

# Score can be forced on our opponent.

$ – Best score so far at a min node: decreases

# Opponent can force a situation no worse than this score.Any move with better score is too good for opponent toallow

Either case: If " % $:

#Stop searching further subtrees of that child.Opponent won’t let you get that high a score.

! Start the process with an infinite window (" = -&, $ = &).

13

alphaBeta (", $)

The top level call: Return alphaBeta (- &, &))

alphaBeta (", $):

! At leaf level (depth limit is reached):

! Assigns estimator function value (in therange (- &.. &) to the leaf node.

! Return this value.

! At a min level (opponent moves):

! For each child, until " % $:

! Set $ = min($, alphaBeta (", $))

! Return $.

! At a max level (our move):

! For each child, until " % $:

! Set " = max(", alphaBeta (", $))

! Return ". 14

10 2 12

10 12

10

Alpha Beta Example

Max

Max

Min

Min

$ =10

" = 12

" ! $!

15

Alpha Beta Example

10 2 12 2 7

10 12 7

10 7

10Max

Max

Min

Min

" = 10

$ = 10

" ! $!

16

Alpha Beta speedup

! Alpha Beta is always a win:

! Returns the same result as Minimax,

! Is minor modification to Minimax

! Claim: The optimal Alpha Beta search treeis O(bd/2) nodes or the square root of thenumber of nodes in the regular Minimaxtree.

! Can enable twice the depth

! In chess branching is about 38. Inpractice Alpha Beta reduces it to about 6and enables 10 ply searches on a PC.

17

Heuristic search techniques

! Alpha-beta is one way to prune thegame tree…

Heuristic = aid to problem-solving

18

Move Ordering

! Explore decisions in order of likely successto get early alpha beta pruning

! Guide search with estimator functionsthat correlate with likely searchoutcomes or

! track which moves tend to cause betacutoff

! Heuristic estimate of the cost (time) ofsearching one move versus another

! Search the easiest move first

19

Timed moves

! Not uncommon to have a limitedtime to make a move. May need toproduce a move in say 2 minutes.

! How do we ensure that we have agood move before the timer goesoff?

20

Iterative Deepening

! Evaluate moves to successively deeper anddeeper depths:

! Start with 1-ply search and get bestmove(s). Fast

! Next do 2-ply search using theprevious best moves to order thesearch.

! Continue to increased depth of search.

! If some depth takes too long, fall back toprevious results (timed moves).

21

Iterative Deepening

! Save best moves of shallower searches tocontrol move ordering.

! Need to search the same part of the treemultiple times but improved moveordering more than makes up for thisredundancy

! Difficulty:

! Time control: each iteration needsabout 6x more time

! What to do when time runs out?22

Transposition Tables

! Minimax and Alpha Beta (implicitly)build a tree. But what is the underlyingstructure of a game?

! Different sequences of moves can leadto the same position.

! Several game positions may befunctionally equivalent (e.g. symmetricpositions).

23

Nim Game Graph

(1,2,3)

(2,3) (1,1,3) (1,2,2) (1,3) (1,1,2) (1,2)

(2,2) (1,2) (2) (1,3) (3) (1,1,2) (1,1,1) (1,1) (1)

(3) (2) (1) (1,1) (1,2) (1,1,1)

(2) (1) loss (1,1)

(1) win

loss

win

Us

Them

Us

Us

Us

Them

Them

24

Transposition Tables

! Memoize: hash table of board positions to get

! Value for the node

! Upper Bound, Lower Bound, or Exact Value

! Be extremely careful with alpha beta as may onlyknow a bound at that position.

! Best move at the position

! Useful for move ordering for greater pruning!

! Which positions to save?

! Sometimes obvious from context

! Ones in which more search time has been invested

! Collisions: Simply overwrite

25

Limited Depth Search Problems

! Horizon effect: push bad news overthe search depth

! Quiescence search: can’t simply stopin the middle of an exchange

26

Alpha Beta Pruning

Theorem: Let v(P) be the value of a positionP. Let X be the value of a call to AB(", $).

Then one of the following holds:

v(P) " " and X " "" < v(P) < $ and X = v(P)

$ " v(p) and X ! $

Suppose we take a chance and reduce thesize of the infinite window.What might happen?

27

Aspiration Window:

! Suppose you had information that value of aposition was probably close to 2 (say fromthe result of shallower search)

! Instead of starting with an infinite window,start with an “aspiration window” around 2(e.g., (1.5, 2.5)) to get more pruning.

! If the result is in that range you aredone.

! If outside the range you don’t knowthe exact value, only a bound.Repeat with a different range.

! How might this technique be use for parallelevaluation?

28

Tricks

! Many tricks and heuristics have beenadded to chess program, including thistiny subset:

! Opening Books

! Avoiding mistakes from earlier games

! Endgame databases (Ken Thompson)

! Singular Extensions

! Think ahead

! Contempt factor

! Strategic time control

Game of

Amazons

30

Game of Amazons

! Invented in 1988 by ArgentinianWalter Zamkauskas.

! Several programming competitionsand yearly championships.

! Distantly related to Go.

! Active area in combinatorial gametheory.

31

Amazons Board

! Chess board 10 x 10

! 4 White and 4 Black chess Queens(Amazons) and Arrows

! Startingconfiguration

! White movesfirst

32

Amazons Rules

! Each move consists of two steps:

1. Move an amazon of own color.

2. This amazon has to throw an arrow toan empty square where it stays.

! Amazons and arrows move as a chessQueen as long as no obstacle blocks theway (amazon or arrow)

! Players alternate moves.

! Player who makes last move wins.

33

Amazons challenges

! Absence of opening theory

! Branching factor of more than 1000

! Often 20 reasonable moves

! Need for deep variations

! Opening book >30,000 moves

34

AMAZONG

! World’s best computer player byJens Lieberum

! Distinguishes three phases of thegame:

! Opening at the beginning

! Middle game

! Filling phase at the end

Seehttp://jenslieberum.de/amazong/amazong.html

35

Amazons Opening

! Main goals in the opening

! Obtain a good distribution ofamazons

! Build large regions of potentialterritory

! Trap opponent’s amazons

36

Amazons Filling Phase

! Filling phase starts when emptysquares can be reached by only oneplayer.

! Happens usually after 50 moves.

! Goal is to have access to moreempty squares than the other player

! Outcome of game determined bycounting number of moves left foreach player.

! But…

37

Defective Territories

! K-defective territory provides kfewer moves than empty squares

38

Zugzwang

! Seems that black has access to 3 emptysquares

! But if black moves first then can only usetwo.

39

More Complex Zugzwang

Player who moves first must either! take their own region and give region C to

the opponent, or! take region C and block off their own

region

40

Chiptest, Deep Thought Timeline

! A VLSI design class project evolves intoF.H.Hsu move-generator chip

! A productive CS-TG results in ChipTest,about 6 weeks before the ACM computerchess championship

! Chiptest participates and playsinteresting (illegal) moves

! Chiptest-M wins ACM CCC

! Redesign becomes DT

! DT participates in human chesschampionships (in addition to CCC)

! DT wins second Fredkin Prize ($10K)

! DT is wiped out by Kasparov

Story continues@ IBM

41

Opinions: Is Computer Chess AI?

From Hans Moravec’s Book “Robot”42

So how does Kasparov win?

! Even the best Chess grandmasters saythey only look 4 or 5 moves ahead eachturn. Deep Junior looks up about 18-25moves ahead. How does it lose!?

! Kasparov has an unbelievable evaluationfunction. He is able to assess strategicadvantages much better than programscan (although this is getting less true).

! The moral, the evaluation function playsa large role in how well your program canplay.

43

State-of-the-art: Backgammon

! Gerald Tesauro (IBM)

! Wrote a program which became“overnight” the best player in theworld

! Not easy!

44

State-of-the-art: Backgammon

! Learned the evaluation function by playing1,500,000 games against itself

! Temporal credit assignment usingreinforcement learning

! Used Neural Network to learn the evaluationfunction

5

9 7

8 2

a b

c d

NeuralNet

9

Learn!Predict!

45

State-of-the-art: Go

! Average branching factor 360

! Regular search methods go bust !

! People use higher level strategies

! Systems use vast knowledge basesof rules… some hope, but still playpoorly

! $2,000,000 for first program todefeat a top-level player