learning positional features for annotating chess games : a case study
DESCRIPTION
CG 2008. Learning Positional Features for Annotating Chess Games : A Case Study. Matej Guid, Martin Mozina, Jana Krivec, Aleksander Sadikov and Ivan Bratko. Faculty of Computer and Information Science University of Ljubljana, Slovenia. Enhanced Knowledge in Chess Programs. - PowerPoint PPT PresentationTRANSCRIPT
Learning Positional Features for Annotating Chess Games:
A Case Study
Matej Guid, Martin Mozina, Jana Krivec, Aleksander Sadikov and Ivan Bratko
CG 2008
Faculty of Computer and Information Science
University of Ljubljana, Slovenia
Enhanced Knowledge in Chess Programs
The improvements of Rybka 3 in evaluation function for chess middlegames:
understanding good and bad bishops, much more advanced understanding of kings attacks, importance of defending king by pieces, “space” (previously addresed only indirectly)
(Larry Kaufmann, designer of Rybka’s evaluation function, RybkaForum.net)
How to “teach” a program to recognise chess concepts?
Knowledge is particularly important when annotating chess games…
The Concept of the Bad Bishop
Chess experts in general understand the concept of bad bishop. Precise formalisation of this concept is difficult.
Traditional definition (John Watson, Secrets of Modern Chess Strategy,
1999)
A bishop that is on the same colour of squares as its own pawns is bad:
its mobility is restricted by its own pawns, it does not defend the squares in front of these pawns. Moreover, centralisation of these pawns is the main factor in
deciding whether the bishop is bad or not.
CRAFTY's Features for Describing Bishops
Feature Description Defficiency for annotating
BLACK_BISHOP_PLUS_PAWNS_ON_COLOR
number of own pawns that are on same colour of square as bishop
all such pawns count the same, regardless of their position and
how badly they restrict the bishop
BLACK_BISHOPS_POSITION
evaluation of bishop's posi- tion based on predefined values for particular squares
such predefined value is not theactual value of bishop's placement in a particular position
BLACK_BISHOPS_MOBILITY
number of squares that bishop attacks
number of attacked squaresand actual bishop's mobilityare not necessarily the same thing
CRAFTY's Features for Describing Bishops
Feature Description Defficiency for annotating
BLACK_BISHOP_PLUS_PAWNS_ON_COLOR
number of own pawns that are on same colour of square as bishop
all such pawns count the same, regardless of their position and
how badly they restrict the bishop
BLACK_BISHOPS_POSITION
evaluation of bishop's posi- tion based on predefined values for particular squares
such predefined value is not theactual value of bishop's placement in a particular position
BLACK_BISHOPS_MOBILITY
number of squares that bishop attacks
number of attacked squaresand actual bishop's mobilityare not necessarily the same thing
CRAFTY's Features for Describing Bishops
Feature Description Defficiency for annotating
BLACK_BISHOP_PLUS_PAWNS_ON_COLOR
number of own pawns that are on same colour of square as bishop
all such pawns count the same, regardless of their position and
how badly they restrict the bishop
BLACK_BISHOPS_POSITION
evaluation of bishop's posi- tion based on predefined values for particular squares
such predefined value is not theactual value of bishop's placement in a particular position
BLACK_BISHOPS_MOBILITY
number of squares that bishop attacks
number of attacked squaresand actual bishop's mobilityare not necessarily the same thing
Static Nature of Positional Features
Positional features in evaluation functions are static in their nature Heuristic search enables to fulfil their purpose – contributing to the
program finding best moves
Static Nature of Positional Features
Positional features in evaluation functions are static in their nature Heuristic search enables to fulfil their purpose – contributing to the
program finding best moves
It is also desirable for positional features for annotating chess games to be of static nature! Heuristic search is important for instructive annotations.
see the paper Automated chess tutor. CG 2006, Turin, Italy, May 29-31, 2006.
Is freeing move e6-e5 possible or not?
Expert-Crafted Rules & Introducing Learning
Data set: 200 middlegame positions from real chess games: bad: 78 bishops not bad: 122 bishops
We randomly selected 100 positions for learning and 100 positions for testing.
CRAFTY’s positional feature values served as attribute values for learning.
Chess experts:
wGM Jana KrivecFM Matej Guid
It is extremely difficult for chess experts to define appropriate rules for determining bad bishops -> knowledge acquisition bottleneck
various expert-crafted rules performed rather poorly (only 59% of examples were correctly classified)-> motivation for introducing machine learning
Data Set for Machine Learning
Bad Bishops from Static Point of View
Our goal: construct a static positional feature BAD_BISHOP Assessing bishops “statically” is counter-intuitive from chess point of
view Special guidelines were chosen for determining “statically” bad
bishops…
The bishop is bad from the static point of view in some position, if 1.
its improvement or exchange is desirable2.
the pawn structure limits the bishop’s chances for taking an active part in the game,3.
its mobility in this position is limited or not important for the evaluation.
Guidelines for Determining Bad Bishops from Static Point of View
Standard Machine Learning Methods' Performance with CRAFTY's features
only Machine learning methods’ performance on initial dataset
Method Classification accuracy
Decision trees (C4.5) 73%
Logistic regression 70%
Rule learning (CN2) 72% The results were obtained on test data set.
The results obtained with CRAFTY’s positional features only are too inaccurate for commenting purposes…
additional information for describing bad bishops is necessary.
see the paper Fighting Knowledge Acquisition Bottleneck with Argument Based Machine Learning. European Conference on Artificial Intelligence, Patras, Greece, July 21-25, 2008. Available at: www.ailab.si/matej
Improved Method
First Critical Example
Rules obtained by ABML method ABCN2 failed to classify this example as "not bad“.
The following question was given to the experts: “Why is the black bishop not bad?“
The experts used their domain knowledge:
“The black bishop is not bad, since its mobility is not seriously restricted by the pawns of both players.”
Introducing new attribute into the domain and adding argument to an
example Experts’ explanation could not be described with current domain attributes.
The argument
“BISHOP=“not bad” because IMPROVED_BISHOP_MOBILITY is high“
was added to the example.
A new attribute, IMPROVED_BISHOP_MOBILITY, was included into the domain:
number of squares accessible to the bishop, taking into account only own and opponent’s pawn structure
Another Critical Example
The ABML-based knowledge elicitation process can be used to induce rules to determine both good (i.e., not bad) and bad bishops.
Now the following question was given to the experts: “Why is the black bishop bad?“
The experts used their domain knowledge:
“The black bishop is bad, since black pawns block its diagonals.”
Introducing new attribute into the domain and adding argument to an
example Experts’ explanation could not be described with current domain attributes.
The argument
“BISHOP=“bad” because BLACK_PAWN_BLOCKS_BISHOP_DIAGONAL is high“
was added to the example.
A new attribute, BLACK_PAWN_BLOCKS_BISHOP_DIAGONAL, was included into the domain:
pawns on the color of the square of the bishop that block the bishop's (front) diagonals
weights of such pawns were taken into account (see the “Look-up table” that follows)
Method failed to explain critical example with given argument.
Counter example was presented to experts:
Experts’ explanation: “The black bishop is not bad, since its mobility is not seriously restricted, taking the pawn structure into account.”
Counter example: “bad”, although BLACK_PAWN_BLOCKS_BISHOP_DIAGONAL is high.
Counter example
"Why is the “green” bishop not bad, compared to the “red” one?"
Critical example: “not bad”, BLACK_PAWN_BLOCKS_BISHOP_DIAGONAL is high.
Improving Arguments with Counter Examples
The argument given to the critical example was extended to
“BISHOP=“bad” because BLACK_PAWN_BLOCKS_BISHOP_DIAGONAL is high and IMPROVED_BISHOP_MOBILITY is low.”
With this argument the method could not find any counter examples anymore.
The new rule covering the critical example became:
if BAD_PAWNS_BLOCK_BISHOP_DIAGONAL > 16 and IMPROVED BISHOP MOBILITY < 4 then BAD_BISHOP= “bad” class distribution [42,0]
Assesing “Bad” Pawns
Experts designed a look-up table (left) with predefined values for pawns that are on color of square of the bishop in order to assign weights to such pawns.
BAD_PAWNS_AHEAD = 16 + 24 + 2 = 42
After the Final Iteration...
Attribute Description
BAD_PAWNSpawns on the color of the square of the bishop - weighted according to their squares (bad pawns)
BAD_PAWNS_AHEAD bad pawns ahead of the bishop
BAD_PAWNS_BLOCK_BISHOP_DIAGONAL
bad pawns that block the bishop's (front) diagonals
BLOCKED_BAD_PAWNS bad pawns blocked by opponent's pawns or pieces
IMPROVED_BISHOP_MOBILITY number of squares accessible to the bishop taking into account only pawns of both opponents
The whole process consisted of 8 iterations. 7 arguments were attached to automatically selected critical
examples 5 new attributes were included into the domain
Classification Accuracy Through Iterations
The accuracies of all methods improved by adding new attributes. ABCN2 (which also used the arguments) outperformed all others.
Arguments suggested useful attributes AND lead to more accurate models.
The Final Model
Rule #2:
if BAD_PAWNS_AHEAD > 18 and IMPROVED_BISHOP_MOBILITY < 3 then BISHOP= “bad” class distribution [46,0]
The Final Model
Possible comment:
“Black bishop is bad, since black pawns on the same colour of squares ahead
of it, and pawns of both opponents restrict its mobility.”
Conclusions
In this case study, Argument Based Machine Learning was shown to be a powerful knowledge-elicitation method for defining deep positional patterns.
ABML offers the following advantages for knowledge elicitation:
explain single example
easier for experts to articulate knowledge
critical examples expert provides only relevant knowledge
arguments constrain learning
hypotheses are consistent with expert knowledge