learning to play: history of machine learning in classic games

HOW MACHINES LEARN TO

PLAY

O F E R EG O Z I

ARTIFICIAL INTELLIGENCELet’s start from the basicsActually, let’s go straight to the cutting edge!Here’s a demo for a multi-layered convolutional neural

network using feed-forward training to perform supervised learning

Well-defined rulesA clear, measurable goalA task one can train forBonus: human opponents!

THE ‘ADDITRON’

Josef Kates1950

Josef Kates1950

EARLY DAYS – MINIMAXBuild a tree of game states

(from current state) Well-defined transition rules

Define a function to score each state

How close are we to the goal (a winning board)?

Choose path that maximizes our gain and minimizes opponent’s gains

Toc-Tac-Toe has only 765 unique states… Solved!!

5 x 1020 (500 billion billion) possible positions

Arthur L. Samuel1956

1989Chinook

1992

Marion Tinsley

199219941995

“With his passing, we lost not only a feared adversary but also a friend. Every member of our team had the deepest respect and admiration for Tinsley. It was a privilege to know him”

2007

10120 possible positions (checkers squared)

1950

1957

160,000 positions per second1980

Alpha-beta pruning

19961997

200 million positions per second

GENERIC LEARNING

So far, humans were central in the learning process

Pre-encoding the allowed movesProviding the winning states

Can machines learn on their own, like real toddlers?

2013

https://arxiv.org/pdf/1312.5602v1.pdf

1) 33600 raw pixels2) Target scoreInput:

2013

Output: player!

• We’re skipping entire ML courses now

• What’s fundamentally different about Deep Learning?• No predefined rules – a generic system• “A bishop moves this way, and a knight this way…”

• No domain knowledge – system “finds” the features• “count #pieces within 3 steps from the king”

DEEP LEARNING

• We’re skipping even more entire ML courses now

• Uses Artificial Neural Network, with LOTS of data• Deep Learning == multiple hidden layers

DEEP LEARNING

Magic happens HERE…Magic built-in

Showtime!

DEEP LEARNING IN 8-BIT

2016#positions > #atoms in universe1202 CPUs and 176 GPUs

GO

HOW ABOUT AI BUILDING THE GAME?...

GENERATIVE LANGUAGE MODELS

• Not done with skipping ML courses just yet • First, let’s divert to literature for a bit, shall we?...

• “Robert Cohn was once middleweight boxi?”

https://medium.com/@ageitgey/machine-learning-is-fun-part-2-a26a10b68df3

GENERATIVE LANGUAGE MODELS

• Feed the book into an RNN, let it train itself…100 iterations:

1000 iterations:

GENERATIVE LANGUAGE MODELS

• Feed the book into an ANN, let it train itself…<10K iterations:

GENERATIVE MODELS

• Now back to games…

GENERATIVE MODELS

• Extracted and encoded all game levels• Initial iterations build junk• After enough training iterations – it works!

THANK YOU!

GENERATIVE ADVERSARIAL NETWORKS (GANS)

GENERATIVE MODELS – MUSIC