jeffrey barratt deep imitation learning for playing chuanbo...

Deep Imitation Learning for Playing Real Time Strategy Games Chuanbo Pan chuanbo@ cs.stanford.edu Jeffrey Barratt jbarratt@ cs.stanford.edu Motivation Competitive Computer Games, despite recent progress in the area, still remain a largely unexplored application of Machine Learning (ML), Artificial Intelligence (AI), and Computer Vision. Real Time Strategy games such as StarCraft II provide an ideal testing environment for AI and ML techniques, as: • They run in real time and involve incomplete information (the player cannot see the whole battlefield at once). • Users must balance making units, controlling those units, executing a strategy, and hiding information from their enemy to successfully win a game of StarCraft II. • Simple actions made early on in the game can greatly impact later stages of the game. Challenges Apart from the reasons mentioned in the motivation, StarCraft II is a challenging game because: • The action space is huge – O(n 4 ) possible selections, and O(n 2 ) possible places to move to. • The game is in real-time – can’t spend forever thinking! • The game is huge and multi-layered. o Have to consider attacking, producing, gathering, etc. o Our mini-game already includes tasks such as moving to beacon (enemy), attacking, and moving back. • Overfitting the data is an easy trap to fall into. Problem Definition 1) The starting position of the PySc2- provided mini-game 2) The marines can start by attacking the more dangerous targets first 3) Splitting the marines mitigates the effect of splash damage 4) Kiting can be used to trade more efficiently with the Zerglings We focus our efforts on one mini-game which involves “splitting”, a process of engaging area-of-effect units with a group of low-health ranged units, where the low-health units are moved apart to mitigate the area of effect damage to as few units at once, as well as other management tactics such as kiting and targeting. We collected data by playing the mini- game repeatedly and recording all actions taken and the state they were made in, yielding around 10,000 state-action pairs. Features and Model We use the newly-released PySc2 [1] framework to interface with the StarCraft II environment. This API allows us the following channels to communicate with StarCraft II: • A state feature vector that represents what the agent currently sees on the screen. o 17 84x84 feature layers, each representing a different aspect of the game (health, alliance status, etc.). o Because they preserve the spatial layout of the screen, feature layers act as “images” of what’s currently seen. • A set of available actions that the agent can legally make in the current state. We wrote code to convert the features and actions from StarCraft II to the network and vice versa. Our approach utilizes Deep Behavioral Cloning. We take advantage of the image-like nature of the state by using a Convolutional Neural Network (CNN). The network outputs scores for each possible action (select, attack, move, noop, etc.) as well as a best guess for each of their parameters for the current state, which will for the most part be coordinates of where to attack, select, move, etc. Results Discussion and Future Work We can observe several things from our results: • Theoretical score ranges from -9 to 750+. Obviously, we are not near maximum potential. • The learning curve indicates that we are improving our score from random but still suffer from overfitting. • The agent sometimes gets stuck moving back and forth between two states. • The agent doesn’t seem to have a full understanding of what it’s doing, just mimicking what looks right. Future work, were we to have 6 months to work: • Using a LSTM Recurrent Neural Network (RNN) to help determine actions, especially in sequence. • Refining our abstraction to ensure stability during training • Gather more training data to cover more possible cases. The graphs on the top show training loss during training with and without regularization. The graphs on the bottom show average scores for the mini-game during training. References [1] Vinyals, Oriol, et al., "StarCraft II: A new challenge for reinforcement learning." arXiv preprint arXiv:1708.04782 (2017). [Online]. Available: https://arxiv.org/abs/1708.04782.

Upload: buiphuc

Post on 04-Nov-2018

219 views

Category:

Documents

0 download

Report

Download

Embed Size (px):

TRANSCRIPT

DeepImitationLearningforPlayingRealTimeStrategyGames

ChuanboPan

[email protected]

JeffreyBarratt

[email protected]

MotivationCompetitiveComputerGames,despiterecentprogressinthearea,stillremainalargelyunexploredapplicationofMachineLearning(ML),ArtificialIntelligence(AI),andComputerVision.RealTimeStrategygamessuchasStarCraftIIprovideanidealtestingenvironmentforAIandMLtechniques,as:•  Theyruninrealtimeandinvolveincompleteinformation

(theplayercannotseethewholebattlefieldatonce).•  Usersmustbalancemakingunits,controllingthoseunits,

executingastrategy,andhidinginformationfromtheirenemytosuccessfullywinagameofStarCraftII.

•  Simpleactionsmadeearlyoninthegamecangreatlyimpactlaterstagesofthegame.

ChallengesApartfromthereasonsmentionedinthemotivation,StarCraftIIisachallenginggamebecause:•  Theactionspaceishuge–O(n4)possibleselections,and

O(n2)possibleplacestomoveto.•  Thegameisinreal-time–can’tspendforeverthinking!•  Thegameishugeandmulti-layered.

o  Havetoconsiderattacking,producing,gathering,etc.o  Ourmini-gamealreadyincludestaskssuchasmoving

tobeacon(enemy),attacking,andmovingback.•  Overfittingthedataisaneasytraptofallinto.

ProblemDefinition

1)ThestartingpositionofthePySc2-providedmini-game

2)Themarinescanstartbyattackingthemoredangeroustargetsfirst

3)Splittingthemarinesmitigatestheeffectofsplashdamage

4)KitingcanbeusedtotrademoreefficientlywiththeZerglings

Wefocusoureffortsononemini-gamewhichinvolves“splitting”,aprocessofengagingarea-of-effectunitswithagroupoflow-healthrangedunits,wherethelow-healthunitsaremovedaparttomitigatetheareaofeffectdamagetoasfewunitsatonce,aswellasothermanagementtacticssuchaskitingandtargeting.Wecollecteddatabyplayingthemini-gamerepeatedlyandrecordingallactionstakenandthestatetheyweremadein,yieldingaround10,000state-actionpairs.

FeaturesandModelWeusethenewly-releasedPySc2[1]frameworktointerfacewiththeStarCraftIIenvironment.ThisAPIallowsusthefollowingchannelstocommunicatewithStarCraftII:•  Astatefeaturevectorthatrepresentswhattheagent

currentlyseesonthescreen.o  1784x84featurelayers,eachrepresentingadifferent

aspectofthegame(health,alliancestatus,etc.).o  Becausetheypreservethespatiallayoutofthescreen,

featurelayersactas“images”ofwhat’scurrentlyseen.•  Asetofavailableactionsthattheagentcanlegallymake

inthecurrentstate.WewrotecodetoconvertthefeaturesandactionsfromStarCraftIItothenetworkandviceversa.OurapproachutilizesDeepBehavioralCloning.Wetakeadvantageoftheimage-likenatureofthestatebyusingaConvolutionalNeuralNetwork(CNN).

Thenetworkoutputsscoresforeachpossibleaction(select,attack,move,noop,etc.)aswellasabestguessforeachoftheirparametersforthecurrentstate,whichwillforthemostpartbecoordinatesofwheretoattack,select,move,etc.

Results

DiscussionandFutureWorkWecanobserveseveralthingsfromourresults:•  Theoreticalscorerangesfrom-9to750+.Obviously,we

arenotnearmaximumpotential.•  Thelearningcurveindicatesthatweareimprovingour

scorefromrandombutstillsufferfromoverfitting.•  Theagentsometimesgetsstuckmovingbackandforth

betweentwostates.•  Theagentdoesn’tseemtohaveafullunderstandingof

whatit’sdoing,justmimickingwhatlooksright.Futurework,werewetohave6monthstowork:•  UsingaLSTMRecurrentNeuralNetwork(RNN)tohelp

determineactions,especiallyinsequence.•  Refiningourabstractiontoensurestabilityduringtraining•  Gathermoretrainingdatatocovermorepossiblecases.

Thegraphsonthetopshowtraininglossduringtrainingwithandwithoutregularization.Thegraphsonthebottomshowaveragescoresforthemini-gameduringtraining.

References[1]Vinyals,Oriol,etal.,"StarCraftII:Anewchallengeforreinforcementlearning."arXivpreprintarXiv:1708.04782(2017).[Online].Available:https://arxiv.org/abs/1708.04782.

INNOVATION NOT IMITATION

Imitation grass

Emulation, imitation, over-imitation and the scope of ...rstb.royalsocietypublishing.org/content/royptb/364/1528/2417.full.pdf · Emulation, imitation, over-imitation and the scope

Brand Imitation

Bandura Imitation

Model-based Imitation Learning by Probabilistic Trajectory ... · Model-based Imitation Learning by Probabilistic ... let the robot imitate this behavior. Such imitation ... learning

Goal-conditioned Imitation Learningpapers.nips.cc/paper/9667-goal-conditioned-imitation-learning.pdf · goal-reaching policies. Learning from Demonstrations, or Imitation Learning

imitation of fountain

Imitation, Empathy,

Imitation of christ

Imitation And Instruction

Imitation Analysis