Learning Deviation Payoffs in Simulation-Based Games

Samuel SokotaSwarthmore Collegesokota@ualberta.ca

Caleb HoSwarthmore College

caleb.yh.ho@gmail.com

Bryce WiedenbeckSwarthmore College

bwieden1@swarthmore.edu

Abstract

We present a novel approach for identifying approximaterole-symmetric Nash equilibria in large simulation-basedgames. Our method uses neural networks to learn a mappingfrom mixed-strategy profiles to deviation payoffs—the ex-pected values of playing pure-strategy deviations from thoseprofiles. This learning can generalize from data about a tinyfraction of a game’s outcomes, permitting tractable analysisof exponentially large normal-form games. We give a proce-dure for iteratively refining the learned model with new dataproduced by sampling in the neighborhood of each candi-date Nash equilibrium. Relative to the existing state of theart, deviation payoff learning dramatically simplifies the taskof computing equilibria and more effectively addresses playerasymmetries. We demonstrate empirically that deviation pay-off learning identifies better approximate equilibria than pre-vious methods and can handle more difficult settings, includ-ing games with many more players, strategies, and roles.

Simultaneous-move games are among the most general toolsfor reasoning about incentives in multi-agent systems. Thestandard mathematical representation of a simultaneous-move game is the normal-form payoff matrix. It describesincentives in a multi-agent system by listing the strategiesavailable to each agent and storing, for every combinationof strategies the agents may jointly select, the resulting util-ity for each agent. This tabular representation grows expo-nentially: the payoff matrix for a normal form game with nplayers andm strategies per player stores utilities for each ofmn pure-strategy profiles. As a result, explicit normal-formrepresentations are typically employed only for games witha very small number of players and strategies.

To study larger games, it is common to replace the explicitpayoff table with a more compact representation of the util-ity function, from which payoff values can be computed ondemand. However, pure-strategy utility functions are gen-erally not sufficient for computing Nash equilibria. Instead,Nash-finding algorithms typically depend on one or more ofthe following quantities being efficiently computable: bestresponses to profiles of mixed strategies, deviation payoffs ofmixed-strategy profiles (the expected utility of deviating to apure-strategy, holding other agents’ mixed strategies fixed),

Copyright c© 2019, Association for the Advancement of ArtificialIntelligence (www.aaai.org). All rights reserved.

or the derivatives of these deviation payoffs. Examples ofalgorithms requiring each of these computations include, re-spectively, fictitious play (Brown 1951), replicator dynam-ics (Taylor and Jonker 1978), and the global Newton method(Govindan and Wilson 2003). Computing any of these quan-tities naively involves summing over all entries in the fully-expanded payoff matrix.

This means that analyzing large games typically requirescarefully selecting a compact representation of the util-ity function that supports efficient computation of mixed-strategy deviation payoffs. For certain classes, like action-graph games (Jiang, Leyton-Brown, and Bhat 2011), theserepresentations are known, but in other cases, no such repre-sentation is available. Moreover, identifying such a compactstructure can be challenging, even when it does exist.

In this paper, we explore a machine learning approach toefficiently approximating deviation payoffs and their deriva-tives, under the hypothesis that sufficiently close approxi-mation of these quantities will tend to yield ε-Nash equi-libria. At the core of the approach is the idea that the ex-pected payoffs of a mixed-strategy profile can be estimatedby the payoffs of pure-strategy profiles sampled from thatmixed-strategy profile. Such sampling is extremely noisy,but by sampling from many different mixed-strategy profilesand employing machine learning algorithms that generalizeacross related mixed strategies, we can estimate a deviationpayoff function for the entire game.

Our methods are applicable in settings where a specificcompact structure is not known in advance, but where pay-off estimates for arbitrary pure-strategy profiles can be ac-quired. This setting is common in simulation-based games(also known as empirical games (Wellman 2006; Tuyls et al.2018)), where game-theoretic models are constructed frompayoffs observed in multi-agent simulations. Simulation-based games were initially used to study trading agent com-petitions (Wellman et al. 2006; Jordan, Kiekintveld, andWellman 2007) and continuous double auctions (Phelps,Marcinkiewicz, and Parsons 2006; Vytelingum, Cliff, andJennings 2008) and have since been employed in numer-ous applications, including designing network routing pro-tocols (Wellman, Kim, and Duong 2013) and understandingthe impacts of high-frequency trading (Wah and Wellman2017). Recently, simulation-based games have arisen in theanalysis of the meta game among AlphaGo variants (Lanc-

tot et al. 2017) and other multi-agent reinforcement learningsettings including sequential social dilemmas (Leibo et al.2017). Many of these applications can involve large num-bers of symmetric players and would benefit from analy-sis that scales efficiently and accurately by leveraging thesesymmetries.

Related WorkMuch recent progress in game-theoretic settings has beenachieved by the application of machine learning to largeextensive-form games. In Go and poker, the extensive formis well-defined, but is far too big to be constructed explicitly,so high-level play (Silver et al. 2017; Moravcık et al. 2017)and approximate equilibrium analysis (Heinrich, Lanctot,and Silver 2015) have required the aid of machine learningbased algorithms. These algorithms run simulations to esti-mate payoffs for various game states and use learning to gen-eralize to unexplored parts of the game tree. Our work em-ploys a similar approach, generating observations by simu-lation and making generalizations by machine learning onlyin the context of normal-form deviation payoffs, instead ofextensive-form state or state-action values.

In the context of simultaneous-move games, a variety ofgame models more compact than the normal-form payoffmatrix have been explored. Graphical games (Kearns 2007)exploit independence among players, action-graph games(Jiang, Leyton-Brown, and Bhat 2011) exploit independenceor partial independence among strategies, and resource-graph games (Jiang, Chan, and Leyton-Brown 2017) exploitindependence among sub-decisions that comprise strategies.All of these models can, in particular contexts, exponen-tially reduce the size of the game representation. But theyall require detailed knowledge of a game’s structure, whichis unlikely to be available in the setting of simulation-basedgames.

When a precise compact model is not known in advance,it may still be possible to induce the parameters of such amodel from simulated or observed data. Duong et al. (2009)construct graphical games from payoff data, and Honorioand Ortiz (2015) construct linear influence games from be-havioral data. An advantage of these approaches is that theresulting models can offer structural insight that may gen-eralize to other related settings. However, they suffer fromsignificant limitations in that only the specifically targetedtype of structure can be learned. So these approaches areonly useful when the analyst has advance knowledge of thepayoff structure. In contrast, deviation payoff learning is ag-nostic to the detailed structure of the payoff functions, mak-ing it applicable to a far wider variety of simulation-basedgames.

We are aware of two papers that have addressed theproblem of learning generic models of simultaneous-movegames from simulation-based data. The first learns payofffunctions that generalize over continuous strategy spaces(Vorobeychik, Wellman, and Singh 2007). The regressiontakes as input a single-parameter description of a player’sstrategy and a single-parameter summary of the profile ofopponent strategies and outputs a utility. For all but the sim-plest regression methods, equilibrium computation requires

using the learned functions to fill in a payoff matrix for adiscretization of the strategy space. This matrix grows expo-nentially with the number of players, prohibiting analysis oflarge games with complex payoff functions.

Closest to the present work, Wiedenbeck, Yang, and Well-man (2018) learn payoff functions of symmetric games.For each strategy, a regressor takes as input a pure-strategyopponent-profile and outputs a utility. They provide an ap-proximation method for deviation payoffs that allows forcomputation of role-symmetric ε-Nash equilibria. This ap-proach can learn complicated payoff functions and scaleswell in the number of players. However, the approxima-tion method, which is only valid for fully-symmetric gameslearned via RBF kernel Gaussian process regression, is pro-hibitively expensive, and thereby limited to small data setsfrom games with small numbers of pure strategies. We com-pare against this method in our experiments and demonstratefavorable results for deviation-payoff learning.

Background and NotationA simultaneous-move game Γ “ pP,S, uq is characterizedby a set of players P “ t1, . . . , nu, a space of pure-strategyprofiles S, and a utility function u. The pure-strategy pro-file space S “

ś

iPP Si is the Cartesian product of strat-egy sets Si for each player i P P . An element of the strat-egy space ~s P S, specifying one strategy for each player,is known as a pure-strategy profile. An n ´ 1 player profile~s´i specifies strategies for all players except i. The utilityfunction u : S ÞÑ Rn maps a profile to a payoff for eachplayer; uip~sq P R is the component payoff for player i. In asimulation-based game, an exogenously-specified simulatorprovides noisy point estimates of the utility function. Thatis, for any given profile ~s, we can observe samples ~v „ D~s,where D~s is some unknown distribution with mean up~sq.

A mixed strategy σ for player i is a probability distri-bution over player i’s strategy set Si. The set of all mixedstrategies for player i is the simplex: ∆pSiq. A mixed strat-egy profile ~σ specifies a mixed strategy for each player.The set of mixed strategy profiles is a simplotope: S “ś

iPP ∆pSiq. Similarly to pure profiles, ~σ´i is the n ´ 1player profile that results from excluding player i from ~σand ~σi is the mixed-strategy of player i.

We define a deviation payoff for player i, strategy s P Si,and mixed profile ~σ as the expected utility to player i forplaying s when all n´ 1 opponents play according to ~σ´i:

devPayips, ~σq :“ E~s´i„~σ´i

uips,~s´iq.

We use devPayip~σq to denote the vector of deviation payoffsfor all strategies of player i. If players jointly play accordingto mixed-strategy profile ~σ, then player i achieves expectedpayoff E~s„~σ uip~sq “ devPayip~σq ¨ ~σi. The regret of a pro-file ~σ is the largest amount any player could have gained bydeviating to a pure strategy:

εp~σq :“ maxiPP

maxsPSi

devPayips, ~σq ´ E~s„~σ

uip~sq

A profile with zero regret is a Nash equilibrium; a profilewith regret below threshold ε is an ε-Nash equilibrium. Ouraim is to efficiently identify low-regret ε-Nash equilibria.

Many common games, especially large simulation-basedgames, exhibit player symmetries. Two players i, j P P aresymmetric if the game created by permuting i and j is equiv-alent to Γ. To aid in representing player symmetries, we par-tition a game’s players into roles. A role R Ď P is a maxi-mal set of symmetric players. A game with one role is fully-symmetric; a game with n roles is fully-asymmetric. The setof roles R induces a partition tSR : R P Ru over the set ofpure strategies S :“

Ů

RPR SR, where SR denotes the purestrategies available to players in role R. We use the notationSR Ă S for the set of mixed-strategy profiles in which allplayers sharing a role also share a mixed strategy (i.e. theset of role-symmetric mixed-strategy profiles). For ~σ P SR,we define ~σ´R as the n ´ 1 player role-symmetric mixed-strategy profile induced by excluding a player in roleR from~σ. Similarly, ~σR is the mixed-strategy in ~σ played by theplayers in role R. We define π : SR Ñ

ś

RPR ∆pSRq to bethe natural projection operator identifying the shared mixed-strategy of each role in a role symmetric mixed-strategy pro-file together (note that π is bijective). To illustrate the differ-ence between SR and πpSRq we can consider a small gamewith role partition R “ tR1 “ t1, 2u, R2 “ t3, 4, 5uu. Inthis case ~σ P SR takes the form p~σR1 , ~σR1 , ~σR2 , ~σR2 , ~σR2q,while πp~σq “ ~σπ takes the form p~σR1 , ~σR2q. For a pure-strategy s P SR and a role-symmetric mixed-strategy profile~σ P SR, a deviation payoff is unambiguously defined by

devPayps, ~σq :“ E~s´R„~σ´R

usp~s´Rq,

where usp~s´Rq denotes the payoff to a player playing strat-egy s in the strategy profile ps,~s´Rq. In this case, we usedevPayRp~σq to denote the vector of deviation payoffs fors P SR. Using this notation, regret can be defined

εp~σq :“ maxRPR

maxsPSR

devPayps, ~σq ´ devPayRp~σq ¨ ~σR.

MethodsDeviation payoff learning uses an exogenous simulator orother data source to estimate deviation payoffs for a role-symmetric mixed-strategy profile by first sampling a pure-strategy profile according to the players’ mixed strategiesand then querying the simulator at the sampled pure-strategyprofile. We construct a data set of such samples and thensolve a regression problem to produce a regressor : S ˆπpSRq Ñ R. For this methodology to be successful, it iscrucial that the simulator possess a significant amount ofknown player symmetry (otherwise the dimensionality ofthe domain of the regressor can be very large, making the re-gression problem difficult). Once this regressor is trained, itallows us to compute candidate role-symmetric ε-Nash equi-libria. We then gather additional samples in the neighbor-hood of each equilibrium candidate and iteratively refine themodel.

Approximating Deviation PayoffsGiven a simulator which draws samples fromD~s given ~s, wecan approximate deviation payoffs for ~σ by first sampling ~sfrom ~σ, and then querying the simulator for payoffs ~v at ~s.The expected values of the payoffs ~v are deviation payoffs

for deviating from ~σ to ~s. Thus, learning deviation payoffscan be cast as a regression problem. In general, finding agood solution to this regression problem can be very diffi-cult, especially when alloted only a small number of queriesfor a large game. In this paper, we are most interested in an-alyzing large games using only a small number of queries,so we ease the difficulty of the regression problem by limit-ing ourselves to only learning the deviation payoff values ofrole-symmetric mixed-strategy profiles. This constraint sig-nificantly reduces the dimensionality of the domain of theregressor in the presence of player symmetries. While thisconstraint also ultimately limits the methods of this paper toapproximating role-symmetric Nash equilibria, it is what al-lows us to consider extremely large games (in the presenceof player symmetries) that would otherwise be intractible toanalyze with a small number of queries.

Formally, we would like to construct a regressor minimiz-ing the quantity,

ÿ

sPS

ż

~σPSR

rdevPayps, ~σq ´ regressorps, πp~σqqs2 .

To approximate the true loss, we useÿ

~σπPΣĂπpSRq

Lp~v,~s, ~σπq where ~v „ D~s and ~s „ π´1p~σπq

as the training loss, where

Lp~v,~s, ~σπq “ÿ

sPt~si:iPP u

r~vs ´ regressorps, ~σπqs2,

and Σ is a finite number of randomly sampled projected role-symmetric mixed-strategy profiles. In the limit as every pro-jected role-symmetric mixed-strategy profile is sampled aninfinite number of times, the minimum of the training losscoincides with the minimum of the true loss at almost everyprofile.

We use a multiheaded neural network as our regres-sor. The network takes a projected role-symmetric mixed-strategy profile ~σπ P πpSRq as input and has a head for eachpure-strategy s P S whose output is the predicted deviationpayoff devPayps, π´1p~σπqq.

Figure 1 shows an example of a learned game with threepure strategies. Each simplex represents the deviation pay-off function of the labeled pure-strategy. Each point on a

Figure 1: A regressor can closely approximate the true devi-ation payoffs in a 100-player symmetric game using payoffsamples.

Figure 2: The black path shows movement of the candi-date equilibrium across two iterations of sampling and re-solving.

simplex (in barycentric coordinates) represents a mixed-strategy. The heat corresponds to the value of the deviationpayoff at that mixed-strategy. The close match between top(true deviation payoffs) and bottom (neural network esti-mates) rows is typical; we find that the network is consis-tently able to learn deviation payoffs with high accuracy.

Approximating Role-Symmetric Nash EquilibriaWe choose to learn deviation payoffs because they providesufficient information to solve for Nash equilibria. Replica-tor dynamics relies solely on deviation payoffs, while otherrole-symmetric Nash equilibrium computation techniquesrely on a combination of deviation payoffs and their deriva-tives (which can easily be provided by differentiable regres-sors such as neural networks).

The existence of a Nash equilibrium depends only on thedeviation payoffs at that mixed-strategy profile, irrespec-tive of deviation payoffs elsewhere in mixed-strategy profilesimplotope. So to find good approximate equilibria it is bet-ter to emphasize learning in areas of the simplotope wherethere are likely to be Nash equilibria, even at the cost ofweaker learning in other areas.

In order to do this in a query-efficient manner, we inte-grate sampling and training with equilibrium computation,as described in Algorithm 1. Specifically, the initial sam-ple of mixed strategies is taken roughly uniformly at ran-dom across the projected mixed-strategy profile simplotope(this can be done with a Dirichlet distribution). A regres-sor trained on these examples is used to solve for equilibria.Then, additional samples are taken in the neighborhood ofthe candidate Nash equilibria and we solve for Nash equi-libria again using a regressor trained on the entire set of ex-amples. The process of resampling and re-solving can be re-peated for several iterations. We find that this process signif-icantly lowers deviation payoff error in the areas of retrain-ing, and is sufficient to find low regret approximate equilib-ria, even with a small number of training examples.

Figure 2 shows an example of the sampling process acrosstwo iterations of resampling and re-solving. A candidateequilibrium is identified using the initial purple samples.

This candidate is revised using the blue samples taken ina neighborhood of the candidate. Finally, the revised can-didate is revised again using the green samples taken in itsneighborhood. The black path shows the movement of thecandidate equilibrium across iterations. The pink star is thefinal guess for the location of the equilibrium.

Algorithm 1 Approximating Role-Symmetric Nash Equi-libria

procedure APPXRSNASHEQUILIBRIA(INITIALQUERIES, RESAMPLEQUERIES, NUMITERS):r~σπs Ð sampleπpSRq

pinitialQueriesqr~ss Ð sampleProfilespr~σπsqr~vs Ð samplePayoffspr~ssqdata Ð pr~σπs, r~ss, r~vsqregressor.fit(data)repeatr~σ˚π s Ð findNashpregressorqr~σπs Ð sampleNbhdpr~σ˚π s, resampleQueriesqr~ss Ð sampleProfilespr~σπsqr~vs Ð samplePayoffspr~ssqdata Ð data` pr~σπs, r~ss, r~vsqregressor.fitpdataq

until numItersreturn findNashpregressorq

In general, sampling the neighborhood of a point on amanifold can be difficult. To do so, we sample from anisotropic Gaussian at the image of each mixed-strategy inthe projected role-symmetric mixed-strategy profile underthe coordinate patch parameterizing the mixed-strategy sim-plex for that role, and then send each sample back to its re-spective simplex under the inverse of its coordinate patch.Sometimes, the samples will not fall within the simplexwhen they are preimaged by the coordinate patch. This prob-lem becomes particularly pronounced as the number of purestrategies grows large (intuitively, this is because simplicesbecome pointier with the number of dimensions). Rejectionsampling quickly becomes highly impractical. To counteractthis, we project the preimage of each sample into its sim-plex (Chen and Ye 2011). Projection sends samples that falloutside the simplotope onto its edges, thereby putting moreemphasis onto the edges than rejection sampling would. Inpractice, we find that overall performance benefits from theextra emphasis on the edges of the simplotope, which areotherwise difficult to learn because there is less nearby vol-ume within the simplotope from which to generalize.

Formally, say we would like to sample a projected role-symmetric mixed-strategy profile ~σ1π in the neighborhoodof ~σπ . Let αR : ∆pSRq Ñ UR be the coordinate patch pa-rameterizing the mixed-stategy simplex for role R. Then themixed-strategy p~σ1πqR for role R in the sampled projectedrole-symmetric mixed-strategy profile is given by

p~σ1πqR “ Ppα´1R pσEqq where σE „ N pαRpp~σπqRq, k¨Iq,

and P is the projection operator described by (Chen and Ye2011). We use the notation σE to capture the intuition thatσE is the Euclidean representation of a mixed-strategy.

Figure 3: Gaussian process regression integration and deviation payoff learning achieve similar performance in congestiongames, but deviation payoff learning significantly outperforms Gaussian process regression integration in more difficult games.

ExperimentsExperimental SpecificationIn the following experiments, we employed a network withthree dense hidden layers of 128, 64, and 32 nodes, followedby a head for each strategy with 16 hidden nodes and a singleoutput. This architecture was tuned based on 100 player, fivepure-strategy games. We held the structure fixed through-out our experiments, other than varying the number of inputnodes and heads to match the number of pure strategies. Wesplit queries half and half between the initial sample and re-sampling for our experiments and used ten iterations of re-sampling. For resampling, we chose the variance k ¨ I of thedistribution we sampled from such that the expected distancebetween a random sample and the mean was about 0.05 (thisrequires a different constant k for each dimension).

To perform our experiments, we generate action-graphgames to serve as a proxy for simulators. We consider threeclasses of games: congestion games, additive sine games,and multiplicative sine games. Additive and multiplicativesine games are action-graph games whose function nodescompute respectively the sum or the product of a low-degreepolynomial and a long-period sine function. Each class ofgame is defined by a distribution over the parameters ofaction-graph games of a particular structure. Using the per-formance on sampled games, we present expectations of themedian and mean performance on each class, with 95% con-fidence intervals. In order to have a basis of comparisonacross different settings, the payoffs of every game are nor-malized to mean zero and standard deviation one. Since weare generating the underlying games, we are able to measurethe true regret of the computed approximate equilibria to useas our metric of performance.

Comparison to Existing WorkWe compare deviation payoff learning to Gaussian pro-cess regression integration (GPRI) (Wiedenbeck, Yang, andWellman 2018). GPRI is the existing state of the art forgeneric model learning of fully-symmetric, simultaneous-move games. However, the means by which GPRI recon-structs deviation payoffs from learned utilities is expensiveand scales poorly both with the number of training examples

and the number of pure strategies in the game. In contrast,we show that deviation payoff learning can handle orders ofmagnitude more training examples and pure strategies, andalso generalizes to games with role-asymmetries.

In our comparison, we use symmetric 100 player gameswith five pure strategies and up to 5,000 queries to the simu-lator. We were unable to make a comparison on larger num-bers of strategies or larger data sets due to the limitations ofGPRI. In Figure 3 we see that both deviation payoff learn-ing and GPRI achieve strong results on congestion games.However, on harder games, the performance of GPRI de-teriorates significantly more than that of deviation payofflearning. Thus, in addition to having a much wider rangeof applicability than GPRI, deviation payoff learning tendsto achieve much lower regret within the range of applica-bility of GPRI. Note that the regret scale on these graphs isdramatically larger than on any of the subsequent plots ex-amining deviation payoff learning.

Scaling in NoiseIt is typical in an empirical game for the simulator to benoisy. In our experimental setup, two different noise mod-els are plausible: noise in player payoffs and noise in pure-strategy payoffs. When there is noise in player payoffs, eachplayer receives their own noisy payoff. This provides a natu-ral form of noise reduction, as we can average over the pay-offs of the players choosing the same pure-strategy. Noisein pure-strategy payoffs, where one noisy payoff is providedfor each pure-strategy supported in the payoff profile, offersa strictly more challenging problem. In our experiment, weconsider the harder case of noise in pure-strategy payoffswith the understanding that our method will only becomemore robust under noise in player payoffs. We use sym-metric 100 player, five pure-strategy additive sine games.The noise is sampled from a mean zero Gaussian of vary-ing width.

We see in Figure 4 that deviation payoff learning can tol-erate large amounts of noise while maintaining strong per-formance. Adding mean zero Gaussian noise with standarddeviation 1/10 appears to have a very marginal effect onlearning. Even with noise having standard deviation 1/2,which is on the same order of magnitude as the standard de-

Figure 4: Deviation payoff learning is robust to noise.

Figure 5: The performance of deviation payoff learning ac-tually improves with the number of players for additive sinegames.

viation of payoffs of the game, deviation payoff learning isstill able to find comparably good mixed strategies to thoseGPRI computed with the same number of queries, but with-out noisy payoffs (see Figure 3). The robustness of devia-tion payoff learning to noise is not surprising, as the centralidea of the learning process is to determine the mean of thedistribution of payoffs. Adding mean zero noise increasesthe variance of this distribution but does not fundamentallychange the problem.

Scaling in the Number of PlayersOne of the advantages of learning deviation payoffs di-rectly is that the learning process scales well with the num-ber of players. In particular, unlike existing methods, devi-ation payoff learning avoids learning the tabular representa-tion of the utility function, which grows exponentially withthe number of players (although the deviation payoff func-tion does tend to become more complex due to it being aweighted sum over the exponentially growing utility func-tion).

In Figure 5, we present results from symmetric additivesine games with five pure strategies. It appears that devia-tion payoff learning actually enjoys increased performancewith larger numbers of players for additive sine games. It is

Figure 6: Deviation payoff learning can scale into gameswith large numbers of pure strategies when there are manyplayers.

possible that this is a result of having more training exam-ples (since pure strategies are less often unsupported whenthere are more players). This is strong evidence that devia-tion payoff learning scales extremely well in the number ofplayers (though we are not suggesting that the performancewill improve with more players for games in general).

Scaling in the Number of Pure-StrategiesPast analyses of simulation-based games with large num-bers of pure strategies, such as Dandekar et al. (2015), havelargely been limited to exploring strategies iteratively tofind small-support Nash equilibria. GPRI is similarly lim-ited due to difficulty handling games with more than fivepure-strategies. In contrast, deviation payoff learning offersa significant advancement in that it allows for direct analy-sis of games with a much larger number of pure strategies.This opens the possibility of discovering large-support Nashequilibria in games with many strategies.

As the number of pure strategies becomes large with re-spect to the number of players, it increases the difficulty ofthe learning problem because there are fewer training ex-amples per pure-strategy deviation payoff function. In orderto minimize this effect and focus on scaling in the numberof pure strategies, we fix a large number of players for thisexperiment. Results are presented from symmetric 10,000player additive sine games.

In Figure 6, we see that, in the case in which the numberof players significantly exceeds the number of pure strate-gies, deviation payoff learning achieves surprisingly strongresults for additive sine games even for large numbers ofstrategies. However, we note that our preliminary experi-ments suggest that performance scaling deteriorates signifi-cantly faster when the number of players in the game is morecomparable to the number of pure strategies.

Scaling in the Number of RolesPerhaps the most significant advantage deviation payofflearning can provide is the ability to handle games with roleasymmetries. In practice, many games of interest are notfully symmetric. Yet existing methods have difficulty rep-resenting multi-role games. We assess the ability of devi-

Figure 7: Validating equilibria can help catch mistakes, but on particularly hard (multiplicative sine) games, the samplingestimate can be overly conservative. Dashed lines indicate 45˝, where true and estimated regrets are equal.

ation payoff learning to handle role asymmetries by fixingthe number of players and strategies and varying the numberof roles. Each role is allotted an equal number of players andan equal number of strategies. Results are from 1,200 player,12 pure-strategy additive sine games.

Figure 8: Deviation payoff learning scales well with roleasymmetries.

We observe in Figure 8 that, for the most part, deviationpayoff learning scales positively with the number of roles.This can likely be attributed to the game size decreasing asthe number of roles increases (holding |S| fixed). In otherwords, the number of terms in the deviation payoff functiondecreases with the number of roles.

Validating Learned ModelsWe have demonstrated that deviation payoff learning tendsto produce low regret mixed strategies. While many resultsfor congestion games and multiplicative sine games wereomitted due to space constraints, we found that performancetended to trend similarly. Even in settings where outlierscause higher expected regret, median regret mostly remainslow. It is desirable to be able to distinguish outlying, high

regret mixed-strategy profiles returned by deviation payofflearning from the more typical, low regret mixed-strategyprofiles. In simulated games, accessing the regret directly isnot possible, but approximating deviation payoffs also al-lows us to compute an approximation of regret (details canbe found in the Background and Notation).

We propose that after equilibria have been identified inthe model, the analyst verify the results by simulating manyprofiles sampled from each equilibrium to estimate its regret.If this estimate agrees with the learned model, analysts canhave confidence in the result, but disagreement may indicatethat additional data is required. We found that a large num-ber of samples is required to achieve reliable estimates, soFigure 7 shows results of 1,000 queries per role-symmetricmixed-strategy profile in symmetric 100 player five pure-strategy games. Each mixed-strategy profile was found byrejection sampling to have low true-game regret, and wecompare against regret estimated by sampling. In conges-tion games, these predictions are quite accurate, while insine games, overestimates are more common.

ConclusionWe have developed a novel methodology for learning ε-Nashequilibria in simulation-based games. We have shown thatdeviation payoff learning outperforms the previous best andsets new performance standards for scaling in noise, num-ber of players, number of strategies, and role asymmetries.We believe that the tools presented here will enable newsimulation-based analysis of many exciting multi-agent in-teraction domains.

ReferencesBrown, G. W. 1951. Iterative solution of games by ficti-tious play. Activity analysis of production and allocation13(1):374–376.Chen, Y., and Ye, X. 2011. Projection onto a simplex. arXivpreprint arXiv:1101.6081.

Dandekar, P.; Goel, A.; Wellman, M. P.; and Wiedenbeck, B.2015. Strategic formation of credit networks. ACM Trans-actions on Internet Technology (TOIT) 15(1):3.Duong, Q.; Vorobeychik, Y.; Singh, S. P.; and Wellman,M. P. 2009. Learning graphical game models. In IJCAI,116–121.Govindan, S., and Wilson, R. 2003. A global newton methodto compute nash equilibria. Journal of Economic Theory110(1):65–86.Heinrich, J.; Lanctot, M.; and Silver, D. 2015. Ficti-tious self-play in extensive-form games. In Proceedingsof the 32nd International Conference on Machine Learning(ICML-15), 805–813.Honorio, J., and Ortiz, L. 2015. Learning the structure andparameters of large-population graphical games from behav-ioral data. Journal of Machine Learning Research.Jiang, A. X.; Chan, H.; and Leyton-Brown, K. 2017. Re-source graph games: A compact representation for gameswith structured strategy spaces. In AAAI, 572–578.Jiang, A. X.; Leyton-Brown, K.; and Bhat, N. a. R. 2011.Action-Graph Games. Games and Economic Behavior71(1):141–173.Jordan, P. R.; Kiekintveld, C.; and Wellman, M. P. 2007.Empirical game-theoretic analysis of the tac supply chaingame. In Proceedings of the 6th international joint confer-ence on Autonomous agents and multiagent systems, 193.ACM.Kearns, M. 2007. Graphical games. In Vazirani, V.; Nisan,N.; Roughgarden, T.; and Tardos, E., eds., Algorithmic gametheory. Cambridge, UK: Cambridge University Press. chap-ter 7, 159–180.Lanctot, M.; Zambaldi, V.; Gruslys, A.; Lazaridou, A.;Tuyls, k.; Perolat, J.; Silver, D.; and Graepel, T. 2017. A uni-fied game-theoretic approach to multiagent reinforcementlearning. In Guyon, I.; Luxburg, U. V.; Bengio, S.; Wallach,H.; Fergus, R.; Vishwanathan, S.; and Garnett, R., eds., Ad-vances in Neural Information Processing Systems 30, 4190–4203. Curran Associates, Inc.Leibo, J. Z.; Zambaldi, V.; Lanctot, M.; Marecki, J.; andGraepel, T. 2017. Multi-agent reinforcement learning in se-quential social dilemmas. In Proceedings of the 16th Confer-ence on Autonomous Agents and MultiAgent Systems, 464–473. International Foundation for Autonomous Agents andMultiagent Systems.Moravcık, M.; Schmid, M.; Burch, N.; Lisy, V.; Morrill, D.;Bard, N.; Davis, T.; Waugh, K.; Johanson, M.; and Bowling,M. 2017. Deepstack: Expert-level artificial intelligence inheads-up no-limit poker. Science 356(6337):508–513.Phelps, S.; Marcinkiewicz, M.; and Parsons, S. 2006. Anovel method for automatic strategy acquisition in n-playernon-zero-sum games. In Proceedings of the fifth interna-tional joint conference on Autonomous agents and multia-gent systems, 705–712. ACM.Silver, D.; Schrittwieser, J.; Simonyan, K.; Antonoglou, I.;Huang, A.; Guez, A.; Hubert, T.; Baker, L.; Lai, M.; Bolton,

A.; et al. 2017. Mastering the game of go without humanknowledge. Nature 550(7676):354.Taylor, P. D., and Jonker, L. B. 1978. Evolutionary stablestrategies and game dynamics. Mathematical biosciences40(1-2):145–156.Tuyls, K.; Perolat, J.; Lanctot, M.; Leibo, J. Z.; and Graepel,T. 2018. A generalised method for empirical game theoreticanalysis. CoRR abs/1803.06376.Vorobeychik, Y.; Wellman, M. P.; and Singh, S. 2007. Learn-ing payoff functions in infinite games. Machine Learning67(1-2):145–168.Vytelingum, P.; Cliff, D.; and Jennings, N. R. 2008. Strate-gic bidding in continuous double auctions. Artificial Intelli-gence 172(14):1700–1729.Wah, E., and Wellman, M. P. 2017. Latency arbitrage infragmented markets: A strategic agent-based analysis. Al-gorithmic Finance 5:69–93.Wellman, M. P.; Jordan, P. R.; Kiekintveld, C.; Miller, J.;and Reeves, D. M. 2006. Empirical game-theoretic analysisof the tac market games. In AAMAS-06 Workshop on Game-Theoretic and Decision-Theoretic Agents.Wellman, M. P.; Kim, T. H.; and Duong, Q. 2013. Analyzingincentives for protocol compliance in complex domains: Acase study of introduction-based routing. In 12th Workshopon the Economics of Information Security.Wellman, M. P. 2006. Methods for empirical game-theoreticanalysis (extended abstract). In Proceedings of the NationalConference on Artificial Intelligence, 1152–1155.Wiedenbeck, B.; Yang, F.; and Wellman, M. P. 2018. Aregression approach for modeling games with many sym-metric players. In 32nd AAAI Conference on Artificial Intel-ligence.