Download - a million bots can't be wrong
![Page 1: a million bots can't be wrong](https://reader034.vdocument.in/reader034/viewer/2022052321/5562234fd8b42ad44d8b4e84/html5/thumbnails/1.jpg)
a million bots can't be wrong @remeniuk, Viaden Media #ScalaSBP, 18-05-2012
![Page 2: a million bots can't be wrong](https://reader034.vdocument.in/reader034/viewer/2022052321/5562234fd8b42ad44d8b4e84/html5/thumbnails/2.jpg)
In Viaden our aim is to make the best poker ever
![Page 3: a million bots can't be wrong](https://reader034.vdocument.in/reader034/viewer/2022052321/5562234fd8b42ad44d8b4e84/html5/thumbnails/3.jpg)
we know that performance tests should be
the first-class citizens
![Page 4: a million bots can't be wrong](https://reader034.vdocument.in/reader034/viewer/2022052321/5562234fd8b42ad44d8b4e84/html5/thumbnails/4.jpg)
and kill 2 birds with one stone, using bots for testing
![Page 5: a million bots can't be wrong](https://reader034.vdocument.in/reader034/viewer/2022052321/5562234fd8b42ad44d8b4e84/html5/thumbnails/5.jpg)
#1 we can emulate 50k players using just one medium EC2 instance #2 bots are interactive, so client teams can use them in development, and QA for testing
![Page 6: a million bots can't be wrong](https://reader034.vdocument.in/reader034/viewer/2022052321/5562234fd8b42ad44d8b4e84/html5/thumbnails/6.jpg)
everyone knows Akka, huh?
![Page 7: a million bots can't be wrong](https://reader034.vdocument.in/reader034/viewer/2022052321/5562234fd8b42ad44d8b4e84/html5/thumbnails/7.jpg)
why Scala and Akka is a perfect choice for making bots?
actors are !(interactive)straightforward remotingsimple scalability/clustering~30 minutes to make a DSL and CLI with Scala and SBT
![Page 8: a million bots can't be wrong](https://reader034.vdocument.in/reader034/viewer/2022052321/5562234fd8b42ad44d8b4e84/html5/thumbnails/8.jpg)
..., but, I have to warn you ...
![Page 9: a million bots can't be wrong](https://reader034.vdocument.in/reader034/viewer/2022052321/5562234fd8b42ad44d8b4e84/html5/thumbnails/9.jpg)
4 dead simple tips for keeping your sanity
when you do asynch with Akka 2.0
![Page 10: a million bots can't be wrong](https://reader034.vdocument.in/reader034/viewer/2022052321/5562234fd8b42ad44d8b4e84/html5/thumbnails/10.jpg)
tip #1: live fast, die young
![Page 11: a million bots can't be wrong](https://reader034.vdocument.in/reader034/viewer/2022052321/5562234fd8b42ad44d8b4e84/html5/thumbnails/11.jpg)
typical approach in Akka 1.x
lobby
desk
botlogin lin
k
tourney
![Page 12: a million bots can't be wrong](https://reader034.vdocument.in/reader034/viewer/2022052321/5562234fd8b42ad44d8b4e84/html5/thumbnails/12.jpg)
Akka 1.x: actors have a long lifecycle
lobby
desk
bot
botplay game
link
linkun
link
tourney
![Page 13: a million bots can't be wrong](https://reader034.vdocument.in/reader034/viewer/2022052321/5562234fd8b42ad44d8b4e84/html5/thumbnails/13.jpg)
lobby
desk
tourney
bot
bot
botplay tournament linkun
link
Akka 1.x: actors have a long lifecycle
![Page 14: a million bots can't be wrong](https://reader034.vdocument.in/reader034/viewer/2022052321/5562234fd8b42ad44d8b4e84/html5/thumbnails/14.jpg)
in Akka 2.0 the world has changed
actors
paths
props
newsupervision
![Page 15: a million bots can't be wrong](https://reader034.vdocument.in/reader034/viewer/2022052321/5562234fd8b42ad44d8b4e84/html5/thumbnails/15.jpg)
now, you're forced to do "the right thing" (c)
lobby
tournament desk
desk
IdleBot
DeskBot
DeskBot
![Page 16: a million bots can't be wrong](https://reader034.vdocument.in/reader034/viewer/2022052321/5562234fd8b42ad44d8b4e84/html5/thumbnails/16.jpg)
all actors are supervised
lobby
desk login
![Page 17: a million bots can't be wrong](https://reader034.vdocument.in/reader034/viewer/2022052321/5562234fd8b42ad44d8b4e84/html5/thumbnails/17.jpg)
easy come
lobby
desk
IdleBotplay game
![Page 18: a million bots can't be wrong](https://reader034.vdocument.in/reader034/viewer/2022052321/5562234fd8b42ad44d8b4e84/html5/thumbnails/18.jpg)
easy go (when supervisor to be changed)
lobby
desk
IdleBotDeskBot
dies
borns
![Page 19: a million bots can't be wrong](https://reader034.vdocument.in/reader034/viewer/2022052321/5562234fd8b42ad44d8b4e84/html5/thumbnails/19.jpg)
class Lobby extends Actor {
case Login(username, password) => context.actorOf(Props(new IdlePokerBot(...)))
case JoinAnyDesk(props) => findDesk ! JoinDesk(props) } class Desk extends Actor {
case JoinDesk(botProps)=> context.actorOf(Props(new DeskPokerBot(botProps)))
} class IdlePokerBot(props: BotProps) extends Actor {
case PlayNow => context.parent ! JoinAnyDesk(props); context.stop(self)
}
![Page 20: a million bots can't be wrong](https://reader034.vdocument.in/reader034/viewer/2022052321/5562234fd8b42ad44d8b4e84/html5/thumbnails/20.jpg)
Props Pattern - "the soul" of an actor
IdleBot
DeskBot
BotProps
BotProps
Props remains alive between actor "reincarnations"
![Page 21: a million bots can't be wrong](https://reader034.vdocument.in/reader034/viewer/2022052321/5562234fd8b42ad44d8b4e84/html5/thumbnails/21.jpg)
case class BotProperties(id: Long, login: String, script: Seq[BotEvent], aggressiveness: Int, sessionId: Protocol.SessionId) class IdlePokerBot(val botProperties: BotProperties) extends Bot[Poker] class DeskPokerBot(val botProperties: BotProperties) extends Bot[Poker]
![Page 22: a million bots can't be wrong](https://reader034.vdocument.in/reader034/viewer/2022052321/5562234fd8b42ad44d8b4e84/html5/thumbnails/22.jpg)
tip #2: think beyond
![Page 23: a million bots can't be wrong](https://reader034.vdocument.in/reader034/viewer/2022052321/5562234fd8b42ad44d8b4e84/html5/thumbnails/23.jpg)
when you know, who's supervising, life's simple
akka://gpdev/user/lobby/player1234 akka://gpdev/user/lobby/desk1/player1234 akka://gpdev/user/lobby/tournament1/desk1/player1234
![Page 24: a million bots can't be wrong](https://reader034.vdocument.in/reader034/viewer/2022052321/5562234fd8b42ad44d8b4e84/html5/thumbnails/24.jpg)
ActorRegistry, actor UUID
but what should I do, now, when I don't know, where to look for my bot?
were removed from Akka
Bad news
![Page 25: a million bots can't be wrong](https://reader034.vdocument.in/reader034/viewer/2022052321/5562234fd8b42ad44d8b4e84/html5/thumbnails/25.jpg)
you can make your own registry (using Extensions, backed with a distributed data structure)...
![Page 26: a million bots can't be wrong](https://reader034.vdocument.in/reader034/viewer/2022052321/5562234fd8b42ad44d8b4e84/html5/thumbnails/26.jpg)
or, use the full power of location transparency
lobby
desk
IdleBot
DeskBot
ProjectionManager
Projectionvar location:
ActorPath
/lobby/desk123/player123/projection/player123
![Page 27: a million bots can't be wrong](https://reader034.vdocument.in/reader034/viewer/2022052321/5562234fd8b42ad44d8b4e84/html5/thumbnails/27.jpg)
class Projection(var container: ActorPath) extends Actor { def receive = { case newContainer: ActorPath => container = newContainer case msg => context.actorFor(container.child(self.path.name)) ! msg } } class ProjectionManager extends Actor { def receive = { case Add(actor) => context.actorOf(Props(new
Projection(actor.path.parent)), actor.path.name) case Forward(msg, path) => context.actorFor(path) ! msg } } projectionManager ! Add(actorRef)projectionManager ! Forward("ping", "actor1")
![Page 28: a million bots can't be wrong](https://reader034.vdocument.in/reader034/viewer/2022052321/5562234fd8b42ad44d8b4e84/html5/thumbnails/28.jpg)
![Page 29: a million bots can't be wrong](https://reader034.vdocument.in/reader034/viewer/2022052321/5562234fd8b42ad44d8b4e84/html5/thumbnails/29.jpg)
class Projection(var container: ActorPath) extends Actor { def receive = { case newContainer: ActorPath => container = newContainer case msg => context.actorFor(container.child(self.path.name)) ! msg } } class ProjectionManager extends Actor { def receive = { case Add(actor) => context.actorOf(Props(new
Projection(actor.path.parent)), actor.path.name) case Forward(msg, path) => context.actorFor(path) ! msg } } projectionManager ! Add(actorRef)system.actorFor(projectionManager.path.child("actor" + i)) ! "ping"
![Page 30: a million bots can't be wrong](https://reader034.vdocument.in/reader034/viewer/2022052321/5562234fd8b42ad44d8b4e84/html5/thumbnails/30.jpg)
![Page 31: a million bots can't be wrong](https://reader034.vdocument.in/reader034/viewer/2022052321/5562234fd8b42ad44d8b4e84/html5/thumbnails/31.jpg)
class ProjectionManager extends Actor { def receive = { case Add(actor) => context.actorOf(Props(new
Projection(actor.path.parent)), actor.path.name) case Forward(msg, path) => context.actorSelection("../*/" + path) ! msg } } val projectionManager = system.actorOf(Props[ProjectionManagerRoutee]
.withRouter(RoundRobinRouter(resizer = Some(DefaultResizer(lowerBound = 10, upperBound = 20)))), "projection")
projectionManager ! Add(actorRef)projectionManager ! Forward("ping", "actor1")
![Page 32: a million bots can't be wrong](https://reader034.vdocument.in/reader034/viewer/2022052321/5562234fd8b42ad44d8b4e84/html5/thumbnails/32.jpg)
![Page 33: a million bots can't be wrong](https://reader034.vdocument.in/reader034/viewer/2022052321/5562234fd8b42ad44d8b4e84/html5/thumbnails/33.jpg)
case class CustomRouter(n: Int, routerDispatcher: String = DefaultDispatcherId, supervisorStrategy: SupervisorStrategy = defaultStrategy) extends RouterConfig { def createRoute(props: Props, provider: RouteeProvider) = { provider.registerRoutees((1 to n).map(i => provider.context.actorOf(Props[ProjectionManager], i.toString))) def destination(sender: ActorRef, path: String) = List(Destination(sender, provider.routees(abs(path.hashCode) % n))) { case m@(sender, Add(actor)) ⇒ destination(sender, actor.path.name) case m@(sender, Forward(_, name)) ⇒ destination(sender, name) } } }
![Page 34: a million bots can't be wrong](https://reader034.vdocument.in/reader034/viewer/2022052321/5562234fd8b42ad44d8b4e84/html5/thumbnails/34.jpg)
![Page 35: a million bots can't be wrong](https://reader034.vdocument.in/reader034/viewer/2022052321/5562234fd8b42ad44d8b4e84/html5/thumbnails/35.jpg)
tip #3: don't do anything stupid
![Page 36: a million bots can't be wrong](https://reader034.vdocument.in/reader034/viewer/2022052321/5562234fd8b42ad44d8b4e84/html5/thumbnails/36.jpg)
you've tried all the options, system load is fine, only 1/10 of the heap is used, but you still can start not more than 1k bots!?
![Page 37: a million bots can't be wrong](https://reader034.vdocument.in/reader034/viewer/2022052321/5562234fd8b42ad44d8b4e84/html5/thumbnails/37.jpg)
ulimit -n <proper value>
![Page 38: a million bots can't be wrong](https://reader034.vdocument.in/reader034/viewer/2022052321/5562234fd8b42ad44d8b4e84/html5/thumbnails/38.jpg)
your actor is lacking of throughput? wait before adding poolsshare responsibility!one fine-grained actor is enough in 99% of the cases
![Page 39: a million bots can't be wrong](https://reader034.vdocument.in/reader034/viewer/2022052321/5562234fd8b42ad44d8b4e84/html5/thumbnails/39.jpg)
100-300 threads are serving 300 bots!?
![Page 40: a million bots can't be wrong](https://reader034.vdocument.in/reader034/viewer/2022052321/5562234fd8b42ad44d8b4e84/html5/thumbnails/40.jpg)
spawn futures, backed with standalone [bounded] pools, for blocking operations
class ThirdPartyWrapper extends Actor { case F(x) => sender ! thirdPartyService.f(x) // call to a function that takes a lot of time to // complete } class ThirdPartyWrapper extends Actor { case F(x) => val _sender = sender Future(thirdPartyService.f(x)).map(_sender ! _) // ugly, but safe, and perfectly right}
![Page 41: a million bots can't be wrong](https://reader034.vdocument.in/reader034/viewer/2022052321/5562234fd8b42ad44d8b4e84/html5/thumbnails/41.jpg)
use separate dispatchers
lobby
desk
DeskBot
ProjectionManager
Projection
projection-manager-dispatcherBalancingDispatcher
projection-dispatcherDispatcher
lobby-dispatcherPinnedDispatcher
container-dispatcherDispatcher
desk-bot-dispatcherDispatcher
![Page 42: a million bots can't be wrong](https://reader034.vdocument.in/reader034/viewer/2022052321/5562234fd8b42ad44d8b4e84/html5/thumbnails/42.jpg)
GOTCHA: Akka successfully bootstraps, even if your dispatcher is not configured, or the config is wrong Always check the logs to make sure that dispatchers are used!
[WARN][gpdev-akka.actor.default-dispatcher-1] [Dispatchers] Dispatcher [bot-system.container-dispatcher] not configured, using default-dispatcher[WARN][gpdev-bot-system.container-dispatcher-1] [PinnedDispatcherConfigurator] PinnedDispatcher [bot-system.lobby-dispatcher] not configured to use ThreadPoolExecutor, falling back to default config. [DEBUG][gpdev-akka.actor.default-dispatcher-24] [akka://gpdev/user/poker/lobby] logged in[DEBUG][gpdev-akka.actor.default-dispatcher-14] [akka://gpdev/user/poker/projeciton/$g/player20013] starting projection...
![Page 43: a million bots can't be wrong](https://reader034.vdocument.in/reader034/viewer/2022052321/5562234fd8b42ad44d8b4e84/html5/thumbnails/43.jpg)
tip #4: analyze that
![Page 44: a million bots can't be wrong](https://reader034.vdocument.in/reader034/viewer/2022052321/5562234fd8b42ad44d8b4e84/html5/thumbnails/44.jpg)
how to measure? Metrics - pushes various collected metrics to GraphiteCarbon and Graphite - gather metrics, and expose them via web interface
![Page 45: a million bots can't be wrong](https://reader034.vdocument.in/reader034/viewer/2022052321/5562234fd8b42ad44d8b4e84/html5/thumbnails/45.jpg)
object BotMetrics { val loggedInCount = new Counter(Metrics.newCounter(classOf[Lobby[_]], "logged-in-count")) GraphiteReporter.enable(1, TimeUnit.MINUTES, "localhost", 2003) } class Lobby extends Actor { case Login(username, pass) => BotMetrics.loggedInCount += 1 }
1. add logged-in user counter 2. update it3. enable reporting to Graphite4. build a graph in Grtaphite
1.
2.
3.
4.
![Page 46: a million bots can't be wrong](https://reader034.vdocument.in/reader034/viewer/2022052321/5562234fd8b42ad44d8b4e84/html5/thumbnails/46.jpg)
what to measure? - mailbox size1
- throughput- time, before the message is processed (both in actor and future)2
- time to process a message- count of threads- actor pool size- heap size
1 requires implementation of a custom mailbox that can expose mailbox size2 every message should be stuffed with a timestamp
![Page 47: a million bots can't be wrong](https://reader034.vdocument.in/reader034/viewer/2022052321/5562234fd8b42ad44d8b4e84/html5/thumbnails/47.jpg)
how to tune dispatchers? VisualVM - thread timeline shows, if thread polls behind dispatchers are used effectively
![Page 48: a million bots can't be wrong](https://reader034.vdocument.in/reader034/viewer/2022052321/5562234fd8b42ad44d8b4e84/html5/thumbnails/48.jpg)
don't neglect old good logging [ERROR][05/06/2012 12:55:43.826] [gpdev-bot-system.desk-bot-dispatcher-7] [akka://gpdev/user/poker/lobby/tournament5382577/desk109129/player20121] unprocessed game event: GameEvent(CHAT,None,None)
![Page 49: a million bots can't be wrong](https://reader034.vdocument.in/reader034/viewer/2022052321/5562234fd8b42ad44d8b4e84/html5/thumbnails/49.jpg)
thanks for listening