akka - developing seda based applications
DESCRIPTION
Developing SEDA Based Applications using Akka.TRANSCRIPT
AkkaDeveloping SEDA Based
Applications
Me Ben Darfler
@bdarfler http://bdarfler.com Senior Software Engineer at Localytics
Localytics Real time mobile analytics platform
40M+ events per day and growing rapidly
3x growth over the past 3 months
Heavy users of Scala/Akka/NoSql We are hiring (seriously, come talk to me)
Localytics
How to keep up with our growth?
Actor Model Lock free approach to concurrency No shared state between actors Asynchronous message passing Mailboxes to buffer incoming messages
AkkaConfigurable● Dispatchers● Mailboxes
Fault Tolerant● Supervisors
Great community● @jboner● @viktorklang
AkkaPerformant
http://blog.jayway.com/2010/08/10/yet-another-akka-benchmark/
SEDAStaged Event Driven Architecture
"Decomposes a complex, event-driven application into a set of stages connected by queues."1
"The most fundamental aspect of the SEDA architecture is the programming model that supports stage-level backpressure and load management."1
1. http://www.eecs.harvard.edu/~mdw/proj/seda/
Backpressure
Whats the big deal?
BackpressureManditory to prevent OutOfMemoryError● Messages backup in memory faster than they
can be processed
Cassandra was seriously bitten by this● Less crappy failure mode when swamped with
inserts than "run out of memory and gc-storm to death" (CASSANDRA-401)
● Add backpressure to StorageProxy (CASSANDRA-685)
BackpressureMailboxes
case class UnboundedMailbox(val blocking: Boolean = false) extends MailboxType
case class BoundedMailbox( val blocking: Boolean = false, val capacity: Int = { if (Dispatchers.MAILBOX_CAPACITY < 0) Int.MaxValue else Dispatchers.MAILBOX_CAPACITY }, val pushTimeOut: Duration = Dispatchers.MAILBOX_PUSH_TIME_OUT) extends MailboxType
BoundedMailbox(false, QUEUE_SIZE, Duration(-1, "millis"))
Backpressure Mailbox
Stages
How do we decompose the problem?
StagesOne actor class per stage
Shared dispatcher
Individually tunable● I/O Bound● CPU Bound
Easier to reason about
Code reuse
DispatchersThreadBasedDispatcher
● Binds one actor to its own thread
ExecutorBasedEventDrivenDispatcher● Must be shared between actors
ExecutorBasedEventDrivenWorkStealingDispatcher● Must be shared between actors of the same type
Queues
SEDA has a queue per stage model
Akka actors have their own mailbox
How do we evenly distribute work?
Work StealingExecutorBasedEventDrivenWorkStealingDispatcher
"Actors of the same type can be set up to share this dispatcher and during execution time the different actors will steal messages from other actors if they have less messages to process"1
1. http://doc.akka.io/dispatchers-scala
Work StealingReally a work "donating" dispatcher
�"I have implemented a work stealing dispatcher for Akka actors. Although its called "work stealing" the implementation actually behaves more as "work donating" because the victim actor takes the initiative. I.e. it actually donates work to its thief, rather than having the thief steal work from the victim."1
1. http://janvanbesien.blogspot.com/2010/03/load-balancing-actors-with-work.html
Work Stealing
Doesn't that conflict with blocking mailboxes?
Work StealingSending actor will block on the receiving actors mailbox before it can "donate"
Might be fixed in Akka 1.1● I owe @viktorklang a test of his latest changes
Load Balancing
Can we distribute work on the sender side?
Load BalancingRouting.loadBalancerActor()● Creates a new actor that forwards
messages in a load balancing fashion
InfiniteIterator● CyclicIterator● SmallestMailboxFirstIterator
Load Balancing
Doesn't the load balancer need a blocking mailbox?
Load BalancingCan't easily change the load balancer's mailbox
Use SmallestMailboxFirstIterator directly
new SmallestMailboxFirstIterator(List(actor, actor, actor))
Fault ToleranceSupervisors
● Restarts actors● Stops after x times within y milliseconds
Restart Strategies● OneForOne● AllForOne
Fault ToleranceGreat for transient issues
● Network failures
Not great for permanent issues● OutOfMemoryError
Final Product// Actor creationval supervisor = Supervisor(SupervisorConfig( OneForOneStrategy(List(classOf[Exception]), RETRIES, WITH_IN_TIME), Supervise(myActors))
def myActors: List[Supervise] = { val mailbox = BoundedMailbox(false, QUEUE_SIZE, Duration(-1, "millis")) val dispatcher = Dispatchers.newExecutorBasedEventDrivenDispatcher( "my-dispatcher", 1, mailbox).setCorePoolSize(POOL_SIZE).build (1 to POOL_SIZE toList).foldRight(List[Supervise]()) { (i, list) => Supervise(actorOf(new MyActor("my-actor-" + i, dispatcher)), Permanent) :: list }}
// Sending a messageval actors = new SmallestMailboxFirstIterator(actorsFor(classOf[MyActor]).toList)def actor = actors.nextactor ! Message()
Thanks
@bdarflerhttp://bdarfler.com