Download - Akka in Production: Our Story
![Page 1: Akka in Production: Our Story](https://reader034.vdocument.in/reader034/viewer/2022051608/540dbc348d7f72747e8b4b1f/html5/thumbnails/1.jpg)
Akka in ProductionOur Story
Evan ChanPNWScala 2013
Saturday, October 19, 13
![Page 2: Akka in Production: Our Story](https://reader034.vdocument.in/reader034/viewer/2022051608/540dbc348d7f72747e8b4b1f/html5/thumbnails/2.jpg)
• Staff Engineer, Compute and Data Services, Ooyala
• Building multiple web-scale real-time systems on top of C*, Kafka, Storm, etc.
• github.com/velvia
• Author of ScalaStorm, Scala DSL for Storm
• @evanfchan
Who is this guy?
2
Saturday, October 19, 13
![Page 3: Akka in Production: Our Story](https://reader034.vdocument.in/reader034/viewer/2022051608/540dbc348d7f72747e8b4b1f/html5/thumbnails/3.jpg)
WANT REACTIVE?
3
event-driven, scalable, resilient and responsive
Saturday, October 19, 13
![Page 4: Akka in Production: Our Story](https://reader034.vdocument.in/reader034/viewer/2022051608/540dbc348d7f72747e8b4b1f/html5/thumbnails/4.jpg)
SCALA AND AKKAAT OOYALA
4
Saturday, October 19, 13
![Page 5: Akka in Production: Our Story](https://reader034.vdocument.in/reader034/viewer/2022051608/540dbc348d7f72747e8b4b1f/html5/thumbnails/5.jpg)
CONFIDENTIAL—DO NOT DISTRIBUTE 5
Founded in 2007
Commercially launch in 2009
230+ employees in Silicon Valley, LA, NYC, London, Paris, Tokyo, Sydney & Guadalajara
Global footprint, 200M unique users,110+ countries, and more than 6,000 websites
Over 1 billion videos played per month and 2 billion analytic events per day
25% of U.S. online viewers watch video powered by Ooyala
COMPANY OVERVIEW
Saturday, October 19, 13
![Page 6: Akka in Production: Our Story](https://reader034.vdocument.in/reader034/viewer/2022051608/540dbc348d7f72747e8b4b1f/html5/thumbnails/6.jpg)
How we started using Scala
• Ooyala was a mostly Ruby company - even MR jobs
• Lesson - don’t use Ruby for big data
• Started exploring Scala for real-time analytics and MR
• Realized a 1-2 orders of magnitude performance boost from Scala
• Today use Scala, Akka with Storm, Spark, MR, Cassandra, all new big data pipelines
Saturday, October 19, 13
![Page 7: Akka in Production: Our Story](https://reader034.vdocument.in/reader034/viewer/2022051608/540dbc348d7f72747e8b4b1f/html5/thumbnails/7.jpg)
Ingesting 2 Billion Events / Day
NginxRaw Log Feeder Kafka
Storm
New Stuff
Consumer watches video
Saturday, October 19, 13
![Page 8: Akka in Production: Our Story](https://reader034.vdocument.in/reader034/viewer/2022051608/540dbc348d7f72747e8b4b1f/html5/thumbnails/8.jpg)
Livelogsd - Akka/Kafka file tailer
Current File
Rotated File
Rotated File 2
File Reader Actor
File Reader Actor
Kafka Feeder
CoordinatorKafka
Saturday, October 19, 13
![Page 9: Akka in Production: Our Story](https://reader034.vdocument.in/reader034/viewer/2022051608/540dbc348d7f72747e8b4b1f/html5/thumbnails/9.jpg)
Storm - with or without Akka?
Kafka Spout
Bolt
Actor
Actor
• Actors talking to each other within a bolt for locality
• Don’t really need Actors in Storm
• In production, found Storm too complex to troubleshoot
• It’s 2am - what should I restart? Supervisor? Nimbus? ZK?
Saturday, October 19, 13
![Page 10: Akka in Production: Our Story](https://reader034.vdocument.in/reader034/viewer/2022051608/540dbc348d7f72747e8b4b1f/html5/thumbnails/10.jpg)
Akka Cluster-based Pipeline
Kafka Consumer
Spray endpoint
Cluster Router
Processing Actors
Kafka Consumer
Spray endpoint
Cluster Router
Processing Actors
Kafka Consumer
Spray endpoint
Cluster Router
Processing Actors
Kafka Consumer
Spray endpoint
Cluster Router
Processing Actors
Kafka Consumer
Spray endpoint
Cluster Router
Processing Actors
Saturday, October 19, 13
![Page 11: Akka in Production: Our Story](https://reader034.vdocument.in/reader034/viewer/2022051608/540dbc348d7f72747e8b4b1f/html5/thumbnails/11.jpg)
Lessons Learned
• Still too complex -- would we want to get paged for this system?
• Akka cluster in 2.1 was not ready for production (newer 2.2.x version is stable)
• Mixture of actors and futures for HTTP requests became hard to grok
• Actors were much easier for most developers to understand
Saturday, October 19, 13
![Page 12: Akka in Production: Our Story](https://reader034.vdocument.in/reader034/viewer/2022051608/540dbc348d7f72747e8b4b1f/html5/thumbnails/12.jpg)
Simplified Ingestion Pipeline
Kafka Partition
1
Kafka SimpleConsumer
Converter Actor
Cassandra Writer Actor
Kafka Partition
2
Kafka SimpleConsumer
Converter Actor
Cassandra Writer Actor
• Kafka used to partition messages
• Single process - super simple!
• No distribution of data
• Linear actor pipeline - very easy to understand
Saturday, October 19, 13
![Page 13: Akka in Production: Our Story](https://reader034.vdocument.in/reader034/viewer/2022051608/540dbc348d7f72747e8b4b1f/html5/thumbnails/13.jpg)
STACKABLE ACTOR TRAITS
13
Saturday, October 19, 13
![Page 14: Akka in Production: Our Story](https://reader034.vdocument.in/reader034/viewer/2022051608/540dbc348d7f72747e8b4b1f/html5/thumbnails/14.jpg)
Why Stackable Traits?
• Keep adding monitoring, logging, metrics, tracing code gets pretty ugly and repetitive
• We want some standard behavior around actors -- but we need to wrap the actor Receive block:
class someActor extends Actor { def wrappedReceive: Receive = { case x => blah } def receive = { case x => println(“Do something before...”) wrappedReceive(x) println(“Do something after...”) }}
Saturday, October 19, 13
![Page 15: Akka in Production: Our Story](https://reader034.vdocument.in/reader034/viewer/2022051608/540dbc348d7f72747e8b4b1f/html5/thumbnails/15.jpg)
Start with a base trait...
trait ActorStack extends Actor { /** Actor classes should implement this partialFunction for standard * actor message handling */ def wrappedReceive: Receive
/** Stackable traits should override and call super.receive(x) for * stacking functionality */ def receive: Receive = { case x => if (wrappedReceive.isDefinedAt(x)) wrappedReceive(x) else unhandled(x) }}
Saturday, October 19, 13
![Page 16: Akka in Production: Our Story](https://reader034.vdocument.in/reader034/viewer/2022051608/540dbc348d7f72747e8b4b1f/html5/thumbnails/16.jpg)
Instrumenting Traits...
trait Instrument1 extends ActorStack { override def receive: Receive = { case x => println("Do something before...") super.receive(x) println("Do something after...") }}
trait Instrument2 extends ActorStack { override def receive: Receive = { case x => println("Antes...") super.receive(x) println("Despues...") }}
Saturday, October 19, 13
![Page 17: Akka in Production: Our Story](https://reader034.vdocument.in/reader034/viewer/2022051608/540dbc348d7f72747e8b4b1f/html5/thumbnails/17.jpg)
Now just mix the Traits in....
class DummyActor extends Actor with Instrument1 with Instrument2 { def wrappedReceive = { case "something" => println("Got something") case x => println("Got something else: " + x) }}
• Traits add instrumentation; Actors stay clean!
• Order of mixing in traits matter
Antes...Do something before...Got somethingDo something after...Despues...
Saturday, October 19, 13
![Page 18: Akka in Production: Our Story](https://reader034.vdocument.in/reader034/viewer/2022051608/540dbc348d7f72747e8b4b1f/html5/thumbnails/18.jpg)
PRODUCTIONIZING AKKA
18
Saturday, October 19, 13
![Page 19: Akka in Production: Our Story](https://reader034.vdocument.in/reader034/viewer/2022051608/540dbc348d7f72747e8b4b1f/html5/thumbnails/19.jpg)
Our Akka Stack
• Spray - high performance HTTP
• SLF4J / Logback
• Yammer Metrics
• spray-json
• Akka 2.x
• Scala 2.9 / 2.10
Saturday, October 19, 13
![Page 20: Akka in Production: Our Story](https://reader034.vdocument.in/reader034/viewer/2022051608/540dbc348d7f72747e8b4b1f/html5/thumbnails/20.jpg)
On distributed systems:“The only thing that matters is
Visibility”
20
Saturday, October 19, 13
![Page 21: Akka in Production: Our Story](https://reader034.vdocument.in/reader034/viewer/2022051608/540dbc348d7f72747e8b4b1f/html5/thumbnails/21.jpg)
Using Logback with Akka
• Pretty easy setup
• Include the Logback jar
• In your application.conf:event-handlers = ["akka.event.slf4j.Slf4jEventHandler"]
• Use a custom logging trait, not ActorLogging
• ActorLogging does not allow adjustable logging levels
• Want the Actor path in your messages?• org.slf4j.MDC.put(“actorPath”, self.path.toString)
Saturday, October 19, 13
![Page 22: Akka in Production: Our Story](https://reader034.vdocument.in/reader034/viewer/2022051608/540dbc348d7f72747e8b4b1f/html5/thumbnails/22.jpg)
Using Logback with Akka
trait Slf4jLogging extends Actor with ActorStack { val logger = LoggerFactory.getLogger(getClass) private[this] val myPath = self.path.toString
logger.info("Starting actor " + getClass.getName)
override def receive: Receive = { case x => org.slf4j.MDC.put("akkaSource", myPath) super.receive(x) }}
Saturday, October 19, 13
![Page 23: Akka in Production: Our Story](https://reader034.vdocument.in/reader034/viewer/2022051608/540dbc348d7f72747e8b4b1f/html5/thumbnails/23.jpg)
Akka Performance Metrics
• We define a trait that adds two metrics for every actor:
• frequency of messages handled (1min, 5min, 15min moving averages)
• time spent in receive block
• All metrics exposed via a Spray route /metricz
• Daemon polls /metricz and sends to metrics service
• Would like: mailbox size, but this is hard
Saturday, October 19, 13
![Page 24: Akka in Production: Our Story](https://reader034.vdocument.in/reader034/viewer/2022051608/540dbc348d7f72747e8b4b1f/html5/thumbnails/24.jpg)
Akka Performance Metrics
trait ActorMetrics extends ActorStack { // Timer includes a histogram of wrappedReceive() duration as well as moving avg of rate of invocation val metricReceiveTimer = Metrics.newTimer(getClass, "message-handler", TimeUnit.MILLISECONDS, TimeUnit.SECONDS)
override def receive: Receive = { case x => val context = metricReceiveTimer.time() try { super.receive(x) } finally { context.stop() } }}
Saturday, October 19, 13
![Page 25: Akka in Production: Our Story](https://reader034.vdocument.in/reader034/viewer/2022051608/540dbc348d7f72747e8b4b1f/html5/thumbnails/25.jpg)
Performance Metrics (cont’d)
Saturday, October 19, 13
![Page 26: Akka in Production: Our Story](https://reader034.vdocument.in/reader034/viewer/2022051608/540dbc348d7f72747e8b4b1f/html5/thumbnails/26.jpg)
Performance Metrics (cont’d)
Saturday, October 19, 13
![Page 27: Akka in Production: Our Story](https://reader034.vdocument.in/reader034/viewer/2022051608/540dbc348d7f72747e8b4b1f/html5/thumbnails/27.jpg)
Flow control
• By default, actor mailboxes are unbounded
• Using bounded mailboxes
• When mailbox is full, messages go to DeadLetters
• mailbox-push-timeout-time: how long to wait when mailbox is full
• Doesn’t work for distributed Akka systems!
• Real flow control: pull, push with acks, etc.
• Works anywhere, but more work
Saturday, October 19, 13
![Page 28: Akka in Production: Our Story](https://reader034.vdocument.in/reader034/viewer/2022051608/540dbc348d7f72747e8b4b1f/html5/thumbnails/28.jpg)
Flow control (Cont’d)
• A working flow control system causes the rate of all actor components to be in sync.
• Witness this message flow rate graph of the start of event processing:
Saturday, October 19, 13
![Page 29: Akka in Production: Our Story](https://reader034.vdocument.in/reader034/viewer/2022051608/540dbc348d7f72747e8b4b1f/html5/thumbnails/29.jpg)
VisualVM and Akka• Bounded mailboxes = time spent enqueueing msgs
Saturday, October 19, 13
![Page 30: Akka in Production: Our Story](https://reader034.vdocument.in/reader034/viewer/2022051608/540dbc348d7f72747e8b4b1f/html5/thumbnails/30.jpg)
VisualVM and Akka
• My dream: a VisualVM plugin to visualize Actor utilization across threads
Saturday, October 19, 13
![Page 31: Akka in Production: Our Story](https://reader034.vdocument.in/reader034/viewer/2022051608/540dbc348d7f72747e8b4b1f/html5/thumbnails/31.jpg)
Tracing Akka Message Flows
• Stack trace is very useful for traditional apps, but for Akka apps, you get this:
at akka.dispatch.Future$$anon$3.liftedTree1$1(Future.scala:195) ~[akka-actor-2.0.5.jar:2.0.5]
at akka.dispatch.Future$$anon$3.run(Future.scala:194) ~[akka-actor-2.0.5.jar:2.0.5]
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:94) [akka-actor-2.0.5.jar:2.0.5]
at akka.jsr166y.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1381) [akka-actor-2.0.5.jar:2.0.5]
at akka.jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:259) [akka-actor-2.0.5.jar:2.0.5]
at akka.jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:975) [akka-actor-2.0.5.jar:2.0.5]
at akka.jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1479) [akka-actor-2.0.5.jar:2.0.5]
at akka.jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104) [akka-actor-2.0.5.jar:2.0.5]
--> trAKKAr message trace <-- akka://Ingest/user/Super --> akka://Ingest/user/K1: Initialize akka://Ingest/user/K1 --> akka://Ingest/user/Converter: Data
• What if you could get an Akka message trace?
Saturday, October 19, 13
![Page 32: Akka in Production: Our Story](https://reader034.vdocument.in/reader034/viewer/2022051608/540dbc348d7f72747e8b4b1f/html5/thumbnails/32.jpg)
Tracing Akka Message Flows
Saturday, October 19, 13
![Page 33: Akka in Production: Our Story](https://reader034.vdocument.in/reader034/viewer/2022051608/540dbc348d7f72747e8b4b1f/html5/thumbnails/33.jpg)
Tracing Akka Message Flows
• Trait sends an Edge(source, dest, messageInfo) to a local Collector actor
• Aggregate edges across nodes, graph and profit!
trait TrakkarExtractor extends TrakkarBase with ActorStack { import TrakkarUtils._
val messageIdExtractor: MessageIdExtractor = randomExtractor
override def receive: Receive = { case x => lastMsgId = (messageIdExtractor orElse randomExtractor)(x) Collector.sendEdge(sender, self, lastMsgId, x) super.receive(x) }}
Saturday, October 19, 13
![Page 34: Akka in Production: Our Story](https://reader034.vdocument.in/reader034/viewer/2022051608/540dbc348d7f72747e8b4b1f/html5/thumbnails/34.jpg)
Good Akka development practices
• Don't put things that can fail into Actor constructor
• Default supervision strategy stops an Actor which cannot initialize itself
• Instead use an Initialize message
• Put your messages in the Actor’s companion object
• Namespacing is nice
Saturday, October 19, 13
![Page 35: Akka in Production: Our Story](https://reader034.vdocument.in/reader034/viewer/2022051608/540dbc348d7f72747e8b4b1f/html5/thumbnails/35.jpg)
PUTTING IT ALL TOGETHER
35
Saturday, October 19, 13
![Page 36: Akka in Production: Our Story](https://reader034.vdocument.in/reader034/viewer/2022051608/540dbc348d7f72747e8b4b1f/html5/thumbnails/36.jpg)
Akka Visibility, Minimal Footprint
trait InstrumentedActor extends Slf4jLogging with ActorMetrics with TrakkarExtractor
object MyWorkerActor { case object Initialize case class DoSomeWork(desc: String)}
class MyWorkerActor extends InstrumentedActor { def wrappedReceive = { case Initialize => case DoSomeWork(desc) => }}
Saturday, October 19, 13
![Page 37: Akka in Production: Our Story](https://reader034.vdocument.in/reader034/viewer/2022051608/540dbc348d7f72747e8b4b1f/html5/thumbnails/37.jpg)
Next Steps
• Name?
• Open source?
• Talk to me if you’re interested in contributing
Saturday, October 19, 13
![Page 38: Akka in Production: Our Story](https://reader034.vdocument.in/reader034/viewer/2022051608/540dbc348d7f72747e8b4b1f/html5/thumbnails/38.jpg)
THANK YOUAnd YES, We’re HIRING!!
ooyala.com/careers
Saturday, October 19, 13