up, up and out - scaling software with akka henrik engstrÖm software engineer - typesafe @h3nk3

Up, up and out - Scaling Software

with Akka

HENRIK ENGSTRÖMSOFTWARE ENGINEER - TYPESAFE

@h3nk3

✦ Typesafe Console Tech Lead

✦ Akka Honorary Team Member

✦ Programming Scala since 2010

$ whoami

✦ Consultant since 1998 - mainly Java

✦ Been with Typesafe since 2011

✦ Arsenal Supporter + Martial Artist

About Typesafe

✦ The Typesafe Platform

‣ Play, Akka, Scala, Scala IDE, Slick, SBT, etc.

✦ Subscription

✦ Training and Consulting

Agenda

✦ Akka Introduction

✦ Core Operations

✦ Let’s Build Something with Actors

✦ Akka Cluster

✦ Typesafe Console

Copyright Ingeborg van Leeuwen

Selection of Akka Production Users

At the core - Actors✦ Carl Hewitt’s definition (1973)

‣ Processing + Storage + Communication

✦ 3 Axioms - an Actor can

- create new actors

- send messages to Actors

- designate how to handle next message

Proven Concept✦ Actors have successfully been

used in the Telecom industry since the 1980s

✦ Unbelievably stable:

‣ 9 nines uptime or ...

‣ 0.999999999% uptime or ...

‣ 32 ms downtime per year

Akka Actors

✦ Akka Actors are extremely lightweight

✦ Each Actor has its own mailbox

✦ Communication done via asynchronous message passing

✦ Very memory efficient

‣ You can have approx. 2.7 million actors/GB

ActorActor MailboxMailbox

Akka Actor in Action

Core Actor Operations

1)DEFINE

2)CREATE

3)SEND

4)SUPERVISE

case class Greeting(who: String)class GreetActor extend Actor with ActorLogging { var counter = 0 def receive = { case Greeting(who) => counter += 1 log.info("Hello %s, counter = %s". format(counter, who)) }}

1) DEFINE

val system = ActorSystem("MyFirstActorSystem")val greetActor: ActorRef = system.actorOf(Props[GreetingActor], "greeter")

2) CREATE

AA

BB

BarBarFooFoo

CC

BBEE

AA

DD

CC

Guardian System Actor

system.actorOf(Props[Foo], "Foo")

context.actorOf(Props[A], "A")

Actors can form hierarchies

AA

BB

BarBarFooFoo

CC

BBEE

AA

DD

CC

/Foo

/Foo/A

/Foo/A/B

/Foo/A/D

Guardian System Actor

Name resolution - like a file-system

Sending✦ Communication done via message

passing

✦ Fire and forget, i.e. asynchronous and non-blocking

✦ Reactive Programming

- messages are kinetic energy

- huge buffered potential energy at rest

Throughput on single box

with 48 cores

+50 million messages per second

greetActor ! Greeting("JavaOneShanghai")// orgreetActor.tell(Greeting("JavaOneShanghai"))

3) SEND

case class Greeting(who: String)class GreetActor extend Actor with ActorLogging { var counter = 0 def receive = { case Greeting(who) => counter += 1 log.info("Hello %s, counter = %s".format(counter, who)) }}val system = ActorSystem("MyFirstActorSystem")val greetActor: ActorRef = system.actorOf(Props[GreetingActor], "greeter")greetActor ! Greeting("JavaOneShanghai")

Full Example

Supervision✦ Manage another Actor’s failures

✦ Notification based

‣ if an Actor crashes a notification will be sent to the supervisor

✦ Clean separation of processing/business logic and error handling

ErrorKernel

class MyActor extends Actor { override val supervisionStrategy = OneForOneStrategy() { case _: ArithmeticException => Resume case _: NullPointerException => Restart case _: IllegalArgumentException => Stop case _: Exception => Escalate } // actor implementation }

4) SUPERVISE

class ActorA extends Actor { val b = context.actorOf(Props[ActorB], "b") context.watch(b) def receive = { case Terminated(a) => // take proper action }}

Death Watch

Akka Configuration

✦ All settings that are programmable are also configurable

‣ A simple restart can totally change the behavior of your application

‣ Will keep your DevOps happy

Let’s Build Something!

✦ Let’s create a hierarchy of words

‣ Each letter becomes one actor

‣ They form a parent child relationship

‣ "hi" becomes Actor("h") as parent to Actor("i")

‣ Each actor has an internal counter to keep track of number of occurrences

Example Hierarchyhhhh

ii22ii22

ee99ee99

llll

ll22ll22

oo44oo44

rooroott

rooroott

ii33ii33

ss1414ss

1414

case class CreateWord(word: String)class LetterActor extends Actor { var count = 0 def receive = { case CreateWord(word) => if(word.isEmpty) count += 1 else { val firstLetter = word.head.toString context.child(firstLetter). getOrElse(context.actorOf(Props[LetterActor], firstLetter)) ! CreateWord(word.tail) } } }

LetterActor

class WordActor extends Actor { def receive = { case CreateWord(word) => val firstLetter = word.head.toString context.child(firstLetter). getOrElse(context.actorOf(Props[LetterActor], firstLetter)) ! CreateWord(word.tail) }}

WordActor

class MainActor extends Actor { val root = context.actorOf(Props[WordActor], "wordActor") def receive = { case cw: CreateWord => root ! cw }}

MainActor

object WordDemo { def main(args: Array[String]) { val system = ActorSystem("WordCount") val main = system.actorOf(Props[MainActor], "mainActor") main ! CreateWord("hello") 1 to 3 foreach { c => main ! CreateWord("hi") } main ! CreateWord("hell") main ! CreateWord("helium") }}

Main Program

Resulting Hierarchy

hhhh eeeellll

ll11ll11

oo11oo11

rooroott

rooroott

ii33ii33

iiii

uuuu

mm11mm11

Akka Cluster

✦ Gossip-based cluster membership

✦ Leader determination

✦ Accrual failure detection

✦ Cluster death watch

✦ Cluster aware routers

Cluster Membership

✦ Node ring style inspired by Riak/Dynamo

✦ Gossip protocol - used for

- membership

- configuration data

- leader determination

✦ Vector clocks to detect convergence

Node Ring with Gossiping Members

MembMemberer

NodeNode

MembMemberer

NodeNode

MembMemberer

NodeNode

MembMemberer

NodeNode

MembMemberer

NodeNode

MembMemberer

NodeNode

MembMemberer

NodeNode

MembMemberer

NodeNode

MembMemberer

NodeNode

MembMemberer

NodeNode

Gossip

Phi Accrual Failure Detector

• B monitors A

• Sample inter-arrival time to expect next beat

• B measures continuum of deadness of A

AA BBregular messages

http://ddg.jaist.ac.jp/pub/HDY+04.pdf

Selective Failure Detection

MembMemberer

NodeNode

MembMemberer

NodeNode

MembMemberer

NodeNode

MembMemberer

NodeNode

MembMemberer

NodeNode

MembMemberer

NodeNode

MembMemberer

NodeNode

MembMemberer

NodeNode

MembMemberer

NodeNode

MembMemberer

NodeNode

Heartbeat

Selective Failure Detection

MembMemberer

NodeNode

MembMemberer

NodeNode

MembMemberer

NodeNode

MembMemberer

NodeNode

MembMemberer

NodeNode

MembMemberer

NodeNode

MembMemberer

NodeNode

MembMemberer

NodeNode

MembMemberer

NodeNode

MembMemberer

NodeNode

Heartbeat

Cluster Death Watch

✦ Triggered by marking node <A> DOWN

‣ Tell parents of their lost children on <A>

‣ Delete all children of actors on <A>

‣ Send TERMINATED for actors on <A>

akka { actor { provider = "akka.cluster.ClusterActorRefProvider" ... } extensions = ["akka.cluster.Cluster"] cluster { seed-nodes = [ "akka://ClusterSystem@127.0.0.1:2551", "akka://ClusterSystem@127.0.0.1:2552" ] auto-down = on }}

Enable Clustering

Typesafe Console

✦ Monitoring tool for Akka (and soon Play) applications

✦ Low overhead - always turned on

✦ Traces everything in your Akka application

✦ UI or REST service to retrieve statistics

✦ Free to use for developers via Activator

Typesafe Console

Dataflow

Akka has much much moreFSM

Transactors

Pub/Sub

ZeroMQ

Microkernel

IO

TestKit

Agents

SLF4J

Durable Mailboxes

EventBus

Camel

Pooling

TypedActor

Extensions

Typed Channels

EOF@h3nk3