reactive streams 1.0 and akka streams

93
Konrad 'ktoso' Malawski GeeCON 2014 @ Kraków, PL Konrad `@ktosopl` Malawski (Channeled by @deanwampler) streams Rise of the 1.0 : reactive streams

Upload: dean-wampler

Post on 06-Apr-2017

1.400 views

Category:

Software


6 download

TRANSCRIPT

Page 1: Reactive Streams 1.0 and Akka Streams

Konrad 'ktoso' MalawskiGeeCON 2014 @ Kraków, PL

Konrad `@ktosopl` Malawski (Channeled by @deanwampler)

streams

Rise of the 1.0 : reactive streams

Page 2: Reactive Streams 1.0 and Akka Streams

Konrad `@ktosopl` Malawski

akka.iotypesafe.comgeecon.org

Java.pl / KrakowScala.plsckrk.com / meetup.com/Paper-Cup @ London

GDGKrakow.pl lambdakrk.pl

(we’re renaming soon!)

Page 4: Reactive Streams 1.0 and Akka Streams

Aside…

Page 5: Reactive Streams 1.0 and Akka Streams

Streams

Page 6: Reactive Streams 1.0 and Akka Streams

A Real Stream…

Page 7: Reactive Streams 1.0 and Akka Streams

An Abstract Stream

Page 8: Reactive Streams 1.0 and Akka Streams

Streams

“You cannot enter the same river twice”~ Heraclitus

http://en.wikiquote.org/wiki/Heraclitus

Page 9: Reactive Streams 1.0 and Akka Streams

Streams

Real Time Stream Processing

When you attach “late” to a Publisher,you may miss initial elements – it’s a river of data.

http://en.wikiquote.org/wiki/Heraclitus

Page 10: Reactive Streams 1.0 and Akka Streams

Reactive Streams 1.0

Page 11: Reactive Streams 1.0 and Akka Streams

Reactive Streams

Stream processing

Page 12: Reactive Streams 1.0 and Akka Streams

Reactive Streams

Back-pressured

Stream processing

Page 13: Reactive Streams 1.0 and Akka Streams

Reactive Streams

Back-pressured Asynchronous

Stream processing

Page 14: Reactive Streams 1.0 and Akka Streams

Reactive Streams

Back-pressured Asynchronous

Stream processing Standardised (!)

Page 15: Reactive Streams 1.0 and Akka Streams

Reactive Streams: Goals

1. Back-pressured Asynchronous Stream processing

2. Standard implemented by many libraries

Page 16: Reactive Streams 1.0 and Akka Streams

Reactive Streams

http://www.reactive-streams.org/

Page 17: Reactive Streams 1.0 and Akka Streams

Reactive Streams

Page 18: Reactive Streams 1.0 and Akka Streams

Reactive Streams - Who?

http://reactive-streams.org

Kaazing Corp.RxJava @ Netflix,

Reactor @ Pivotal (SpringSource),Vert.x @ Red Hat,

Twitter,Akka Streams, Slick @ Typesafe,

Spray @ Spray.io,Oracle,

OpenJDK (Java 9) – Doug Lea - SUNY Oswego …

Page 19: Reactive Streams 1.0 and Akka Streams

Reactive Streams - Inter-op

http://reactive-streams.org

We want to make different implementations co-operate with each other.

Page 20: Reactive Streams 1.0 and Akka Streams

Reactive Streams - Inter-op

http://reactive-streams.org

The different implementations “talk to each other”using the Reactive Streams protocol.

Page 21: Reactive Streams 1.0 and Akka Streams

Reactive Streams - Inter-op

http://reactive-streams.org

The Reactive Streams SPI is NOT meant to be user-api.You should use one of the implementing libraries.

Page 22: Reactive Streams 1.0 and Akka Streams

Reactive Streams - Inter-op exampleimport ratpack.rx.RxRatpackimport ratpack.test.embed.EmbeddedAppimport ratpack.handling.Handlerimport ratpack.handling.Contextimport rx.Observableimport scala.collection.JavaConverters._import akka.stream.scaladsl.Flowimport akka.stream.scaladsl.Sourceimport rx.RxReactiveStreamsimport akka.stream.scaladsl.Sinkimport akka.actor.ActorSystemimport akka.stream.FlowMaterializerimport ratpack.http.ResponseChunksimport java.util.function.Consumerimport ratpack.test.http.TestHttpClientimport reactor.rx.Streams

Page 23: Reactive Streams 1.0 and Akka Streams

import ratpack.rx.RxRatpackimport ratpack.test.embed.EmbeddedAppimport ratpack.handling.Handlerimport ratpack.handling.Contextimport rx.Observableimport scala.collection.JavaConverters._import akka.stream.scaladsl.Flowimport akka.stream.scaladsl.Sourceimport rx.RxReactiveStreamsimport akka.stream.scaladsl.Sinkimport akka.actor.ActorSystemimport akka.stream.FlowMaterializerimport ratpack.http.ResponseChunksimport java.util.function.Consumerimport ratpack.test.http.TestHttpClientimport reactor.rx.Streams

object ScalaMain extends App {

Page 24: Reactive Streams 1.0 and Akka Streams

import akka.actor.ActorSystemimport akka.stream.FlowMaterializerimport ratpack.http.ResponseChunksimport java.util.function.Consumerimport ratpack.test.http.TestHttpClientimport reactor.rx.Streams

object ScalaMain extends App { val system = ActorSystem("InteropTest") implicit val mat = FlowMaterializer()(system) RxRatpack.initialize() val handler = new Handler { override def handle(ctx: Context): Unit ={ // RxJava Observable val intObs = Observable.from((1 to 10).asJava) // Reactive Streams Publisher

Page 25: Reactive Streams 1.0 and Akka Streams

import akka.actor.ActorSystemimport akka.stream.FlowMaterializerimport ratpack.http.ResponseChunksimport java.util.function.Consumerimport ratpack.test.http.TestHttpClientimport reactor.rx.Streams

object ScalaMain extends App { val system = ActorSystem("InteropTest") implicit val mat = FlowMaterializer()(system) RxRatpack.initialize() val handler = new Handler { override def handle(ctx: Context): Unit ={ // RxJava Observable val intObs = Observable.from((1 to 10).asJava) // Reactive Streams Publisher

Page 26: Reactive Streams 1.0 and Akka Streams

import akka.actor.ActorSystemimport akka.stream.FlowMaterializerimport ratpack.http.ResponseChunksimport java.util.function.Consumerimport ratpack.test.http.TestHttpClientimport reactor.rx.Streams

object ScalaMain extends App { val system = ActorSystem("InteropTest") implicit val mat = FlowMaterializer()(system) RxRatpack.initialize() val handler = new Handler { override def handle(ctx: Context): Unit ={ // RxJava Observable val intObs = Observable.from((1 to 10).asJava) // Reactive Streams Publisher

Page 27: Reactive Streams 1.0 and Akka Streams

RxRatpack.initialize() val handler = new Handler { override def handle(ctx: Context): Unit ={ // RxJava Observable val intObs = Observable.from((1 to 10).asJava) // Reactive Streams Publisher val intPub = RxReactiveStreams.toPublisher(intObs) // Akka Streams Source val stringSource = Source(intPub).map(_.toString) // Reactive Streams Publisher val stringPub = stringSource.runWith( Sink.fanoutPublisher(1, 1)) // Reactor Stream

Page 28: Reactive Streams 1.0 and Akka Streams

// Reactive Streams Publisher val intPub = RxReactiveStreams.toPublisher(intObs) // Akka Streams Source val stringSource = Source(intPub).map(_.toString) // Reactive Streams Publisher val stringPub = stringSource.runWith( Sink.fanoutPublisher(1, 1)) // Reactor Stream val reactor.function.Function val linesStream = Streams.create(stringPub).map[String]( new Function[String, String] { override def apply(in: String) = in+"\n" }) // and now render the HTTP response

Page 29: Reactive Streams 1.0 and Akka Streams

val stringPub = stringSource.runWith( Sink.fanoutPublisher(1, 1)) // Reactor Stream val reactor.function.Function val linesStream = Streams.create(stringPub).map[String]( new Function[String, String] { override def apply(in: String) = in+"\n" }) // and now render the HTTP response ctx.render( ResponseChunks.stringChunks(linesStream)) } })

EmbeddedApp.fromHandler(handler).test( new Consumer[TestHttpClient] {

Page 30: Reactive Streams 1.0 and Akka Streams

// and now render the HTTP response ctx.render( ResponseChunks.stringChunks(linesStream)) } })

EmbeddedApp.fromHandler(handler).test( new Consumer[TestHttpClient] { override def accept( client: TestHttpClient): Unit = { val text = client.getText() println(text) system.shutdown() } })}

Page 31: Reactive Streams 1.0 and Akka Streams

// and now render the HTTP response ctx.render( ResponseChunks.stringChunks(linesStream)) } })

EmbeddedApp.fromHandler(handler).test( new Consumer[TestHttpClient] { override def accept( client: TestHttpClient): Unit = { val text = client.getText() println(text) system.shutdown() } })}

Page 32: Reactive Streams 1.0 and Akka Streams

Reactive Streams - Inter-op

public interface Subscriber<T> { public void onSubscribe(Subscription s); public void onNext(T t); public void onError(Throwable t); public void onComplete();}

public interface Publisher<T> { public void subscribe(Subscriber<? super T> s);}

public interface Subscription { public void request(long n); public void cancel();}

Page 33: Reactive Streams 1.0 and Akka Streams

What is back-pressure?

Page 34: Reactive Streams 1.0 and Akka Streams

Back-pressure? Example Without

Publisher[T] Subscriber[T]

Page 35: Reactive Streams 1.0 and Akka Streams

Back-pressure? Example Without

Fast Publisher Slow Subscriber

Page 36: Reactive Streams 1.0 and Akka Streams

Back-pressure? “Why would I need that!?”

Page 37: Reactive Streams 1.0 and Akka Streams

Back-pressure? Push + NACK model

Page 38: Reactive Streams 1.0 and Akka Streams

Back-pressure? Push + NACK model

Subscriber usually has some kind of buffer.

Page 39: Reactive Streams 1.0 and Akka Streams

Back-pressure? Push + NACK model

Page 40: Reactive Streams 1.0 and Akka Streams

Back-pressure? Push + NACK model

Page 41: Reactive Streams 1.0 and Akka Streams

Back-pressure? Push + NACK model

What if the buffer overflows?

Page 42: Reactive Streams 1.0 and Akka Streams

Back-pressure? Push + NACK model (a)

Use bounded buffer, drop messages + require re-sending

Page 43: Reactive Streams 1.0 and Akka Streams

Back-pressure? Push + NACK model (a)

Kernel does this!Routers do this!

(TCP)

Use bounded buffer, drop messages + require re-sending

Page 44: Reactive Streams 1.0 and Akka Streams

Back-pressure? Push + NACK model (b)Increase buffer size… Well, while you have memory available!

Page 45: Reactive Streams 1.0 and Akka Streams

Back-pressure? Push + NACK model (b)

Page 46: Reactive Streams 1.0 and Akka Streams

NACKing is NOT enough!

Negative ACKnowledgement

Page 47: Reactive Streams 1.0 and Akka Streams

Back-pressure? Example NACKing

Buffer overflow is imminent!

Page 48: Reactive Streams 1.0 and Akka Streams

Back-pressure? Example NACKingTelling the Publisher to slow down / stop sending…

Page 49: Reactive Streams 1.0 and Akka Streams

Back-pressure? Example NACKing

NACK did not make it in time, because M was in-flight!

Page 50: Reactive Streams 1.0 and Akka Streams

Back-pressure?

speed(publisher) < speed(subscriber)

Page 51: Reactive Streams 1.0 and Akka Streams

Back-pressure? Fast Subscriber, No Problem

No problem!

Page 52: Reactive Streams 1.0 and Akka Streams

Back-pressure? Reactive-Streams

= “Dynamic Push/Pull”

Page 53: Reactive Streams 1.0 and Akka Streams

Just push – not safe when Slow Subscriber

Just pull – too slow when Fast Subscriber

Back-pressure? RS: Dynamic Push/Pull

Page 54: Reactive Streams 1.0 and Akka Streams

Solution: Dynamic adjustment

Back-pressure? RS: Dynamic Push/Pull

Just push – not safe when Slow Subscriber

Just pull – too slow when Fast Subscriber

Page 55: Reactive Streams 1.0 and Akka Streams

Back-pressure? RS: Dynamic Push/PullSlow Subscriber sees it’s buffer can take 3 elements. Publisher will never blow up it’s buffer.

Page 56: Reactive Streams 1.0 and Akka Streams

Back-pressure? RS: Dynamic Push/PullFast Publisher will send at-most 3 elements. This is pull-based-backpressure.

Page 57: Reactive Streams 1.0 and Akka Streams

Back-pressure? RS: Dynamic Push/Pull

Fast Subscriber can issue more Request(n), before more data arrives!

Page 58: Reactive Streams 1.0 and Akka Streams

Back-pressure? RS: Dynamic Push/PullFast Subscriber can issue more Request(n), before more data arrives.

Publisher can accumulate demand.

Page 59: Reactive Streams 1.0 and Akka Streams

Back-pressure? RS: Accumulate demand

Publisher accumulates total demand per subscriber.

Page 60: Reactive Streams 1.0 and Akka Streams

Back-pressure? RS: Accumulate demandTotal demand of elements is safe to publish. Subscriber’s buffer will not overflow.

Page 61: Reactive Streams 1.0 and Akka Streams

Back-pressure? RS: Requesting “a lot”

Fast Subscriber can issue arbitrary large requests, including “gimme all you got” (Long.MaxValue)

Page 62: Reactive Streams 1.0 and Akka Streams

streams

Page 63: Reactive Streams 1.0 and Akka Streams

Akka

• Fault tolerant

• Supervision hierarchies

• Failure detection

• Asynchronous data processing

• Optimised for high performance

• both in-jvm and across network

• Adaptive Cluster

• Load balancing among Cluster Nodes

Actor 131 Actor 132

Supervisor 1

Actor 12 Actor 13

Actor 111 Actor 112

Supervisor 11

Page 64: Reactive Streams 1.0 and Akka Streams

Akka

Akka has multiple modules:

akka-actor: actors (concurrency abstraction)akka-camel: integrationakka-remote: remote actorsakka-cluster: clusteringakka-persistence: CQRS / Event Sourcingakka-streams: stream processing…

Page 65: Reactive Streams 1.0 and Akka Streams

AkkaAkka is a high-performance concurrency library for Scala and Java.

At it’s core it focuses on the Actor Model:

Page 66: Reactive Streams 1.0 and Akka Streams

An Actor can only: • Send and receive messages• Create Actors• Change it’s behaviour

AkkaAkka is a high-performance concurrency library for Scala and Java.

At it’s core it focuses on the Actor Model:

Page 67: Reactive Streams 1.0 and Akka Streams

class Player extends Actor {

def receive = { case NextTurn => sender() ! decideOnMove() }

def decideOnMove(): Move = ???}

Akka Actors

Page 68: Reactive Streams 1.0 and Akka Streams

Akka

Actors are: a distribution and concurrency abstraction.

Streams are: a way to describe how data flows through a system.

Page 69: Reactive Streams 1.0 and Akka Streams

Akka Streams – Linear Flow

Page 70: Reactive Streams 1.0 and Akka Streams

Akka Streams – Linear Flow

Page 71: Reactive Streams 1.0 and Akka Streams

Akka Streams – Linear Flow

Page 72: Reactive Streams 1.0 and Akka Streams

Akka Streams – Linear Flow

Page 73: Reactive Streams 1.0 and Akka Streams

Akka Streams – Linear Flow

Flow[Double].map(_.toInt). [...]

No Source attached yet.“Pipe ready to work with Doubles”.

Page 74: Reactive Streams 1.0 and Akka Streams

Akka Streams – Linear Flow

implicit val sys = ActorSystem()implicit val mat = ActorMaterializer()

Source(1 to 3).runWith(Sink.foreach(println))

Page 75: Reactive Streams 1.0 and Akka Streams

Akka Streams – Linear Flow

Source(1 to 3).runWith(Sink.foreach(println))

// sugar for runWithSource(1 to 3).foreach(println)

implicit val sys = ActorSystem()implicit val mat = ActorMaterializer()

Page 76: Reactive Streams 1.0 and Akka Streams

Akka Streams – Linear Flow

Source(1 to 3).runWith(Sink.foreach(println))

// sugar for runWithSource(1 to 3).foreach(println)

Sink.foldSink.headSink.ignoreSink.publisherSink.cancelled// your own Sink…

implicit val sys = ActorSystem()implicit val mat = ActorMaterializer()

Page 77: Reactive Streams 1.0 and Akka Streams

Akka Streams – Flows are reusable

sink.runWith(Source(1 to 10)) sink.runWith(Source(1 to 100)) sink.runWith(Source(1 to 1000))

source.runWith(Sink.ignore) source.runWith(Sink.foreach(println))

Multiple materializations

val ready = source.to(Sink.ignore)

ready.run()ready.run()

Page 78: Reactive Streams 1.0 and Akka Streams

Akka Streams <-> Actors – Advancedval subscriber = ActorSubscriber( system.actorOf(Props[SubStreamParent], ”parent”))

Source(1 to 100) .map(_.toString) .filter(_.length == 2) .drop(2) .conflate(seed => seed)((acc, i) => acc + i) .groupBy(_.last) .runWith(subscriber)

All the usual ops available for Linear Flows.

Page 79: Reactive Streams 1.0 and Akka Streams

Akka Streams <-> Actors – Advancedval subscriber = ActorSubscriber( system.actorOf(Props[SubStreamParent], ”parent”))

Source(1 to 100) .map(_.toString) .filter(_.length == 2) .drop(2) .conflate(seed => seed)((acc, i) => acc + i) .groupBy(_.last) .runWith(subscriber)

Aggregating values until downstream demand comes.

Page 80: Reactive Streams 1.0 and Akka Streams

Akka Streams <-> Actors – Advancedval subscriber = ActorSubscriber( system.actorOf(Props[SubStreamParent], ”parent”))

Source(1 to 100) .map(_.toString) .filter(_.length == 2) .drop(2) .conflate(seed => seed)((acc, i) => acc + i) .groupBy(_.last) .runWith(subscriber)

Creates a stream of streams:Source[(Int, Source[String])]

Page 81: Reactive Streams 1.0 and Akka Streams

Akka Streams: Graphs

val p: Publisher[String] = FlowGraph.closed(out) { implicit b ⇒ o ⇒ val merge = b.add(new StrictRoundRobin[String])

in1 ~> merge.in(0) in2 ~> merge.in(1) merge.out ~> o.inlet }.run()

val in1 = Source(List("a", "b", "c", "d")) val in2 = Source(List("e", "f"))

val out = Sink.publisher[String]

Page 82: Reactive Streams 1.0 and Akka Streams

Akka Streams: Graphs

val p: Publisher[String] = FlowGraph.closed(out) { implicit b ⇒ o ⇒ val merge = b.add(new StrictRoundRobin[String])

in1 ~> merge.in(0) in2 ~> merge.in(1) merge.out ~> o.inlet }.run()

val in1 = Source(List("a", "b", "c", "d")) val in2 = Source(List("e", "f"))

val out = Sink.publisher[String]

Sink[String, Publisher[String]]

imports Graphs

Page 83: Reactive Streams 1.0 and Akka Streams

Akka Streams: Graphs

val p: Publisher[String] = FlowGraph.closed(out) { implicit b ⇒ o ⇒ val merge = b.add(new StrictRoundRobin[String])

in1 ~> merge.in(0) in2 ~> merge.in(1) merge.out ~> o.inlet }.run()

val in1 = Source(List("a", "b", "c", "d")) val in2 = Source(List("e", "f"))

val out = Sink.publisher[String]

Sink[String, Publisher[String]]

materializes a Publisher[String]

Page 84: Reactive Streams 1.0 and Akka Streams

Reactive Streams

Bigger than Scala-ecosystem - JDK-wide (and wider).

Inter-operable back-pressure protocol.

Future work: reactive-streams-io, reactive-streams-js

Akka Streams - one of the leading Reactive Streams impls. Complex in-memory stream processing.

SlickR - provides Reactive Stream from DB queries

Akka Http - Akka Streams based; “Spray 2.0”

Page 85: Reactive Streams 1.0 and Akka Streams

Wrapping up

Page 86: Reactive Streams 1.0 and Akka Streams

Available Sources

•FutureSource•IterableSource•IteratorSource•PublisherSource•SubscriberSource•ThunkSource•SynchronousFileSource•TickSource (timer based)•… easy to add your own!

Page 87: Reactive Streams 1.0 and Akka Streams

Available operations• drop / dropWithin• take / takeWithin• filter• groupBy• grouped• transform• buffer• collect• expand• splitWhen / splitAfter• map• scan• prefixAndTail• … easy to add your own!

“Rate – detaching” operations:• buffer• collect• concat• conflate

Page 88: Reactive Streams 1.0 and Akka Streams

Available Sinks• BlackHoleSink• FoldSink• ForeachSink• FutureSink• OnCompleteSink• UdpSink [next release]• SynchronousFileSink• PublisherSink / FanoutPublisherSink• SubscriberSink• FileSink• … easy to add your own!

Page 89: Reactive Streams 1.0 and Akka Streams

Available Junctions• Broadcast• Merge

• FlexiMerge• Route

• FlexiRoute• Zip

• ZipWith• Unzip

• UnzipWith• Concat• … easy to add your own!

Page 90: Reactive Streams 1.0 and Akka Streams

Community ecosystem• Kafka • Spark (soon?)• Slick • S3• Files• MongoDB• …

Page 91: Reactive Streams 1.0 and Akka Streams

Links• The projects: • http://akka.io • http://reactive-streams.org

• Akka User - mailing list: • https://groups.google.com/group/akka-user

Page 92: Reactive Streams 1.0 and Akka Streams

Tänan! Dzięki! Thanks!

ありがとう!

ktoso @ typesafe.com t: ktosopl / g: ktoso blog: project13.pl

Page 93: Reactive Streams 1.0 and Akka Streams

©Typesafe 2015 – All Rights Reserved

The company soon to be

previously known as