reactive database access with slick3

Reactive database access with Slick3

Naoki Takezoe@takezoen

BizReach, Inc

Who am I?

● Naoki Takezoe: @takezoen● Scala programmer at BizReach, Inc● Long time OSS programmer and technical

writer

Slick3 (a.k.a. Reactive Slick)http://slick.typesafe.com/

What's Reactive Slick

● Data access framework for Scala● Powerful type-safe query API● Parallel execution and streaming

What's Reactive Slick

● Data access framework for Scala● Powerful type-safe query API● Parallel execution and streaming

History of Slick

● 2010 ScalaQuery 0.9● 2012 Slick 0.11 (Move to Typesafe)● 2013 Slick 1.0● 2014 Slick 2.0 (Statically Session)● 2015 Slick 3.0 (Reactive)

History of Slick

● 2010 ScalaQuery 0.9● 2012 Slick 0.11 (Move to Typesafe)● 2013 Slick 1.0● 2014 Slick 2.0 (Statically Session)● 2015 Slick 3.0 (Reactive)

Large API changes.Migration was very hard...

Background

● C10K problem● Non-blocking I/O based web framework● Microservice architecture● Big data

Parallel execution

Example 1: Simple query

// Define query actionval select: DBIO[Seq[String]] = Coffee // Table .filter(_.price <= 100.bind) // Where clause .map(_.name) // Select clause .result

// Execute querydb.run(select) // => Future[Seq[String]]

Example 2: Parallel execution

// Define insert actionsval inserts: Seq[DBIO[Int]] = Seq( Coffee("Colombian", 101, 7.99, 0, 0).insert, Coffee("French_Roast", 49, 8.99, 0, 0).insert, Coffee("Espresso", 150, 9.99, 0, 0).insert, Coffee("Colombian_Decaf", 101, 8.99, 0, 0).insert, Coffee("French_Roast_Decaf", 49, 9.99, 0, 0).insert)

// Combine above actionsval combined: DBIO[Seq[Int]] = DBIO.sequence(inserts)

// Run action transactionallydb.run(combined.transactionally) // => Future[Seq[Int]]

Example 3: Work with other aync tasks

def index = Action.async { for { res <- ws.url("http://xxx").get() name <- db.run( Coffee .filter(_.id === res.body.bind) .map(_.name).result.head ) } yield Ok(name)}

Future[WSResponse]

Future[String]Future[Result]

But JDBC is blocking.Why Slick3?

Separate ExecutionContextController Thread

(Play)I/O Thread

(Slick)

JDBC

Separate ExecutionContextController Thread

(Play)I/O Thread

(Slick)

JDBC

for CPU bound tasks

for I/O bound tasks

by separating ExecutionContext

● Use server resources effectively● Avoid exhausting threads by I/O tasks

Future: Real non-blocking database access?

● postgresql-async● mysql-async

https://github.com/mauricio/postgresql-async

Streaming results(Reactive Streams)

Streaming results is

● for large data processing● based on Reactive Streams

Reactive Streams

public interface Publisher<T> { public void subscribe(Subscriber<? super T> s);}

public interface Subscriber<T> { public void onSubscribe(Subscription s); public void onNext(T t); public void onError(Throwable t); public void onComplete();}

public interface Subscription { public void request(long n); public void cancel();}

public interface Processor<T, R> extends Subscriber<T>, Publisher<R> {}

Example: Slick API

val publisher: DatabasePublisher[String] = db.stream(Coffee.map(_.message).result)

// Iterate stream using Slick APIpublisher.foreach { name => println(name)}

DataPublisher implementationdef foreach[U](f: T => U)(implicit ec: ExecutionContext): Future[Unit] = { val p = Promise[Unit]() @volatile var lastMsg: Future[Any] = null @volatile var subscr: Subscription = null subscribe(new Subscriber[T] { def onSubscribe(s: Subscription): Unit = { subscr = s s.request(1L) } def onComplete(): Unit = { ... } def onError(t: Throwable): Unit = { ... } def onNext(t: T): Unit = { lastMsg = Future(f(t)) lastMsg.onComplete { case Success(v) => subscr.request(1L) case Failure(t) => ... } } }) p.future}

Implemented based on Reactive Streams API

Missing piece

Need both of the publicher and the subscriber

Publisher Subscriber / Publisher Subscriber

One idea: Slick3 + Akka Streams?

http://www.slideshare.net/kazukinegoro5/akka-streams-100-scalamatsuri

Problem

Problem 1: Functionallity

Problem 2: Difficulty

● Programming○ Monadic API is hard for non functional programmers

● Configuration○ Connection pool and ExecutionContext

Sumary

Summary

● Slick3 provides 2 reactives:○ Parallel database processing○ Streaming large data based on Reactive Sreams

● Suitabe usecase for Slick are:○ Composite database tasks and async tasks○ Large data processing using Reactive Streams