reactive database access with slick3
TRANSCRIPT
Reactive database access with Slick3
Naoki Takezoe@takezoen
BizReach, Inc
Who am I?
● Naoki Takezoe: @takezoen● Scala programmer at BizReach, Inc● Long time OSS programmer and technical
writer
Slick3 (a.k.a. Reactive Slick)http://slick.typesafe.com/
What's Reactive Slick
● Data access framework for Scala● Powerful type-safe query API● Parallel execution and streaming
What's Reactive Slick
● Data access framework for Scala● Powerful type-safe query API● Parallel execution and streaming
History of Slick
● 2010 ScalaQuery 0.9● 2012 Slick 0.11 (Move to Typesafe)● 2013 Slick 1.0● 2014 Slick 2.0 (Statically Session)● 2015 Slick 3.0 (Reactive)
History of Slick
● 2010 ScalaQuery 0.9● 2012 Slick 0.11 (Move to Typesafe)● 2013 Slick 1.0● 2014 Slick 2.0 (Statically Session)● 2015 Slick 3.0 (Reactive)
Large API changes.Migration was very hard...
Background
● C10K problem● Non-blocking I/O based web framework● Microservice architecture● Big data
Parallel execution
Example 1: Simple query
// Define query actionval select: DBIO[Seq[String]] = Coffee // Table .filter(_.price <= 100.bind) // Where clause .map(_.name) // Select clause .result
// Execute querydb.run(select) // => Future[Seq[String]]
Example 2: Parallel execution
// Define insert actionsval inserts: Seq[DBIO[Int]] = Seq( Coffee("Colombian", 101, 7.99, 0, 0).insert, Coffee("French_Roast", 49, 8.99, 0, 0).insert, Coffee("Espresso", 150, 9.99, 0, 0).insert, Coffee("Colombian_Decaf", 101, 8.99, 0, 0).insert, Coffee("French_Roast_Decaf", 49, 9.99, 0, 0).insert)
// Combine above actionsval combined: DBIO[Seq[Int]] = DBIO.sequence(inserts)
// Run action transactionallydb.run(combined.transactionally) // => Future[Seq[Int]]
Example 3: Work with other aync tasks
def index = Action.async { for { res <- ws.url("http://xxx").get() name <- db.run( Coffee .filter(_.id === res.body.bind) .map(_.name).result.head ) } yield Ok(name)}
Future[WSResponse]
Future[String]Future[Result]
But JDBC is blocking.Why Slick3?
Separate ExecutionContextController Thread
(Play)I/O Thread
(Slick)
JDBC
Separate ExecutionContextController Thread
(Play)I/O Thread
(Slick)
JDBC
for CPU bound tasks
for I/O bound tasks
by separating ExecutionContext
● Use server resources effectively● Avoid exhausting threads by I/O tasks
Future: Real non-blocking database access?
● postgresql-async● mysql-async
https://github.com/mauricio/postgresql-async
Streaming results(Reactive Streams)
Streaming results is
● for large data processing● based on Reactive Streams
Reactive Streams
public interface Publisher<T> { public void subscribe(Subscriber<? super T> s);}
public interface Subscriber<T> { public void onSubscribe(Subscription s); public void onNext(T t); public void onError(Throwable t); public void onComplete();}
public interface Subscription { public void request(long n); public void cancel();}
public interface Processor<T, R> extends Subscriber<T>, Publisher<R> {}
Example: Slick API
val publisher: DatabasePublisher[String] = db.stream(Coffee.map(_.message).result)
// Iterate stream using Slick APIpublisher.foreach { name => println(name)}
DataPublisher implementationdef foreach[U](f: T => U)(implicit ec: ExecutionContext): Future[Unit] = { val p = Promise[Unit]() @volatile var lastMsg: Future[Any] = null @volatile var subscr: Subscription = null subscribe(new Subscriber[T] { def onSubscribe(s: Subscription): Unit = { subscr = s s.request(1L) } def onComplete(): Unit = { ... } def onError(t: Throwable): Unit = { ... } def onNext(t: T): Unit = { lastMsg = Future(f(t)) lastMsg.onComplete { case Success(v) => subscr.request(1L) case Failure(t) => ... } } }) p.future}
Implemented based on Reactive Streams API
Missing piece
Need both of the publicher and the subscriber
Publisher Subscriber / Publisher Subscriber
One idea: Slick3 + Akka Streams?
http://www.slideshare.net/kazukinegoro5/akka-streams-100-scalamatsuri
Problem
Problem 1: Functionallity
Problem 2: Difficulty
● Programming○ Monadic API is hard for non functional programmers
● Configuration○ Connection pool and ExecutionContext
Sumary
Summary
● Slick3 provides 2 reactives:○ Parallel database processing○ Streaming large data based on Reactive Sreams
● Suitabe usecase for Slick are:○ Composite database tasks and async tasks○ Large data processing using Reactive Streams