an embedded query language in scala · an embedded query language in scala amirshaikhha school of...

An Embedded Query Language in Scala

Amir Shaikhha

School of Computer and Communication Sciences, EPFLTypesafe, Lausanne

Master Thesis, August 2013

1 / 60

Introduction

Outline

1 Introduction

2 Lifted Embedding

3 Direct Embedding

4 Shadow Embedding

5 Evaluation

1 / 60

Introduction

Introduction

Problem StatementWrite the code to access database

2 / 60

Introduction

Introduction

Instead of writing database code in SQL

select c.NAME from COFFEES c where c.ID = 10

Write database code in Scala

for (c <- coffees if c.id == 10) yield c.name

3 / 60

Introduction

Slick

Scala Language-Integrated Connection Kit

4 / 60

Introduction

SlickArchitecture

5 / 60

Lifted Embedding

Outline

1 Introduction

2 Lifted Embedding

3 Direct Embedding

4 Shadow Embedding

5 Evaluation

5 / 60

Lifted Embedding

Lifted EmbeddingArchitecture

Uses standard ScalaNot Scala-Virtualized

6 / 60

Lifted Embedding


Uses standard Scala

Not Scala-Virtualized

6 / 60

Lifted Embedding


Uses standard ScalaNot Scala-Virtualized

6 / 60

Lifted Embedding

Lifted EmbeddingExample

Lifted Embedding Example

Query(Coffees) filter(c => c.id === 10) map(c => c.name)

7 / 60

Lifted Embedding




Scala for-comprehension

for (c <- Query(Coffees) if c.id === 10)yield c.name

7 / 60

Lifted Embedding




SQL Statement


7 / 60

Lifted Embedding

Lifted EmbeddingType Information


8 / 60

Lifted Embedding

Lifted EmbeddingType Information

Query(Coffees) filter(c => c.id:Rep[Int] === 10:Rep[Int]) map(c => c.name:Rep[String])

8 / 60

Lifted Embedding

Lifted EmbeddingProblem 1


How to create Lifted Embedding Table?

9 / 60

Lifted Embedding

Lifted Embedding

object Coffees extends Table[(Int , String , Double ,String , Int)]("COFFEES") {

def id = column[Int]("ID", O.PrimaryKey)def name = column[String ]("NAME")//...

}

Boilerplate!

10 / 60

Lifted Embedding

Type Providers

Generate the types

Existing SchemaAnnotated classes

11 / 60

Lifted Embedding

Type Providers

Generate the types out of:Existing Schema

Annotated classes

11 / 60

Lifted Embedding

Type Providers

Generate the types out of:Existing SchemaAnnotated classes

11 / 60

Lifted Embedding

Type ProvidersArchitecture

Type Macros are in macro paradiseCode Generation uses standard Scala

12 / 60

Lifted Embedding


Type Macros are in macro paradise

Code Generation uses standard Scala

12 / 60

Lifted Embedding


Type Macros are in macro paradiseCode Generation uses standard Scala

12 / 60

Lifted Embedding


How to catch the errors?

13 / 60

Lifted Embedding

Lifted EmbeddingType Errors - Good Part

Query(Coffees) map(c => c.id.toDouble)

Compile Error

value toDouble is not a member of scala.slick.lifted.Column[Int]

14 / 60

Lifted Embedding

Lifted EmbeddingType Errors - Bad Part

Query(Coffees) map(c => c.id substring 2)

Compile Error

value substring is not a member of scala.slick.lifted.Column[Int]

15 / 60

Lifted Embedding

Lifted EmbeddingType Errors - Even Worse!

Query(Coffees) map (c =>if(c.origin == "Iran")

"Good"else

c.quality)

Compile ErrorDon’t know how to unpack Any to T and pack to Gnot enough arguments for method map: (implicit shape:scala.slick.lifted.Shape[Any,T,G])scala.slick.lifted.Query[G,T].Unspecified value parameter

Scala-Virtualized has not this problem

16 / 60

Lifted Embedding

Lifted EmbeddingType Errors - Even Worse!

Query(Coffees) map (c =>if(c.origin == "Iran")

"Good"else

c.quality)

Compile ErrorDon’t know how to unpack Any to T and pack to Gnot enough arguments for method map: (implicit shape:scala.slick.lifted.Shape[Any,T,G])scala.slick.lifted.Query[G,T].Unspecified value parameter

Scala-Virtualized has not this problem16 / 60

Lifted Embedding

Lifted EmbeddingType Errors

Adapted from http://thumbs.dreamstime.com/z/old-bus-desert-7703223.jpg

17 / 60

Lifted Embedding


How to have high performance?

18 / 60

Lifted Embedding

Lifted EmbeddingPerformance

19 / 60

Lifted Embedding


Caching invokers

Query templates

20 / 60

Lifted Embedding


Caching invokersQuery templates

20 / 60

Lifted Embedding

Lifted EmbeddingQuery Template

Lifted Embedding Query Template

val getCoffee = for {id <- Parameters[Int]c <- Query(Coffees) if c.id === id

} yield c.name

getCoffee (10)

JDBC Prepared Statement

"select c.NAME from COFFEES c where c.ID = ?"

21 / 60

Lifted Embedding

Lifted EmbeddingSummary

Problem 1How to create Lifted Embedding Table?Type Providers

Problem 2How to catch the errors?Comprehensive type errorsNonunderstandable type errors

Problem 3How to have high performance?Caching Invokers and Query TemplatesUser effort needed

22 / 60

Lifted Embedding


Problem 1How to create Lifted Embedding Table?

Type Providers



22 / 60

Lifted Embedding





22 / 60

Lifted Embedding



Problem 2How to catch the errors?

Comprehensive type errorsNonunderstandable type errors


22 / 60

Lifted Embedding



Problem 2How to catch the errors?Comprehensive type errors

Nonunderstandable type errors


22 / 60

Lifted Embedding





22 / 60

Lifted Embedding




Problem 3How to have high performance?

Caching Invokers and Query TemplatesUser effort needed

22 / 60

Lifted Embedding




Problem 3How to have high performance?Caching Invokers and Query Templates

User effort needed

22 / 60

Lifted Embedding





22 / 60

Lifted Embedding

Is it possible to have comprehensible type errors?

23 / 60

Direct Embedding

Outline

1 Introduction

2 Lifted Embedding

3 Direct Embedding

4 Shadow Embedding

5 Evaluation

23 / 60

Direct Embedding

Direct Embedding

Query expression to Scala AST (compile-time)Scala AST to Slick AST (run time)Similar to LINQA prototype

24 / 60

Direct Embedding

Direct Embedding

Query expression to Scala AST (compile-time)

Scala AST to Slick AST (run time)Similar to LINQA prototype

24 / 60

Direct Embedding

Direct Embedding

Query expression to Scala AST (compile-time)Scala AST to Slick AST (run time)

Similar to LINQA prototype

24 / 60

Direct Embedding

Direct Embedding

Query expression to Scala AST (compile-time)Scala AST to Slick AST (run time)Similar to LINQ

A prototype

24 / 60

Direct Embedding

Direct Embedding

Query expression to Scala AST (compile-time)Scala AST to Slick AST (run time)Similar to LINQA prototype

24 / 60

Direct Embedding

Direct EmbeddingExample

Direct Embedding Example

Query[Coffee] filter(c => c.id == 10) map(c => c.name)

SQL Statement


25 / 60

Direct Embedding

Direct EmbeddingType Information


26 / 60

Direct Embedding

Direct EmbeddingType Information

Query[Coffee] filter(c => c.id:Int == 10:Int) map(c => c.name:String)

26 / 60

Direct Embedding

Direct EmbeddingType Errors - Good Part

Query[Coffee] map(c => c.id substring 2)

Compile Errorvalue substring is not a member of Int

27 / 60

Direct Embedding

Direct EmbeddingType Errors - Bad Part

Query[Coffee] map(c => c.id.toDouble)

Compiles!Run time error!

28 / 60

Direct Embedding



Compiles!

Run time error!

28 / 60

Direct Embedding



Compiles!Run time error!

28 / 60

Direct Embedding

Direct EmbeddingType Errors

Adapted from http://r32argent.ca/R32%20information_files/VW%20ads/vw_bus.jpg

29 / 60

Direct Embedding

Direct EmbeddingSummary

Problem 2 (recap)


Comprehensible type errorsIncomprehensive type errors

30 / 60

Direct Embedding


Problem 2 (recap)

How to catch the errors?Comprehensible type errors

Incomprehensive type errors

30 / 60

Direct Embedding


Problem 2 (recap)

How to catch the errors?Comprehensible type errorsIncomprehensive type errors

30 / 60

Direct Embedding

Is it possible to have comprehensive and comprehensibletype errors at the same time?

31 / 60

Shadow Embedding

Outline

1 Introduction

2 Lifted Embedding

3 Direct Embedding

4 Shadow Embedding

5 Evaluation

31 / 60

Shadow Embedding

Shadow EmbeddingArchitecture

32 / 60

Shadow Embedding


shadow = shallow + deep

32 / 60

Shadow Embedding


32 / 60

Shadow Embedding

Shadow EmbeddingShallow Interface

33 / 60

Shadow Embedding

Shadow EmbeddingShallow Interface

Query interface:

class Query[T] {def map[S](f: T => S): Query[S]def filter(f: T => Boolean): Query[T]def flatMap[S](f: T => Query[S]): Query[S]def groupBy[S](f: T => S): Query[(S, Query[T])]def union(q2: Query[T]): Query[T]def join[S](q2: Query[S]): JoinQuery[T, S]// ...

}

34 / 60

Shadow Embedding

Shadow EmbeddingExample

Shallow Embedding Example

stage {Query[Coffee] filter

(c => c.id == 10) map(c => c.name)

}

35 / 60

Shadow Embedding

Shadow EmbeddingType Information


36 / 60

Shadow Embedding

Shadow EmbeddingType Information

Query[Coffee] filter(c => c.id:Int == 10:Int) map(c => c.name:String)

36 / 60

Shadow Embedding

Shadow EmbeddingYin-Yang Transformation

37 / 60

Shadow Embedding


Shallow Query

stage {Query (1) filter (x => x == 10)

}

38 / 60

Shadow Embedding


After Language Virtualization

Query (1) filter (x => x __== 10)

38 / 60

Shadow Embedding


After Ascription

Query(1:Int) filter((x:Int) => (x:Int) __== (10:Int))

38 / 60

Shadow Embedding


After Lifting

Query(lift(1):Int) filter((x:Int) => (x:Int) __== (lift(10):Int))

38 / 60

Shadow Embedding


After Type Transformation

Query(lift (1):this.Int) filter((x:this.Int) =>

(x:this.Int) __== (lift (10):this.Int))

38 / 60

Shadow Embedding


After Scope Injection

new ShadowDSLComponent {this.Query(lift (1):this.Int) filter

((x:this.Int) =>(x:this.Int) __== (lift (10):this.Int))

}

38 / 60

Shadow Embedding


Lifted Embedding Query

new ShadowDSLComponent {this.Query(lift (1):this.Int) filter

((x:this.Int) =>(x:this.Int) __== (lift (10):this.Int))

}

38 / 60

Shadow Embedding

Shadow EmbeddingLifted Embedding

39 / 60

Shadow Embedding


No need to convert from Scala AST to Slick AST

Interoperable with Lifted Embedding

40 / 60

Shadow Embedding


No need to convert from Scala AST to Slick ASTInteroperable with Lifted Embedding

40 / 60

Shadow Embedding

Shadow EmbeddingA Problem similar to Problem 1

Problem 1 (recap)


stage {Query[Coffee] map (c => c.id)

}

How to create Shadow Embedding Table?Reuse Type Providers of Lifted Embedding!

41 / 60

Shadow Embedding


Problem 1 (recap)



}

How to create Shadow Embedding Table?

Reuse Type Providers of Lifted Embedding!

41 / 60

Shadow Embedding


Problem 1 (recap)



}

How to create Shadow Embedding Table?Reuse Type Providers of Lifted Embedding!

41 / 60

Shadow Embedding

Shadow EmbeddingProblem 2

Problem 2 (recap)


42 / 60

Shadow Embedding

Shadow EmbeddingType Errors - Good Part

stage {Query[Coffee] map

(c => c.id substring 2)}

Compile Errorvalue substring is not a member of Int

43 / 60

Shadow Embedding

Shadow EmbeddingType Errors - Good Part Again!

stage {Query[Coffee] map

(c => c.id.toDouble)}

Compile Errorin Slick method toDouble is not a member of Int

44 / 60

Shadow Embedding

Shadow EmbeddingType Errors - Surprise!

stage {Query[Coffee] map (c =>

if(c.origin == "Iran")"Good"

elsec.quality

)}

Compiles and works!

45 / 60

Shadow Embedding

Shadow EmbeddingType Errors

Adapted from http://www.littlerocktours.com/images/vehicles/buses/109-lg.jpg

46 / 60

Shadow Embedding

Shadow EmbeddingProblem 3

Problem 3 (recap)


47 / 60

Shadow Embedding

Shadow EmbeddingShadow Interpreter

48 / 60

Shadow Embedding

Shadow EmbeddingShadow Interpreter

49 / 60

Shadow Embedding

Shadow EmbeddingQuery Template

Shadow Embedding Query Template

def getCoffee(id: Int) = stage {for {

c <- Query[Coffee] if c.id == id} yield c.name

}

getCoffee (10)

JDBC Prepared Statement

"select c.NAME from COFFEES c where c.ID = ?"

50 / 60

Shadow Embedding

Shadow EmbeddingQuery Template - Shadow vs. Lifted

def getCoffee(id: Int) = stage {for {

c <- Query[Coffee] if c.id == id} yield c.name

}

getCoffee (10)

vs.val getCoffee = for {

id <- Parameters[Int]c <- Query(Coffees) if c.id === id

} yield c.name

getCoffee (10)

51 / 60

Shadow Embedding

Shadow EmbeddingComposability

val query: Query[Coffee] = stage {Query[Coffee] filter (_.origin == "Iran")

}

stage {query map (_.name)

}

52 / 60

Shadow Embedding

Shadow EmbeddingSummary

Problem 1 (recap)

How to create Lifted Embedding Table?Type Providers

Problem 2 (recap)

How to catch the errors?Shallow Interface makes it comprehensibleYin-Yang makes it comprehensive

Problem 3 (recap)

How to have high performance?Shadow Interpreter reduces the user effort

53 / 60

Shadow Embedding


Problem 1 (recap)


Type Providers

Problem 2 (recap)


Problem 3 (recap)


53 / 60

Shadow Embedding


Problem 1 (recap)


Problem 2 (recap)


Problem 3 (recap)


53 / 60

Shadow Embedding


Problem 1 (recap)


Problem 2 (recap)


Shallow Interface makes it comprehensibleYin-Yang makes it comprehensive

Problem 3 (recap)


53 / 60

Shadow Embedding


Problem 1 (recap)


Problem 2 (recap)

How to catch the errors?Shallow Interface makes it comprehensible

Yin-Yang makes it comprehensive

Problem 3 (recap)


53 / 60

Shadow Embedding


Problem 1 (recap)


Problem 2 (recap)


Problem 3 (recap)


53 / 60

Shadow Embedding


Problem 1 (recap)


Problem 2 (recap)


Problem 3 (recap)


Shadow Interpreter reduces the user effort

53 / 60

Shadow Embedding


Problem 1 (recap)


Problem 2 (recap)


Problem 3 (recap)


53 / 60

Evaluation

Outline

1 Introduction

2 Lifted Embedding

3 Direct Embedding

4 Shadow Embedding

5 Evaluation

53 / 60

Evaluation

Correctness

Several basic tests

All Direct Embedding test suitesImportant Lifted Embedding test suites

54 / 60

Evaluation

Correctness

Several basic testsAll Direct Embedding test suites

Important Lifted Embedding test suites

54 / 60

Evaluation

Correctness

Several basic testsAll Direct Embedding test suitesImportant Lifted Embedding test suites

54 / 60

Evaluation

PerformanceMicrobenchmarking

Shadow Embedding Simple Selection

for (i <- range) {stage {

for (c <- Query[Coffee] if c.id == 1) yield c}

}

55 / 60

Evaluation


55 / 60

Evaluation


Shadow Embedding Parameterized Selection

for (i <- range) {stage {

for (c <- Query[Coffee] if c.id < i) yield c}

}

55 / 60

Evaluation


55 / 60

Evaluation

PerformanceDatabench

50,000 accounts500,000 transactions20% updating80% reading

56 / 60

Evaluation

PerformanceDatabench

57 / 60

Evaluation

Conclusion

User-friendly

Shallow InterfaceType ProvidersComposabilityShadow Interpreter

Comprehensive and comprehensible type errors

Shallow InterfaceYin-Yang

Highly performant

Shadow Interpreter


Reusing Lifted Embedding

Maintainable


58 / 60

Evaluation

Conclusion

User-friendlyShallow Interface

Type ProvidersComposabilityShadow Interpreter



Highly performant

Shadow Interpreter



Maintainable


58 / 60

Evaluation

Conclusion

User-friendlyShallow InterfaceType Providers

ComposabilityShadow Interpreter



Highly performant

Shadow Interpreter



Maintainable


58 / 60

Evaluation

Conclusion

User-friendlyShallow InterfaceType ProvidersComposability

Shadow InterpreterComprehensive and comprehensible type errors


Highly performant

Shadow Interpreter



Maintainable


58 / 60

Evaluation

Conclusion

User-friendlyShallow InterfaceType ProvidersComposabilityShadow Interpreter



Highly performant

Shadow Interpreter



Maintainable


58 / 60

Evaluation

Conclusion


Comprehensive and comprehensible type errorsShallow Interface

Yin-YangHighly performant

Shadow Interpreter



Maintainable


58 / 60

Evaluation

Conclusion


Comprehensive and comprehensible type errorsShallow InterfaceYin-Yang

Highly performant

Shadow Interpreter



Maintainable


58 / 60

Evaluation

Conclusion



Highly performant

Shadow InterpreterInteroperable with Lifted Embedding


Maintainable


58 / 60

Evaluation

Conclusion



Highly performantShadow Interpreter



Maintainable


58 / 60

Evaluation

Conclusion





Reusing Lifted EmbeddingMaintainable


58 / 60

Evaluation

Conclusion




Interoperable with Lifted EmbeddingReusing Lifted Embedding

Maintainable


58 / 60

Evaluation

Conclusion




Interoperable with Lifted EmbeddingReusing Lifted Embedding

MaintainableReusing Lifted Embedding

58 / 60

Evaluation

Future Work

Macro annotations

Shadow Programming

Yin-YangType providers

59 / 60

Evaluation

Future Work

Macro annotationsShadow Programming


59 / 60

Evaluation

Future Work


Yin-Yang

Type providers

59 / 60

Evaluation

Future Work



59 / 60

Evaluation

Thank You

Thank You!

60 / 60

an embedded query language in scala · an embedded query language in scala amirshaikhha school of...

Documents