building dsls with scala
TRANSCRIPT
![Page 1: Building DSLs with Scala](https://reader033.vdocument.in/reader033/viewer/2022042722/588a33d31a28abc6168b532f/html5/thumbnails/1.jpg)
Building DSLs with ScalaMohit Jaggi
Code Ninja and Big (Data) Troublemaker
Ayasdi
![Page 2: Building DSLs with Scala](https://reader033.vdocument.in/reader033/viewer/2022042722/588a33d31a28abc6168b532f/html5/thumbnails/2.jpg)
Who am I?• Software engineer and architect
• Past life: built networking, security and application delivery appliances using ASICs, microcode, C, some C++. Operating system optional.
• This life: distributed systems, (big) data analytics, machine learning using Java and Scala. And real servers with a proper operating system!
![Page 3: Building DSLs with Scala](https://reader033.vdocument.in/reader033/viewer/2022042722/588a33d31a28abc6168b532f/html5/thumbnails/3.jpg)
Why I love Scala?• Less red tape
• Multi-paradigm
• Exploits the Java ecosystem
• Static typing and type inference
• Same language for everybody from casual script writer to application programmer to library designer
![Page 4: Building DSLs with Scala](https://reader033.vdocument.in/reader033/viewer/2022042722/588a33d31a28abc6168b532f/html5/thumbnails/4.jpg)
Who will find this useful?
• Beginner to intermediate scala programmers
• If you write code, you have APIs
• Library designers
![Page 5: Building DSLs with Scala](https://reader033.vdocument.in/reader033/viewer/2022042722/588a33d31a28abc6168b532f/html5/thumbnails/5.jpg)
Agenda
• Motivation for DSLs
• Scala Constructs useful for DSLs
• Lessons from my limited experience
![Page 6: Building DSLs with Scala](https://reader033.vdocument.in/reader033/viewer/2022042722/588a33d31a28abc6168b532f/html5/thumbnails/6.jpg)
Motivation for DSLs
• API Design Considerations
• DSL
• Types of DSLs and tradeoffs
![Page 7: Building DSLs with Scala](https://reader033.vdocument.in/reader033/viewer/2022042722/588a33d31a28abc6168b532f/html5/thumbnails/7.jpg)
Considerations for a good API
• sufficient for current needs
• extensible for anticipated needs
• does not preclude unanticipated needs
• forward/backward compatibility
• easy to use
• little or no documentation required, but
• complete documentation available
![Page 8: Building DSLs with Scala](https://reader033.vdocument.in/reader033/viewer/2022042722/588a33d31a28abc6168b532f/html5/thumbnails/8.jpg)
DSL
• Domain Specific Language
• helps with ease of use part
• e.g. SQL, R, jooq, bigdf
![Page 9: Building DSLs with Scala](https://reader033.vdocument.in/reader033/viewer/2022042722/588a33d31a28abc6168b532f/html5/thumbnails/9.jpg)
Types of DSLs
• External or standalone
• Internal or embedded
![Page 10: Building DSLs with Scala](https://reader033.vdocument.in/reader033/viewer/2022042722/588a33d31a28abc6168b532f/html5/thumbnails/10.jpg)
External DSL• New language from “scratch”
• Write a parser using something like lex/yacc, antlr
• Write an interpreter or
• Write a code generator
• Manage a run-time environment(Java?, native?, REPL?)
![Page 11: Building DSLs with Scala](https://reader033.vdocument.in/reader033/viewer/2022042722/588a33d31a28abc6168b532f/html5/thumbnails/11.jpg)
Internal DSL• “internal” or “embedded”
in a general purpose programming language
• Designed to “feel like” a different language more intuitive to the domain
• e.g. Jooq in Java/Scala feels like SQL
![Page 12: Building DSLs with Scala](https://reader033.vdocument.in/reader033/viewer/2022042722/588a33d31a28abc6168b532f/html5/thumbnails/12.jpg)
Tradeoffs
• Less work
• More completeness
• Less syntax flexibility
• “global optimizations” hard
• Error messages less meaningful
• More freedom in syntax
• More “global optimizations” possible
• Possible to have better error messages
• More work
• Typically slower without good code generation
Internal
External
Benefit Cost
![Page 13: Building DSLs with Scala](https://reader033.vdocument.in/reader033/viewer/2022042722/588a33d31a28abc6168b532f/html5/thumbnails/13.jpg)
Why Scala for DSLs?
select(BOOK.ID) from BOOK where BOOK.TITLE === “Scala scala”
select(BOOK.ID).from(BOOK).where(BOOK.TITLE.equal(“Java java”))
![Page 14: Building DSLs with Scala](https://reader033.vdocument.in/reader033/viewer/2022042722/588a33d31a28abc6168b532f/html5/thumbnails/14.jpg)
Scala Constructs Useful For DSLs
![Page 15: Building DSLs with Scala](https://reader033.vdocument.in/reader033/viewer/2022042722/588a33d31a28abc6168b532f/html5/thumbnails/15.jpg)
Vanishing Method Call
• apply() and update() can be called without naming them
df(“age”) == df.apply(“age”) == df.column(“age”)
df(“age”) = df(“age”) + 1
df.update(df(“age”), df(“age”) + 1)
![Page 16: Building DSLs with Scala](https://reader033.vdocument.in/reader033/viewer/2022042722/588a33d31a28abc6168b532f/html5/thumbnails/16.jpg)
Simulated dynamic typing• extend the Dynamic trait
• hides compile errors, is it worth paying that cost?
df.age == df(“age”) == df.selectDynamic(“age”)
def selectDynamic(colName: String) = { val col = self.column(colName) if (col == null) logger.error(s"$colName does not match any DF API or column name") col }
def updateDynamic(colName: String)(that: Column): Unit = update(colName, that)
df.age = df.age + 1
df.updateDynamic(“age”)(age + 1)
![Page 17: Building DSLs with Scala](https://reader033.vdocument.in/reader033/viewer/2022042722/588a33d31a28abc6168b532f/html5/thumbnails/17.jpg)
Dial 0 for operator• dot is optional, so methods looks like operators
• any symbol can be used as operator
• for example to list people that can buy beer you can use
df.where(df(“age”).gt(21))
df where (df(“age”) > 21)
![Page 18: Building DSLs with Scala](https://reader033.vdocument.in/reader033/viewer/2022042722/588a33d31a28abc6168b532f/html5/thumbnails/18.jpg)
• operator ending in colon applies to right operand, this provides more freedom (~active vs passive voice)
object orange { def eat_:(who: String):Unit =
println(s"$who is eating orange”) }
“monkey” eat_: orange
• careful with operators like ==, use === instead
![Page 19: Building DSLs with Scala](https://reader033.vdocument.in/reader033/viewer/2022042722/588a33d31a28abc6168b532f/html5/thumbnails/19.jpg)
Free companion object
• hide “new” to make an object
val df = DF.fromCSVFile(…)
• also a good place to put implicit conversions in
![Page 20: Building DSLs with Scala](https://reader033.vdocument.in/reader033/viewer/2022042722/588a33d31a28abc6168b532f/html5/thumbnails/20.jpg)
Reading between the lines• implicit conversion to “enrich” types
• can also use “implicit class”
• newer construct “Value Classes” can sometimes replace this
2 MB == ByteSize(2*1024*1024)
class BytesMaker(n: Int) {
def MB = new Bytes(n * 1024 * 1024)
def GB = new Bytes(n * 1024 * 1024 * 1024)
}
case class Bytes(val n: Int) { def print = println(s”bytes=$n") }
object Bytes {
import scala.language.implicitConversions
implicit def intToBytesMaker(n: Int) = new BytesMaker(n)
}
![Page 21: Building DSLs with Scala](https://reader033.vdocument.in/reader033/viewer/2022042722/588a33d31a28abc6168b532f/html5/thumbnails/21.jpg)
My name is Bond
• Pass by name parameters are useful to pass in “blocks of code”
def doTwice(action: => Unit) { action action }
doTwice { println("hey") }
![Page 22: Building DSLs with Scala](https://reader033.vdocument.in/reader033/viewer/2022042722/588a33d31a28abc6168b532f/html5/thumbnails/22.jpg)
Multiple Parameter Lists
• Help avoid unwanted commas and more intuitive parentheses/braces def when(predicate: => Boolean)(action: => Unit) = {
if(predicate) action
}
when(1 == 1) { println(“1”) }
instead of
when(1 == 1, println(“1”))
![Page 23: Building DSLs with Scala](https://reader033.vdocument.in/reader033/viewer/2022042722/588a33d31a28abc6168b532f/html5/thumbnails/23.jpg)
FWIW - API
• Don't provide unnecessary options, if needed you can always add more; like "salt in a dish", you can't remove it
• Names meaningful to user not the coder
• Consistent coding style
• Option[T] if truly optional, else exception
![Page 24: Building DSLs with Scala](https://reader033.vdocument.in/reader033/viewer/2022042722/588a33d31a28abc6168b532f/html5/thumbnails/24.jpg)
FWIW - DSL• Don’t take it to extremes (like “Baysick”)
• Assume users know some(or lot of) Scala
• Try to generate useful error messages
• Careful with ==, right associative operators, implicit search order
• Aim to provide gentle slope to Scala instead of none; basic Scala is simple, your API users will appreciate it
![Page 25: Building DSLs with Scala](https://reader033.vdocument.in/reader033/viewer/2022042722/588a33d31a28abc6168b532f/html5/thumbnails/25.jpg)
Thanks!We are hiring
http://engineering.ayasdi.com http://www.ayasdi.com/careers