The Evolution Of Scala Martin Odersky

EPFL and Typesafe

10 Years of Scala

Pre History

1980s Modula-2, Oberon 1990-95 Functional Programming 1995-98 Pizza 1998-99 GJ, javac 2000-02 Functional Nets, Funnel

3

4

Minimal programming language based on type members and functional nets (a variant of join calculus)

Analogous to Pict (Pierce and Turner 2001) for Pi-calculus.

A Minimal Language

•  Idea of Funnel: Show that we can build a general programming language that can be understood as thin syntactic sugar over a core calculus. –  General: OO, functional + imperative, concurrent –  Core calculus: Functional nets –  Sugar: Records, Lambdas, Type members.

•  Wrote some programs (including parts of the Funnel library) in Funnel.

•  Quickly became apparent that encodings suck: –  Confusing for beginners –  Boring to do them over and over again for experts

5

Motivation for Scala

•  Grew out of Funnel •  Wanted to show that we can do a practical combination

of OOP and FP. •  What got dropped:

–  Concurrency was relegated to libraries –  No tight connection between language and core calculus

(fragments were studied in the νObj paper and others.)

•  What got added: –  Native object and class model, Java interop, XML literals (!).

6

Why a New Language?

•  The OO dogma ruled then: Encapsulate mutable data with methods. –  Infamous example: Java beans. –  There was no place for functional

programming in this.

•  New at the time: Webservices that process immutable (semi-)structured data. –  Service sees the data “from the outside”. –  Functional programming supports that view,

e.g. using pattern matching, recursion.

•  Rationale given: Would be good to have a new FP language for webservices

7

8

Really, Why a new Language? The work on Scala was motivated by two hypotheses: Hypothesis 1: A general-purpose language needs to be scalable; the same concepts should describe small as well as large parts. Hypothesis 2: Scalability can be achieved by unifying and generalizing functional and object-oriented programming concepts.

How That Worked Out

9

(from:James Iry: A Brief, Incomplete, and Mostly Wrong History of Programming Languages)

Scala and Pizza

•  Pizza (Odersky and Wadler 96) was another language on the JVM that added functional elements to Java: –  algebraic datatypes and pattern matching –  function values –  generics

•  Scala was more ambitious: –  More innovation on the OOP side –  More functional, e.g. immutable values, by-name parameters, –  Better integration of functional/oop, e.g. case classes. –  Not backwards compatible with Java

10

Java Features Not kept in Scala

public  static  void  Enumerations Annotation Syntax Wildcard types Raw types Primitive types Array types Definite assignment rules

11

Statements: break  continue  synchronized  assert  for  (C-style) try (resource) super(...)

Expressions: primitive operators cast syntax conditional x  ?  y  :  z   array selection a[i]  

Scala Beginnings

2003: First internal use –  to teach “Functional and Logic Programming Course” at EPFL.

(2nd year, ~ 150 participants), –  despite being not really ready for the task.

2004: Official announcement of Scala 1.0

–  First vocal outside users: Miles Sabin, John Pretty @ Sygneca –  Together with Iulian Dragos and myself these are probably the

only people who have used Scala continuously for 10 years.

12

Scala Reloaded

2006: Scala 2.0 released –  Compiler written in Scala –  Followed the cake-pattern described “Scalable Component

Abstractions [Odersky&Zenger 05].

A few new features: –  Semicolon inference (!) –  Generalization of implicits and traits –  Automatically added empty parameter lists ()

Additions in 2.1, 2.2: –  Qualified access: private[C],  protected[C]  –  Multi-line string literals: ”””this  is  a  line  

                                                       and  this  is  another”””        –  Procedure syntax: def  sort(xs:  Array[T])  {...}    

13

Scala Reloaded

2006: Scala 2.0 released –  Compiler written in Scala –  Followed the cake-pattern described “Scalable Component

Abstractions [Odersky&Zenger 05].

A few new features: –  Semicolon inference (!) –  Generalization of implicits and traits –  Automatically added empty parameter lists ()

Additions in 2.1, 2.2: –  Qualified access: private[C],  protected[C]  –  Multi-line string literals: ”””this  is  a  line  

                                                       and  this  is  another”””        –  Procedure syntax: def  sort(xs:  Array[T])  {...}    

14

Learning from Experience

Scala 1.x had –  Parameterless methods supporting the uniform access principle.

def  length:  Int  =  ...    

–  Partially applied functions that are always eta-expanded: def  sum(f:  Int  =>  Int)(bounds:  Range)  =  ...  val  sumSquares  =  sum(x  =>  x*x)    

The combination of these two was a source of common pitfalls:

println(“abc”.length)      //  prints:  <function>    

15

Avoiding the Pitfalls

1.  Auto-add () for references f is to nullary functions def  f()  =  ...  

2.  Eta-expand only if –  expected type is a function or –  missing parameters are specified with `_’

16

The Growth Year

2007: Scala 2.3-2.7 add lots of new features: Extractors object  Email  {  def  unapply  ...  }  

case  Email(name,  domain)  =>  ...  Tuples (1,  “a”,  true)  Assignment operators +=,  *=,  ...  “_” notation for functions (_  +  1)  Early initialization    object  Foo  extends  {    

           val  x  =  3            }  with  SomeTrait  

Lazy values lazy  val  rest  =  f()  Higher-kinded types class  Functor[F[_]]  {  ...  }  Structural types {  val  key:  String  }  Existential types    Map[T,  T]  forSome  {  type  T  }  

17

Why The Rapid Growth?

•  People asked for it –  “If Scala only had this one new feature, I could use it in my

organization”

•  People volunteered to do it –  Lots of thoughtful suggestions on the mailing list. –  PhD students were keen to see their thesis work applied.

18

Community Formation

2007: Lift web framework launched. 2008: First Scala liftoff unconference (50 particants)

–  Twitter goes public with Scala, hype starts

2009: More Scala liftoffs. 2010-14: Scala Days

–  2010 EPFL 180 participants –  2011 Stanford 280 –  2012 London 400 –  2013 New York 500 Scala Workshop Montellier –  2014 Berlin 800 Scala Symposium Uppsala

Lots of other meetups and conferences

19

Scala 2.8 and 2.9: Consolidation

2010: Scala 2.8, with –  New collections with bitrot prevention. –  Fixed leaky array model. –  New semantics of nested packages. –  Better type inference for implicit resolution –  Lots of bug-fixes

2011: Scala 2.9, with –  Parallel collections –  Special trait DelayedInit, used in App –  Trait Dynamic, to interface with dynamic languages

20

Scala 2.10: Differentiation

2012: Scala 2.10, with •  New features, added through the Scala Improvement

Process (SIPs): –  Value classes class  Meter(x:  Long)  

       extends  AnyVal  

–  Implicit classes  implicit  class  StringOps(s:  String)  –  String interpolation    s”you  have  $n  new  calls”  

•  Experimental features –  Macros def  await(x:  Future[T])  =  macro  ...  –  Reflection These are only enabled when compiling with –Xexperimental

•  Language imports require explicit enabling of some features available previously.

21

Scala Improvement Process

22

Design Tradeoffs

The Scala way: Provide few constructs of maximal generality.

Implicit conversions > implicit classes > extension methods

where “>” means more general.

Implicit conversions are very powerful But they can be misused,

in particular if there are too many of them.

23

General Problem

•  Scala is geared for orthogonality and expressiveness •  I believe that in the end, that’s the most productive

combination. •  But there are challenges.

–  Some combinations of language features might be less desirable than others.

–  How to avoid feature misuse?

•  Idea: Have a mechanism that demands that some problematic features are explicitly imported (Haskell uses something similar).

24

SIP 18: Language Imports

•  Say you have:

object letsSimulateJS { implicit def foo(x: String): Int = Integer.parseInt(x)}

•  Compiling gives:

warning:  there  were  1  feature  warnings;  re-­‐run  with  -­‐feature  for  details  

one  warning  found    

 25

SIP 18: Language Imports

Say you have:

object letsSimulateJS { implicit def foo(x: String): Int = Integer.parseInt(x)}

Compiling with –feature gives: letsSimulateJS.scala:8:  warning:  implicit  conversion  method  foo  should  be  enabled  by  making  the  implicit  value  language.implicitConversions  visible.  This  can  be  achieved  by  adding  the  import  clause  'import  

scala.language.implicitConversions'  or  by  setting  the  compiler  option  -­‐language:implicitConversions.  See  the  Scala  docs  for  value  scala.language.implicitConversions  for  a  discussion  why  the  feature  should  be  explicitly  enabled.      implicit  def  foo(x:  String):  Int  =  Integer.parseInt(x)                                ^  one  warning  found  

 

26

Turning off the Warnings

You turn off the warning by bringing the identifier

scala.language.implicitConversions    

into scope, usually, using an import:

import language.implicitConversions

27

Features Controlled by SIP-18

From language: –  Implicit Conversions –  Dynamic –  Postfix Operators –  Dynamic dispatch on structural types –  Existential types –  Higher-kinded types

From language.experimental  –  Macros

28

Now: Scala 2.11

•  Smaller: –  broke out parts of libraries into separate modules

•  Faster –  Better incremental compilation

•  Stronger: –  Lots of bug fixes, tooling improvements

29

Now: Scala.JS

Why a Scala for Javascript? –  JS is becoming ubiquitous. –  Desire to use the same language on

client and server. –  But not everybody likes Javascript or

dynamic languages.

Scala.JS profits from Scala’s tradition of interoperating with a host language through very general abstractions. Can combine JS DOM and Scala collections. For the young age of the project, very mature and well-received.

30

Invariants

In all this evolution, what stays constant? What are some of the essential traits that make Scala what it is?

31

1st Invariant: A Scalable Language

•  Instead of providing lots of features in the language, have the right abstractions so that they can be provided in libraries.

•  This has worked quite well so far. •  It implicitly trusts programmers and library designers to

“do the right thing”, or at least the community to sort things out.

32

Libraries on top of Scala

33

SBT Chisel Spark

Spray Kafka

Akka

ScalaTest Squeryl Specs

shapeless

Scalaz

Slick

Growable = Good?

In fact, it’s a double edged sword.

–  DSLs can fracture the user community (“The Lisp curse”)

–  Besides, no language is liked by everyone, no matter whether its a DSL or general purpose.

–  Host languages get the blame for the DSLs they embed.

Growable is great for experimentation. But it demands conformity and discipline for large scale production use.

34

•  Scala’s core is its type system. •  Most of the advanced types concepts are about

flexibility, less so about safety.

2nd Invariant: It’s about the Types

35 Flexibility / Ease of Use

Safety

Scala

Trend in Type-systems

Goals of PL design

Stunted Evolution

null  -­‐  “The Million Dollar Mistake” •  Why does Scala not have null-safety? •  We had plans to do it

you can see the traces in the stdlib with marker trait NotNull.  

 •  But by then everybody was using already Option. •  So NPEs are actually quite rare in Scala code. •  Don’t want two ways to do the same thing.

36

What’s Next?

•  Scala 2.12 will be a fairly conservative evolution of 2.11

•  Main feature: Java 8 interop. –  Scala and Java lambdas can understand each other –  SAM method convention added to Scala –  Should make use of Java 8 streams –  Default methods for traits?

37

And After That?

Main Goals: Make the language and its libraries •  simpler to understand, •  more robust, •  better performing Want to continue to make it the language of choice for smart kids.

38

Scala “Aida”

Will concentrate on the standard library. –  Reduce reliance on inheritance –  Make all default collections immutable (e.g. scala.Seq will be an

alias of scala.immutable.Seq) –  Other small cleanups that are possible with a rewriting step (e.g.

rename mapValues)

Projects which might make it if they mature fast enough: –  scala.meta, the new, simplified approach to macros and

reflection. –  Collection fusion in the style of ScalaBlitz –  Better specialization through miniboxing.

39

Scala “Don Giovanni”

Concentrates on the language •  Simple foundations:

–  A single fundamental concept - type members – can give precise meaning to generics, existential types, and higher-kinded types.

–  Intersection and union types. –  Theoretical foundations given by minimal core calculus (DOT).

•  Cleaned-up syntax: –  Trait parameters instead of early definition syntax –  XML string interpolation instead of XML literals –  Procedure syntax is dropped. –  Simplified and unified type syntax for all forms of information

elision, forSome syntax is eliminated. 40

Scala “Don Giovanni”

•  Removing puzzlers: –  Result types mandatory for implicit definitions. –  Inherited explicit result types take precedence over locally-

inferred ones. –  String “+” needs explicit enabling. –  Avoid surprising behavior of auto-tupling.

•  Backwards compatibility: –  A migration tool will upgrade sources automatically. –  Should work for almost all commonly used code. –  Will not generally work for code using –Xexperimental –  But we aim to have features that can support analogous

functionality.

41

The Growth Year, Revisited

Extractors object  Email  {  def  unapply  ...  }  ✔   case  Email(name,  domain)  =>  ...  

Tuples (1,  “a”,  true)                                      ✔  Assignment operators +=,  *=,  ++=                                            ✔

Annotations @volatile,  @deprecated                      ✔ “_” notation for functions (_  +  1) ✔

Early initialization    object  Foo  extends  {                ✗              val  x  =  3            }  with  SomeTrait  

Higher-kinded types class  Functor[F[_]]  {  ...  }    ≈  Structural types {  val  key:  String  }                ≈  Lazy values lazy  val  rest  =  f()              ✔  Existential types    Map[T,  T]  forSome  {  type  T  } ✗  

42

Conclusion

•  Languages are not cast in stone; they evolve whether you like it or not.

•  Community matters •  Community will take a language where you never

expected it to go.

In the end languages are as much social phenomena as technical ones.

43