managing binary compatibility in scala (scala lift off 2011)

22
Managing Binary Compatibility in Scala Mirco Dotta Typesafe October 13, 2011 Mirco Dotta Managing Binary Compatibility in Scala

Upload: mircodotta

Post on 11-Nov-2014

1.052 views

Category:

Technology


3 download

DESCRIPTION

Slides of my Scala Lift Off 2011 talk. The content of the presentation is mostly similar to the one presented at Scala Days 2011, with a few additions. Particularly, lazy values are discussed.

TRANSCRIPT

Page 1: Managing Binary Compatibility in Scala (Scala Lift Off 2011)

Managing Binary Compatibility in Scala

Mirco Dotta

Typesafe

October 13, 2011

Mirco Dotta Managing Binary Compatibility in Scala

Page 2: Managing Binary Compatibility in Scala (Scala Lift Off 2011)

Introduction Sources of Incompatibility Conclusion

Outline

IntroductionExampleScala vs. Java

Sources of IncompatibilityType InferencerType ParametersTraitLazy Fields

Conclusion

Mirco Dotta Managing Binary Compatibility in Scala

Page 3: Managing Binary Compatibility in Scala (Scala Lift Off 2011)

Introduction Sources of Incompatibility Conclusion Example Scala vs. Java

Example

Assume Analyzer is part of a library we produce. We decide thatits API has to evolve as follows:

class Analyzer { // old versiondef analyze(issues: HashMap[ , ]) {...}

}

class Analyzer { // new versiondef analyze(issues: Map[ , ]) {...}

}

Further, assume the next expression was compiled against the oldlibrary

new Analyzer().analyze(new HashMap[Any,Any])

Would the compiled code work if run against the new library?

The answer lies in the bytecode...

Mirco Dotta Managing Binary Compatibility in Scala

Page 4: Managing Binary Compatibility in Scala (Scala Lift Off 2011)

Introduction Sources of Incompatibility Conclusion Example Scala vs. Java

Example: Bytecode

Let’s have a look at the method signature for the two versions:

class Analyzer { // old versionanalyze(Lscala/collection/immutable/HashMap);V}

class Analyzer { // new versionanalyze(Lscala/collection/immutable/Map);V}

The expression compiled against the old library would look like:

...invokevirtual #9;// #9 == Analyzer.analyze:(Lscala/collection/immutable/HashMap;)V

⇒ The method’s name has been statically resolved atcompile-time.

Running it against the new library would result in the JVMthrowing a NoSuchMethodException.

⇒ The evolution of class Analyzer breaks compatibility withpre-existing binaries.

Mirco Dotta Managing Binary Compatibility in Scala

Page 5: Managing Binary Compatibility in Scala (Scala Lift Off 2011)

Introduction Sources of Incompatibility Conclusion Example Scala vs. Java

Is Binary Compatibility a Scala issue?

The short answer is No. The discussed example can be easilyported in Java or other languages targeting the JVM.

Scala shares with Java many sources of binary incompatibility.

But Scala offers many language features not available in Java:

I First-class functions.

I Type Inferencer.

I Multiple inheritance via mixin composition (i.e., traits).

I Lazy values.

. . . Just to cite a few.

⇒ Scala code has new “unique” sources of binary incompatibility.

Mirco Dotta Managing Binary Compatibility in Scala

Page 6: Managing Binary Compatibility in Scala (Scala Lift Off 2011)

Introduction Sources of Incompatibility Conclusion Type Inferencer Type Parameters Trait Lazy Fields

Type Inferencer: Member Signature

Does the following evolution break binary compatibility?

class TextAnalyzer { // old versiondef analyze(text: String) = {

val issues = Map[String,Any]()// ...issues

}}

class TextAnalyzer { // new versiondef analyze(text: String) = {

val issues = new HashMap[String,Any]// ...issues

}}

QuestionWhat is the inferred return type of analyze?

Let’s compare the two methods’ signature.

class TextAnalyzer { // old versionpublic scala.collection.immutable.Map

analyze(java.lang.String);}

class TextAnalyzer { // new versionpublic scala.collection.immutable.HashMap

analyze(java.lang.String);}

Mirco Dotta Managing Binary Compatibility in Scala

Page 7: Managing Binary Compatibility in Scala (Scala Lift Off 2011)

Introduction Sources of Incompatibility Conclusion Type Inferencer Type Parameters Trait Lazy Fields

Type Inferencer: Member Signature (2)

QuestionCan we prevent the method’s signature change?

That’s easy! The method’s return type has to be explicitlydeclared:

class TextAnalyzer { // bytecode compatible new versiondef analyze(text: String): Map[String,Any] = {

val issues = new HashMap[String,Any]// ...issues

}}

Take Home Message

Always declare the member’s type. If you don’t, you mayinadvertently change the members’ signature.

Mirco Dotta Managing Binary Compatibility in Scala

Page 8: Managing Binary Compatibility in Scala (Scala Lift Off 2011)

Introduction Sources of Incompatibility Conclusion Type Inferencer Type Parameters Trait Lazy Fields

Type Parameters & Type Erasure

Assume NodeImpl <: Node. Does the following evolution breakbinary compatibility?

class Tree[T <: NodeImpl] { // old versiondef contains(e: T): Boolean = {...}

}

class Tree[T <: Node] { // new versiondef contains(e: T): Boolean = {...}

}

Type parameters are erased during compilation, but the erasure ofTree.contains is different for the two declarations.

boolean contains(NodeImpl) {...} boolean contains(Node) {...}

Remember that the JVM looks up methods based on the textualrepresentation of the signature, along with the method name.

Take Home Message

Type parameters may change the members’ signature and hencebreak binary compatibility.

Mirco Dotta Managing Binary Compatibility in Scala

Page 9: Managing Binary Compatibility in Scala (Scala Lift Off 2011)

Introduction Sources of Incompatibility Conclusion Type Inferencer Type Parameters Trait Lazy Fields

Trait Compilation

Traits are a powerful language construct that enablesmultiple-inheritance on top of a runtime – the JVM – that doesnot natively support it.

Understanding how traits are compiled is crucial if you need toensure release-to-release binary compatibility.

So, how does the Scala compiler generate the bytecode of a trait?

There are two key elements:

I A trait is compiled into an interface plus an abstract classcontaining only static methods.

I “Forwarder” methods are injected in classes inheriting traits.

Mirco Dotta Managing Binary Compatibility in Scala

Page 10: Managing Binary Compatibility in Scala (Scala Lift Off 2011)

Introduction Sources of Incompatibility Conclusion Type Inferencer Type Parameters Trait Lazy Fields

Trait Compilation Explained

An example will help visualize how traits get compiled:

// declared in a librarytrait TypeAnalyzer {

def analyze(prog: Program) {// the trait’s method impl code

}}

// client codeclass TypingPhase extends TypeAnalyzer

The following is the (pseudo-)bytecode generated by scalac:

interface TypeAnalyzer {void analyze(Program prog);

}abstract class TypeAnalyzer$class {

static void analyze(TypeAnalyzer $this,Program prog) {

// the trait’s method impl code}

}

class TypingPhase implements TraitAnalyzer {// forwarder method injected by scalacvoid analyze(Program prog) {

// delegates to implementationTypeAnalyzer$class.analyze(this,prog)

}}

Mirco Dotta Managing Binary Compatibility in Scala

Page 11: Managing Binary Compatibility in Scala (Scala Lift Off 2011)

Introduction Sources of Incompatibility Conclusion Type Inferencer Type Parameters Trait Lazy Fields

Trait: Adding a concrete method

QuestionCan we add a member in a trait without breaking compatibilitywith pre-existing binaries?

Mirco Dotta Managing Binary Compatibility in Scala

Page 12: Managing Binary Compatibility in Scala (Scala Lift Off 2011)

Introduction Sources of Incompatibility Conclusion Type Inferencer Type Parameters Trait Lazy Fields

Trait: Adding a concrete method

trait TypeAnalyzer { // new versiondef analyze(prog: Program) {...}def analyze(clazz: ClassInfo) {...}

}

//TypeAnalyzer trait compiledinterface TypeAnalyzer {

void analyze(Program prog);void analyze(ClassInfo clazz);

}abstract class TypeAnalyzer$class {

static void analyze(TypeAnalyzer $this,Program prog) {...}

static void analyze(TypeAnalyzer $this,ClassInfo clazz) {...}

}

// compiled against the old versionclass TypingPhase implements TraitAnalyzer {

// forwarder method injected by scalacvoid analyze(Program prog) {

// delegates to implementationTypeAnalyzer$class.analyze(this,prog)

}// missing concrete implementation!??analyze(ClassInfo clazz)??

}

Take Home Message

Adding a concrete method in a traitbreaks binary compatibility.

Mirco Dotta Managing Binary Compatibility in Scala

Page 13: Managing Binary Compatibility in Scala (Scala Lift Off 2011)

Introduction Sources of Incompatibility Conclusion Type Inferencer Type Parameters Trait Lazy Fields

Lazy values decompiled

I Lazy values are a Scala language feature that has nocorrespondent in the JVM.

I ⇒ Their semantic has to be encoded in terms of the availableJVM’s primitives.

I A lazy val is encoded into a private field and a getter plus abitmap.

I The bitmap is used by the getter for ensuring that the privatefield gets initialized only once.

I The bitmap is of type Int, meaning that a maximum of 32lazy value can be correctly handled by it.

I The bitmap is shared with subclasses, so that a new bitmapfield is created only if strictly necessary.

Mirco Dotta Managing Binary Compatibility in Scala

Page 14: Managing Binary Compatibility in Scala (Scala Lift Off 2011)

Introduction Sources of Incompatibility Conclusion Type Inferencer Type Parameters Trait Lazy Fields

Lazy to Eager

QuestionCan we transform a lazy field into an eager one?

Mirco Dotta Managing Binary Compatibility in Scala

Page 15: Managing Binary Compatibility in Scala (Scala Lift Off 2011)

Introduction Sources of Incompatibility Conclusion Type Inferencer Type Parameters Trait Lazy Fields

Lazy to Eager

Let’s take a simple example and look at the generated bytecode:

class ClassAnalyzer { // old versionlazy val superclasses: List[Class] = {

// ...}

}

class ClassAnalyzer { // new versionval superclasses: List[Class] = {

// ...}

}

And here is the bytecode:

class ClassAnalyzer { // old versionprivate List[Class] superclasses;volatile int bitmap$0public int superclasses();// ...

}

class ClassAnalyzer { // new versionprivate final List[Class] superclasses;public int superclasses();// ...

}

Take Home Message

Transforming a lazy value into an eager one does preservecompatibility with pre-existing binaries.

Mirco Dotta Managing Binary Compatibility in Scala

Page 16: Managing Binary Compatibility in Scala (Scala Lift Off 2011)

Introduction Sources of Incompatibility Conclusion Type Inferencer Type Parameters Trait Lazy Fields

Eager to Lazy

QuestionCan we transform an eager field into a lazy one?

Mirco Dotta Managing Binary Compatibility in Scala

Page 17: Managing Binary Compatibility in Scala (Scala Lift Off 2011)

Introduction Sources of Incompatibility Conclusion Type Inferencer Type Parameters Trait Lazy Fields

Eager to Lazy

The previous example demonstrates that lazy and eager fields aresyntactically equivalent, from a client perspective.

Does that mean that the transformation is safe?

It depends.

I If you are in a closed world, then the transformation is safe(e.g., your class is final).

I Otherwise, you are taking a high risk on breaking the semanticof lazy, and no good can come out of that. Remember thatthe bitmap is shared between subclasses.

Take Home Message

Transforming an eager value into a lazy one may not preservecompatibility with pre-existing binaries.

Mirco Dotta Managing Binary Compatibility in Scala

Page 18: Managing Binary Compatibility in Scala (Scala Lift Off 2011)

Introduction Sources of Incompatibility Conclusion Conclusion Future Work Scala Migration Manager

Conclusion

I Ensuring release-to-release binary compatibility of Scalalibraries is possible.

I Though, sometimes it can be difficult to tell if a change in theAPI of a class/trait will break pre-existing binaries.

I In the discussed examples we have seen that:I Type inferencer may be at the root of changes in the member’s

signature.

I Type parameters may also modify members’ signature.

I Traits are a sensible source of binary incompatibilities.

I Transforming an eager field into a lazy one can break semantic.

It really looks like library’s maintainers’ life ain’t that easy...

Mirco Dotta Managing Binary Compatibility in Scala

Page 19: Managing Binary Compatibility in Scala (Scala Lift Off 2011)

Introduction Sources of Incompatibility Conclusion Conclusion Future Work Scala Migration Manager

Scala Migration Manager (MiMa)

A few months ago we released the Scala Migration Manager!

I It’s free!!

I It can tell you, library maintainers, if your next release isbinary compatible with the current one.

I It can tell you, libraries users, if two releases of a library arebinary compatible.

MiMa can collect and report all sources of “syntactic” binaryincompatibilities between two releases of a same library.

I “Syntactic” means NO LinkageError (e.g.,NoSuchMethodException) will ever be thrown at runtime.

Now, it’s time for a short demo!

Mirco Dotta Managing Binary Compatibility in Scala

Page 20: Managing Binary Compatibility in Scala (Scala Lift Off 2011)

Introduction Sources of Incompatibility Conclusion Conclusion Future Work Scala Migration Manager

Future Work

Reporting binary incompatibilities is only half of the story. We areworking on a “companion” tool that will help you migrate binaryincompatibilities.

For the reporting there are many ideas spinning around. Yourfeedback will help us decide what brings you immediate value

One that I believe is useful:

I Maven/Sbt integration.

Mirco Dotta Managing Binary Compatibility in Scala

Page 21: Managing Binary Compatibility in Scala (Scala Lift Off 2011)

Introduction Sources of Incompatibility Conclusion Conclusion Future Work Scala Migration Manager

Scala Migration Manager

Visit http://typesafe.com/technology/migration-manager

I More information about the Migration Manager

I Download it and try it out, it’s free!

We want to hear back from you.

I Success stories

I Request new features

I Report bugs

Want to know more, make sure to get in touch!

email: [email protected], twitter: @mircodotta

Mirco Dotta Managing Binary Compatibility in Scala

Page 22: Managing Binary Compatibility in Scala (Scala Lift Off 2011)

Introduction Sources of Incompatibility Conclusion Conclusion Future Work Scala Migration Manager

One last thing I didn’t mention...

We will make the sources public!

Mirco Dotta Managing Binary Compatibility in Scala