programming languages #devcon2013
DESCRIPTION
TRANSCRIPT
![Page 1: Programming Languages #devcon2013](https://reader034.vdocument.in/reader034/viewer/2022051323/54b7363d4a7959404e8b45b9/html5/thumbnails/1.jpg)
Programming Languages
Telefónica Digital DevCon 2013
Iván Montes - @drslump
![Page 2: Programming Languages #devcon2013](https://reader034.vdocument.in/reader034/viewer/2022051323/54b7363d4a7959404e8b45b9/html5/thumbnails/2.jpg)
Disclaimer• Talk is focused on mainstream languages
• Many of the stuff we will talk about is already available
• I know your favourite language probably does something similar, or you know a hack to make it do it.
• Scripting languages do have a compiler, they just happen to include an evaluator too :-)
• C++ is purposely left out of any comparison (Guys you need to simplify the spec!)
• If you don’t understand some terminology, ask!
![Page 3: Programming Languages #devcon2013](https://reader034.vdocument.in/reader034/viewer/2022051323/54b7363d4a7959404e8b45b9/html5/thumbnails/3.jpg)
What is the best?• Good news! You’re probably using it already
• Check list:
• You can focus on solving your business problem
• You are proficient with it
• You can deploy your product without fear
• You have tooling available (editors, testing, packaging…)
• You are not the only soul who knows how it works :-)
![Page 4: Programming Languages #devcon2013](https://reader034.vdocument.in/reader034/viewer/2022051323/54b7363d4a7959404e8b45b9/html5/thumbnails/4.jpg)
But…
![Page 5: Programming Languages #devcon2013](https://reader034.vdocument.in/reader034/viewer/2022051323/54b7363d4a7959404e8b45b9/html5/thumbnails/5.jpg)
Can it be improved?• Given:
• Expected functionality is ever growing
• Solutions complexity is ever growing
• When:
• We add layers and layers to keep it under control
• Those layers are usually libraries and frameworks
• Then:
• Your really nice language is not that important anymore
• You end up programming for the framework (does it pass the check list?)
![Page 6: Programming Languages #devcon2013](https://reader034.vdocument.in/reader034/viewer/2022051323/54b7363d4a7959404e8b45b9/html5/thumbnails/6.jpg)
The problem• We can separate a language in:
• Specification, Compiler, Runtime and Libraries
• Of those we are only in control of the Libraries
• So we abuse them to solve our problems
• But we can end up creating others:
• Support
• Versioning
• Troubleshooting
![Page 7: Programming Languages #devcon2013](https://reader034.vdocument.in/reader034/viewer/2022051323/54b7363d4a7959404e8b45b9/html5/thumbnails/7.jpg)
What’s wrong with Libraries?• They are meant for code reuse
• We (ab)use them as language customisation and extensibility tools
• The cycle of doom then begins:
• Our cool language toolchain breaks
• We expend time modifying editors, build tools, …
• The library gets updated or we want to support a new tool
• Cycle ends with lots of abandonware even if you don’t notice it:
• Check SourceForge, GitHub and similar sites
• The guy now maintaining your 2009 project has megabytes of legacy source code, with no expertise to hack it and nobody to offer support.
![Page 8: Programming Languages #devcon2013](https://reader034.vdocument.in/reader034/viewer/2022051323/54b7363d4a7959404e8b45b9/html5/thumbnails/8.jpg)
Language design• Disclaimer: It’s a hard discipline, requiring lots of talent
and resources.
• Basically the current situation in the mainstream market sucks.
• It’s slowly changing though (guys at Microsoft Research and Universities)
• Designers are realising (again) that we are not totally stupid!
• What we need is a very old tech, we just need it refined.
![Page 9: Programming Languages #devcon2013](https://reader034.vdocument.in/reader034/viewer/2022051323/54b7363d4a7959404e8b45b9/html5/thumbnails/9.jpg)
Taking control• Premise: Any non-tech-savvy person assumes
that a developer is someone intelligent.
• Think highly of yourself!
• We want/need more control over our primary tool.
• Make yourself this question:
What do you think has evolved more in the last 25 years: compilers or cad software?
![Page 10: Programming Languages #devcon2013](https://reader034.vdocument.in/reader034/viewer/2022051323/54b7363d4a7959404e8b45b9/html5/thumbnails/10.jpg)
but I already can!• No, you don’t.
• Some compilers support writing add ons for them.
• The problem is neither being able to write an extension in C to be called from the host language.
• The Language itself must define how this extension mechanisms work. It must be part of its specification.
• This way we get extensibility without sacrificing portability and risking vendor lock-in.
• It helps with versioning, you extend a version of the language not a version of a given compiler.
![Page 11: Programming Languages #devcon2013](https://reader034.vdocument.in/reader034/viewer/2022051323/54b7363d4a7959404e8b45b9/html5/thumbnails/11.jpg)
Extensibility• We should be able to extend pretty much anything in our
language.
• Extending the syntax and semantics (specification) is doable but not very mature yet.
• Extending the runtime is expensive. You wan’t something tested and battle tested and then tested again.
• However extending the compiler is ready for prime time!
• Preprocessors and lispy macros are cool, don’t resent me.
• But we need something more refined, almost fool-proof.
![Page 12: Programming Languages #devcon2013](https://reader034.vdocument.in/reader034/viewer/2022051323/54b7363d4a7959404e8b45b9/html5/thumbnails/12.jpg)
A Compiler• A compiler basically does three tasks:
• Parse (Ast construction)
• Resolve (name resolution, type binding)
• Emit (machine code, bytecode).
• Keep in mind that a bit is a measure for information!
Memory usage (Kb)
CodeParse
ResolveEmit
0 300 600 900 1200
![Page 13: Programming Languages #devcon2013](https://reader034.vdocument.in/reader034/viewer/2022051323/54b7363d4a7959404e8b45b9/html5/thumbnails/13.jpg)
Info!• Compilers produce a lot of information because they need
to understand our code.
• Unfortunately this pretty cool information is trashed once the compilation is done.
• Some languages support reflection, at least not everything is lost (re Java fan boys, type erasure is not cool).
• Given that a lot of very clever people put a lot of effort over several years creating a compiler.
• It seems like a huge waste of resources to not use it more.
CodeParse
ResolveEmit
![Page 14: Programming Languages #devcon2013](https://reader034.vdocument.in/reader034/viewer/2022051323/54b7363d4a7959404e8b45b9/html5/thumbnails/14.jpg)
Compiler Service• Almost a reality (llvm, mono mcs, roslyn, …).
• Note: You know that an IDE basically implements an incremental compiler nowadays, right? Wonder why they feel slow now?
• Use cases:
• Better editor support
• Optimised/Incremental builds
• Documentation generation
• Quality/Security audits
![Page 15: Programming Languages #devcon2013](https://reader034.vdocument.in/reader034/viewer/2022051323/54b7363d4a7959404e8b45b9/html5/thumbnails/15.jpg)
Compiler Service (II)• But we want more!
• If only we could hook our logic into the compiler like we do with build systems…
• The following would be dirty cheap to have:
• Injection of cross-concerns (logging, aspects, …)
• Statically checked/expanded DSLs
• Quality, Security and Business specific enforcements
![Page 16: Programming Languages #devcon2013](https://reader034.vdocument.in/reader034/viewer/2022051323/54b7363d4a7959404e8b45b9/html5/thumbnails/16.jpg)
Compiler Service (III)• All of the sudden, half our
libraries functionality is now managed by the compiler!
• Wait! won’t we suffer the “cycle of doom” too?
• No! :-)
• The logic is encapsulated under the compiler public interface.
• All the tooling using the compiler will automatically support our use cases!
• That means for example:
• Editors will autocomplete and report errors for expanded DSLs
• Build systems will notify errors generated by our custom enforcements
• Besides, we reduced some heavy weight from the runtime, pretty cool for restrained mobile apps for example.
![Page 17: Programming Languages #devcon2013](https://reader034.vdocument.in/reader034/viewer/2022051323/54b7363d4a7959404e8b45b9/html5/thumbnails/17.jpg)
Macros• Extending the compiler is nice but could be made
more user friendly.
• When you have a pattern coming again and again in your code, it will probably be better solved with a macro.
• Macros are a pretty old concept but they mostly refer to “text macros”. What we need are AST Macros.
• Instead of replacing chunks of text we operate on the syntax tree of the compiler. Much safer and powerful!
![Page 18: Programming Languages #devcon2013](https://reader034.vdocument.in/reader034/viewer/2022051323/54b7363d4a7959404e8b45b9/html5/thumbnails/18.jpg)
Examplemacro using(expr as Expression): yield [| try: $(using.Body) ensure: $(expr).close() |]
fp = open(‘path/to/file.ext’) using fp: fp.write(‘foobar') # Note that the file resource is closed automatically
![Page 19: Programming Languages #devcon2013](https://reader034.vdocument.in/reader034/viewer/2022051323/54b7363d4a7959404e8b45b9/html5/thumbnails/19.jpg)
Quasi-Quotations• Normal language syntax constructs that are
somehow quoted (`[| .. |]` in the example).
• Their purpose is to make creating complex AST structures a breeze.
• The compiler generates a separate AST for them. It’s like a template.
• When we inject them somewhere the compiler will apply that template in that AST node.
![Page 20: Programming Languages #devcon2013](https://reader034.vdocument.in/reader034/viewer/2022051323/54b7363d4a7959404e8b45b9/html5/thumbnails/20.jpg)
Splicing• Injection points in a quasi-quotation (`$(..)` in the
example).
• Their purpose is to parametrize quasi-quotations to dynamically build arbitrary AST structures.
• Resolves to a syntax node, the compiler will insert that node in the template tree.
• They keep their lexical information once inserted.
![Page 21: Programming Languages #devcon2013](https://reader034.vdocument.in/reader034/viewer/2022051323/54b7363d4a7959404e8b45b9/html5/thumbnails/21.jpg)
Why are they better?• Syntax is integrated and validated as normal code.
• Most of the lexical information is kept, thus reported errors after expansion contain valid information.
• Debuggable!
• Toolchain friendly. They are transparent to consuming tools.
• DSL friendly. Capture common patterns to solve your problems and offer a statically checked interface to fit your needs!
![Page 22: Programming Languages #devcon2013](https://reader034.vdocument.in/reader034/viewer/2022051323/54b7363d4a7959404e8b45b9/html5/thumbnails/22.jpg)
Can’t libraries do this?• Yes, but they are not the right tool for the job.
• Lots of stuff can be done at compile time, no need to bloat the runtime with stuff that just makes developing easier.
• Unified error reporting. Find out incorrect usages at compile time instead of uncontrolled runtime exceptions.
• The `assert` example shows pretty well the difference:
#!/bin/env python !foo = 10 assert foo != 10 # AssertionError:
#!/bin/env booi !foo = 10 assert foo != 10 # AssertionError: foo != 10
![Page 23: Programming Languages #devcon2013](https://reader034.vdocument.in/reader034/viewer/2022051323/54b7363d4a7959404e8b45b9/html5/thumbnails/23.jpg)
SyntaxA very quick overview
![Page 24: Programming Languages #devcon2013](https://reader034.vdocument.in/reader034/viewer/2022051323/54b7363d4a7959404e8b45b9/html5/thumbnails/24.jpg)
Parsers• Disclaimer: Not saying it’s a good
idea for a general language
• The technology allows it and current systems have enough memory to pull it.
• Traditionally parsing involved a lexer generating tokens and a parser (either a hand-coded recursive descent or a generated table based).
• PEG style parsers (and other pattern matching techniques) operate the same on a stream of chars (source code in a file), a tree of XML nodes or any other object structure.
• They can be made to have constant parsing time using memoization.
• OMeta is an example of PEG parser with a built in extension model.
• Parsing rules (productions) can be inherited and modified at will using a pretty simple syntax.
![Page 25: Programming Languages #devcon2013](https://reader034.vdocument.in/reader034/viewer/2022051323/54b7363d4a7959404e8b45b9/html5/thumbnails/25.jpg)
Making the language ours• Use cases:
• Any sort of DSL would benefit from having its specific syntax
• Units: delay = 10h + 30m, weight = 20Kg
• Reduce verbosity: for i in 1..10
• Short comings:
• Editors won’t know how to colorise the new syntax
• It’s tempting to abuse it
• Makes your code less portable
• Tightly coupled to the standard grammar
![Page 26: Programming Languages #devcon2013](https://reader034.vdocument.in/reader034/viewer/2022051323/54b7363d4a7959404e8b45b9/html5/thumbnails/26.jpg)
User eXperienceHave you checked clang recently?
![Page 27: Programming Languages #devcon2013](https://reader034.vdocument.in/reader034/viewer/2022051323/54b7363d4a7959404e8b45b9/html5/thumbnails/27.jpg)
Language design• Syntax of a language is the first and
most important point of interaction.
• Language design is still mainly driven by the target audience expectations (oh those curly braces everywhere).
• The syntax must be very carefully drafted to avoid ambiguities. Even if designers still care very much about the aesthetics.
• When asked if they performed user tests when designing C#, Hejlsberg answered "only for the integration in Visual Studio”. More generally, check how it integrates with the tools you’re going to use.
• Very terse syntaxes tend to be good for DSLs and small and targeted projects. Code is read and reasoned about much more often than written!
• Semantics are as important as the syntax, don’t overlook its implications.
• Stay away from languages that release new keywords and constructs every other minor version. A syntax either works or doesn’t but can target every use case.
![Page 28: Programming Languages #devcon2013](https://reader034.vdocument.in/reader034/viewer/2022051323/54b7363d4a7959404e8b45b9/html5/thumbnails/28.jpg)
Reporting• The compiler is our main tool
but still, it’s generally speaking, a bitch reporting errors.
• Error reporting is the main interface, besides the language syntax, between the compiler and the developer.
• IDEs partially solve this by implementing their own heuristics on the code.
• But it should be the compiler! I don’t want my IDE bloated with these things.
• clang has done a great job with this. Other compilers should follow their lead.
• Compilers ought to be extensible in this aspect too, if we can extend them with our own logic we must have a clear way to give feedback to the user.
![Page 29: Programming Languages #devcon2013](https://reader034.vdocument.in/reader034/viewer/2022051323/54b7363d4a7959404e8b45b9/html5/thumbnails/29.jpg)
Parsing errors• Language newcomers tend to have syntax errors until they get fluent.
• The common approach is to extend the grammar to include invalid productions with custom errors.
• Grammars tend to get out of hand and difficult to refactor.
• A pretty cool way to improve the errors is to use example snippets of incorrect syntax.
• The snippets are fed to the parser and the state of it when it errors is recorded.
• Recorded states are then used to build the final parser, so when hits an error that matches a custom message can be displayed.
class Foo: def foo() # Note: I forgot the colon pass !
Before: <stdin>(3,13): BCE0043: Unexpected token: <INDENT>. <stdin>(3,13): BCE0044: expecting "DEDENT", found '<EOL>'. <stdin>(3,13): BCE0044: expecting "EOF", found '<DEDENT>'.
After: <stdin>(3,13): BCE0043: Methods must use a ‘:’ for their body.
![Page 30: Programming Languages #devcon2013](https://reader034.vdocument.in/reader034/viewer/2022051323/54b7363d4a7959404e8b45b9/html5/thumbnails/30.jpg)
Type SafetyWishing it could be extended
![Page 31: Programming Languages #devcon2013](https://reader034.vdocument.in/reader034/viewer/2022051323/54b7363d4a7959404e8b45b9/html5/thumbnails/31.jpg)
Type Safety• Major families:
• Nominal (Java, C#, …)
• Structural (Go, TypeScript, …)
• Duck typing (Python, JavaScript, …)
• Some languages offer a mix of them (ie. Scala, ObjC, Boo, …)
• Many allow for runtime duck typing.
• Pattern matching solves a lot of type safety derived issues (bye bye Visitor)
• The problem with extending them is that the semantics of the language change.
• People can easily reason and adapt to syntax changes but for semantic ones…
• In the current state of the art is probably better switching languages completely if the problem at hand requires a different model of type safety.
![Page 32: Programming Languages #devcon2013](https://reader034.vdocument.in/reader034/viewer/2022051323/54b7363d4a7959404e8b45b9/html5/thumbnails/32.jpg)
API Versioning• If your language uses nominal typing you actually have to embrace
it. Carefully plan your Interfaces to get the job done.
• C# does it right. If you want to be confident about exposing an API you can’t make every method virtual (*cough* Java *cough*).
• public, protected and private are visibility modifiers, not extension point indicators.
• Structural typing really shines in this regard. But it must allow it from the call site not only the definition site (Go vs Scala). The library author doesn’t have a clue if his API works as designed or not :-)
• Dynamic languages are flawed on this. Rapid prototyping but zero confidence when upgrading.
![Page 33: Programming Languages #devcon2013](https://reader034.vdocument.in/reader034/viewer/2022051323/54b7363d4a7959404e8b45b9/html5/thumbnails/33.jpg)
Cool stuffOld tech sold as a bright future
![Page 34: Programming Languages #devcon2013](https://reader034.vdocument.in/reader034/viewer/2022051323/54b7363d4a7959404e8b45b9/html5/thumbnails/34.jpg)
Live Coding• Imagine you could tap into the
runtime environment and modify the executing instructions on realtime.
• Many projects deploy nowadays into a Virtual Machine, which technically simplify it.
• Still, the VM expects byte code, we want to write in a higher level syntax.
• Extensible compilers would allow bringing Live Coding to languages that are not interpreted.
• We ask the compiler to regenerate the changes in our function, the result is a chunk of byte code to fed into the VMs.
• No need to run an interpreter and very few limitations.
• Still, this is the kind of thing that would flourish with an extensible compiler design. No big vendor is going to release something like this.
![Page 35: Programming Languages #devcon2013](https://reader034.vdocument.in/reader034/viewer/2022051323/54b7363d4a7959404e8b45b9/html5/thumbnails/35.jpg)
Type Providers• Popularised recently by F#
• They are basically specialised macros, whose purpose is to generate new types in the program.
• Remember when you had to preprocess some XML schema files to generate Java classes? What if the compiler could do that for you automatically?
• Using Json? Just feed an example message file to the type provider macro and it will generate a typed interface for it.
• This is available in many languages at runtime via Reflection. What’s cool is that now you have it at compile time. Type checking and auto completion as obvious benefits.
![Page 36: Programming Languages #devcon2013](https://reader034.vdocument.in/reader034/viewer/2022051323/54b7363d4a7959404e8b45b9/html5/thumbnails/36.jpg)
What now?
![Page 37: Programming Languages #devcon2013](https://reader034.vdocument.in/reader034/viewer/2022051323/54b7363d4a7959404e8b45b9/html5/thumbnails/37.jpg)
Is your responsibility• To get better tools for our job
we have to earn it. Advocate these concepts to gain traction in the profession.
• Language designers do listen to its users even if for the mainstream languages they are slow reacting to it.
• If you have to choose between an extensible language and one that’s not, weight the former appropriately in the comparison.
• Don’t be afraid of moulding a language to your needs. Know your tool and help it help you!
• If deploying into a VM (JVM, CLR, JavaScript) there is little risk in mixing languages in a project that boost our productivity.
• Forget the idea that only functional languages are extensible. Imperative language can do it too!
![Page 38: Programming Languages #devcon2013](https://reader034.vdocument.in/reader034/viewer/2022051323/54b7363d4a7959404e8b45b9/html5/thumbnails/38.jpg)
Conclusions
![Page 39: Programming Languages #devcon2013](https://reader034.vdocument.in/reader034/viewer/2022051323/54b7363d4a7959404e8b45b9/html5/thumbnails/39.jpg)
Conclusions• Languages should be
designed for extensibility.
• Anyone that can create a library should be also able to extend the compiler/language. Not everything must be done at runtime!
• We need better compilers in every aspect.
• Statically typed languages are closing the gap with dynamic ones in ease of use and user experience. We have to review our current stand points on this.
• If you have the chance to learn a new framework or a new language, chose the later, it’s much more rewarding.
• Extensible compilers will foster innovation, no need to wait for Oracle or Microsoft to come up with new ideas every 3 years. People will be able to shape the future trying new things and introducing those that work as part of the standard languages.
• Good drivers don’t need to know about the internals of the car, but it certainly doesn’t hurt :-)
![Page 40: Programming Languages #devcon2013](https://reader034.vdocument.in/reader034/viewer/2022051323/54b7363d4a7959404e8b45b9/html5/thumbnails/40.jpg)
–Ludwig Wittgenstein
“The limits of my language are the limits of my world.”