systems programming & beyond using c++ and d hello,...

c© 2012– Andrei Alexandrescu. 1 / 51

Systems Programming & Beyond using C++ and D

Hello, WorlD!

Andrei Alexandrescu

[email protected]

Prepared for LASER Summer School 2012

[email protected]

What’s the Deal??


D is. . .


• Efficient when it can

• Convenient when it should

• Safe when it must

Basics


• Statically typed

Use deduction wherever possible

• Sininimp-Mulinint object-oriented style

• Gray-box modularity:

◦ Python-inspired modules for “classic” entities

◦ White-box parameterized types and functions

• Heterogeneous translation of parameterized code

Hello, World!


#!/usr/bin/rdmd

import std.stdio;

void main()

{

writeln("Hello, world!");

}

• “Meh”worthy

• However:

◦ Simple

◦ Correct

◦ Scriptable


bool binarySearch(R, T)(R input, T value)

if (isRandomAccessRange!R

&& is(ElementType!R == T)) {

while (!input.empty) {

auto i = input.length / 2;

auto mid = input[i];

if (mid > value) input = input[0 .. i];

else if (mid < value) {

input = input[i + 1 .. $];

}

else return true;

}

return false;

}

Approach to Safety


Memory Safety


• All data is read with the type it was written

• Eliminates unpleasant class of errors

• Makes bugs reproducible

• Preserves the integrity of the type system

Challenge


• System languages must:

◦ Forge pointers (linkers/loaders)

◦ Magically change type of memory (allocators/GCs)

◦ Recycle memory “early”

◦ Circumvent type system

Attack


• All of D can’t be safe

• Define a safe subset

• Safe subset usable for large portions or entire programs

• Tagged with @safe attribute

@safe double cube(double x) {

return x * x * x;

}

• Unsafe functions tagged with @system

@system void free(void*);

But wait


• Important case: safe functions with

unknown/unprovable implementation

◦ fopen, malloc, isAlpha

◦ Small buffer optimization

◦ Tagged union implementation

• Define attribute @trusted

Rules


• @safe functions cannot:

◦ call @system functions

◦ escape addresses of locals

◦ forge pointers

◦ cast pointers around

◦ do pointer arithmetic

• N.B. Not all rules completely implemented

◦ Some on the conservative side

◦ A few details in flux

Resulting setup


• Low notational overhead: entire modules or portions

thereof can be annotated

• Application divided in:

◦ Safe part, uses GC, easy to write and debug

◦ Trusted part, encapsulated unsafe manipulation,

proven by means of code review

◦ System part, non-encapsulated unsafe

manipulation, only when solidly justified

Approach to Purity


Thesis


• Writing entire programs in pure style difficult

• Writing fragments of programs in pure style easy, useful

• + Easier to verify useful properties, debug

• + Better code generation

• − Challenge: interfacing pure and impure code

Functional Factorial (yawn)


ulong factorial(uint n) {

return n <= 1 ? 1 : n * factorial(n - 1);

}

• It’s PSPACE!

• Somebody should do hard time for this

However, it’s pure


pure ulong factorial(uint n) {

return n <= 1 ? 1 : n * factorial(n - 1);

}

• Pure is good

Functional Factorial, Fixed


pure ulong factorial(uint n) {

ulong crutch(uint n, ulong result) {

return n <= 1

? result

: crutch(n - 1, n * result);

}

return crutch(n, 1);

}

• Threads state through as parameters

• You know what? I don’t care for it

Honest Factorial



ulong result = 1;

for (; n > 1; --n) {

result *= n;

}

return result;

}

• But no longer pure!

• Well allow me to retort

Pure is as pure does


• “Pure functions always return the same result for the

same arguments”

• No reading and writing of global variables

◦ Global constants okay!

• No calling of impure functions

• Who said anything about local, transient state inside the

function?

Transitive State


pure void reverse(T)(T[] a) {

foreach (i; 0 .. data.length / 2) {

swap(data[i], data[$ - i - 1]);

}

}

• Possibility: disallow

• More useful: relaxed rule

• Operate with transitive closure of state reachable

through parameter

• Not functional pure, but an interesting superset

• No need for another annotation, it’s all in the signature!

(BTW, my int is purer than your int)


pure BigInt factorial(uint n) {

BigInt result = 1;

for (; n > 1; --n) {

result *= n;

}

return result;

}

• Works, but not in released version

• Not all stdlib “purified” yet

Aftermath


• If parameters reach mutable state:

◦ Relaxed pure—no globals, no I/O, no impure calls

• If parameters can’t reach mutable state:

◦ “Haskell-grade” observed purity

◦ Yet imperative implementation possible

◦ As long as it’s local only

Generic Programming


Generic Programming


• “Write once, instantiate anywhere”

◦ Specialized implementations welcome

• Prefers static type information and static expansion

(macro style)

• Fosters strong mathematical underpinnings

◦ Minimizing assumptions common theme in math

• Tenuous relationship with binary interfaces

• Starts with algorithms, not interfaces or objects

• “Premature encapsulation is the root of some

derangement”

Warmup: the “dream min”


• Should work at efficiency comparable to hand-written

code

• Avoid nonsensical calls min("hello", 42) etc.

• Work for all ordered types and conversions

• Decline incompatible types, without prejudice

First prototype


auto min(L, R)(L lhs, R rhs) {

return rhs < lhs ? rhs : lhs;

}

auto a = min(x, y);

• This function is as efficient as any specialized,

handwritten version (true genericity)

• Deduction for argument types and result type

Reject nonsensical calls


• min("hello", 42) client code error

auto min(L, R)(L lhs, R rhs)

if (typeof(rhs < lhs) == bool) {


}

• Rejects calls with 0 or 1 arguments

• Allows other overloads to take over, e.g. min element

over a collection

Naming names


template ordered(L, R) {

enum ordered =

is(typeof(L.init < R.init) == bool);

}

auto min(L, R)(L lhs, R rhs)

if (ordered!(L, R)) {


}

• The “eponymous template trick”

How about min over many elements?


auto min(R)(R range)

if (isInputRange!R &&

is(typeof(range.front < range.front) == bool))

{

auto result = range.front;

range.popFront();

foreach (e; range) {

if (e < result) result = e;

}

return result;

}

auto m = [ 1, 5, 2, 0, 7, 9 ].min();

• Works over anything that can be iterated

How about argmin now?


auto argmin(alias fun, R)(R r)

if (isInputRange!R &&

is(typeof(fun(r.front) < fun(r.front)) == bool))

{

auto result = r.front;

auto cache = fun(result);

r.popFront();

foreach (e; r) {

auto cand = fun(e);

if (cand > cache) continue;

result = e;

cache = cand;

}

return result;

}

argmin


auto s = ["abc", "a", "xz"];

auto m = s.argmin!((x) => x.length);

• Works on anything iterable and any predicate

• Predicate is passed by alias

• No loss of efficiency

The Generative Connection


Generative programming


• In brief: code that generates code

• Generic programming often requires algorithm

specialization

• Specification often present in a DSL

Embedded DSLs


Force into host language’s syntax?

Embedded DSLs


• Formatted printing?

Embedded DSLs



• Regular expressions?

Embedded DSLs




• EBNF?

Embedded DSLs




• EBNF?

• PEG?

Embedded DSLs




• EBNF?

• PEG?

• SQL?

Embedded DSLs




• EBNF?

• PEG?

• SQL?

• . . . Pasta for everyone!

Embedded DSLs


Here: use with native grammar

Process during compilation

Generate D code accordingly

Compile-Time Evaluation


• A large subset of D available for compile-time evaluation


ulong result = 1;

for (; n > 1; --n) result *= n;

return result;

}

...

auto f1 = factorial(10); // run-time

static f2 = factorial(10); // compile-time

Code injection with mixin


mixin("writeln(\"hello, world\");");

mixin(generateSomeCode());

• Not as glamorous as AST manipulation but darn

effective

• Easy to understand and debug

• Now we have compile-time evaluation AND mixin. . .


Wait a minute!

Example: bitfields in library


struct A {

int a;

mixin(bitfields!(

uint, "x", 2,

int, "y", 3,

uint, "z", 2,

bool, "flag", 1));

}

A obj;

obj.x = 2;

obj.z = obj.x;

Parser


import pegged.grammar; // by Philippe Sigaud

mixin(grammar("

Expr < Factor AddExpr*

AddExpr < (’+’/’-’) Factor

Factor < Primary MulExpr*

MulExpr < (’*’/’/’) Primary

Primary < Parens / Number / Variable

/ ’-’ Primary

Parens < ’(’ Expr ’)’

Number <~ [0-9]+

Variable <- Identifier

"));

Usage


// Parsing at compile-time:

static parseTree1 = Expr.parse(

"1 + 2 - (3*x-5)*6");

pragma(msg, parseTree1.capture);

// Parsing at run-time:

auto parseTree2 = Expr.parse(readln());

writeln(parseTree2.capture);

Scaling up


1000 lines of D grammar →

3000 lines D parser


Highly integrated lex+yacc


What about regexen?

Compile-time regular expressions


• GSoC project by Dmitry Olshansky: FReD

• Fully UTF capable, no special casing for ASCII

• Two modes sharing the same backend:

auto r1 = regex("^.*/([^/]+)/?$");

static r2 = ctRegex!("^.*/([^/]+)/?$");

• Run-time version uses intrinsics in a few places

• Static version generates specialized automaton, then

compiles it

Summary


Summary


• Aim: efficiency, convenience, safety

• Safe subset

• Pure functions

• Generic programming

• Generative programming

systems programming & beyond using c++ and d hello,...

Documents