strategies for rule-based program transformation

Strategies for Rule-Based Program TransformationSoftware Engineering Techniques for Program Transformation

Eelco Visser

Center for Software TechnologyDepartment of Information & Computing Sciences

Faculty of ScienceUtrecht UniversityThe Netherlands

April 18, 2005Dagstuhl Seminar

Transformation Techniques in Software Engineering

Sources

A Survey of Strategies in Rule-Based Program TransformationSystemsE. Visser (JSC 2005)

Program Transformation with Stratego/XTE. Visser (DSPG’03)

Program Transformation with Scoped Dynamic Rewrite RulesM. Bravenboer, A. van Dam, K. Olmos, and E. Visser (FI)

Composing Source-to-Source Data-Flow Transformations withRewriting Strategies and Dependent Dynamic Rewrite RulesK. Olmos and E. Visser. (CC’05)

Sources

Outline

1 Program Transformation

2 Transformation Rules

3 Transformation Strategies

4 Dynamic Rules

Part I

Program Transformation

Computing with Programs

Program transformation is the domain of computing whereprograms are the data

Improving Programs

Program transformation is the mechanical manipulation of aprogram in order to improve it relative to some cost function Csuch that

C (tr(P)) < C (P)

[Pettorossi 1996]

Dimensions of improvement

I Performance

I Memory usage

I Understandability

I Flexibility

I Maintainability

I Portability

I Correctness

I Satisfaction of requirements

Improving Programs

Program transformation is the mechanical manipulation of aprogram in order to improve it relative to some cost function Csuch that

C (tr(P)) < C (P)

[Pettorossi 1996]

Dimensions of improvement

I Performance

I Memory usage

I Understandability

I Flexibility

I Maintainability

I Portability

I Correctness

I Satisfaction of requirements

Application: Software Development

Software development is concerned with transformation fromrequirements to specification to implementation

Supporting abstraction

Approaches have varying degrees of automation

Transformational programming [Partsch 1986, Feather 1987]

Formal development of implementations from specifications

I Properties

I Mechanizable and traceableI ‘Correct by construction’

I Fold/Unfold [Burstall & Darlington 1977]

I Calculating efficient programs from general onesI Application-specific

I Program refinement [Green, Paige, Smith, ...]

I Kids: Specialization of solution theories to domain theories

I Pragmas [Peyton Jones et al. 2001]

I Programmer directives to compiler

I Draco, Popart, CIP, ...

Compilation [ASU 1986, ...]

Fully automatic transformation from high-level language tolow-level language

I Instruction selection [Fraser et al. 1992]

I cost-driven translation from IR to machine instructions

I Compilation by transformation [Peyton Jones & Santos 1998]

I compilation as sequence of small transformation steps

I Program optimization [Appel 1998, Muchnik 1997, ...]

I improve run-time, memory usage, power consumptionI inlining, constant propagation, dead code elimination, ...

I Application generation [Smaragdakis & Batory 2000]

I Compilation for domain-specific languages

Application: Software Evolution

Software evolution is concerned with the understanding andmaintenance of existing legacy programs

Recovering abstractions

Reverse Engineering [Chikofski & Cross 1990, V.d. Brand et al. 1997]

Extract from a low-level program a high-level program orspecification

I Decompilation

I Architecture extraction

I Documentation generation

I Software Visualization

I Decompilation

Software Renovation

Change the extensional behaviour of a program

I Error repair (Y2K)

I Changing requirements (Euro)

I Refactoring [Fowler 1999]

I improving the design of existing programs

I Obfuscation [Collberg et al. 1998]

I make a program harder to understand

Migration

Transformation to another language at the same level ofabstraction

I Dialect upgrading

I Porting (Pascal to C, Java to C#)

Realizing Program Transformations

Transformations are useful in software engineering if they are

I Correct

I Reproducable

I Automatic

I Scalable

How to compose transformation systems that have theseproperties?

What are the right abstractions for building transformationsystems?

A component-based approach to program transformation in whichbasic transformation components can be reused in many differentcompositions.

I Correct

I Reproducable

I Automatic

I Scalable

I Correct

I Reproducable

I Automatic

I Scalable

I Correct

I Reproducable

I Automatic

I Scalable

Program Transformation Mechanics

Basic Components

Transformation rule: one step transformation of part of a program

Perspective

A high-level, language parametric, rule-based programtransformation system, which supports a wide range oftransformations, admitting efficient implementations that scale tolarge programs

Practice

Many different systems with different trade-offs

I expressivity of rule specification

I strategies for control

I efficiency of implementation

I object languages supported

I range of transformations

Basic Components

Perspective

Practice

Basic Components

Perspective

Practice

Architecture of Transformation Systems

program

transform

parse pretty-print

transformtree tree

Part II

Transformation Rules

4 Dynamic Rules

Example: Desugaring

for i := 1 to n dofor j := 1 to n doc[i,j] := sum k = 1 to n (a[i,k] * b[k,j])

for i := 1 to n dofor j := 1 to n do

c[i,j] := let var d := 0in for k := 1 to n do

d := d + a[i,k] * b[k,j];d

let var din for i := 1 to n do

for j := 1 to n do(d := 0;for k := 1 to n do d := d + a[i,k] * b[k,j];c[i,j] := d)

Example: Desugaring

for i := 1 to n dofor j := 1 to n doc[i,j] := let var d := 0

in for k := 1 to n dod := d + a[i,k] * b[k,j];

Example: Desugaring

for i := 1 to n dofor j := 1 to n doc[i,j] := let var d := 0

in for k := 1 to n dod := d + a[i,k] * b[k,j];

Programs as Terms

Abstract Syntax

Programs can be represented as terms

sum k = 1 to n (a[i,k] * b[k,j])

Sum([Index("k",Int("1"),Var("n"))],Times(Subscript(Var("a"),[Var("i"),Var("k")]),

Subscript(Var("b"),[Var("k"),Var("j")])))

Patterns

Term patterns can be used to analyze and compose programs

Sum([idx | idx*], e)

matches

Sum([Index("k",Int("1"),Var("n"))],Times(Subscript(Var("a"),[Var("i"),Var("k")]),

Subscript(Var("b"),[Var("k"),Var("j")])))

Rewrite Rules

SumSplit :Sum([idx | idx*], e) -> Sum([idx], Sum(idx*, e))where <not([])> idx*

DefSum :Sum([Index(x, e1, e2)], e3) ->Let([VarDec(y, NoTp(), Int("0"))],

[For(Var(x), e1, e2,Assign(Var(y), Plus(Var(y), e3))),

Var(y)])where new => y

LetFromAssign :Assign(lv, Let(d*, e*)) ->Let(d*, [Assign(lv, Seq([e*]))])

Term Rewriting

Term rewriting = normalization with respect to a set of rules

normalization = exhaustive application

normal form : no sub-term can be rewritten

Patterns in Concrete Syntax

I Problem: terms are hard to read, especially larger terms

I Observation: one-to-one correspondence between abstractsyntax and concrete syntax

I Solution: use concrete syntax for patterns

SumSplit :|[ sum idx; idx* (e) ]| -> |[ sum idx(sum idx*(e)) ]|where <not([])> idx*

DefSum :|[ sum x = e1 to e2 ( e3 ) ]| ->|[ let var y := 0

in for x := e1 to e2 do y := y + e3; y

end ]|where new => y

Systems that support Term Rewriting

I ASF+SDF

I Elan

I Maude

Extensions of Term Rewriting

I Matching modulo equations

I Associative, commutative, identity, ...I List matchingI Combinations: A, AC, ACI, AI

I Object variable bindings

I De Bruijn indicesI Higher-order abstract syntaxI FreshML

Part III

Transformation Strategies

4 Dynamic Rules

Example: Compilation by Transformation

sum k = 1 to n(a[i,k] * b[k,j])

a_0 := 0;k := 1;d_0 := n;t_3 := k <= d_0;label f_0;i_0 := not(t_3);if i_0 goto g_0;t_1 := a[i,k];t_2 := b[k,j];t_0 := t_1 * t_2;a_0 := a_0 + t_0;k := k + 1;t_3 := k <= d_0;goto f_0;label g_0

a_0 := 0;for k := 1 to n do(t_1 := a[i,k];t_2 := b[k,j];t_0 := t_1 * t_2;a_0 := a_0 + t_0)

a_0 := 0;k := 1;d_0 := n;t_3 := k <= d_0;while t_3 do(t_1 := a[i,k];t_2 := b[k,j];t_0 := t_1 * t_2;a_0 := a_0 + t_0;k := k + 1;t_3 := k <= d_0)

Controlling Application of Rules

Limitations of Term Rewriting

I Non-Termination

I Exhaustive application may not terminate

I Non-Confluence

I There may be diverging paths in rewrite relation

I Non-Selection

I All rules are applied everywhere

Program transformation requires control over application of rules

Common Solution: Functional Rewriting

Use extra constructors (called ‘functions’) to define control-point

I Traversal overhead: one rule for each constructor

I Tangling of rules and strategy: rules not reusable

Compile(e) ->CollectDecls(SimpleExpressions(ForToWhile(Desugar(e))))

Desugar(|[ sum idx; idx+ (e) ]|) ->Desugar(|[ sum idx(sum idx+ (e)) ]|)

Desugar(|[ sum x = e1 to e2 ( e3 ) ]|) ->Desugar(|[ ... ]|

Desugar(|[ let d1* in e1* ]|) -> |[ let d2* in e2* ]|where Desugar(d1*) => d2*; Desugar(e1*) => e2*

Strategic Control

I Application of rules controlled by strategies

I Select rules and strategy

I Separation of rules and strategies

I reuse of rules in multiple transformationsI instantiate strategies with different rules

TAMPR [Boyle et al. 1997]

I Transformation Assisted Multiple Program Realization

I Normalization with selection of rules

I Sequence of normal forms: divide transformation in stages

Similar to layered graph grammars?

Staged Transformation

compile =for-with-while; return-value; mark-procedure-calls; simple-expressions; control-flow-to-goto; collect-declarations; use-return-register; vars-on-stack; add-stack-machine; flatten-sequences; unmark-procedure-calls

Compilation by sequence oftransformations

for-with-while =topdown(try(ForToWhile))

simple-expressions =innermost(Simplify <+ LiftNonAtomicArgument <+ Desugar

<+ LetSplit)

Strategic Rewriting [Luttik & Visser 1997]

I Strategies are composed using strategy combinators

I Generic traversal reduces traversal overhead

Strategic Programming [Laemmel, Visser & Visser 2003]

Language independent paradigm for programming with strategies,in particular, generic traversals.

Instantiated for rewriting, logic programming, functionalprograming, object-oriented programming

Strategic Rewriting [Luttik & Visser 1997]

I Strategies are composed using strategy combinators

I Generic traversal reduces traversal overhead

Strategic Programming [Laemmel, Visser & Visser 2003]

Language independent paradigm for programming with strategies,in particular, generic traversals.

Instantiated for rewriting, logic programming, functionalprograming, object-oriented programming

Sequential Composition

I Syntax: s1; s2I Apply s1, then s2I Fails if either s1 or s2 fails

I Variable bindings are propagated

Plus(Var("a"),Int("3"))stratego> ?Plus(e1, e2); !Plus(e2, e1)Plus(Int("3"),Var("a"))

Deterministic Choice

Deterministic Choice (Left Choice)

I Syntax: s1 <+s2I First apply s1, if that fails apply s2I Note: local backtracking

PlusAssoc :Plus(Plus(e1, e2), e3) -> Plus(e1, Plus(e2, e3))

EvalPlus :Plus(Int(i),Int(j)) -> Int(k) where <addS>(i,j) => k

Plus(Int("14"),Int("3"))stratego> PlusAssoccommand failedstratego> PlusAssoc <+ EvalPlusInt("17")

Identity and Failure

Identity

I Syntax: id

I Always succeedI Some laws

I id ; s ≡ sI s ; id ≡ sI id <+ s ≡ idI s <+ id 6≡ s

Failure

I Syntax: fail

I Always failI Some laws

I fail <+ s ≡ sI s <+ fail ≡ sI fail ; s ≡ failI s ; fail 6≡ fail

Defined Combinators

try(s) = s <+ id

repeat(s) = try(s; repeat(s))

while(c, s) = if c then s; while(c,s) end

do-while(s, c) = s; if c then do-while(s, c) end

Traversal

Traversal Styles

Full traversal

I Folds: functional programming

I Visitors: object-oriented programming

Strategic programming

I One-level traversal combinator

I descend to direct subterms

I Combine into multiple full traversal strategies

I Flavours

I Congruence operator: specific for signatureI Generic: independent of signature

Traversal Strategies

Visiting All Subterms

I Syntax: all(s)

I Apply strategy s to all direct sub-terms

Plus(Int("14"),Int("3"))stratego> all(!Var("a"))Plus(Var("a"),Var("a"))

bottomup(s) = all(bottomup(s)); stopdown(s) = s; all(topdown(s))downup(s) = s; all(downup(s)); salltd(s) = s <+ all(alltd(s))innermost(s) = bottomup(try(s; innermost(s)))

for-with-while = topdown(try(ForToWhile))simple-expressions =

innermost(Simplify <+ LiftNonAtomicArgument <+ Desugar<+ LetSplit)

I Syntax: all(s)

for-with-while = topdown(try(ForToWhile))simple-expressions =

innermost(Simplify <+ LiftNonAtomicArgument <+ Desugar<+ LetSplit)

I Syntax: all(s)

for-with-while = topdown(try(ForToWhile))simple-expressions =innermost(Simplify <+ LiftNonAtomicArgument <+ Desugar

<+ LetSplit)

Congruence Operator: Data-type Specific Traversal

I Syntax: c(s1,...,sn)for each n-ary constructor c

I Apply strategies to direct sub-terms of a c term

Plus(Int("14"),Int("3"))stratego> Plus(!Var("a"), id)Plus(Var("a"),Int("3"))

mark-procedure-calls =rec mark(alltd(Assign(id, id)<+ While(id, mark)<+ For(id, id, id, mark)<+ If(id, mark, mark)<+ VarDec(id, id, id)<+ !Proc(<Call(id,id)>)))

History of Strategies

A Slice of History

I LCF etc: tactics in theorem provers

I ASF+SDF language processing environment: rewriting(later added traversal functions)

I Rewriting logic

I ELAN specification system: rewriting & strategy expressions

I Stratego: generic traversal & dynamic rules

I Strafunski: generic traversal in Haskell

I JJTraveler: visitor combinators in Java

Other Slices

I TXL: generic traversal function

Part IV

Dynamic Rules

4 Dynamic Rules

Context-Sensitive Transformations

Problem: Rewrite Rules are Context-free

Rewrite rules can only access information in term that is matched

Many Transformations are Context-Sensitive

I Constant propagation

I Copy propagation

I Common-subexpression elimination

I Partial evaluation

I Function inlining

I Dead code elimination

Solution: Dynamic Rewrite Rules

Define rewrite rules during transformation

I Copy propagation

I Function inlining

I Copy propagation

I Function inlining

Constant Folding

Constant folding

y := x * (3 + 4) ⇒ y := x * 7

Constant folding rules

EvalAdd : |[ i + j ]| -> |[ k ]| where <add>(i, j) => k

EvalMul : |[ i * j ]| -> |[ k ]| where <mul>(i, j) => k

AddZero : |[ 0 + e ]| -> |[ e ]|

Constant folding strategy (bottom-up)

EvalBinOp = EvalAdd <+ AddZero <+ EvalMul <+ EvalOther

try(s) = s <+ id

constfold = all(constfold); try(EvalBinOp)

Constant Folding

Constant folding

y := x * (3 + 4) ⇒ y := x * 7

Constant folding rules

EvalAdd : |[ i + j ]| -> |[ k ]| where <add>(i, j) => k

EvalMul : |[ i * j ]| -> |[ k ]| where <mul>(i, j) => k

AddZero : |[ 0 + e ]| -> |[ e ]|

Constant folding strategy (bottom-up)

EvalBinOp = EvalAdd <+ AddZero <+ EvalMul <+ EvalOther

try(s) = s <+ id

constfold = all(constfold); try(EvalBinOp)

Defining and Undefining Rules Dynamically

Constant Propagation and Folding in Straight-Line Code

b := 1;c := b + 3;b := foo();a := b + c

b -> 1b -> 1 & c -> 4b - & c -> 4b - & c -> 4 & a -

prop-const =PropConst <+ prop-const-assign<+ (all(prop-const); try(EvalBinOp))

prop-const-assign =|[ x := <prop-const => e> ]|; if <is-value> e then

rules( PropConst : |[ x ]| -> |[ e ]| )elserules( PropConst :- |[ x ]| )

b := 1;c := b + 3;b := foo();a := b + c

b -> 1b -> 1 & c -> 4b - & c -> 4b - & c -> 4 & a -

b := 1;c := b + 3;b := foo();a := b + c

b -> 1b -> 1 & c -> 4b - & c -> 4b - & c -> 4 & a -

b := 1;c := b + 3;b := foo();a := b + c

b -> 1