3. annotating the abstract syntax tree ― the context from: chapter 3, modern compiler design, by...

3. Annotating the Abstract Syntax Tree ― the Context

From: Chapter 3, Modern Compiler Design, by Dick Grun et al.

Annotating the AST` 2

Background

• The lexical analyzer supplies the initial attributes of the terminals in the leaf nodes of the AST.

• Lexical analysis and parsing together perform the context-free processing of the source program.

• Context handling is required for two different purposes:– to check context condition imposed by the language specification and

– to collect information for semantic processing.


3.1 Attribute grammars

• The computation required by context handling can be specified inside the CFG that is already being used for parsing; this results in an attribute grammar.

• The CFG is extended with two features:– For each grammar symbol, S, terminal or non-terminal, zero or more

attributes are specified, each with a name and a type, like the fields in a record; these are formal attributes, since, like formal parameters, they consists of a name and a type only.

• Room for the actual attributes is allocated automatically in each node that is created for S in the AST.

• The attributes are used to hold information about the semantics attached to that specific node.

• All nodes in the AST correspond to the same grammar symbol S have the same formal attributes, but their values – the actual attributes – mat differ.

(Another feature is to be continued in the next page…)



– With each production rule NM1…Mn , a set of computation rules are associated – the attribute evaluation rules – which express some of the attribute values of the LHS N and the members of the RHS Mi in terms of other attribute values of these.

• These evaluation rules also check the context conditions and issue warning and error messages.



• Requirements to be fulfilled by the attributes:– The attributes of each grammar symbol N are divided into two groups:

synthesis attributes and inherited attributes.

– The evaluation rules for all production rules of N can count on the values of the inherited attributes of N to be set by the parent node, and have themselves the obligation to set synthesized attributes of N.

• The division of attributes into synthesized and inherited is not a logical necessity, but it is very useful and is an integral part of all theory about attribute grammars.



• It is the task of attribute grammar to activate the evaluation rules in such an order as to set all attribute values in a given AST, without using a value before it has been computed.

• The paradigm of the attribute evaluator is that of a data-flow machine: – A computation is performed only when all the values it depends on have been

determined.– Initially, the only attributes that have values belong to the terminal symbols;

theses are synthesized attributes and their values directly from the program text.– These synthesized attributes then become accessible to the evaluation rules of

their parent nodes, where they allow further computation, both for the synthesized attributes of the parent and for the inherited values of the children of the parent.

– The attribute evaluator continues to propagate the values until all attributes have obtained their values.

• This will happen eventually, provided there is no cycle in the computation.



• Detailed explanations required for Fig. 3.21. The execution order of the evaluation rules is not determined by their

textual position but rather by the availability of their operands.

2. The non-terminal Defined_identifier is used rather than just Identifier.

– An instance of the former only has its name; while– An instance of the latter has in addition to its name, scope information,

type, kind, possibly a value, allocating information, etc.

3. The use of Checked type of Constant_definition(Expression .type)rather than just Expression .type.



• Disagreement on whether the start symbol and terminal symbols are different from other symbol with respect to attributes.– From the original theory as published by Knuth (1968), 1) the start

symbol has no inherited attributes, and 2) the terminal symbols has no attributes at all.

1) The AST has a certain semantics, which would emerge as the synthesized attribute of the start symbol; and this semantics was independent of the environment, there was nothing to inherit.

2) Terminal symbols serve syntactic purpose most of the time and then have no semantics.



• Good reasons to allow both types of attributes to both 1) the start symbol and 2) terminal symbols1) The start symbol may need inherited attributes to supply, for example,

definitions from standard libraries, or details about the machine for which to generate code; and

2) Terminal symbols already have synthesized attributes in the form of their representation.



• Now we look into means of evaluating the attributes.• Problem:

– an infinite loop in the computation

• Normally it is the responsibility of the compiler writer not to write infinite loops, but– when one provides a high-level mechanism, one hopes to be able to

give a bit more support.

– There is indeed possibly an algorithm for loop detection in attribute grammars.

• To understand these algorithms, we need to understand the dependency graph.


3.1.1 Dependency graphs

• It is useful to depict the data flow in a node for a given production rule of non-terminal N by a simple diagram: dependency graph.– The inherited attributes of N are represented by named boxes on the left

of the label and synthesized attributes by named boxed on the right.

– The diagram consists of two levels, the top depicting the LHS of the grammar rule and the bottom the RHS.

– Data flow is indicated by arrows leading from the source attributes to the destination attributes.


3.1.2 Attribute allocation

• To make the above approach work, we need a system that will– create the AST,

– allocate space for the attributes in each node in the tree,

– fill the attributes of terminals in the tree with values derived from the representations of the terminals,

– execute evaluation rules of nodes to assign values to attributes until no new values can be assigned, and do this in the right order, so that no attribute value will be used before it is available and that each attribute will get a value once,

– detect when it cannot do so.

• Such a system is called an attribute evaluator.



• The naive and most general way of implementing attribute evaluation is just to implement the data-flow machine.

• We use the following technique:– visit all nodes of the data-flow graph,

– performing all possible assignments in each node when visiting it, and

– repeat this process until all synthesized attributes of the root have been given a value.

• An assignment is possible when all attributes needed for the assignment have already given a value.– This algorithm looks wasteful of computer time, but good for

educational purposes.1. Algorithmically possible

2. Easy to implement

3. A good stepping stone to more realistic attribute evaluation



• The above method is an example of dynamic attribute evaluation, since the order in which the attributes are evaluated is determined dynamically, at the run time of compiler.

• This is opposed to static attribute evaluation, when the order in which the attributes are evaluated is fixed in advance during compiler evaluation.


3.1.2.1 A dynamic attribute evaluator

• The strength of attribute grammars lie in the fact that they transport information from anywhere in the parse tree to anywhere else, in a controlled way.

• We use a simple attribute grammar to demonstrate the attribute evaluator.



• The attribute grammar code in Fig. 3.8 is very heavy and verbose.• Practical attribute grammars have abbreviation techniques for these

and other repetitive code structures.

Digit_Seq(INH base, SYN value) Digit_Seq(base, value) Digit(base, value)ATTRIBUTE RULES:

SET value TO Digit_Seq .value * base + Digit .value;|

Digit(base, value)



• To implement the data-flow machine in the way explained above, we have to visit all nodes of the data dependency graph.

• Visiting all nodes of a graph usually requires some care to avoid infinite loops, but a simple solution is available in this case since the nodes are also linked in the parse tree, which is loop-free.

• By visiting all nodes in the parse tree we automatically visit all nodes in the data dependency graph, and we can visit all nodes in the parse tree by traversing it recursively.



• Now the algorithm at each node is very simple:– try to perform all the assignments in the rules section of that node,

– traverse the children, and

– when returning from them again try to perform all the assignments in the rule section.

• The pre-visit assignments propagate inherited attribute values downwards;– the post-visit assignment harvest the synthesized attributes of the

children and propagate them upwards.


Input: 567BOutput:Evaluate for Number calledEvaluate for Number calledNumber .value = 375


Number value

Digit_Seq valuebase

Digit valuebase

Base_Tag base

Digit_Seq valuebase

Digit valuebaseDigit_Seq valuebase

Digit valuebase

B

7

6

7

1

1

1

1

2

2

2

2

3

3

3

3


3.1.3 Cycle handling

• To prevent the attribute evaluator from looping, cycle in the evaluation must be detected.

• Dynamic cycle detection– the cycle is detected during the evaluation of the attributes in an actual

syntax tree;

– it shows that there is a cycle in a particular tree.

• Static cycle detection– looks at the attribute grammar and from it deduces whether any tree

that it produces can ever exhibit a cycle: it covers all trees.


3.1.3.1 Dynamic cycle detection

• A simple way to dynamically detect a cycle in the above data-flow implementation (but inelegant):– if the syntax tree has N attributes and more than N rounds are found to

be required for obtaining an answer, there must be a cycle.


3.1.3.2 Static cycle detection

• First of all, we must know how such a cycle can exist at all.• A cycle cannot originate directly from a dependency graph of a

production rule P because



• For an attribute dependency cycle to exist, – the data flow has to leave the node,

– pass through some part of the tree and return to the node,

– perhaps repeat this process several times to different parts of the tree and then return to the attribute it started from.



• From Fig. 3.17, there are two kinds of dependencies between the attributes of a non-terminal N:– from inherited to synthesized (IS-dependency)

– from synthesized to inherited (SI-dependency)

• The summary of the dependencies between the attributes of a non-terminal can be collected in an IS-SI graph.



• The IS-SI graphs are used to find cycles in the attribute dependencies of a grammar.

• Suppose we are given – the dependency graph for a production rule NPQ and

– the complete IS-SI graphs of children P and Q in it

– then

– we can obtain the IS-dependencies of N cause by NPQ by adding the dependencies in the IS-SI graphs of P and Q to the dependency graph of NPQ and taking the transitive closure of the dependencies.



• If we had all dependency graphs of all production rules in which N is a child, and– the complete IS-SI graphs of all the other non-terminals in those

production rules,

– we could in the same manner as above detect any cycle that runs through a tree of which N is a child, and

– obtain all SI-graphs of N.

• Together this leads to the IS-SI graph of N and the detection of all cycles involving N.



• The data-flow technique described so far enables us to create very general attribute evaluators easily, and the circularity test shown here allows us to make sure that they will not loop.

• It is, however, felt that this full generality is not always necessary and that there is room for less general but much more efficient attribute evaluation methods.

• We will cover three levels of simplification:– multi-visit attribute grammars,

– L-attributed grammars, and

– S-attributed grammars.



• So far we have assumed that the attributes of a node are allocated in that node, like fields in a record.– For simple attributes integers, pointers, etc. this id satisfactory, but

– for large values, e.g., the environment, this is clearly undesirable.

• The easiest solution to the environment problem– implementing the routine that updates the environments such that it

delivers a pointer to the new environment.

• Another problem:– many attributes are just copies of other attributes on a higher or lower

level in the syntax tree, and that

– much information is replicated many times, requiring time for the copying and using up memory.


3.1.5 Multi-visit attribute grammars

• We have seen a solution to the cyclicity problem for attribute grammars, we turn to their efficiency problems.

• The dynamic evaluation of attributes exhibits some serious inefficiencies:– values must repeatedly be tested for availability;

– the complicated flow of control causes much overhead; and

– repeated traversals over the syntax tree may be needed to obtain all desired attribute values.


3.1.5.1 Multi-visits

• The above problems can be avoided by having a fixed evaluation sequence,– implemented as program code,

– for each production rule of each non-terminal N;

– this implements a form of static attribute evaluation.

• The task of such a code sequence is to evaluate the attributes of a node P, which represented production rule N→M1M2…

• The attribute values needed to do so can be obtained in two ways:– The code can visit a child C of P to obtain the values of some C’s

synthesized attributes while supplying some of C’s inherited values to enable C to compute those synthesized attributes.

– It can leave for the parent of P to obtain the values of some of P’s own inherited attributes while supplying some of P’s own synthesized attributes to enable the parent to compute those inherited attributes.



• Since there is no point in computing an attribute before it is needed, the computation of the required attributes can be placed just before the point at which the flow of control leaves the node for the parent or for a child.

• So there are basically two kinds of visits:Supply a set of inherited attribute values to a child Mi

Visit Mi

Harvest a set of synthesized attribute values supplied by Mi

Supply a set of synthesized attribute values to parentVisit the parentHarvest a set of inherited attribute values supplied by the parent

and

• This reduces the possibilities for the visiting code of a production rule N→M1M2 …Mn to the outline shown in Figure 3.24.



• This scheme is called multi-visit attribute evaluation:– The flow of control pays multiple visit to each node according to a

scheme fixed at compiler generation time.

• It can be implemented as a tree-walker, which executes the code sequentially and moves the flow of control to the children or the parent as indicated;– It will need a stack to leave to the correct position in the parent.

• Alternatively, and more usually, multi-visit attribute evaluation is implemented by recursive descent.– Each visit from the parent is then implemented as a separate routine, a

visiting routine, which evaluates the appropriate attribute rules and calls the appropriate visit routine of the children.

– The ‘leave to parent’ at the end of each visit is implemented as a return statement and the leave stack is accommodated in the return stack.



• An important observation about the sets IN1..n and SN1..n

– INi is associated with the start of the i-th visit by the parent and SNi with the i-th leave to the parent.

– The parent of the node N must of course adhere to this interface, but the parent does not know which production rule for N has produced the child it is about to visit.

– So the set IN1..n and SN1..n must be the same for all production rules for N;

• they are a property of the non-terminal N rather than of each separate production rule for N.

• Similarly, all visiting routines for production rules in the grammar that contain non-terminal N in the RHS must call the visiting routines of N in the same order 1..n.



• To obtain a multi-visit attribute evaluator, we will first show that once we know acceptable IN and SN sets for all non-terminals we can construct a multi-visit attribute evaluator, and we will then see how to obtain such sets.


3.1.5.2 Attribute partitionings

• The above outline of the multiple visit to a node for a production rule N→M1M2…partitions the attributes of N into a list of pairs of sets of attributes: (IN1,SN1), (IN2,SN2), …, (INn,SNn) for what is called an n-visit.– Visit i uses the attributes in INi, which were set by the parent, visits som

e children some number of ties in some order, and returns after having set the attributes in SNi.

– The sets IN1..n must contain all inherited attributes of N, and SN1..n all its synthesized attributes, since each attribute must in the end receives a value some way or another.

• None of the INi and SNi can be empty, except IN1 and perhaps SNn.



• Given an acceptable partitioning (INi,SNi) i=1..n, it is relatively simple to generate the corresponding multi-visit attribute evaluator.

• We now consider – how this can be done, and

– at the same time see what properties of an “acceptable” partitioning are.



• The evaluator we are about to construct consists of a set of recursive routines.

• There are n routines for each production rule P N→M1M2…for non-terminal N, one for each of the n visits, with n determined by N.

– So if there are p production rules for N, there will be a total of pn visit routines for N.

– Assuming that P is the k-th alternative of N, a possible name for the routine for the i-th visit to that alternative might be Visit_i to N alternative k.



• Now discuss how we can determine which visit routines to call in which order inside a visiting routine Visit_i to N alternative k(), based on information gathered during the generation of the routines Visit_h to N() for 1hi, and knowledge of INi.

– We therefore skip the remaining details of Sections 3.1.5.2 and the whole Section of 3.1.5.3, but they can be found from pages 222 to 229.

– We give the summary of the types of attribute grammars then.


3.1.5.3 Ordered attribute grammars


3.1.6 Summary of the types of attribute grammars

• There are series of restrictions that reduce the most general attribute grammars to ordered attribute grammars.– They increase considerably the algorithmic tractability of the grammars

but

– Are almost no obstacles to the compiler writer who uses the attribute grammar

• The first restriction– All synthesized attributes of a production rule and all inherited attribute

of its children must get values assigned to them in the production.

• The second restriction– No tree produced by the grammar may have a cycle in the attribute

dependencies.

– The test for this property is exponential time in the number of attributes in a non-terminal.



• The third restriction– The grammar is still non-cyclic even if a single IS-SI graph is used per non-termin

al rather than an IS-SI graph set. – The test for this property is linear.

• The fourth restriction– The attributes can be evaluated using the fixed multi-visit scheme.– This leads to multi-visit attribute grammars. (3.1.5.1-2)– Such grammars have a partitioning for the attributes of each non-terminals.– Testing whether an attribute grammar is multi-visit is exponential in the total num

ber of attributes.

• The fifth restriction– The partitioning is constructed heuristically using the late evaluation criterion.– This leads to ordered attribute grammar. (3.1.5.3)– The test is reduced to O(n2) and O(n ln n).



• Two classes of attribute grammars resulting from far more serious restrictions– L-attributed grammars

• An inherited attribute of a child of a non-terminal N may depend only on – synthesized attributes of children to the left of it in the production rule of N and on

– the inherited attributes of N itself.

– S-attribute grammars• Can not have inherited attributes at all.


3.2 Manual methods

• Although attribute grammars are the only means we have at the moment for generating context processing programs automatically, – the more advanced attribute evaluation techniques for them are still in

their infancy, and– Much context processing programming is still done at a lower level, by

writing code in a traditional language like C or C++.

• We will give two methods to collect context information from the AST– Symbolic interpretation– Data-flow equations

• Both start from the AST,– both require more flow-of-control information

• It is much more convenient to have the flow-of-control available at each node in the form of successor pointers– The control flow graph


3.2.1 Threading the AST

• The control flow graph can be constructed statically by threading the tree, as follows.– A threading routine exists for each node type;– The threading routine for a node type N

• gets a pointer to the node to be processed as a parameter, • determines which production rule of N describes the node, and• calls the threading routines of its children, in a recursive traversal of the

AST.

– Using this technique, the threading routine for a binary expression could be the following form:PROCEDURE Thread binary expression (Expr node pointer) Thread expression (Expr node pointer .left operand); Thread expression (Expr node pointer .right operand); //link this node to the dynamically last node SET Last node pointer .successor TO Expr node pointer; //make this node the new dynamically last node SET Last node pointer to Expr node pointer;



• A complication arises if the flow of control exits in more than one place from the tree below a node.– E.g., the if-statement

– Problems and solutions• The node corresponding to the run-time then/else decision has two

successors rather than one.– Sol: Just storing two successor pointers in the if-node, make the if-node different

from other nodes

• When we reach the node following the entire if-statement, its address must be recorded in the last nodes of both the then-part and the else-part.

– Sol: Construct a special join node to merge the diverging flow of control

• Fig. 3-38:Sample threading routine for if-statement• Figs.3.39 and 40: AST before and after threading



• Threading the AST can also be expressed by means of an attribute grammar.



• It is often useful to implement the control flow graph as a doubly-linked graph.



• We have seen means to construct the complete control flow graph of a program, we are in a position to discuss two manual methods of context handling:– Symbolic interpretation, which tries too mimic the behavior of the

program at run time in order to collect context information

– Data-flow equations, which is semi-automated restricted form of symbolic interpretation.


3.2.2 Symbolic interpretation

• The run-time behavior of the code at each node is determined by values of the variables it finds at run time upon entering the code, and the behavior determines these values again upon leaving the node.

• Much contextual information about variables can be deduced statically by simulating this run-time process at compile time in a technique called symbolic interpretation or simulation on the stack.

• The technique of symbolic interpretation– A stack representation is attached to each arrow in the control flow graph.– It holds an entry for each identifier visible at that point in the program.– We are mostly interested in variables and constants.– The entry summarizes all compile-time information we have about the variable or

the constant.– Such information could, for example, tell whether it has been initialized or not, or

even what its value is.– The stack representation at the entry to a node and at its exit are connected

by the semantics of that node.


3.2.2 Symbolic interpretation

• It will be clear that many properties can be propagated in this way through the control flow graph, and that the information obtained can be very useful both for doing context checks and for doing optimization.– In fact, this is how some implementation of the C context checking progr

am lint operates.

• Using two variants of symbolic interpretation to check for uninitialized variables– Simple symbolic interpretation

• Works in one scan from routine entrance to routine exit• Applies to structured programs and specific properties only

– Full symbolic interpretation• Works in the presence of any kind of flow of control and for wide range of pro

perties.


3.2.2.1 Simple symbolic interpretation

• To check for the use of uninitialized variables using simple symbolic interpretation, we make a compile-time representation of the local stack of a routine and follow this representation through the entire routine.

• Such a representation can be implemented conveniently as a linked list of names and properties pairs, a ‘property list’.

• The list starts off as empty, or, – if there are parameters, Initialized for IN and INOUT parameters and Unini

tialized for OUT parameters

• We maintain a return list, in which we combine the stack representation as found at return statements and routine exit.

• We then follow the arrows in the control flow graph, all the while updating our list.

– The precise actions required at each node type depend on the semantics of the source language.

– We therefore indicate them briefly.



• When a declaration is met– The declared name is added to the list, with the appropriate status, Ini

tialized and Uninitialized.

• When the flow of control splits, e.g., if-statement– One copy of the original list is used to go through the then-part;

– Another is for the then-part; and

– Merging is made at the end-if node.

– Merging is trivial, except that a variable obtained a value in one branch but not in the other.

• In this case: the status of the variable is made May be initialized.

• When an assignment is met– The destination variable is set to Initialized, after processing the so

urce expression first, because … .



• When the value of a variable is used, usually in an expression, – Its status is checked

• An error message is given if it is not Initialized• A warning message is given if it is May be initialized

• When a node describing a routine call is met– Need not do anything at all in principle

– If it has IN and/or INOUT parameters, treat them as if they were used in an expression

– If it has any IONOUT and OUT parameters, treat them as if they were used in an assignment

• When a for-statement is met



• When we find an exit-loop statement, …• When we find a return statement, …• When we reach the end node of a routine, …• If the bounds in for-statement are constants, the loop will be

performed at least once. – The merging of the original list and the exit list must not perform to avoid

inappropriate messages.

– The same applies to the infinite loops in Cfor(;;)…

while(1)…



• Once we have a system of symbolic interpretation in place in our compiler, we can easily extend it to fit special requirements of and possibilities offered by the source language.1. Do the same thing to see if a variable, constant, field selector, etc. is used at all.

2. Replace the status Initialized by the value, the range or even the set of values the variable may hold, a technique called constant propagation.

• Problem occurs when implementing constant propagation.



• Requirements for simple symbolic interpretation to work1. The program must consist of flow-of-control structures with one entry

point and one exit point only.

2. The value of the property must form a lattice, which means that• the values can be ordered in a sequence 1.. n such that there is no

operation that will transform j into i with i j; we will write i j for all ij.

3. The result of merging two values must be at least as large as the smaller of the two.

4. An action taken on i in a given situation must make any action taken on j in

that same situation superfluous, for ij.

Explanations of the requirements are given further in the following...



• The first requirement allows each control structure to be treated in isolation, – with the property being analyzed well-defined at the entry point of the

structure and at its exit.

• The other three requirements allow us to ignore the jump back to the beginning of looping control structures.– We call the value of the property at the entrance of the loop body in

and that at the exit is out.

– Req. 2 guarantees that inout.

– Req. 3 guarantees that when we merge the out from the end of the first round

through the loop back into in to obtain a value new at the start of a second

round, then newin.

• To take the second loop, we would undertake actions based on new.

• However, Req. 4 says that all these actions are superfluous.Explanations of the requirements are given further in the following...



• The initialization property with values 1=Uninitialized, 2=May be initialized, and 1=Initialized, fulfills these requirements, since– the initialization status can only progress from left to right over these val

ues and

– the actions on Uninitialized (error message) render those on May be initialized superfluous (warning message), which again supersedes those on Initialized (none).

• If these four requirements are not fulfilled, it is necessary to perform full symbolic interpretation.


3.2.2.1 Full symbolic interpretation

• Goto statements can not be handled by simple symbolic interpretation, since they violate requirement 1.

• To handle goto statement need full symbolic interpretation.• Full symbolic interpretation consists of performing the simple symbol

ic interpretation algorithm repeatedly until no more changes in the values of the properties occur, in closure algorithm fashion.

• For the remainder of Full symbolic interpretation see pp. 251-253.