programming language implementation lexical and syntax analysis part ii

Dr. M Shafique Introduction 1

Programming Language Implementation

Lexical and Syntax Analysis

Part II


Outline

• Overview of parsing• Introduction

• Parsing

• Some more details • Lexical analysis

• Parsing


Reference

• Compilers: Principles, Techniques, and ToolsA. V. Aho, R. Sethi, and J. D. UllmanAddison-Wesley Publishing Company 1988

Chapters 1, 2, 3, 4, and 5


Introduction

• A programming language can be defined by describing its• Syntax and

• Semantics

• Grammar-oriented compilation technique • Syntax-directed translation

• Example• Infix expressions translated to post-fix expressions

• Input to output mapping

• 9 -5 +2 to 95-2+


Example

• Syntax1. e e + d

2. e e – d

3. e d

4. d 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9

• Tokens• + - 0 1 2 3 4 5 6 7 8 9


Parsing

• Parsing is a process to determine if a string of tokens can be generated by a grammar

• Parsing methods• Top-down parsing

• Bottom-up parsing


Parsing• Top-down parsing

1. At node n (labeled with nonterminal A), select one of the productions for A and construct children at n for the symbols on the RHS of the production

2. Find the next node at which a subtree is to be constructed

• Recursive-descent parsing is a top-down syntax analysis method in which a set of recursive procedures are executed to process the input• A procedure is associated with each non terminal of a

grammar• Left recursive rules can loop forever


Parsing

• Bottom-up parsing

Bottom-up parsing constructs a aprse tree for an input string of tokens beginning at the leaves and working up towards the root.• Shift-reduce parsing

• Operator precedence parsing

• LR parsing


Parsing

• Removing left recursion• Example

• Left-recursive grammar

A A α | β

• Equivalent grammar without left recursion

A β R

R αR | ε


Some Important Basic Definitions

lexical: Of or relating to the morphemes of a language.

morpheme: A meaningful linguistic unit that cannotbe divided into smaller meaningful parts.

lexical analysis: The task concerned with breaking aninput into its smallest meaningful units, called tokens.



syntax: The way in which words are put togetherto form phrases, clauses, or sentences. The rulesgoverning the formation of statements in a programminglanguage.

syntax analysis: The task concerned with fitting asequence of tokens into a specified syntax.

parsing: To break a sentence down into its componentparts of speech with an explanation of the form, function,and syntactical relationship of each part.



parsing = lexical analysis + syntax analysis

semantic analysis: The task concerned with calculating the program’s meaning.


Regular Expressions

Symbol: a A regular expression formed by a.

Alternation:M | N A regular expression formed by M or N.

Concatenation:M • N A regular expression formed by M followed by N.

Epsilon: The empty string.

Repetition:M* A regular expression formed by zero or

more repetitions of M.


Building a Recognizer for a Language

General approach:

1. Build a deterministic finite automaton (DFA) from regular expression E

2. Execute the DFA to determine whether an input string belongs to L(E)

Note: The DFA construction is done automatically by a tool such as lex.


Finite Automata

A nondeterministic finite automaton A = {S, , s0, F, move }consists of:1. A set of states S2. A set of input symbols (the input symbol alphabet)3. A state s0 that is distinguished as the start state4. A state F distinguished as the accepting state5. A transition function move that maps state-symbol pairs into sets of state.

In a Deterministic Finite State Automata (DFA), the functionmove maps each state-symbol pair into a unique state.


Finite Automata

A Deterministic Finite Automaton (DFA):

A Nondeterministic Finite Automaton (NFA):

0 1 2 3a

b

b bstart

0 1 2 3

a

a

b

b bstart

What languages areaccepted by theseautomata?

b*abb

(a|b)*abb

(Aho,Sethi,Ullman, pp. 114)


Another NFA

start

a

b

a

b

An -transition is taken without consuming any character from the input.

What does the NFA above accepts?

aa*|bb*



Constructing NFA

It is very simple. Remember that a regular expression is formed by the use of alternation, concatenation, and repetition.

How do we define an NFA that accepts a regular expression?

Thus all we need to do is to know how to build the NFAfor a single symbol, and how to compose NFAs.


Composing NFAs with Alternation

The NFA for a symbol a is: ai fstart

Given two NFA N(s) and N(t)

N(s)

N(t)


starti

f

, the NFA N(s|t) is:


Composing NFAs with Concatenation

start

Given two NFA N(s) and N(t), the NFA N(st) is:

N(s) N(t)i f



Composing NFAs with Repetition

The NFA for N(s*) is

N(s)

fi



Properties of the NFA

• Following this construction rules, we obtain an NFA N(r) with these properties:• N(r) has at most twice as many states as the number of

symbols and operators in r;• N(r) has exactly one starting and one accepting state;• Each state of N(r) has at most one outgoing transition on a

symbol of the alphabet or at most two outgoing -transitions.



How to Parse a Regular Expression?

Given a DFA, we can generate an automaton that recognizes the longest substring of an inputthat is a valid token.

Using the three simple rules presented, it is easyto generate an NFA to recognize a regular expression.

Given a regular expression, how do we generatean automaton to recognize tokens?

Create an NFA and convert it to a DFA.


a An ordinary character stands for itself.

The empty string.

Another way to write the empty string.

M | N Alternation, Choosing from M or N.

M N Concatenation, an M followed by an N.

M* Repetition (zero or more times).

M+ Repetition (one or more times).

M? Optional, zero or one occurrence of M.

[a -zA -Z] Character set alternation.

. Stands for any single character except newline.

“a.+*” Quotation, a string in quotes stands for itself

literally.

Regular expression notation: An Example

(Appel, pp. 20)


if {return IF;}

[a - z] [a - z0 - 9 ] * {return ID;}

[0 - 9] + {return NUM;}

([0 - 9] + “.” [0 - 9] *) | ([0 - 9] * “.” [0 - 9] +) {return REAL;}

(“--” [a - z]* “\n”) | (“ ” | “ \n ” | “ \t ”) + {/* do nothing*/}

. {error ();}

(Appel, pp. 20)

Regular expressions for some tokens


Building Finite Automatas for Lexical Tokens

(Appel, pp. 21)

The NFA for a symbol i is: i1 2start

The NFA for the regular expression if is:

f 31start 2i

The NFA for a symbol f is: f 2start 1

IF

if {return IF;}



(Appel, pp. 21)

a-z 21start

ID

[a-z] [a-z0-9 ] * {return ID;}

0-9

a-z



(Appel, pp. 21)

0-9 21start

NUM

[0 - 9] + {return NUM;}

0-9



(Appel, pp. 21)

1start

REAL

([0 - 9] + “.” [0 - 9] *) | ([0 - 9] * “.” [0 - 9] +) {return REAL;}

0-9

0-9

2 3.

0-9

0-950-94

.



(Appel, pp. 21)

1start

/* do nothing */

(“--” [a - z]* “\n”) | (“ ” | “ \n ” | “ \t ”) + {/* do nothing*/}

- 2

a-z

- 3 4\n

\n\t

5blank \n

\tblank


ID

1 20 - 9 0 - 9

NUM

0 - 9

1 2 3

4 5

0 - 9

0 - 9 0 - 9

0 - 9

REAL

1 2 43

5

a-z\n- -

blank, etc.blank, etc.

White space

21any but \n

error

IF

1 2a-z a-z

0-9


1 2i f

3

.

.

(Appel, pp. 21)


Conversion of NFA into DFA

(Appel, pp. 27)

What states can be reached from state 1 without consuming a character?

2 3 84 5 6 7

139 10 11 1214 15

1

a-z

0-90-9

a-z

0-9i

f

IF

error

NUM

ID

anycharacter



What states can be reached from state 1 without consuming a character?

{1,4,9,14} form the -closure of state 1

(Appel, pp. 27)

2 3 84 5 6 7

139 10 11 1214 15

1

a-z

0-90-9

a-z

0-9i

f

IF

error

NUM

ID

anycharacter



What are all the state closures in this NFA?

closure(1) = {1,4,9,14}closure(5) = {5,6,8}closure(8) = {6,8}closure(7) = {7,8}

(Appel, pp. 27)

closure(10) = {10,11,13}closure(13) = {11,13}closure(12) = {12,13}

2 3 84 5 6 7

139 10 11 1214 15

1

a-z

0-90-9

a-z

0-9i

f

IF

error

NUM

ID

anycharacter



Given a set of NFA states T, the -closure(T) is theset of states that are reachable through -transiton from

any state s T.

Given a set of NFA states T, move(T, a) is theset of states that are reachable on input a

from any state sT.



Problem Statement for Conversion of NFA into DFA

Given an NFA find the DFA with the minimum number of states that has the same behavior as the NFA for all inputs.

If the initial state in the NFA is s0, then theset of states in the DFA, Dstates, is initialized with a

state representing -closure(s0).




(Appel, pp. 27)

Dstates = {1-4-9-14}

1-4-9-14

Now we need to compute:

move(1-4-9-14,a-h) = ?

2 3 84 5 6 7

139 10 11 1214 15

1

a-z

0-90-9

a-z

0-9i

f

IF

error

NUM

ID

anycharacter



(Appel, pp. 27)

Dstates = {1-4-9-14}

1-4-9-14


move(1-4-9-14,a-h) = {5,15}

-closure({5,15}) = ?

2 3 84 5 6 7

139 10 11 1214 15

1

a-z

0-90-9

a-z

0-9i

f

IF

error

NUM

ID

anycharacter



(Appel, pp. 27)

Dstates = {1-4-9-14}

1-4-9-14


move(1-4-9-14,a-h) = {5,15}

-closure({5,15}) = {5,6,8,15}

a-h 5-6-8-15

2 3 84 5 6 7

139 10 11 1214 15

1

a-z

0-90-9

a-z

0-9i

f

IF

error

NUM

ID

anycharacter



(Appel, pp. 27)

Dstates = {1-4-9-14}move(1-4-9-14, i) = ?

2 3 84 5 6 7

139 10 11 1214 15

1

a-z

0-90-9

a-z

0-9i

f

IF

error

NUM

ID

anycharacter

1-4-9-14

a-h 5-6-8-15



(Appel, pp. 27)

Dstates = {1-4-9-14}move(1-4-9-14, i) = {2,5,15}

-closure({2,5,15}) = ?

2 3 84 5 6 7

139 10 11 1214 15

1

a-z

0-90-9

a-z

0-9i

f

IF

error

NUM

ID

anycharacter

1-4-9-14

a-h 5-6-8-15



(Appel, pp. 27)

Dstates = {1-4-9-14}move(1-4-9-14, i) = {2,5,15}

-closure({2,5,15}) = {2,5,6,8,15}

2 3 84 5 6 7

139 10 11 1214 15

1

a-z

0-90-9

a-z

0-9i

f

IF

error

NUM

ID

anycharacter

1-4-9-14

a-h 5-6-8-15

2-5-6-8-15i



(Appel, pp. 27)

Dstates = {1-4-9-14}move(1-4-9-14, j-z) = ?

2 3 84 5 6 7

139 10 11 1214 15

1

a-z

0-90-9

a-z

0-9i

f

IF

error

NUM

ID

anycharacter

1-4-9-14

a-h 5-6-8-15

2-5-6-8-15i



(Appel, pp. 27)

Dstates = {1-4-9-14}move(1-4-9-14, j-z) = {5,15}

-closure({5,15}) = ?

2 3 84 5 6 7

139 10 11 1214 15

1

a-z

0-90-9

a-z

0-9i

f

IF

error

NUM

ID

anycharacter

1-4-9-14

a-h 5-6-8-15

2-5-6-8-15i



(Appel, pp. 27)

Dstates = {1-4-9-14}move(1-4-9-14, j-z) = {5,15}

-closure({5,15}) = {5,6,8,15}

2 3 84 5 6 7

139 10 11 1214 15

1

a-z

0-90-9

a-z

0-9i

f

IF

error

NUM

ID

anycharacter

1-4-9-14

a-h 5-6-8-15

2-5-6-8-15i

j-z



(Appel, pp. 27)

Dstates = {1-4-9-14}move(1-4-9-14, 0-9) = {10,15}

-closure({10,15}) = {10,11,13,15}

2 3 84 5 6 7

139 10 11 1214 15

1

a-z

0-90-9

a-z

0-9i

f

IF

error

NUM

ID

anycharacter

1-4-9-14

a-h 5-6-8-15

2-5-6-8-15i

j-z10-11-13-15

0-9



(Appel, pp. 27)

Dstates = {1-4-9-14}move(1-4-9-14, other) = {15}

-closure({15}) = {15}

2 3 84 5 6 7

139 10 11 1214 15

1

a-z

0-90-9

a-z

0-9i

f

IF

error

NUM

ID

anycharacter

1-4-9-14

a-h 5-6-8-15

2-5-6-8-15i

j-z10-11-13-15

0-9

15other



(Appel, pp. 27)

Dstates = {1-4-9-14}

The analysis for 1-4-9-14is complete. We mark it andpick another state in the DFAto analyse.

2 3 84 5 6 7

139 10 11 1214 15

1

a-z

0-90-9

a-z

0-9i

f

IF

error

NUM

ID

anycharacter

1-4-9-14

a-h 5-6-8-15

2-5-6-8-15i

j-z10-11-13-15

0-9

15other


The corresponding DFA

5-6-8-15

2-5-6-8-15

10-11-13-15

3-6-7-8

11-12-13

6-7-8

15

1-4-9-14

a-e, g-z, 0-9

a-z,0-9

a-z,0-9

0-9

0-9

f

i

a-h

j-z

0-9

other

ID

ID

NUM NUM

IF

error

ID

a-z,0-9

(Appel, pp. 29)

See pp. 118 of Aho-Sethi-Ullmanand pp. 29 of Appel.


Lexical Analyzer and Parser

lexicalanalyzer

Syntaxanalyzer

symboltable

get nexttoken


token: smallest meaningful sequence of characters of interest in source program

SourceProgram

get nextchar

next char next token

(Contains a record for each identifier)


Definition of Context-Free Grammars

A context-free grammar G = (T, N, S, P) consists of:1. T, a set of terminals (scanner tokens).2. N, a set of nonterminals (syntactic variables generated

by productions).3. S, a designated start nonterminal.4. P, a set of productions. Each production has the form,

A::= , where A is a nonterminal and is a sentential form , i.e., a string of zero or more grammar symbols (terminals/nonterminals).


Syntax Analysis

Syntax Analysis Problem Statement: To find a derivation sequence in a grammar G for the input token stream (or say that none exists).


Tree nodes represent symbols of the grammar (nonterminals or terminals) and tree edges represent derivation steps.

Parse trees

A parse tree is a graphical representation of a derivation sequence of a sentential form.


Derivation

E E + E | E E | ( E ) | - E | id

Given the following grammar:

Is the string -(id + id) a sentence in this grammar?

Yes because there is the following derivation:

E -E -(E) -(E + E) -(id + id)

Where reads “derives in one step”.



Derivation

E E + E | E E | ( E ) | - E | id

Lets examine this derivation:

E -E -(E) -(E + E) -(id + id)

E E

E-

E

E-

E( )

E

E-

E( )

+E E

E

E-

E( )

+E E

id idThis is a top-down derivationbecause we start building theparse tree at the top parse tree


Dr. M Shafique Introduction 56Which derivation tree is correct?

Another Derivation Example

Find a derivation for the expression: id + id idE E

+E E

E

+E E

E E

E

+E E

E E

id id

id

E E

E E

E

E E

+E E

E

E E

+E E

id id

id

E E + E | E E | ( E ) | - E | id



According to the grammar, both are correct.

Another Derivation Example

Find a derivation for the expression: id + id idE

+E E

E E

id id

id

E

+E E

E E

id id

id

A grammar that produces more than oneparse tree for any input sentence is saidto be an ambiguous grammar.

E E + E | E E | ( E ) | - E | id



Left Recursion

Consider the grammar:E E + T | TT T F | FF ( E ) | id

A top-down parser might loop forever when parsingan expression using this grammar

E E

+E T

E

+E T

+E T

E

+E T

+E T

+E T



Left Recursion

Consider the grammar:E E + T | TT T F | FF ( E ) | id

A grammar that has at least one production of the formA A is a left recursive grammar.

Top-down parsers do not work with left-recursivegrammars.

Left-recursion can often be eliminated by rewriting thegrammar.



Left Recursion

This left-recursivegrammar:

E E + T | TT T F | FF ( E ) | id

Can be re-written to eliminate the immediate left recursion:

E TE’E’ +TE’ | T FT’T’ FT’ | F ( E ) | id



Predictive Parsing

Consider the grammar:

stm if expr then stmt else stmt | while expr do stmt | begin stmt_list end

A parser for this grammar can be written with the following simple structure: switch(gettoken())

{ case if: …. break;

case while: …. break;

case begin: …. break;

default: reject input;}

Based only on the first token,the parser knows which rule to use to derive a statement.

Therefore this is called apredictive parser.



Left Factoring

The following grammar:

stmt if expr then stmt else stmt | if expr then stmt

Cannot be parsed by a predictive parser that looksone element ahead.

But the grammar can be re-written:

stmt if expr then stmt stmt’stmt‘ else stmt |

Where is the empty string.


Rewriting a grammar to eliminate multiple productionsstarting with the same token is called left factoring.


A Predictive Parser


Grammar:

INPUT SYMBOL NON- TERMINAL id + * ( ) $

E E TE’ E TE’ E’ E’ +TE’ E’ E’ T T FT’ T FT’ T’ T’ T’ *FT’ T’ T’ F F id F (E)

ParsingTable:



A Predictive Parser



STACK:

id idid+ INPUT:

Predictive ParsingProgram

E

$

$ OUTPUT:

E

T

E’

$

T E’

PARSINGTABLE:


T

E’

$

T

E’

$

A Predictive Parser



STACK:

id idid+ INPUT:


$ OUTPUT:

E

F

T’

E’

$

F T’

T E’

PARSINGTABLE: (Aho,Sethi,

Ullman, pp. 186)



T

E’

$

T

E’

$

A Predictive Parser



STACK:

id idid+ INPUT:


$ OUTPUT:

E

F

T’

E’

$

F T’

T E’

id

T’

E’

$id

PARSINGTABLE:


A Predictive Parser



STACK:

id idid+ INPUT:


$ OUTPUT:

E

F

T’

E’

$

F T’

T E’

id

T’

E’

$id

Action when Top(Stack) = input $ : Pop stack, advance input.


Ullman, pp. 188)


A Predictive Parser



STACK:

id idid+ INPUT:


$ OUTPUT:

E

F T’

T E’

id

T’

E’

$

E’

$


Ullman, pp. 188)


A Predictive Parser

E

F T’

T E’

id

T+ E’

F T’

id F T’

id

The predictive parser proceedsin this fashion emiting thefollowing productions:

E’ +TE’T FT’F idT’ FT’F idT’ E’

When Top(Stack) = input = $the parser halts and accepts the

input string. (Aho,Sethi,Ullman, pp. 188)


LL(k) Parser

This parser parses from left to right, and does aleftmost-derivation. It looks up 1 symbol ahead to choose its next action. Therefore, it is known asa LL(1) parser.

An LL(k) parser looks k symbols ahead to decideits action.


The Parsing Table


Given this grammar:



PARSINGTABLE:

How is this parsing table built?


FIRST and FOLLOW

We need to build a FIRST set and a FOLLOW setfor each symbol in the grammar.

FIRST() is the set of terminal symbols that can begin any string derived from .

The elements of FIRST and FOLLOW areterminal symbols.

FOLLOW() is the set of terminal symbols that can follow :

t FOLLOW() derivation containing t



Rules to Create FIRST


GRAMMAR:

1. If X is a terminal, FIRST(X) = {X}

FIRST(id) = {id}FIRST() = {}FIRST(+) = {+}

SETS:

2. If X , then FIRST(X)3. If X Y1Y2 ••• Yk

FIRST(() = {(}FIRST()) = {)}

FIRST rules:

*and Y1 ••• Yi-1 and a FIRST(Yi)

then a FIRST(X)

FIRST(F) = {(, id}FIRST(T) = FIRST(F) = {(, id}FIRST(E) = FIRST(T) = {(, id}

FIRST(E’) = {} {+, }FIRST(T’) = {} {, }



Rules to Create FOLLOW


GRAMMAR:

1. If S is the start symbol, then $ FOLLOW(S)

FOLLOW(E) = {$}

FOLLOW(E’) = { ), $}

SETS:

2. If A B, and a FIRST() and a then a FOLLOW(B)3. If A B and a FOLLOW(A) then a FOLLOW(B)

FOLLOW rules:

{ ), $}

3a. If A B and and a FOLLOW(A) then a FOLLOW(B)

* FOLLOW(T) = { ), $}

FIRST(F) = {(, id}FIRST(T) = {(, id}FIRST(E) = {(, id}

FIRST(E’) = {+, }FIRST(T’) = { , }

A and B are non-terminals, and are strings of grammar symbols





GRAMMAR:


FOLLOW(E) = {), $}

FOLLOW(E’) = { ), $}

SETS: 3. If A B and a FOLLOW(A) then a FOLLOW(B)

FOLLOW rules:


* FOLLOW(T) = { ), $}


FIRST(E’) = {+, }FIRST(T’) = { , }

2. If A B, and a FIRST() and a then a FOLLOW(B)

{+, ), $}





GRAMMAR:


FOLLOW(E) = {), $}

FOLLOW(E’) = { ), $}

SETS:

FOLLOW rules:

FOLLOW(T) = {+, ), $}


FIRST(E’) = {+, }FIRST(T’) = { , }


FOLLOW(T’) = {+, ), $}


*





GRAMMAR:


FOLLOW(E) = {), $}

FOLLOW(E’) = { ), $}

SETS:

FOLLOW rules:

FOLLOW(T) = {+, ), $}


FIRST(E’) = {+, }FIRST(T’) = { , }


FOLLOW(T’) = {+, ), $}


* FOLLOW(F) = {+, ), $}





GRAMMAR:


FOLLOW(E) = {), $}

FOLLOW(E’) = { ), $}

SETS:

FOLLOW rules:

FOLLOW(T) = {+, ), $}


FIRST(E’) = {+, }FIRST(T’) = { , }

3. If A B and a FOLLOW(A) then a FOLLOW(B)

FOLLOW(T’) = {+, ), $}


* FOLLOW(F) = {+, ), $}

2. If A B, and a FIRST() and a then a FOLLOW(B)

{+, , ), $}



Rules to Build Parsing Table


GRAMMAR:

FOLLOW(E) = {), $}

FOLLOW(E’) = { ), $}

FOLLOW SETS:

FOLLOW(T) = {+, ), $}

FOLLOW(T’) = {+, ), $}FOLLOW(F) = {+, , ), $}


FIRST(E’) = {+, }FIRST(T’) = { , }

FIRST SETS:

PARSINGTABLE:

1. If A : if a FIRST(), add A to M[A, a]






GRAMMAR:

FOLLOW(E) = {), $}

FOLLOW(E’) = { ), $}

FOLLOW SETS:

FOLLOW(T) = {+, ), $}

FOLLOW(T’) = {+, ), $}FOLLOW(F) = {+, , ), $}


FIRST(E’) = {+, }FIRST(T’) = { , }

FIRST SETS:

PARSINGTABLE:








GRAMMAR:

FOLLOW(E) = {), $}

FOLLOW(E’) = { ), $}

FOLLOW SETS:

FOLLOW(T) = {+, ), $}

FOLLOW(T’) = {+, ), $}FOLLOW(F) = {+, , ), $}


FIRST(E’) = {+, }FIRST(T’) = { , }

FIRST SETS:

PARSINGTABLE:

1. If A : if a FIRST(), add A to M[A, a]2. If A : if FIRST(), add A to M[A, b] for each terminal b FOLLOW(A),







GRAMMAR:

FOLLOW(E) = {), $}

FOLLOW(E’) = { ), $}

FOLLOW SETS:

FOLLOW(T) = {+, ), $}

FOLLOW(T’) = {+, ), $}FOLLOW(F) = {+, , ), $}


FIRST(E’) = {+, }FIRST(T’) = { , }

FIRST SETS:

PARSINGTABLE:

1. If A : if a FIRST(), add A to M[A, a]2. If A : if FIRST(), add A to M[A, b] for each terminal b FOLLOW(A), 3. If A : if FIRST(), and $ FOLLOW(A), add A to M[A, $]






Bottom-Up and Top-Down Parsers

Top-down parsers: starts constructing the parse tree at thetop (root) of the tree and move down towards the leaves.• Easy to implement by hand, but • Works with restricted grammars.

example: predictive parsers

Bottom-up parsers: builds the nodes on the bottom of theparse tree first and moves up towards the root.• Suitable for automatic parser generation• Handle a larger class of grammars.

examples: shift-reduce parser (or LR(k) parsers)



Bottom-Up Parser

A bottom-up parser, or a shift-reduce parser, beginsat the leaves and works up to the top of the tree.

The reduction steps trace a rightmost derivationin reverse.

S aABeA Abc | bB d

Consider the Grammar:

We want to parse the input string abbcde.



Bottom-Up Parser Example

a dbb cINPUT:

Bottom-Up ParsingProgram

e OUTPUT:$

ProductionS aABeA Abc

A bB d




a dbb cINPUT:


e OUTPUT:

A

b

$


A bB d




a dbA cINPUT:


e OUTPUT:

A

b

$


A bB d




a dbA cINPUT:


e OUTPUT:

A

b

$


A bB d

We are not reducing here in this example.

A parser would reduce, get stuck and then backtrack!




a dbA cINPUT:


e OUTPUT:

A

b

$


A bB d

c

A

b




a dAINPUT:


e OUTPUT:

A c

A

b

$


A bB d

b




a dAINPUT:


e OUTPUT:

A c

A

b

$


A bB d

b

B

d




a BAINPUT:


e OUTPUT:

A c

A

b

$


A bB d

b

B

d




a BAINPUT:


e OUTPUT:

A c

A

b

$


A bB d

b

B

d

a

S

e




SINPUT:


OUTPUT:

A c

A

b

$


A bB d

b

B

d

a

S

e

This parser is known as an LR Parser because it scans the input from Left to right, and it constructs

a Rightmost derivation in reverse order.




The scanning of productions for matching withhandles in the input string, and backtracking makesthe method used in the previous example veryinefficient.

Can we do better?


LR Parser Example

Input

Stack

LR ParsingProgram

action goto

Output



LR Parser Example

action goto State id + * ( ) $ E T F

0 s5 s4 1 2 3 1 s6 acc 2 r2 s7 r2 r2 3 r4 r4 r4 r4 4 s5 s4 8 2 3 5 r6 r6 r6 r6 6 s5 s4 9 3 7 s5 s4 10 8 s6 s11 9 r1 s7 r1 r1 10 r3 r3 r3 r3 11 r5 r5 r5 r5

The following grammar:

(1) E E + T(2) E T(3) T T F(4) T F(5) F ( E ) (6) F id

Can be parsed with this actionand goto table



LR Parser Example

id idid +INPUT: $

STACK: E0


LR ParsingProgram



GRAMMAR:

OUTPUT:



OUTPUT:

LR Parser Example

id idid +INPUT: $

STACK:


LR ParsingProgram

E5

id

0



F

id

GRAMMAR:



OUTPUT:

0

LR Parser Example

id idid +INPUT: $

STACK:


LR ParsingProgram



F

id

GRAMMAR:



OUTPUT:

E3

F

0

LR Parser Example

id idid +INPUT: $

STACK:


LR ParsingProgram



T

F

id

GRAMMAR:



OUTPUT:

0

LR Parser Example

id idid +INPUT: $

STACK:


LR ParsingProgram



T

F

id

GRAMMAR:



OUTPUT:

LR Parser Example

id idid +INPUT: $

STACK:


LR ParsingProgram

E2

T

0



T

F

id

GRAMMAR:



OUTPUT:

LR Parser Example

id idid +INPUT: $

STACK:

(1) E E + T(2) E’ T(3) T T F(4) T F(5) F ( E ) (6) F id

LR ParsingProgram



E7

2

T

0

T

F

id

GRAMMAR:



OUTPUT:

LR Parser Example

id idid +INPUT: $

STACK:


LR ParsingProgram

E5

id

7

2

T

0



T

F

id

F

id

GRAMMAR:



OUTPUT:

LR Parser Example

id idid +INPUT: $

STACK:


LR ParsingProgram

E7

2

T

0action goto State

id + * ( ) $ E T F 0 s5 s4 1 2 3 1 s6 acc 2 r2 s7 r2 r2 3 r4 r4 r4 r4 4 s5 s4 8 2 3 5 r6 r6 r6 r6 6 s5 s4 9 3 7 s5 s4 10 8 s6 s11 9 r1 s7 r1 r1 10 r3 r3 r3 r3 11 r5 r5 r5 r5

T

F

id

F

id

GRAMMAR:



OUTPUT:

LR Parser Example

id idid +INPUT: $

STACK:


LR ParsingProgram

E10

F

7

2

T

0



T

T F

F

id

id

GRAMMAR:



OUTPUT:

0

LR Parser Example

id idid +INPUT: $

STACK:


LR ParsingProgram



T

T F

F

id

id

GRAMMAR:



OUTPUT:

LR Parser Example

id idid +INPUT: $

STACK:


LR ParsingProgram

2

T

0

T

T F

F

id

idaction goto State

id + * ( ) $ E T F 0 s5 s4 1 2 3 1 s6 acc 2 r2 s7 r2 r2 3 r4 r4 r4 r4 4 s5 s4 8 2 3 5 r6 r6 r6 r6 6 s5 s4 9 3 7 s5 s4 10 8 s6 s11 9 r1 s7 r1 r1 10 r3 r3 r3 r3 11 r5 r5 r5 r5

E

GRAMMAR:



OUTPUT:

0

LR Parser Example

id idid +INPUT: $

STACK:


LR ParsingProgram



T

T F

F

id

id

E

GRAMMAR:



OUTPUT:

LR Parser Example

id idid +INPUT: $

STACK:


LR ParsingProgram

1

E

0



T

T F

F

id

id

E

GRAMMAR:



OUTPUT:

LR Parser Example

id idid +INPUT: $

STACK:


LR ParsingProgram



T

T F

F

id

id

E

6

+

1

E

0

GRAMMAR:



LR Parser Example

id idid +INPUT: $

STACK:


LR ParsingProgram



OUTPUT:

T

T F

F

id

id

E

5

id

6

+

1

E

0

F

id

GRAMMAR:



LR Parser Example

id idid +INPUT: $

STACK:


LR ParsingProgram

OUTPUT:

T

T F

F

id

id

E

6

+

1

E

0

F

id

GRAMMAR:





LR Parser Example

id idid +INPUT: $

STACK:


LR ParsingProgram



OUTPUT:

T

T F

F

id

id

E

3

F

6

+

1

E

0

F

id

GRAMMAR:

T



LR Parser Example

id idid +INPUT: $

STACK:


LR ParsingProgram

OUTPUT:

T

T F

F

id

id

E

6

+

1

E

0

F

id

GRAMMAR:





LR Parser Example

id idid +INPUT: $

STACK:


LR ParsingProgram



OUTPUT:

T

T F

F

id

id

E

9

T

6

+

1

E

0

F

id

GRAMMAR:

T

E

+



LR Parser Example

id idid +INPUT: $

STACK:


LR ParsingProgram

0

GRAMMAR:



OUTPUT:

T

T F

F

id

id

E

F

id

T

E

+



LR Parser Example

id idid +INPUT: $

STACK:


LR ParsingProgram



OUTPUT:

T

T F

F

id

id

E

1

E

0

F

id

GRAMMAR:

T

E

+



Constructing Parsing Tables

All LR parsers use the same parsing program thatwe demonstrated in the previous slides.

What differentiates the LR parsers are the action and the goto tables:

Simple LR (SLR): succeeds for the fewest grammars, but is the easiest to implement.

Canonical LR: succeeds for the most grammars, but is the hardest to implement. It splits states when necessary to prevent reductions that would get the parser stuck.

Lookahead LR (LALR): succeeds for most common syntacticconstructions used in programming languages, but producesLR tables much smaller than canonical LR.

(See AhoSethiUllman pp. 221-230).





Using Lex

Lexcompiler

Lexsource

programlex.l

lex.yy.c

Ccompiler

lex.yy.c a.out

a.outInput

stream

sequenceof

tokens

(Aho-Sethi-Ullman, pp. 258)


Using Yacc

Yacccompiler

YaccSpecificationtranslate.y

y.tab.c

Ccompiler

y.tab.c a.out

a.outInput

output


yacc translate.y

cc y.tab.c -ly


Parsing Action Conflicts

If the grammar specified is ambiguous, yacc willreport parsing action conflicts.

These conflicts can be reduce/reduce conflicts orshift/reduce conflicts.

Yacc has rules to resolve such conflicts automatically(see AhoSethiUllman, pp. 262-264), but the resultingparser might not have the behavior intended by thegrammar writer.

Whenever you see a conflict report, rerun yacc withthe -v flag, examine the y.output file, and re-writeyour grammar to eliminate the conflicts.



Three-Address StatementsA popular form of intermediate code used in optimizing compilers is three-address statements (or variations, such as quadruples).

Source statement:x = a + b c + d

Three address statements with temporaries t1 and t2:

t1 = b ct2 = a + t1

x = t2 + d

• Intermediate Code GenerationRead Aho-Sethi-Ullman: Chapter 8.1 ~ 8.3, 8.7



Lexical Analyzer (Scanner)+

Syntax Analyzer (Parser)+ Semantic Analyzer

Abstract Syntax Tree with attributes

Intermediate-code Generator

Non-optimized Intermediate Code

FrontEnd

ErrorMessage

Front End of a Compiler


Component-Based Approach to Building Compilers

Target-1 Code Generator Target-2 Code Generator

Intermediate-code Optimizer

Language-1 Front End

Source programin Language-1

Language-2 Front End

Source programin Language-2

Non-optimized Intermediate Code

Optimized Intermediate Code

Target-1 machine code Target-2 machine code


Advantages of Using an Intermediate Language

1. Retargeting - Build a compiler for a new machine by attaching a new code generator to an existing front-end.

2. Optimization - reuse intermediate code optimizers in compilers for different languages and different machines.

Note: the terms “intermediate code”, “intermediate language”, and “intermediate representation” are all used interchangeably.


position := initial + rate * 60

Th

e P

has

es o

f a

Co

mp

iler

lexical analyzer

id1 := id2 + id3 * 60

syntax analyzer

:=

id1 +

id2 *

id3 60

semantic analyzer

:=

id1 +

id2 *

id3 inttoreal

60

intermediate code generator

temp1 := inttoreal (60)temp2 := id3 * temp1temp3 := id2 + temp2id1 := temp3

code optimizer

temp1 := id3 * 60.0id1 := id2 + temp1

code generator

MOVF id3, R2MULF #60.0, R2MOVF id2, R1ADDF R2, R1MOVF R1, id1

programming language implementation lexical and syntax analysis part ii

Documents

syntax analysis method

na regular expression

aa regular expression

regular expression e2

input string of tokens

specified syntax

fitting asequence of

parsingat node n