compiler principles fall 2014-2015 compiler principles lecture 3: parsing part 2 roman manevich...

Fall 2014-2015 Compiler PrinciplesLecture 3: Parsing part 2

Roman ManevichBen-Gurion University

2

Tentative syllabusFrontEnd

Scanning

Top-downParsing (LL)

Bottom-upParsing (LR)

AttributeGrammars

IntermediateRepresentation

Lowering

Optimizations

Local Optimizations

DataflowAnalysis

LoopOptimizations

Code Generation

RegisterAllocation

InstructionSelection

mid-term exam

3

Previously• Role of syntax analysis

• Context-free grammars refresher

• Top-down (predictive) parsing– Recursive descent

4

Functions for nonterminalsE LIT | (E OP E) | not ELIT true | falseOP and | or | xor

E() {if (current {TRUE, FALSE}) LIT();else if (current == LPAREN) match(LPARENT); E(); OP(); E(); match(RPAREN);else if (current == NOT) match(NOT); E();else error;

}

LIT() {if (current == TRUE) match(TRUE);else if (current == FALSE) match(FALSE);else error;

}

OP() {if (current == AND) match(AND);else if (current == OR) match(OR);else if (current == XOR) match(XOR);else error;

}

5

Technical challengeswith recursive descent

• With lookahead 1, the function for indexed_elem will never be tried… – What happens for input of the form ID[expr]

term ID | indexed_elemindexed_elem ID [ expr ]

Recursive descent: problem 1

6

Recursive descent: problem 2

int S() { return A() && match(token(‘a’)) && match(token(‘b’));}int A() { return match(token(‘a’)) || 1;}

S A a bA a |

What happens for input “ab”? What happens if you flip order of alternatives and try “aab”?

7

Recursive descent: problem 3

int E() { return E() && match(token(‘-’)) && term();}

E E - term | term

What happens when we execute this procedure? Recursive descent parsers cannot handle left-recursive grammars

p. 127

8

9

Agenda• Predicting productions via

FIRST/FOLLOW/NULLABLE sets

• Handling conflicts

• LL(k) via pushdown automata

10

How do we predict?

• How can we decide which production of ‘E’ to take?

E LIT | (E OP E) | not ELIT true | falseOP and | or | xor

FIRST sets

• For a nonterminal A, FIRST(A) is the set of terminals that can start in a sentence derived from A– Formally: FIRST(A) = {t | A * t ω}

• For a sentential form α, FIRST(α) is the set of terminals that can start in a sentence derived from α– Formally: FIRST(α) = {t | α * t ω}

11

FIRST sets example

• FIRST(E) = …?• FIRST(LIT) = …?• FIRST(OP) = …?

12

E LIT | (E OP E) | not ELIT true | falseOP and | or | xor

FIRST sets example

• FIRST(E) = FIRST(LIT) FIRST(( E OP E )) FIRST(not E)• FIRST(LIT) = { true, false }• FIRST(OP) = {and, or, xor}

• A set of recursive equations• How do we solve them?

13

E LIT | (E OP E) | not ELIT true | falseOP and | or | xor

14

Computing FIRST sets

• This is known as a fixed-point algorithm• We will see such iterative methods later in the course

and learn to reason about them

Assume no null productions (A )1. Initially, for all nonterminals A, set

FIRST(A) = { t | A t ω for some ω }2. Repeat the following until no changes occur:

for each nonterminal A for each production A α1 | … | αk

FIRST(A) = FIRST(α1) … FIRST(∪ ∪ αk)

15

Exercise: compute FIRST

STMT if EXPR then STMT | while EXPR do STMT | EXPR ;EXPR TERM -> id | zero? TERM | not EXPR | ++ id | -- idTERM id | constant

TERM EXPR STMT

16

1. Initialization

STMT if EXPR then STMT | while EXPR do STMT | EXPR ;EXPR TERM -> id | zero? TERM | not EXPR | ++ id | -- idTERM id | constant

TERM EXPR STMT

idconstant

zero?Not++--

ifwhile

17

2. Iterate 1

STMT if EXPR then STMT | while EXPR do STMT | EXPR ;EXPR TERM -> id | zero? TERM | not EXPR | ++ id | -- idTERM id | constant

TERM EXPR STMT

idconstant

zero?Not++--

ifwhile

zero?Not++--

18

2. Iterate 2

STMT if EXPR then STMT | while EXPR do STMT | EXPR ;EXPR TERM -> id | zero? TERM | not EXPR | ++ id | -- idTERM id | constant

TERM EXPR STMT

idconstant

zero?Not++--

ifwhile

idconstant

zero?Not++--

19

2. Iterate 3 – fixed-point

STMT if EXPR then STMT | while EXPR do STMT | EXPR ;EXPR TERM -> id | zero? TERM | not EXPR | ++ id | -- idTERM id | constant

TERM EXPR STMT

idconstant

zero?Not++--

ifwhile

idconstant

zero?Not++--

idconstant

20

Reasoning about the algorithm

• Is the algorithm correct?• Does it terminate? (complexity)

Assume no null productions (A )

1. Initially, for all nonterminals A, setFIRST(A) = { t | A t ω for some ω }

2. Repeat the following until no changes occur:for each nonterminal A for each production A α1 | … | αk

FIRST(A) = FIRST(α1) … FIRST(∪ ∪ αk)

21

Reasoning about the algorithm

• Termination:

• Correctness:

22

LL(1) Parsing of grammars without epsilon productions

Using FIRST sets

• Assume G has no epsilon productions and for every non-terminal X and every pair of productions X and X we have thatFIRST() FIRST() = {}

• No intersection between FIRST sets => can always pick a single rule

23

Using FIRST sets

• In our Boolean expressions example– FIRST( LIT ) = { true, false }– FIRST( ( E OP E ) ) = { ‘(‘ }– FIRST( not E ) = { not }

• If the FIRST sets intersect, may need longer lookahead– LL(k) = class of grammars in which production rule

can be determined using a lookahead of k tokens– LL(1) is an important and useful class

• What if there are epsilon productions?24

25

Extending LL(1) Parsingfor epsilon productions

26

FIRST, FOLLOW, NULLABLE sets

• For each non-terminal X• FIRST(X) = set of terminals that can start in a

sentence derived from X– FIRST(X) = {t | X * t ω}

• NULLABLE(X) if X * • FOLLOW(X) = set of terminals that can follow X

in some derivation– FOLLOW(X) = {t | S * X t }

27

Computing the NULLABLE set

• Lemma: NULLABLE(1 … k) = NULLABLE(1) … NULLABLE(k)

1. Initially NULLABLE(X) = false2. For each non-terminal X if exists a production

X then NULLABLE(X) = true

3. Repeatfor each production Y 1 … k if NULLABLE(1 … k) then

NULLABLE(Y) = trueuntil NULLABLE stabilizes

28

Exercise: compute NULLABLE

S A a bA a | B A B | CC b |

NULLABLE(S) = NULLABLE(A) NULLABLE(a) NULLABLE(b)NULLABLE(A) = NULLABLE(a) NULLABLE()NULLABLE(B) = NULLABLE(A) NULLABLE(B) NULLABLE(C)NULLABLE(C) = NULLABLE(b) NULLABLE()

29

FIRST with epsilon productions

• How do we compute FIRST(1 … k) when epsilon productions are allowed?– FIRST(1 … k) = ?

30

FIRST with epsilon productions

• How do we compute FIRST(1 … k) when epsilon productions are allowed?– FIRST(1 … k) =

if not NULLABLE(1) then FIRST(1) else FIRST(1) FIRST (2 … k)

31

Exercise: compute FIRST

S A c bA a |

NULLABLE(S) = NULLABLE(A) NULLABLE(c) NULLABLE(b)NULLABLE(A) = NULLABLE(a) NULLABLE()

FIRST(S) = FIRST(A) FIRST(cb)FIRST(A) = FIRST(a) FIRST ()

FIRST(S) = FIRST(A) {c}FIRST(A) = FIRST(a)

FOLLOW sets

• if X α Y then FOLLOW(Y) ?

if NULLABLE() or = thenFOLLOW(Y) ?

p. 189

32

FOLLOW sets

• if X α Y then FOLLOW(Y) FIRST()

if NULLABLE() or = thenFOLLOW(Y) ?

p. 189

33

FOLLOW sets

• if X α Y then FOLLOW(Y) FIRST()

if NULLABLE() or = thenFOLLOW(Y) FOLLOW(X)

p. 189

34

35

FOLLOW sets

• if X α Y then FOLLOW(Y) FIRST()

if NULLABLE() or = thenFOLLOW(Y) FOLLOW(X)

• Allows predicting epsilon productions:X when the lookahead token is in FOLLOW(X)

p. 189

S A c bA a |

What should we predict for input “cb”? What should we predict for input “acb”?

36

LL(k) grammars

37

Conflicts

• FIRST-FIRST conflict– X α and X and– If FIRST(α) FIRST(β) {}

• FIRST-FOLLOW conflict– NULLABLE(X)– If FIRST(X) FOLLOW(X) {}

38

LL(1) grammars

• A grammar is in the class LL(1) when it can be derived via:– Top-down derivation– Scanning the input from left to right (L)– Producing the leftmost derivation (L)– With lookahead of one token– For every two productions Aα and Aβ we have

FIRST(α) ∩ FIRST(β) = {}and if NULLABLE(A) then FIRST(A) FOLLOW(A) = {}

• A language is said to be LL(1) when it has an LL(1) grammar

LL(k) grammars

• Generalizes LL(1) for k lookahead tokens• Need to generalize FIRST and FOLLOW for k

lookahead tokens

39

40

Agenda• Predicting productions via

FIRST/FOLLOW/NULLABLE sets

• Handling conflicts

• LL(k) via pushdown automata

41

Handling conflicts

Back to problem 1

• FIRST(term) = { ID }• FIRST(indexed_elem) = { ID }

• FIRST-FIRST conflict

term ID | indexed_elemindexed_elem ID [ expr ]

42

Solution: left factoring• Rewrite the grammar to be in LL(1)

Intuition: just like factoring in algebra: x*y + x*z into x*(y+z)

term ID | indexed_elemindexed_elem ID [ expr ]

term ID after_IDAfter_ID [ expr ] |

43

New grammar is more complex – has epsilon production

S if E then S else S | if E then S | T

Exercise: apply left factoring

44

S if E then S else S | if E then S | T

S if E then S S’ | TS’ else S |

Exercise: apply left factoring

45

Back to problem 2

• FIRST(S) = { a } FOLLOW(S) = { } • FIRST(A) = { a } FOLLOW(A) = { a }

• FIRST-FOLLOW conflict

S A a bA a |

46

Solution: substitution

S A a bA a |

S a a b | a b

Substitute A in S

47

48

Solution: substitution

S A a bA a |

S a a b | a b

Substitute A in S

S a after_A after_A a b | b

Left factoring

Back to problem 3

• Left recursion cannot be handled with a bounded lookahead

• What can we do?

E E - term | term

49

Left recursion removal

• L(G1) = β, βα, βαα, βααα, …

• L(G2) = same

N Nα | β N βN’ N’ αN’ |

G1 G2

E E - term | term E term TE | termTE - term TE |

For our 3rd example:

p. 130

Can be done algorithmically.Problem 1: grammar becomes mangled beyond recognitionProblem 2: grammar may not be LL(1)

50

51

Recap

• Given a grammar• Compute for each non-terminal– NULLABLE– FIRST using NULLABLE– FOLLOW using FIRST and NULLABLE

• Compute FIRST for each sentential form appearing on right-hand side of a production

• Check for conflicts– If exist: attempt to remove conflicts by rewriting

grammar

52

Agenda• Predicting productions via

FIRST/FOLLOW/NULLABLE sets

• Handling conflicts

• LL(k) via pushdown automata

53

LL(1) parsing:the automata approach

By MG (talk · contribs) (Own work) [GFDL (http://www.gnu.org/copyleft/fdl.html) or CC-BY-SA-3.0 (http://creativecommons.org/licenses/by-sa/3.0/)], via Wikimedia Commons

54

Marking “end-of-file”

• Sometimes it will be useful to transform a grammar G with start non-terminal S into a grammar G’ with a new start non-terminal S‘ and a new production rule S’ S $where $ is not part of the set of tokens

• To parse an input α with G’ we change it intoα $

• Simplifies top-down parsing with null productions and LR parsing

55

Another convention

• We will assume that all productions have been consecutively numbered(1) S E $(2) E T(3) E E + T(4) T id (5) T ( E )

LL(1) Parsers

• Recursive Descent– Manual construction

(parsing combinators make this easier, but…)– Uses recursion

• Wanted– A parser that can be generated automatically– Does not use recursion

56

• Pushdown automaton uses– Input stream– Prediction stack– Parsing table

• Nonterminal token production rule• Entry indexed by nonterminal N and token t contains the

alternative of N that must be predicated when current input starts with t

• Essentially, classic conversion from CFG to PDA– The only difference is that we replace non-

deterministic choice with the parsing table

LL(1) parsing via pushdown automata

57

58

Model of non-recursivepredictive parser

Predictive Parsing program

Parsing Table

X

Y

Z

$

Stack

$ b + a

Output

59

LL(1) parsing algorithm• Set stack=S$• While true

– Prediction• When top of stack is nonterminal N• pop N, lookup table[N,t]• If table[N,t] is not empty, push table[N,t] on prediction stack• Otherwise: return syntax error

– Match• When top of prediction stack is a terminal t, must be equal to next input

token t’. If (t = t’), pop t and consume t’.If (t ≠ t’): return syntax error

– End• When prediction stack is empty• If input is empty at that point: return success• Otherwise: return syntax error

( ) not true false and or xor $

E 2 3 1 1LIT 4 5OP 6 7 8

(1) E → LIT(2) E → ( E OP E ) (3) E → not E(4) LIT → true(5) LIT → false(6) OP → and(7) OP → or(8) OP → xor

Non

term

inal

s

Input tokens

Which rule should be used

Example transition table

60

( FIRST(E)

a b c

A A aAb A c

A aAb | caacbb$

Input suffix Stack content Move

aacbb$ A$ predict(A,a) = A aAbaacbb$ aAb$ match(a,a)

acbb$ Ab$ predict(A,a) = A aAbacbb$ aAbb$ match(a,a)

cbb$ Abb$ predict(A,c) = A ccbb$ cbb$ match(c,c)

bb$ bb$ match(b,b)

b$ b$ match(b,b)

$ $ match($,$) – success

Running parser example

61

a b c

A A aAb A c

A aAb | cabcbb$

Input suffix Stack content Move

abcbb$ A$ predict(A,a) = A aAbabcbb$ aAb$ match(a,a)

bcbb$ Ab$ predict(A,b) = ERROR

Illegal input example

62

63

Creating the prediction table

• Let G be an LL(1) grammar• Compute FIRST/NULLABLE/FOLLOW• Check for conflicts• For non-terminal N and token t predict: …

Top-down parsing summary

• Recursive descent• LL(k) grammars• LL(k) parsing with pushdown automata• Cannot deal with left recursion– Left-recursion removal might result with

complicated grammar

64

Next lecture:Bottom-up parsing