language processing systemshamada/lp/l05-lp.pdf · 2018-10-16 · a predictive parser 2. apply the...
TRANSCRIPT
![Page 1: Language Processing Systemshamada/LP/L05-LP.pdf · 2018-10-16 · A Predictive Parser 2. Apply the predictive parsing algorithm to construct the parse tree The following algorithm](https://reader034.vdocument.in/reader034/viewer/2022042022/5e79815e45e3cb10cf4512b3/html5/thumbnails/1.jpg)
Language Processing Systems
Prof. Mohamed Hamada
Software Engineering Lab. The University of Aizu
Japan
![Page 2: Language Processing Systemshamada/LP/L05-LP.pdf · 2018-10-16 · A Predictive Parser 2. Apply the predictive parsing algorithm to construct the parse tree The following algorithm](https://reader034.vdocument.in/reader034/viewer/2022042022/5e79815e45e3cb10cf4512b3/html5/thumbnails/2.jpg)
Syntax Analysis (Parsing)
![Page 3: Language Processing Systemshamada/LP/L05-LP.pdf · 2018-10-16 · A Predictive Parser 2. Apply the predictive parsing algorithm to construct the parse tree The following algorithm](https://reader034.vdocument.in/reader034/viewer/2022042022/5e79815e45e3cb10cf4512b3/html5/thumbnails/3.jpg)
lexical analyzer
Syntax analyzer
symbol table
get next token
Source Program
get next char
next char next token
(Contains a record for each identifier)
1. Uses Regular Expressions to define tokens
2. Uses Finite Automata to recognize tokens
Uses Top-down parsing or Bottom-up parsing
To construct a Parse tree
![Page 4: Language Processing Systemshamada/LP/L05-LP.pdf · 2018-10-16 · A Predictive Parser 2. Apply the predictive parsing algorithm to construct the parse tree The following algorithm](https://reader034.vdocument.in/reader034/viewer/2022042022/5e79815e45e3cb10cf4512b3/html5/thumbnails/4.jpg)
Top Down Parsing
Parsing
Bottom Up Parsing
Predictive Parsing Shift-reduce Parsing
LL(k) Parsing LR(k) Parsing
Left Recursion
Left Factoring
![Page 5: Language Processing Systemshamada/LP/L05-LP.pdf · 2018-10-16 · A Predictive Parser 2. Apply the predictive parsing algorithm to construct the parse tree The following algorithm](https://reader034.vdocument.in/reader034/viewer/2022042022/5e79815e45e3cb10cf4512b3/html5/thumbnails/5.jpg)
Top Down Parsing
Parsing
Bottom Up Parsing
Predictive Parsing Shift-reduce Parsing LL(k) Parsing
LR(k) Parsing Left Recursion Left Factoring
Top-down parsers: starts constructing the parse tree at the top (root) of the tree and move down towards the leaves. Easy to implement by hand, but work with restricted grammars. Example: predictive parsers
![Page 6: Language Processing Systemshamada/LP/L05-LP.pdf · 2018-10-16 · A Predictive Parser 2. Apply the predictive parsing algorithm to construct the parse tree The following algorithm](https://reader034.vdocument.in/reader034/viewer/2022042022/5e79815e45e3cb10cf4512b3/html5/thumbnails/6.jpg)
A Predictive Parser How it works?
1. Construct the parsing table from the given grammar
2. Apply the predictive parsing algorithm to construct the parse tree
![Page 7: Language Processing Systemshamada/LP/L05-LP.pdf · 2018-10-16 · A Predictive Parser 2. Apply the predictive parsing algorithm to construct the parse tree The following algorithm](https://reader034.vdocument.in/reader034/viewer/2022042022/5e79815e45e3cb10cf4512b3/html5/thumbnails/7.jpg)
A Predictive Parser 1. Construct the parsing table from the given grammar
The following algorithm shows how we can construct the parsing table:
Input: a grammar G
Output: the corresponding parsing table M
Method: For each production A à α of the grammar do the following steps:
1. For each terminal a in FIRST(α), add A à α to M[A,a].
2. If λ in FIRST(α), add A à α to M[A,b] for each terminal b in FOLLOW(A).
3. If λ FIRST(α) and $ in FOLLOW(A), add A à α to M[A,$]
![Page 8: Language Processing Systemshamada/LP/L05-LP.pdf · 2018-10-16 · A Predictive Parser 2. Apply the predictive parsing algorithm to construct the parse tree The following algorithm](https://reader034.vdocument.in/reader034/viewer/2022042022/5e79815e45e3cb10cf4512b3/html5/thumbnails/8.jpg)
A Predictive Parser 2. Apply the predictive parsing algorithm to construct the parse tree
The following algorithm shows how we can construct the move parsing table for an input string w$ with respect to a given grammar G.
set ip to point to the first symbol of the input string w$ repeat
if Top(stack) is a terminal or $ then if Top(stack) = Current-Input(ip) then
Pop(stack) and advance ip else null
else if M[X,a]= Xà Y1Y2 …Yk then
begin Pop(stack); Push Y1; Y2;… ; Yk onto the stack, with Y1 on top; Output the production Xà Y1Y2 …Yk
end else null
until Top(stack) = $ (i.e. the stack become empty)
![Page 9: Language Processing Systemshamada/LP/L05-LP.pdf · 2018-10-16 · A Predictive Parser 2. Apply the predictive parsing algorithm to construct the parse tree The following algorithm](https://reader034.vdocument.in/reader034/viewer/2022042022/5e79815e45e3cb10cf4512b3/html5/thumbnails/9.jpg)
A Predictive Parser
E → TE’ E’ → +TE’ | λ T → FT’ T’ → *FT’ | λ F → ( E ) | id
Grammar:
NON- TERMINAL
INPUT SYMBOL id + * ( ) $
E E → TE’ E → TE’ E’ E’ → +TE’ E’ → λ E’ → λ T T → FT’ T → FT’ T’ T’→ λ T’ → *FT’ T’ → λ T’ → λ F F → id F → (E)
Parsing Table:
Example
2. Apply the predictive parsing algorithm to construct the parse tree
![Page 10: Language Processing Systemshamada/LP/L05-LP.pdf · 2018-10-16 · A Predictive Parser 2. Apply the predictive parsing algorithm to construct the parse tree The following algorithm](https://reader034.vdocument.in/reader034/viewer/2022042022/5e79815e45e3cb10cf4512b3/html5/thumbnails/10.jpg)
STACK:
id id id + *INPUT:
Predictive Parsing Program
E $
$ OUTPUT: E
T E’ $
T E’
PARSING TABLE:
ip
Set ip to point to the first symbol of the input string w$ repeat
if Top(stack) is a terminal or $ then if Top(stack) = Current-Input(ip) then
Pop(stack) and advance ip else null else if M[X,a]= Xà Y1Y2 …Yk then begin
Pop(stack); Push Y1; Y2;… ; Yk onto the stack, with Y1 on top; Output the production Xà Y1Y2 …Yk end
else null until Top(stack) = $ (i.e. the stack become empty)
Set ip to point to the first symbol of the input string w$
if Top(stack) is a terminal or $ then
else if M[X,a]= Xà Y1Y2 …Yk then
Pop(stack); Push Y1; Y2;… ; Yk onto the stack, with Y1 on top; Output the production Y1; Y2;… ; Yk ;
Top(stack)=$
NON- TERMINAL
INPUT SYMBOL id + * ( ) $
E E → TE’ E → TE’ E’ E’ → +TE’ E’ → λ E’ → λ T T → FT’ T → FT’ T’ T’→ λ T’ → *FT’ T’ → λ T’ → λ F F → id F → (E)
![Page 11: Language Processing Systemshamada/LP/L05-LP.pdf · 2018-10-16 · A Predictive Parser 2. Apply the predictive parsing algorithm to construct the parse tree The following algorithm](https://reader034.vdocument.in/reader034/viewer/2022042022/5e79815e45e3cb10cf4512b3/html5/thumbnails/11.jpg)
T E’ $
T E’ $
A Predictive Parser
STACK:
id id id + *INPUT:
Predictive Parsing Program
$ OUTPUT: E
F T’ E’ $
F T’
T E’
PARSING TABLE:
NON- TERMINAL
INPUT SYMBOL id + * ( ) $
E E → TE’ E → TE’ E’ E’ → +TE’ E’ → λ E’ → λ T T → FT’ T → FT’ T’ T’→ λ T’ → *FT’ T’ → λ T’ → λ F F → id F → (E)
![Page 12: Language Processing Systemshamada/LP/L05-LP.pdf · 2018-10-16 · A Predictive Parser 2. Apply the predictive parsing algorithm to construct the parse tree The following algorithm](https://reader034.vdocument.in/reader034/viewer/2022042022/5e79815e45e3cb10cf4512b3/html5/thumbnails/12.jpg)
T E’ $
T E’ $
A Predictive Parser
STACK:
id id id + *INPUT:
Predictive Parsing Program
$ OUTPUT: E
F T’ E’ $
F T’
T E’
id T’ E’ $
id
PARSING TABLE:
NON- TERMINAL
INPUT SYMBOL id + * ( ) $
E E → TE’ E → TE’ E’ E’ → +TE’ E’ → λ E’ → λ T T → FT’ T → FT’ T’ T’→ λ T’ → *FT’ T’ → λ T’ → λ F F → id F → (E)
![Page 13: Language Processing Systemshamada/LP/L05-LP.pdf · 2018-10-16 · A Predictive Parser 2. Apply the predictive parsing algorithm to construct the parse tree The following algorithm](https://reader034.vdocument.in/reader034/viewer/2022042022/5e79815e45e3cb10cf4512b3/html5/thumbnails/13.jpg)
A Predictive Parser
STACK:
id id id + *INPUT:
Predictive Parsing Program
$ OUTPUT: E
F T’ E’ $
F T’
T E’
id T’ E’ $
id
Action when Top(Stack) = input ≠ $ : Pop stack, advance input.
PARSING TABLE:
NON- TERMINAL
INPUT SYMBOL id + * ( ) $
E E → TE’ E → TE’ E’ E’ → +TE’ E’ → λ E’ → λ T T → FT’ T → FT’ T’ T’→ λ T’ → *FT’ T’ → λ T’ → λ F F → id F → (E)
![Page 14: Language Processing Systemshamada/LP/L05-LP.pdf · 2018-10-16 · A Predictive Parser 2. Apply the predictive parsing algorithm to construct the parse tree The following algorithm](https://reader034.vdocument.in/reader034/viewer/2022042022/5e79815e45e3cb10cf4512b3/html5/thumbnails/14.jpg)
A Predictive Parser
STACK:
id id id + *INPUT:
Predictive Parsing Program
$ OUTPUT: E
F T’
T E’
id λ
T’ E’ $
E’ $
PARSING TABLE:
NON- TERMINAL
INPUT SYMBOL id + * ( ) $
E E → TE’ E → TE’ E’ E’ → +TE’ E’ → λ E’ → λ T T → FT’ T → FT’ T’ T’→ λ T’ → *FT’ T’ → λ T’ → λ F F → id F → (E)
![Page 15: Language Processing Systemshamada/LP/L05-LP.pdf · 2018-10-16 · A Predictive Parser 2. Apply the predictive parsing algorithm to construct the parse tree The following algorithm](https://reader034.vdocument.in/reader034/viewer/2022042022/5e79815e45e3cb10cf4512b3/html5/thumbnails/15.jpg)
A Predictive Parser
E
F T’
T E’
id λ
T + E’
F T’
id F * T’
id λ
λ
The predictive parser proceeds in this fashion emiting the following productions:
E’ → +TE’ T → FT’ F → id T’ → * FT’ F → id T’ → λ
E’ → λ
When Top(Stack) = input = $ the parser halts and accepts the input string.
![Page 16: Language Processing Systemshamada/LP/L05-LP.pdf · 2018-10-16 · A Predictive Parser 2. Apply the predictive parsing algorithm to construct the parse tree The following algorithm](https://reader034.vdocument.in/reader034/viewer/2022042022/5e79815e45e3cb10cf4512b3/html5/thumbnails/16.jpg)
LL(k) Parser This parser parses from left to right, and does a leftmost-derivation. It looks up 1 symbol ahead to choose its next action. Therefore, it is known as a LL(1) parser.
An LL(k) parser looks k symbols ahead to decide its action.
LL(1) A grammar whose parsing table has no multiply-defined entries
LL(1) grammars enjoys several nice properties: for example they are not ambiguous and not left recursive.
![Page 17: Language Processing Systemshamada/LP/L05-LP.pdf · 2018-10-16 · A Predictive Parser 2. Apply the predictive parsing algorithm to construct the parse tree The following algorithm](https://reader034.vdocument.in/reader034/viewer/2022042022/5e79815e45e3cb10cf4512b3/html5/thumbnails/17.jpg)
LL(k) Parser
E → TE’ E’ → +TE’ | λ T → FT’ T’ → *FT’ | λ F → ( E ) | id Whose PARSINGTABLE:
Example 1 The grammar
Is LL(1) grammar
LL(1) A grammar whose parsing table has no multiply-defined entries
NON- TERMINAL
INPUT SYMBOL id + * ( ) $
E E → TE’ E → TE’ E’ E’ → +TE’ E’ → λ E’ → λ T T → FT’ T → FT’ T’ T’→ λ T’ → *FT’ T’ → λ T’ → λ F F → id F → (E)
![Page 18: Language Processing Systemshamada/LP/L05-LP.pdf · 2018-10-16 · A Predictive Parser 2. Apply the predictive parsing algorithm to construct the parse tree The following algorithm](https://reader034.vdocument.in/reader034/viewer/2022042022/5e79815e45e3cb10cf4512b3/html5/thumbnails/18.jpg)
NON- TERMINAL
INPUT SYMBOL a b e i t $
S S→ a S → iEtSS’
S’ S’ → λ S’ →eS
S’ → λ
E E →b
LL(k) Parser
S → iEtSS`| a S’ → eS | λ E → Fb Whose PARSINGTABLE:
Example 2 The grammar
Is NOT LL(1) grammar
LL(1) A grammar whose parsing table has no multiply-defined entries
![Page 19: Language Processing Systemshamada/LP/L05-LP.pdf · 2018-10-16 · A Predictive Parser 2. Apply the predictive parsing algorithm to construct the parse tree The following algorithm](https://reader034.vdocument.in/reader034/viewer/2022042022/5e79815e45e3cb10cf4512b3/html5/thumbnails/19.jpg)
Top Down Parsing
Parsing
Bottom Up Parsing
Predictive Parsing Shift-reduce Parsing
LL(k) Parsing LR(k) Parsing
Left Recursion
Left Factoring
![Page 20: Language Processing Systemshamada/LP/L05-LP.pdf · 2018-10-16 · A Predictive Parser 2. Apply the predictive parsing algorithm to construct the parse tree The following algorithm](https://reader034.vdocument.in/reader034/viewer/2022042022/5e79815e45e3cb10cf4512b3/html5/thumbnails/20.jpg)
Bottom-Up Parsers
Bottom-up parsers: build the nodes on the bottom of the parse tree first. Suitable for automatic parser generation, handle a larger class of grammars. Examples: shift-reduce parser (or LR(k) parsers)
![Page 21: Language Processing Systemshamada/LP/L05-LP.pdf · 2018-10-16 · A Predictive Parser 2. Apply the predictive parsing algorithm to construct the parse tree The following algorithm](https://reader034.vdocument.in/reader034/viewer/2022042022/5e79815e45e3cb10cf4512b3/html5/thumbnails/21.jpg)
Bottom-up Parsing
• No problem with left-recursion • Widely used in practice • LR(1), SLR(1), LALR(1)
![Page 22: Language Processing Systemshamada/LP/L05-LP.pdf · 2018-10-16 · A Predictive Parser 2. Apply the predictive parsing algorithm to construct the parse tree The following algorithm](https://reader034.vdocument.in/reader034/viewer/2022042022/5e79815e45e3cb10cf4512b3/html5/thumbnails/22.jpg)
Non-ambiguous CFG
CLR(1)
LALR(1)
SLR(1)
LL(1)
Grammar Hierarchy
![Page 23: Language Processing Systemshamada/LP/L05-LP.pdf · 2018-10-16 · A Predictive Parser 2. Apply the predictive parsing algorithm to construct the parse tree The following algorithm](https://reader034.vdocument.in/reader034/viewer/2022042022/5e79815e45e3cb10cf4512b3/html5/thumbnails/23.jpg)
Bottom-up Parsing
• Works from tokens to start-symbol • Repeat:
– identify handle - reducible sequence: • non-terminal is not constructed but • all its children have been constructed
– reduce - construct non-terminal and update stack
• Until reducing to start-symbol
![Page 24: Language Processing Systemshamada/LP/L05-LP.pdf · 2018-10-16 · A Predictive Parser 2. Apply the predictive parsing algorithm to construct the parse tree The following algorithm](https://reader034.vdocument.in/reader034/viewer/2022042022/5e79815e45e3cb10cf4512b3/html5/thumbnails/24.jpg)
Bottom-up Parsing 1 + (2) + (3)
E + (E) + (3)
+
E → E + (E) E → i
E
1 2 + 3
E
E + (3)
E
( ) ( )
E + (E)
E
E
E
E + (2) + (3) i = 0,1, 2, …, 9
![Page 25: Language Processing Systemshamada/LP/L05-LP.pdf · 2018-10-16 · A Predictive Parser 2. Apply the predictive parsing algorithm to construct the parse tree The following algorithm](https://reader034.vdocument.in/reader034/viewer/2022042022/5e79815e45e3cb10cf4512b3/html5/thumbnails/25.jpg)
Bottom-up Parsing
• Is the following grammar LL(1) ?
1 + (2) 1 + (2) + (3)
❚ But this is a useful grammar
E → E + (E) E → i
❚ NO
![Page 26: Language Processing Systemshamada/LP/L05-LP.pdf · 2018-10-16 · A Predictive Parser 2. Apply the predictive parsing algorithm to construct the parse tree The following algorithm](https://reader034.vdocument.in/reader034/viewer/2022042022/5e79815e45e3cb10cf4512b3/html5/thumbnails/26.jpg)
Bottom-Up Parser
A bottom-up parser, or a shift-reduce parser, begins at the leaves and works up to the top of the tree.
The reduction steps trace a rightmost derivation on reverse.
S → aABe A → Abc | b B → d
Consider the Grammar:
We want to parse the input string abbcde.
![Page 27: Language Processing Systemshamada/LP/L05-LP.pdf · 2018-10-16 · A Predictive Parser 2. Apply the predictive parsing algorithm to construct the parse tree The following algorithm](https://reader034.vdocument.in/reader034/viewer/2022042022/5e79815e45e3cb10cf4512b3/html5/thumbnails/27.jpg)
Bottom-Up Parser Example
a d b b c INPUT:
Bottom-Up Parsing Program
e OUTPUT: $
Production S → aABe A → Abc A → b B → d
![Page 28: Language Processing Systemshamada/LP/L05-LP.pdf · 2018-10-16 · A Predictive Parser 2. Apply the predictive parsing algorithm to construct the parse tree The following algorithm](https://reader034.vdocument.in/reader034/viewer/2022042022/5e79815e45e3cb10cf4512b3/html5/thumbnails/28.jpg)
Bottom-Up Parser Example
a d b b c INPUT:
Bottom-Up Parsing Program
e OUTPUT:
A
b
$
Production S → aABe A → Abc A → b B → d
![Page 29: Language Processing Systemshamada/LP/L05-LP.pdf · 2018-10-16 · A Predictive Parser 2. Apply the predictive parsing algorithm to construct the parse tree The following algorithm](https://reader034.vdocument.in/reader034/viewer/2022042022/5e79815e45e3cb10cf4512b3/html5/thumbnails/29.jpg)
Bottom-Up Parser Example
a d b A c INPUT:
Bottom-Up Parsing Program
e OUTPUT:
A
b
$
Production S → aABe A → Abc A → b B → d
![Page 30: Language Processing Systemshamada/LP/L05-LP.pdf · 2018-10-16 · A Predictive Parser 2. Apply the predictive parsing algorithm to construct the parse tree The following algorithm](https://reader034.vdocument.in/reader034/viewer/2022042022/5e79815e45e3cb10cf4512b3/html5/thumbnails/30.jpg)
Bottom-Up Parser Example
a d b A c INPUT:
Bottom-Up Parsing Program
e OUTPUT:
A
b
$
Production S → aABe A → Abc A → b B → d
We are not reducing here in this example. A parser would reduce, get stuck and then backtrack!
![Page 31: Language Processing Systemshamada/LP/L05-LP.pdf · 2018-10-16 · A Predictive Parser 2. Apply the predictive parsing algorithm to construct the parse tree The following algorithm](https://reader034.vdocument.in/reader034/viewer/2022042022/5e79815e45e3cb10cf4512b3/html5/thumbnails/31.jpg)
Bottom-Up Parser Example
a d b A c INPUT:
Bottom-Up Parsing Program
e OUTPUT:
A
b
$
Production S → aABe A → Abc A → b B → d
c
A
b
![Page 32: Language Processing Systemshamada/LP/L05-LP.pdf · 2018-10-16 · A Predictive Parser 2. Apply the predictive parsing algorithm to construct the parse tree The following algorithm](https://reader034.vdocument.in/reader034/viewer/2022042022/5e79815e45e3cb10cf4512b3/html5/thumbnails/32.jpg)
Bottom-Up Parser Example
a d A INPUT:
Bottom-Up Parsing Program
e OUTPUT:
A c
A
b
$
Production S → aABe A → Abc A → b B → d
b
![Page 33: Language Processing Systemshamada/LP/L05-LP.pdf · 2018-10-16 · A Predictive Parser 2. Apply the predictive parsing algorithm to construct the parse tree The following algorithm](https://reader034.vdocument.in/reader034/viewer/2022042022/5e79815e45e3cb10cf4512b3/html5/thumbnails/33.jpg)
Bottom-Up Parser Example
a d A INPUT:
Bottom-Up Parsing Program
e OUTPUT:
A c
A
b
$
Production S → aABe A → Abc A → b B → d
b
B
d
![Page 34: Language Processing Systemshamada/LP/L05-LP.pdf · 2018-10-16 · A Predictive Parser 2. Apply the predictive parsing algorithm to construct the parse tree The following algorithm](https://reader034.vdocument.in/reader034/viewer/2022042022/5e79815e45e3cb10cf4512b3/html5/thumbnails/34.jpg)
Bottom-Up Parser Example
a B A INPUT:
Bottom-Up Parsing Program
e OUTPUT:
A c
A
b
$
Production S → aABe A → Abc A → b B → d
b
B
d
![Page 35: Language Processing Systemshamada/LP/L05-LP.pdf · 2018-10-16 · A Predictive Parser 2. Apply the predictive parsing algorithm to construct the parse tree The following algorithm](https://reader034.vdocument.in/reader034/viewer/2022042022/5e79815e45e3cb10cf4512b3/html5/thumbnails/35.jpg)
Bottom-Up Parser Example
a B A INPUT:
Bottom-Up Parsing Program
e OUTPUT:
A c
A
b
$
Production S → aABe A → Abc A → b B → d
b
B
d
a
S
e
![Page 36: Language Processing Systemshamada/LP/L05-LP.pdf · 2018-10-16 · A Predictive Parser 2. Apply the predictive parsing algorithm to construct the parse tree The following algorithm](https://reader034.vdocument.in/reader034/viewer/2022042022/5e79815e45e3cb10cf4512b3/html5/thumbnails/36.jpg)
Bottom-Up Parser Example
S INPUT:
Bottom-Up Parsing Program
OUTPUT:
A c
A
b
$
Production S → aABe A → Abc A → b B → d
b
B
d
a
S
e
This parser is known as an LR Parser because it scans the input from Left to right, and it constructs
a Rightmost derivation in reverse order.
![Page 37: Language Processing Systemshamada/LP/L05-LP.pdf · 2018-10-16 · A Predictive Parser 2. Apply the predictive parsing algorithm to construct the parse tree The following algorithm](https://reader034.vdocument.in/reader034/viewer/2022042022/5e79815e45e3cb10cf4512b3/html5/thumbnails/37.jpg)
Bottom-Up Parser Example
The scanning of productions for matching with handles in the input string, and backtracking makes the method used in the previous example very inefficient.
Can we do better?
See next lecture