lexical analysis - information sciences institute · 2016-01-03 · 2 csci 565 - compiler design...

1

Spring 2016CSCI 565 - Compiler Design

Pedro [email protected]

Lexical Analysis

Regular Expressions & DFA

Copyright 2016, Pedro C. Diniz, all rights reserved.Students enrolled in the Compilers class at the University of Southern California have explicit permission to make copies of these materials for their personal use.

2



2

Outline

• What is a Lexical Analyzer?

• Regular Expressions

• Matching regular expressions using Nondeterministic Finite Automata (NFA)

• Transforming an NFA to a DFA

3



3

What is a Lexical Analyzer?

• Example of Tokens• Operators = + - > ( { := == <>• Keywords if while for int double• Numeric literals 43 5.65 -3.6e10 0x13F3A• Character literals ‘a’ ‘~’ ‘\’’• String literals “565” “Fall 10” “\”\” = empty”

• Example of non-tokens• White space space(‘ ‘) tab(‘\t’) end-of-line(‘\n’)• Comments /*this is not a token*/

Source Program Text Tokens

4



4

f o r v a r 1 = 1 0 v a r 1 < =

Lexical Analyzer in Action

5



5

for_key ID(“var1”) eq_op Num(10) ID(“var1”) leq_op

Lexical Analyzer in Action

f o r v a r 1 = 1 0 v a r 1 < =

6



6

Lexical Analyzer Needs To...

• Partition Input Program Text into Subsequence of Characters Corresponding to Tokens

• Attach the Corresponding Attributes to the Tokens

• Eliminate White Space and Comments

7



7

• Precisely identify the type of token that matches the input string

• 603 Num(603)• CSCI565 ID(“CSCI565”)

• Precisely describe different types of tokens• FORTRAN DO I=1,10 • C++ for(int i=1; i <= 10; i++)• C-shell foreach i (1 2 3 4 5 6 7 8 9 10)

• Use Regular Expressions to precisely describe what strings each type of token can recognize

Lexical Analyzer Needs To...

8



8

Outline



• Matching Regular expressions using Nondeterministic Finite Automata (NFA)


9



9

Examples of Regular ExpressionsRegular Expression Strings Matched

a “a”a · b “ab”a | b “a” “b”ε “”

a* “” “a” “aa” “aaa”…(a | ε) · b “ab” “b”num = 0|1|2|3|4|5|6|7|8|9 “0”, “1”, …posint = num · num* “8” “6035” …int = (ε | -) · posint “-42” “1024” …real = int · (ε | (. · posint)) “-12.56” “12” “1.414”...

10



10

Definition: Formal Languages

• Alphabet Σ = finite set of symbols– Σ = { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 }

• String s = finite sequence of symbols from alphabet– s = 6004

• Empty string ε = special string of length zero• Language L = set of strings over an alphabet

– L = { 6001, 6002, 6003, 6004, 6035 6891 … }

11



11

• For a regular expression r, the language L(r) = { all the strings that match r }– L((a | ε) · b) = {“ab” “b”}

• Suppose r and s are Regular Expressions denoting languages L(r) and L(s)– L(r | s) = L(r) ∪ L(s)– L(r · s) = { xy | x ∈ L(r) and y ∈ L(s) }– L(r*) = { x1 x2 ... xk | xi ∈ L(r) and k >= 0 }– L(ε) = {}

Definition: Formal Languages

12



12

More Regular Expressions• We know:

– L(r | s) is the union of L(r) and L(s)– L(r · s) is the concatenation of L(r) and L(s)– L(r*) is the Kleene closure of L(r)

• “zero or more occurrence of”

• Few additional ones– “one or more occurrence of”r+ = r · r*– “zero or one occurrence of” r? = r | ε

13



13

Question

• What regular expression best identifies USC course numbers?

num = 0|1|2|3|4|5|6|7|8|9

1) class = num · num*

2) class = num · . · num*

3) class = num | . | num*

4) class = (num · . · num)*

14



14

Outline• What is a Lexical Analyzer?




15



15

Reg. Expression to NFA Construction

a a

εε

rr

ss

If r and s are regular expressions with the NFA’s

16



16

rr

ss

r · sr s

r | s r

s

ε

ε ε

ε

r*r

ε

ε

εε


17



17

r ?sε ε

ε

r+r

εε


18



18

Construction Example(-| ε)· (0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

19



19

ε

-

(-| ε)· (0|1|2|3|4|5|6|7|8|9)+· (. ·(0|1|2|3|4|5|6|7|8|9)*)?

Construction Example

20



20

(-| ε)· (0|1|2|3|4|5|6|7|8|9)+· (. ·(0|1|2|3|4|5|6|7|8|9)*)?

ε

-ε


21



21

(-| ε)· (0|1|2|3|4|5|6|7|8|9)+· (. ·(0|1|2|3|4|5|6|7|8|9)*)?

ε

ε

-ε


22



22

(-| ε)· (0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

ε

ε

0123456789

-ε


23



23

(-| ε)· (0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

ε

ε

0123456789

-ε ε


24



24

(-| ε)· (0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

ε

ε

0123456789

-ε ε

ε

ε


25



25

(-| ε)· (0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

ε

ε

0123456789

-ε . ε

ε


26



26

(-| ε)· (0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

ε

ε

0123456789

-ε . εε

εε

ε


27



27

(-| ε)· (0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

ε

ε

0123456789

ε

0123456789

-ε . εε

εε


28



28

ε

ε

0123456789

ε

0123456789

-ε . εε

εε

- 1 2 . 8

String Matching

29



29

ε

ε

0123456789

ε

0123456789

-ε . εε

εε

- 1 2 . 8

String Matching

30



30

ε

ε

0123456789

ε

0123456789

-ε . εε

εε

- 1 2 . 8

String Matching

31



31

ε

ε

0123456789

ε

0123456789

-ε . εε

εε

- 1 2 . 8

String Matching

32



32

ε

ε

0123456789

ε

0123456789

-ε . εε

εε

- 1 2 . 8

String Matching

33



33

ε

ε

0123456789

ε

0123456789

-ε . εε

εε

- 1 2 . 8

String Matching

34



34

ε

ε

0123456789

ε

0123456789

-ε . εε

εε

- 1 2 . 8

String Matching

35



35

ε

ε

0123456789

ε

0123456789

-ε . εε

εε

- 1 2 . 8

String Matching

36



36

ε

ε

0123456789

ε

0123456789

-ε . εε

εε

- 1 2 . 8

String Matching

37



37

ε

ε

0123456789

ε

0123456789

-ε . εε

εε

- 1 2 . 8

String Matching

38



38

ε

ε

0123456789

ε

0123456789

-ε . εε

εε

- 1 2 . 8

String Matching

39



39

ε

ε

0123456789

ε

0123456789

-ε . εε

εε

- 1 2 . 8

String Matching

40



40

ε

ε

0123456789

ε

0123456789

-ε . εε

εε

- 1 2 . 8

String Matching

41



41

ε

ε

0123456789

ε

0123456789

-ε . εε

εε

- 1 2 . 8

String Matching

42



42

ε

ε

0123456789

ε

0123456789

-ε . εε

εε

- 1 2 . 8

String Matching

43



43

ε

ε

0123456789

ε

0123456789

-ε . εε

εε

- 1 2 . 8

String Matching

44



44

ε

ε

0123456789

ε

0123456789

-ε . εε

εε

- 1 2 . 8

String Matching

45



45

ε

ε

0123456789

ε

0123456789

-ε . εε

εε

- 1 2 . 8

String Matching

46



46

ε

ε

0123456789

ε

0123456789

-ε . εε

εε

- 1 2 . 8

String Matching

47



47

Implementing a Lexical Analyzer

• Need to find which strings match a Regular Expression• Create a NFA for to match the Regular Expression• Unfortunately, NFA does not have a simple

implementation• Need to create a Deterministic Finite Automaton

(DFA) from a NFA

48



48

Outline





49



49

Constructing a DFA from a NFA

• Why do we need a DFA?– Easy to implement– Current state + input symbol uniquely identifies the next state

• How do you construct a DFA from a NFA?

50



50

• Why do we need a DFA?– Easy to implement– Current state + input symbol uniquely identifies the next state

• How do you construct a DFA from a NFA?– DFA keeps track of which states the NFA would be in– Each state of the DFA is in fact a subset of the states of the NFA

NFA DFA

aA

B

A

B

Constructing a DFA from a NFA

aa

ε a1

2

3

4 5

aa

ε a1

2

3

4 5

51



51

State ε-closure• The ε-closure of a state s is the set of states that can

be reached from that state without consuming any of the input– ε-Closure(S) is the smallest set T such that

• Algorithm (fixed-point)

€

T = S edge(s,ε)s∈T$

% & &

'

( ) )

€

T← Srepeat

T '←T

T←T ' edge(s,ε)s∈T '%

& ' '

(

) * *

until T =T '

52



52

ε

-ε . εε

ε

1 4 5 8

ε0

123

4567

8

9

32

ε0

123

4567

8

9

76

ε

S = {1}T = {}T’ = {}

(-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

ε-closure({1})

€

T← Srepeat

T '←T


& ' '

(

) * *

until T =T '

53



53

ε

-ε . εε

ε

1 4 5 8

ε0

123

4567

8

9

32

ε0

123

4567

8

9

76

ε

S = {1}T = {1}T’ = {}

€

T← Srepeat

T '←T


& ' '

(

) * *

until T =T '

(-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

ε-closure({1})

54



54

ε

-ε . εε

ε

1 4 5 8

ε0

123

4567

8

9

32

ε0

123

4567

8

9

76

ε

S = {1}T = {1}T’ = {1}

(-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

ε-closure({1})

€

T← Srepeat

T '←T


& ' '

(

) * *

until T =T '

55



55

ε

-ε . εε

ε

1 4 5 8

ε0

123

4567

8

9

32

ε0

123

4567

8

9

76

ε

S = {1}T = {1, 2}T’ = {1}

(-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

ε-closure({1})

€

T← Srepeat

T '←T


& ' '

(

) * *

until T =T '

56



56

ε

-ε . εε

ε

1 4 5 8

ε0

123

4567

8

9

32

ε0

123

4567

8

9

76

ε

S = {1}T = {1, 2}T’ = {1}

(-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

ε-closure({1})

€

T← Srepeat

T '←T


& ' '

(

) * *

until T =T '

57



57

ε

-ε . εε

ε

1 4 5 8

ε0

123

4567

8

9

32

ε0

123

4567

8

9

76

ε

S = {1}T = {1, 2}T’ = {1, 2}

(-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

ε-closure({1})

€

T← Srepeat

T '←T


& ' '

(

) * *

until T =T '

58



58

ε

-ε . εε

ε

1 4 5 8

ε0

123

4567

8

9

32

ε0

123

4567

8

9

76

ε

S = {1}T = {1, 2}T’ = {1, 2}

(-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

ε-closure({1})

€

T← Srepeat

T '←T


& ' '

(

) * *

until T =T '

59



59

ε

-ε . εε

ε

1 4 5 8

ε0

123

4567

8

9

32

ε0

123

4567

8

9

76

ε

S = {1}T = {1, 2}T’ = {1, 2}

(-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

ε-closure({1})

€

T← Srepeat

T '←T


& ' '

(

) * *

until T =T '

60



60

ε

-ε . εε

ε

1 4 5 8

ε0

123

4567

8

9

32

ε0

123

4567

8

9

76

ε

S = {1}T = {1, 2}T’ = {1, 2}

(-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

ε-closure({1})

€

T← Srepeat

T '←T


& ' '

(

) * *

until T =T '

61



61

What is ε-closure({3})?

ε

-ε . εε

ε

1 4 8

ε0

123

4567

8

9

32

ε0

123

4567

8

9

76

ε

S = {3}T = ??

(-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

5

62



62

ε

-ε . εε

ε

1 4 5 8

ε0

123

4567

8

9

32

ε0

123

4567

8

9

76

ε

S = {3}T = {2, 3, 4, 8}

(-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

What is ε-closure({3})?

63



63

DFAedge• Given symbol c and a set of states S, what states can you reach?

• First find the states you can reach on the symbol c• Then, compute ε-closure to determine what other states are

reachable from each new state following ε-transitions.

€

DFAedge(S,c) = ε − closure edge(s,c)s∈S%

& ' '

(

) * *

64



64

ε

-ε . εε

ε

1 4 5 8

ε0

123

4567

8

9

32

ε0

123

4567

8

9

76

ε

S = {1}

DFAedge({1}, 3) = ??

(-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

What is DFAedge({1}, 3)?

65



65

ε

-ε . εε

ε

1 4 8

ε0

123

4567

8

9

32

ε0

123

4567

8

9

76

ε

S = {1}

edge({1},3) = {}DFAedge({1},3) = {}

(-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?


5

66



66

ε

-ε . εε

ε

1 4 8

ε0

123

4567

8

9

32

ε0

123

4567

8

9

76

ε

S = {2}

edge({2}, 3) = {3}DFAedge({2}, 3) = ε-closure({3})

(-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?


5

67



67

ε

-ε . εε

ε

1 4 8

ε0

123

4567

8

9

32

ε0

123

4567

8

9

76

ε

S = {4}

DFAedge({4}, .) = ??

(-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

What is DFAedge({4}, .)?

5

68



68

ε

-ε . εε

ε

1 4 8

ε0

123

4567

8

9

32

ε0

123

4567

8

9

76

ε

S = {4}

DFAedge({4}, .) = {5,6,8}

(-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

What is DFAedge({4}, .)?

5

69



69

ε

-ε . εε

ε

1 4 5 8

ε0

123

4567

8

9

32

ε0

123

4567

8

9

76

ε

S = {2}

ε-closure({3}) ={2,3,4,8}DFAedge({2}, 3) = {2,3,4,8}

(-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?


70



70

NFA to DFA: the Subset Construction

€

states[0] = s1states[1] = ε − closure({s1})p =1j = 0while ( j ≤ p) doforeach c∈ ∑ doe = DFAedge(states[ j],c)if (e = states[i] for some i ≤ p) thentrans[ j,c] = ielsep = p+1states[p] = etrans[ j,c] = pj = j +1

end ifend foreachend while

• Approach– Use Subset Construction– Mimics the Set of states the NFA

should be in if it were to operate non-deterministically

– Label the states as accepting if they have at least one of the accepting states of the NFA

71



71

ε

-ε . εε

ε

1 4 5 8

ε0

123

4567

8

9

32

ε0

123

4567

8

9

76

ε

RE = (-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

72



72

ε

-ε . εε

ε

1 4 5 8

ε0

123

4567

8

9

32

ε0

123

4567

8

9

76

ε

RE = (-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

Let's simplify the diagram first...

73



73

ε

-ε . εε

ε

1 4 5 8

ε

0...9

32

ε

76

ε

RE = (-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

0...9 just means any character in the range from 0 to 9

0...9

74



74

ε

-ε . εε

ε

1 4 5 8

ε

0...9

32

ε

76

ε

RE = (-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

0...9

ε-closure(1) = ?

75



75

ε

-ε . εε

ε

1 4 5 8

ε

0...9

32

ε

76

ε

RE = (-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

0...9

ε-closure(1) = {1, 2}

76



76

ε

-ε . εε

ε

1 4 5 8

ε

0...9

32

ε

76

ε

RE = (-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

0...9

ε-closure(1) = {1, 2}

This corresponds to the first state in the DFA...

77



77

ε

-ε . εε

ε

1 4 5 8

ε

0...9

32

ε

76

ε

RE = (-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

0...9

DFA State NFA Statesε-closure after transition on...

0...9 − .0 {1, 2}

Simplest approach for subset construction is to build a table

78



78

ε

-ε . εε

ε

1 4 5 8

ε

0...9

32

ε

76

ε

RE = (-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

0...9


0...9 − .0 {1, 2}

From NFA{1, 2}, if input is 0...9

79



79

ε

-ε . εε

ε

1 4 5 8

ε

0...9

32

ε

76

ε

RE = (-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

0...9


0...9 − .0 {1, 2}


80



80

ε

-ε . εε

ε

1 4 5 8

ε

0...9

32

ε

76

ε

RE = (-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

0...9


0...9 − .0 {1, 2}


81



81

ε

-ε . εε

ε

1 4 5 8

ε

0...9

32

ε

76

ε

RE = (-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

0...9


0...9 − .0 {1, 2}


82



82

ε

-ε . εε

ε

1 4 5

ε

0...9

32

ε

76

ε

RE = (-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

0...9


0...9 − .0 {1, 2} {2, 3, 4, 8}


8

83



83

ε

-ε . εε

ε

1 4 5

ε

0...9

32

ε

76

ε

RE = (-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

0...9


0...9 − .0 {1, 2} {2, 3, 4, 8}

1 {2, 3, 4, 8}

NFA{2, 3, 4, 8} is a new combination, so add a DFA state

8

84



84

ε

-ε . εε

ε

1 4 5

ε

0...9

32

ε

76

ε

RE = (-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

0...9


0...9 − .0 {1, 2} {2, 3, 4, 8}

1 {2, 3, 4, 8}

From NFA{1, 2}, if input is −

8

85



85

ε

-ε . εε

ε

1 4 5

ε

0...9

32

ε

76

ε

RE = (-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

0...9


0...9 − .0 {1, 2} {2, 3, 4, 8} {2}1 {2, 3, 4, 8}

ε-closure doesn't lead anywhere else

8

86



86

ε

-ε . εε

ε

1 4 5

ε

0...9

32

ε

76

ε

RE = (-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

0...9


0...9 − .0 {1, 2} {2, 3, 4, 8} {2}1 {2, 3, 4, 8}

2 {2}

NFA{2} is a new combination, so add a DFA state

8

87



87

ε

-ε . εε

ε

1 4 5

ε

0...9

32

ε

76

ε

RE = (-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

0...9


0...9 − .0 {1, 2} {2, 3, 4, 8} {2}1 {2, 3, 4, 8}

2 {2}

From NFA{1, 2}, if input is .

8

88



88

ε

-ε . εε

ε

1 4 5

ε

0...9

32

ε

76

ε

RE = (-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

0...9


0...9 − .0 {1, 2} {2, 3, 4, 8} {2} error1 {2, 3, 4, 8}

2 {2}

From NFA{1, 2}, if input is .

8

89



89

ε

-ε . εε

ε

1 4 5

ε

0...9

32

ε

76

ε

RE = (-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

0...9


0...9 − .0 {1, 2} {2, 3, 4, 8} {2} error

1 {2, 3, 4, 8}

2 {2}

From NFA{2, 3, 4, 8}, if input is 0...9

8

90



90

ε

-ε . εε

ε

1 4 5

ε

0...9

32

ε

76

ε

RE = (-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

0...9


0...9 − .0 {1, 2} {2, 3, 4, 8} {2} error

1 {2, 3, 4, 8}

2 {2}


8

91



91

ε

-ε . εε

ε

1 4 5

ε

0...9

32

ε

76

ε

RE = (-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

0...9


0...9 − .0 {1, 2} {2, 3, 4, 8} {2} error

1 {2, 3, 4, 8}

2 {2}


8

92



92

ε

-ε . εε

ε

1 4 5

ε

0...9

32

ε

76

ε

RE = (-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

0...9


0...9 − .0 {1, 2} {2, 3, 4, 8} {2} error

1 {2, 3, 4, 8} {2, 3, 4, 8}2 {2}


8

93



93

ε

-ε . εε

ε

1 4 5

ε

0...9

32

ε

76

ε

RE = (-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

0...9


0...9 − .0 {1, 2} {2, 3, 4, 8} {2} error

1 {2, 3, 4, 8} {2, 3, 4, 8} error2 {2}

From NFA{2, 3, 4, 8}, if input is –

8

94



94

ε

-ε . εε

ε

1 4 5

ε

0...9

32

ε

76

ε

RE = (-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

0...9


0...9 − .0 {1, 2} {2, 3, 4, 8} {2} error

1 {2, 3, 4, 8} {2, 3, 4, 8} error

2 {2}

From NFA{2, 3, 4, 8}, if input is .

8

95



95

ε

-ε . εε

ε

1 4 5

ε

0...9

32

ε

76

ε

RE = (-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

0...9


0...9 − .0 {1, 2} {2, 3, 4, 8} {2} error

1 {2, 3, 4, 8} {2, 3, 4, 8} error

2 {2}


8

96



96

ε

-ε . εε

ε

1 4 5

ε

0...9

32

ε

76

ε

RE = (-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

0...9


0...9 − .0 {1, 2} {2, 3, 4, 8} {2} error

1 {2, 3, 4, 8} {2, 3, 4, 8} error {5, 6, 8}2 {2}


8

97



97

ε

-ε . εε

ε

1 4 5

ε

0...9

32

ε

76

ε

RE = (-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

0...9


0...9 − .0 {1, 2} {2, 3, 4, 8} {2} error

1 {2, 3, 4, 8} {2, 3, 4, 8} error {5, 6, 8}2 {2}

3 {5, 6, 8}

NFA{5, 6, 8} is a new combination, so add a DFA state

8

98



98

ε

-ε . εε

ε

1 4 5

ε

0...9

32

ε

76

ε

RE = (-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

0...9


0...9 − .0 {1, 2} {2, 3, 4, 8} {2} error

1 {2, 3, 4, 8} {2, 3, 4, 8} error {5, 6, 8}

2 {2}

3 {5, 6, 8}

From NFA{2}, if input is 0...9

8

99



99

ε

-ε . εε

ε

1 4 5

ε

0...9

32

ε

76

ε

RE = (-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

0...9


0...9 − .0 {1, 2} {2, 3, 4, 8} {2} error

1 {2, 3, 4, 8} {2, 3, 4, 8} error {5, 6, 8}

2 {2} {2, 3, 4, 8}3 {5, 6, 8}

From NFA{2}, if input is 0...9

8

100



100

ε

-ε . εε

ε

1 4 5

ε

0...9

32

ε

76

ε

RE = (-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

0...9


0...9 − .0 {1, 2} {2, 3, 4, 8} {2} error

1 {2, 3, 4, 8} {2, 3, 4, 8} error {5, 6, 8}

2 {2} {2, 3, 4, 8} error3 {5, 6, 8}

From NFA{2}, if input is –

8

101



101

ε

-ε . εε

ε

1 4 5

ε

0...9

32

ε

76

ε

RE = (-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

0...9


0...9 − .0 {1, 2} {2, 3, 4, 8} {2} error

1 {2, 3, 4, 8} {2, 3, 4, 8} error {5, 6, 8}

2 {2} {2, 3, 4, 8} error error3 {5, 6, 8}

From NFA{2}, if input is .

8

102



102

ε

-ε . εε

ε

1 4 5

ε

0...9

32

ε

76

ε

RE = (-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

0...9


0...9 − .0 {1, 2} {2, 3, 4, 8} {2} error

1 {2, 3, 4, 8} {2, 3, 4, 8} error {5, 6, 8}

2 {2} {2, 3, 4, 8} error error

3 {5, 6, 8}

From NFA{5, 6, 8}, if input is 0...9

8

103



103

ε

-ε . εε

ε

1 4 5

ε

0...9

32

ε

76

ε

RE = (-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

0...9


0...9 − .0 {1, 2} {2, 3, 4, 8} {2} error

1 {2, 3, 4, 8} {2, 3, 4, 8} error {5, 6, 8}

2 {2} {2, 3, 4, 8} error error

3 {5, 6, 8}


8

104



104

ε

-ε . εε

ε

1 4 5

ε

0...9

32

ε

76

ε

RE = (-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

0...9


0...9 − .0 {1, 2} {2, 3, 4, 8} {2} error

1 {2, 3, 4, 8} {2, 3, 4, 8} error {5, 6, 8}

2 {2} {2, 3, 4, 8} error error

3 {5, 6, 8} {6, 7, 8}


8

105



105

ε

-ε . εε

ε

1 4 5

ε

0...9

32

ε

76

ε

RE = (-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

0...9


0...9 − .0 {1, 2} {2, 3, 4, 8} {2} error

1 {2, 3, 4, 8} {2, 3, 4, 8} error {5, 6, 8}

2 {2} {2, 3, 4, 8} error error

3 {5, 6, 8} {6, 7, 8}4 {6, 7, 8}

NFA{6, 7, 8} is a new combination, so add a DFA state

8

106



106

ε

-ε . εε

ε

1 4 5

ε

0...9

32

ε

76

ε

RE = (-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

0...9


0...9 − .0 {1, 2} {2, 3, 4, 8} {2} error

1 {2, 3, 4, 8} {2, 3, 4, 8} error {5, 6, 8}

2 {2} {2, 3, 4, 8} error error

3 {5, 6, 8} {6, 7, 8} error4 {6, 7, 8}

From NFA{5, 6, 8}, if input is –

8

107



107

ε

-ε . εε

ε

1 4 5

ε

0...9

32

ε

76

ε

RE = (-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

0...9


0...9 − .0 {1, 2} {2, 3, 4, 8} {2} error

1 {2, 3, 4, 8} {2, 3, 4, 8} error {5, 6, 8}

2 {2} {2, 3, 4, 8} error error

3 {5, 6, 8} {6, 7, 8} error error4 {6, 7, 8}

From NFA{5, 6, 8}, if input is .

8

108



108

ε

-ε . εε

ε

1 4 5

ε

0...9

32

ε

76

ε

RE = (-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

0...9


0...9 − .0 {1, 2} {2, 3, 4, 8} {2} error

1 {2, 3, 4, 8} {2, 3, 4, 8} error {5, 6, 8}

2 {2} {2, 3, 4, 8} error error

3 {5, 6, 8} {6, 7, 8} error error

4 {6, 7, 8}


8

109



109

ε

-ε . εε

ε

1 4 5

ε

0...9

32

ε

76

ε

RE = (-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

0...9


0...9 − .0 {1, 2} {2, 3, 4, 8} {2} error

1 {2, 3, 4, 8} {2, 3, 4, 8} error {5, 6, 8}

2 {2} {2, 3, 4, 8} error error

3 {5, 6, 8} {6, 7, 8} error error

4 {6, 7, 8}


8

110



110

ε

-ε . εε

ε

1 4 5

ε

0...9

32

ε

76

ε

RE = (-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

0...9


0...9 − .0 {1, 2} {2, 3, 4, 8} {2} error

1 {2, 3, 4, 8} {2, 3, 4, 8} error {5, 6, 8}

2 {2} {2, 3, 4, 8} error error

3 {5, 6, 8} {6, 7, 8} error error

4 {6, 7, 8} {6, 7, 8}


8

111



111

ε

-ε . εε

ε

1 4 5

ε

0...9

32

ε

76

ε

RE = (-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

0...9


0...9 − .0 {1, 2} {2, 3, 4, 8} {2} error

1 {2, 3, 4, 8} {2, 3, 4, 8} error {5, 6, 8}

2 {2} {2, 3, 4, 8} error error

3 {5, 6, 8} {6, 7, 8} error error

4 {6, 7, 8} {6, 7, 8} error error

Last two are errors...

8

112



112

ε

-ε . εε

ε

1 4 5

ε

0...9

32

ε

76

ε

RE = (-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

0...9


0...9 − .0 {1, 2} {2, 3, 4, 8} {2} error

1 {2, 3, 4, 8} {2, 3, 4, 8} error {5, 6, 8}

2 {2} {2, 3, 4, 8} error error

3 {5, 6, 8} {6, 7, 8} error error

4 {6, 7, 8} {6, 7, 8} error error

No cells left to fill – DONE!

8

113



113

ε

-ε . εε

ε

1 4 5

ε

0...9

32

ε

76

ε

RE = (-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

0...9


0...9 − .0 {1, 2} {2, 3, 4, 8} {2} error

1 {2, 3, 4, 8} {2, 3, 4, 8} error {5, 6, 8}

2 {2} {2, 3, 4, 8} error error

3 {5, 6, 8} {6, 7, 8} error error

4 {6, 7, 8} {6, 7, 8} error error

In this case, NFA state 8 is an accepting state, so any DFA state whichcontains NFA state 8 should also be accepting.

8

114



114

ε

-ε . εε

ε

1 4 5

ε

0...9

32

ε

76

ε

RE = (-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

0...9


0...9 − .0 {1, 2} {2, 3, 4, 8} {2} error

1* {2, 3, 4, 8} {2, 3, 4, 8} error {5, 6, 8}

2 {2} {2, 3, 4, 8} error error

3* {5, 6, 8} {6, 7, 8} error error

4* {6, 7, 8} {6, 7, 8} error error

In this case, NFA state 8 is an accepting state, so any DFA state whichcontains NFA state 8 should also be accepting.

8

115



115


0...9 − .0 {1, 2} {2, 3, 4, 8} {2} error

1* {2, 3, 4, 8} {2, 3, 4, 8} error {5, 6, 8}

2 {2} {2, 3, 4, 8} error error

3* {5, 6, 8} {6, 7, 8} error error

4* {6, 7, 8} {6, 7, 8} error error

Now use the table as a guide to construct the DFA diagram

116



116


0...9 − .0 {1, 2} {2, 3, 4, 8} {2} error

1* {2, 3, 4, 8} {2, 3, 4, 8} error {5, 6, 8}

2 {2} {2, 3, 4, 8} error error

3* {5, 6, 8} {6, 7, 8} error error

4* {6, 7, 8} {6, 7, 8} error error

Now use the table as a guide to construct the DFA diagram

0 1 3 4

2

0...9

–

.

0...9

0...9

0...9 0...9

117



117

0 1 3 4

2

0...9

–

.

0...9

0...9

0...9 0...9

ε

-ε . εε

ε

1 4 5

ε

0...9

32

ε

76

ε

RE = (-| ε) ·(0|1|2|3|4|5|6|7|8|9)+ · (. ·(0|1|2|3|4|5|6|7|8|9)*)?

0...9

8NFA

DFA

118



118

NFA vs DFA: Complexity

• Matching time and space used depends on the length of the regular expression |r| and length of the input string |x|

• NFA matching time is O(|r|x|x|) and used space is O(r)

• DFA matching time is O(|x|) and used space is O(2|r|)– The number of states may grow exponential (cf. subset construction)– (a|b)*a (a|b) (a|b)… (a|b)

• Using lazy transition evaluation only states really used in practice are computed.– Optimization that overcomes or mitigates issues with space

119



119

Summary

• Lexical Analyzer create tokens out of a text stream

• Tokens defined using Regular Expressions (REs)

• Regular Expressions can be mapped to Nondeterministic Finite Automata (NFA) – by simple Thompson’s construction

• NFA is transformed to a DFA – Transformation Algorithm: the Subset construction– Executing a DFA is straightforward

lexical analysis - information sciences institute · 2016-01-03 · 2 csci 565 - compiler design...

Documents