20130530-pegjs
TRANSCRIPT
PEG.js- Javascript Parser Generator - PEG.js- Javascript Parser Generator -
Hidetomo Suzuki
2013 / 5 / 30
ScopeScope
Page 2
This MTG Target
Understand What’s PEG.js
Know How to Use PEG.js
Make New DSL and Parser
2013 / 5 / 30
AgendaAgenda
Page 3
Table of Contents
1.How to make parser
2. What’s PEG.js
3. Try to used to be PEG
4. Let’s make DSL
5.Applications
2013 / 5 / 30
AgendaAgenda
Page 4
Table of Contents
1.How to make parser
2. What’s PEG.js
3. Try to used to be PEG
4. Let’s make DSL
5.Applications
2013 / 5 / 30 Page 5
1. How to make parser 1. How to make parser
1. Lexical Analysis(字句解析 )
2. Syntactic Parsing( 構文解析 )
3. Semantic Analysis( 意味解析 )
4. Intermidiate Code Generation( 中間コード生成 )
5. Code Optimization( コード最適化 )
6. Code Generation( コード生成)
General Step of Processing with Compiler
Parser
2013 / 5 / 30 Page 6
1. How to make parser 1. How to make parser
1. Lexical Analysis(字句解析 )
2. Syntactic Parsing( 構文解析 )
3. Semantic Analysis( 意味解析 )
Analysis for Make Parser
Make strings(minimum unit of string has semantic) from characters
Make tree structure from strings which result of Lexical Analysis
Type Check
※ A string of Lexical Analysis is called “Token”.
2013 / 5 / 30 Page 7
1. How to make parser 1. How to make parser
1. Lexical Analysis
position_x = position_x + 2.0 * time
Identifier : position_x Substitution Symbol : = Identifier : position_x Addition Symbol : + Number : 2.0 Multiple Symbol : * Identifier : time
2013 / 5 / 30 Page 8
1. How to make parser 1. How to make parser
2. Syntactic Parsing Identifier : position_x Substitution Symbol : = Identifier : position_x Addition Symbol : + Number : 2.0 Multiple Symbol : * Identifier : time
Substitution Symbol
Identifier
position_x
Expression
ExpressionExpression
Expression ExpressionIdentifier
position_x
Number
2.0
Identifier
time
=
+
*
2013 / 5 / 30 Page 9
1. How to make parser 1. How to make parser
3. Semantic Analysis
Substitution Symbol
Identifier
position_x
Expression
ExpressionExpression
Expression ExpressionIdentifier
position_x
Number Identifier
=
+
Correct Case : Real Number * Integer Number
Wrong Case : Real Number * Function Pointer
2.0 time
*
2013 / 5 / 30
AgendaAgenda
Page 10
Table of Contents
1.How to make parser
2. What’s PEG.js
3. Try to used to be PEG
4. Let’s make DSL
5.Applications
2013 / 5 / 30 Here comes your footer
Page 11
2. What‘s PEG.js?2. What‘s PEG.js?
Parser Generator for JavaScript
Use PEG(Parsing Expression Grammar)
Easy to try with the web page
Outline of PEG.js
http://pegjs.majda.cz/
2013 / 5 / 30 Here comes your footer
Page 12
2. What‘s PEG.js?2. What‘s PEG.js?
PEG(Parsing Expression Grammar) is one of grammar for artificial language.
Focus point is how input will be analyzed.
What‘s PEG
Ex) Simple Expression Grammar which Has Four Arithmetic Operations
expression <- addexp
addexp <- multiexp (“+” multiexp / “-” multiexp)*
multiexp <- number (“*” number / “/” number)*
number <- [0-9]+
Accept : 2*3+6/2-5*3 -> multiexp + multiexp – multiexp -> ・・・Decline : (1+1)*3+6/2-(7-2)*3 -> ?*3+6/2-?*3 -> can’t recognize…
2013 / 5 / 30 Here comes your footer
Page 13
2. What‘s PEG.js?2. What‘s PEG.js?
• Nothing Confusion
• Different : “a b / a”, “a / a b”
• Easy to use compared with CFG
• Don’t need to implement scanner (Lexical Analyzer)
• Don’t Express Left Recursive
PEG vs CFG(Context Free Grammar)
• Confusion Existing
• Same : “a b | a”, “a | a b”
• Need to implement scanner
• Can Express Left Recursive
PEG CFG
2013 / 5 / 30 Page 14
2. What‘s PEG.js?2. What‘s PEG.js?
1. Make Grammar with PEG
2. Generate Parser with “pegjs” Command
3. Use the Parser Loaded as a Library
Step of Make Parser with PEG.js
vim filename.pegjs
Pegjs filename.pegjs parser.js
<script src=“./parser.js” type=“text/javascript”></script><script type=“text/javascript”> parser.parse(“something”);
2013 / 5 / 30 Page 15
2. What‘s PEG.js?2. What‘s PEG.js?
Install PEG.js
# Install Pythonbrew $ curl -kL http://xrl.us/pythonbrewinstall | bash$ vim ~/.bash_profile[[ -s $HOME/.pythonbrew/etc/bashrc ]] && source $HOME/.pythonbrew/etc/bashrc$ source ~/.bashrc$ pythonbrew install 2.7.2$ pythonbrew switch 2.7.2
# Install Node$ git clone git://github.com/creationix/nvm.git ~/.nvm$ nvm install v0.8.7$ source ~/.nvm/nvm.sh$ nvm use v0.8.7$ vim ~/.npmrc $ registry = http://registry.npmjs.org/
2013 / 5 / 30 Page 16
2. What‘s PEG.js?2. What‘s PEG.js?
1. Make Grammar with PEG
2. Generate Parser with “pegjs” Command
3. Use the Parser Loaded as a Library
Let’s use PEG.js a little bit
$ vim ffirst.pegjsstart = addexpaddexp = integer ("+" integer / "-" integer)*integer = [0-9]+
$ ./node_modules/pegjs/bin/pegjs first.pegjs parser.js
$ vim parser.js1st line: module.export = … -> var parser = …$ vim index.html<script type=“text/javascript” src=“./parser.js”></script><script type=“text/javascript”> alert(parser.parse(“1+2+3-5”));</script>
2013 / 5 / 30
AgendaAgenda
Page 17
Table of Contents
1.How to make parser
2. What’s PEG.js
3. Try to used to be PEG
4. Let’s make DSL
5.Applications
2013 / 5 / 30 Page 18
3.Try to used to be PEG3.Try to used to be PEG
PEG.js Online Version (http://pegjs.majda.cz/online)
2013 / 5 / 30 Page 19
3. Try to used to be PEG3. Try to used to be PEG
Very Simple PEG.js Grammar(calc : integer)
Grammar
Test Input
Test Input Result
start = integerinteger = digits:[0-9]+ { return parseInt(digits.join(“”), 10); }
123456789
123456789
2013 / 5 / 30 Page 20
3. Try to used to be PEG3. Try to used to be PEG
Very Simple PEG.js Grammar(calc : add two integer)
Grammar
Test Input
Test Input Result
start = target1:integer "+" target2:integer { return target1+target2; }integer = digits:[0-9]+ { return parseInt(digits.join(""), 10); }
81+19
100
2013 / 5 / 30 Page 21
3. Try to used to be PEG3. Try to used to be PEG
Very Simple PEG.js Grammar(calc : add and minus two integer)
Grammar
Test Input
Test Input Result
start = target1:integer "+" target2:integer { return target1+target2; } / target1:integer "-" target2:integer { return target1-target2; } integer = digits:[0-9]+ { return parseInt(digits.join(""), 10); }
100-19
81
2013 / 5 / 30 Page 22
3. Try to used to be PEG3. Try to used to be PEG
Very Simple PEG.js Grammar(calc : add and minus integers)
Grammar
Test Input
Test Input Result
start = expressionexpression = ope1:integer "+" ope2:expression { return ope1+ope2; } / ope1:integer "-" ope2:expression { return ope1-ope2; } / ope:integer { return ope; }integer = digits:[0-9]+ { return parseInt(digits.join(""), 10); }
1+2+3+4+5-6+7-8+9-10
9
2013 / 5 / 30 Page 23
3. Try to used to be PEG3. Try to used to be PEG
Very Simple PEG.js Grammar(calc : four arithmetic)
Grammar
Test Input
Test Input Result
start = expressionexpression = ope1:integer "+" ope2:expression { return ope1+ope2; } / ope1:integer "-" ope2:expression { return ope1-ope2; } / ope1:integer "*" ope2:expression { return ope1*ope2; } / ope1:integer "/" ope2:expression { return ope1/ope2; } / ope:integer { return ope; }integer = digits:[0-9]+ { return parseInt(digits.join(""), 10); }
2*3/2
3
2013 / 5 / 30 Page 24
3. Try to used to be PEG3. Try to used to be PEG
Very Simple PEG.js Grammar(calc : four arithmetic with priority)
Grammar
Test Input
Test Input Result
start = expressionexpression = ope1:multiple "+" ope2:expression { return ope1+ope2; } / ope1:multiple "-" ope2:expression { return ope1-ope2; } / ope:multiple { return ope; }multiple = ope1:integer "*" ope2:multiple { return ope1*ope2; } / ope1:integer "/" ope2:multiple { return ope1/ope2; } / ope:integer { return ope; }integer = digits:[0-9]+ { return parseInt(digits.join(""), 10); }
2*3/2+8/4
5
2013 / 5 / 30 Page 25
3. Try to used to be PEG3. Try to used to be PEG
Very Simple PEG.js Grammar(calc : four arithmetic with priority)
Grammar
Test Input
Test Input Result
start = expressionexpression = ope1:multiple "+" ope2:expression { return ope1+ope2; } / ope1:multiple "-" ope2:expression { return ope1-ope2; } / ope:multiple { return ope; }multiple = ope1:bracket "*" ope2:multiple { return ope1*ope2; } / ope1:bracket "/" ope2:multiple { return ope1/ope2; } / ope:bracket { return ope; }bracket = "(" exp:expression ")" {return exp; } / ope:integer {return ope; }integer = digits:[0-9]+ { return parseInt(digits.join(""), 10); }
(7+3)*3/2+8/(6-2)
17
2013 / 5 / 30
AgendaAgenda
Page 26
Table of Contents
1.How to make parser
2. What’s PEG.js
3. Try to used to be PEG
4. Let’s make DSL
5.Applications
2013 / 5 / 30 Page 27
4. Let‘s make DSL4. Let‘s make DSL
Config File Format
Parser Result
Web Page Image
Target
<LogicalExpression?>Y---[function param]N---[function param]
if (isLogicalExpression) function(param);else function(param);
Ex)<Hydea?>Y---[show ‘hydea’]N---[show ‘hydeb’]
Ex)if (isHydea) show(‘hydea’);else show(‘hydeb’);
URL: ex.com/?p=hydea URL: ex.com/?p=hydeb
2013 / 5 / 30 Page 28
4. Let‘s make DSL4. Let‘s make DSL
Just write over specification
Make Grammar
start = "<" logic "?>Y---[" function " " param "]\nN---[" function " " param "]"logic = [a-zA-Z]+function = [a-zA-Z]+param = "'" [a-zA-Z]+ "'"
2013 / 5 / 30 Page 29
4. Let‘s make DSL4. Let‘s make DSL
Refactoring a little bit
Make Grammar
start = "<" identifier "?>Y---[" identifier " " param "]\nN---[" identifier " " param "]"param = "'" identifier "'"identifier = [a-zA-Z]+
2013 / 5 / 30 Page 30
4. Let‘s make DSL4. Let‘s make DSL
Finish to make param and identifier part
Make Grammar
start = "<" identifier "?>Y---[" identifier " " param "]\nN---[" identifier " " param "]"param = "'" id:identifier "'" {return "'" + id + "'"; }identifier = chars:[a-zA-Z]+ {return chars.join(""); }
2013 / 5 / 30 Page 31
4. Let‘s make DSL4. Let‘s make DSL
Organize Function Part
Make Grammar
start = "<" identifier "?>Y---[" function "]\nN---[" function "]"function = id:identifier " " param:param {return id + "(" + param + ")"; }param = "'" id:identifier "'" {return "'" + id + "'"; }identifier = chars:[a-zA-Z]+ {return chars.join(""); }
2013 / 5 / 30 Page 32
4. Let‘s make DSL4. Let‘s make DSL
Finish to make
Make Grammar
start = "<" id:identifier "?>Y---[" f1:function "]\nN---[" f2:function "]" {return "if (is" + id + ") " + f1 + ";\nelse " + f2 + ";"; }function = id:identifier " " param:param {return id + "(" + param + ")"; }param = "'" id:identifier "'" {return "'" + id + "'"; }identifier = chars:[a-zA-Z]+ {return chars.join(""); }
2013 / 5 / 30
AgendaAgenda
Page 33
Table of Contents
1.How to make parser
2. What’s PEG.js
3. Try to used to be PEG
4. Let’s make DSL
5.Applications
2013 / 5 / 30 Page 34
5. Application5. Application
Show My Demo(Demo’s grammar)start = treetree = stat:statement? { return stat; }statement = stat:if_statement { return stat.toString(); }/ stat:proc_statement { return stat.toString(); }if_statement = fac:if_factor 'Y' edge branch1:proc_statement '\n''N' '\n' edge branch2:statement{ return branch2.toString().match(/if/) ?"if (" + fac + ")\n\t" + branch1 + '\nelse ' + branch2: 'if (' + fac + ')\n\t' + branch1 + '\nelse\n\t' + branch2;}proc_statement = fac1:proc_factor [-|]+ fac2:proc_statement { return fac1 + '.then(' + fac2.slice(0,fac2.length-1) + ');'; }/ factor:proc_factor { return factor + ';'; }proc_factor = '[' procexp:expression ']' { return procexp; }if_factor = '<' ifexp:expression '>' { return ifexp; }expression = elem:element ' ' cdr:arguments { return elem.toString() + '(' +cdr + ')'; }/ elem:element { return elem.toString(); }arguments = elem:element ',' arg:arguments { return elem.toString() + ',' + arg; }/ elem:element { return elem.toString(); }element= characters:[a-zA-Z0-9-_."]+ { return characters.join('') }edge= symbols:[-|\n]+ { return symbols.join('') }
2013 / 5 / 30 Page 35
5. Application5. Application
Recommendation
Conf1 Conf2 Conf3 ・・・
RecommendationEngine
Choose a config file
2013 / 5 / 30
ReferenceReference
Page 36
Famous Book for Compiler
2013 / 5 / 30
Thank youThank you
Page 37
Thank you for your listening