building a python ide with xtext
TRANSCRIPT
![Page 1: Building a Python IDE with Xtext](https://reader033.vdocument.in/reader033/viewer/2022052301/55abe6c81a28abba678b4594/html5/thumbnails/1.jpg)
B U I L D I N G A P Y T H O N I D E W I T H X T E X T
S E B A S T I A N Z A R N E K O W
![Page 2: Building a Python IDE with Xtext](https://reader033.vdocument.in/reader033/viewer/2022052301/55abe6c81a28abba678b4594/html5/thumbnails/2.jpg)
T H I S I S N O T A B O U T
![Page 3: Building a Python IDE with Xtext](https://reader033.vdocument.in/reader033/viewer/2022052301/55abe6c81a28abba678b4594/html5/thumbnails/3.jpg)
T H I S I S A B O U T
![Page 4: Building a Python IDE with Xtext](https://reader033.vdocument.in/reader033/viewer/2022052301/55abe6c81a28abba678b4594/html5/thumbnails/4.jpg)
A N D A N T L R
![Page 5: Building a Python IDE with Xtext](https://reader033.vdocument.in/reader033/viewer/2022052301/55abe6c81a28abba678b4594/html5/thumbnails/5.jpg)
A N D
[\n\r\t ]
![Page 6: Building a Python IDE with Xtext](https://reader033.vdocument.in/reader033/viewer/2022052301/55abe6c81a28abba678b4594/html5/thumbnails/6.jpg)
I N D E N TAT I O N A W A R E
Block: OPEN stmt+=Statement+ CLOSE;
![Page 7: Building a Python IDE with Xtext](https://reader033.vdocument.in/reader033/viewer/2022052301/55abe6c81a28abba678b4594/html5/thumbnails/7.jpg)
C H A L L E N G E SI N D E N TA T I O N A W A R E L A N G U A G E S
![Page 8: Building a Python IDE with Xtext](https://reader033.vdocument.in/reader033/viewer/2022052301/55abe6c81a28abba678b4594/html5/thumbnails/8.jpg)
PA R S I N G C O N T E N T A S S I S T F O R M AT T I N G F O L D I N G O U T L I N E
C H A L L E N G E SI N D E N TA T I O N A W A R E L A N G U A G E S
![Page 9: Building a Python IDE with Xtext](https://reader033.vdocument.in/reader033/viewer/2022052301/55abe6c81a28abba678b4594/html5/thumbnails/9.jpg)
PA R S I N GI N D E N TA T I O N A W A R E L A N G U A G E S - C H A L L E N G E S
![Page 10: Building a Python IDE with Xtext](https://reader033.vdocument.in/reader033/viewer/2022052301/55abe6c81a28abba678b4594/html5/thumbnails/10.jpg)
PA R S I N GI N D E N TA T I O N A W A R E L A N G U A G E S - C H A L L E N G E S
Stateless, Upfront Lexing Reentrant Lexing CommonToken Oddities Token Types Only Basic Logic in Terminal Rules
![Page 11: Building a Python IDE with Xtext](https://reader033.vdocument.in/reader033/viewer/2022052301/55abe6c81a28abba678b4594/html5/thumbnails/11.jpg)
S O M E P Y T H O N E S Q U E C O D E
if name==“Sebastian”: println “Hello”
![Page 12: Building a Python IDE with Xtext](https://reader033.vdocument.in/reader033/viewer/2022052301/55abe6c81a28abba678b4594/html5/thumbnails/12.jpg)
S O M E P Y T H O N E S Q U E C O D E
if.name==“Sebastian”:\n ....println.“Hello”\n
![Page 13: Building a Python IDE with Xtext](https://reader033.vdocument.in/reader033/viewer/2022052301/55abe6c81a28abba678b4594/html5/thumbnails/13.jpg)
PA R S I N G - T O K E N I Z I N G
i f . n a m e = = “ S e b a s t i an ” : \n . . . . p r i n t l n . “ H e l l o ” \n
![Page 14: Building a Python IDE with Xtext](https://reader033.vdocument.in/reader033/viewer/2022052301/55abe6c81a28abba678b4594/html5/thumbnails/14.jpg)
PA R S I N G - T O K E N I Z I N G
i f . n a m e = = “ S e b a s t i a n ” : \n . . . . p r i n t l n . “ H e l l o ” \n
CharStream
o.a.IntStream
AntlrStringStream
![Page 15: Building a Python IDE with Xtext](https://reader033.vdocument.in/reader033/viewer/2022052301/55abe6c81a28abba678b4594/html5/thumbnails/15.jpg)
PA R S I N G - T O K E N I Z I N G if.name==“Sebastian”:\n....println.“Hello”\n
CharStream
o.a.IntStream
AntlrStringStream
![Page 16: Building a Python IDE with Xtext](https://reader033.vdocument.in/reader033/viewer/2022052301/55abe6c81a28abba678b4594/html5/thumbnails/16.jpg)
PA R S I N G - T O K E N I Z I N G if.name==“Sebastian”:\n....println.“Hello”\n
CharStream
o.a.IntStream
AntlrStringStream
1 2
9
1 0
7
8
6
9
1 0
9
9
8
- 1
![Page 17: Building a Python IDE with Xtext](https://reader033.vdocument.in/reader033/viewer/2022052301/55abe6c81a28abba678b4594/html5/thumbnails/17.jpg)
PA R S I N G - T O K E N I Z I N G if.name==“Sebastian”:\n....println.“Hello”\n
CharStream
o.a.IntStream
TokenSource
Lexer
AntlrStringStream
1 2
9
1 0
7
8
6
9
1 0
9
9
8
- 1
![Page 18: Building a Python IDE with Xtext](https://reader033.vdocument.in/reader033/viewer/2022052301/55abe6c81a28abba678b4594/html5/thumbnails/18.jpg)
PA R S I N G - T O K E N I Z I N G if.name==“Sebastian”:\n....println.“Hello”\n
1 2
9
1 0
7
8
6
9
1 0
9
9
8
TokenStream
- 1
TokenSource
Lexer
o.a.IntStream
![Page 19: Building a Python IDE with Xtext](https://reader033.vdocument.in/reader033/viewer/2022052301/55abe6c81a28abba678b4594/html5/thumbnails/19.jpg)
if.name==“Sebastian”:\n....println.“Hello”\n
1 2
9
1 0
7
8
6
9
1 0
9
9
8
- 1
Parser
TokenStream
TokenSource
PA R S I N G - T O K E N I Z I N G
![Page 20: Building a Python IDE with Xtext](https://reader033.vdocument.in/reader033/viewer/2022052301/55abe6c81a28abba678b4594/html5/thumbnails/20.jpg)
Parser
TokenStream
TokenSource
PA R S I N G - T O K E N I Z I N G
9
9
9
9
- 1
if.name==“Sebastian”:\n....println.“Hello”\n
![Page 21: Building a Python IDE with Xtext](https://reader033.vdocument.in/reader033/viewer/2022052301/55abe6c81a28abba678b4594/html5/thumbnails/21.jpg)
if.name==“Sebastian”:\n....println.“Hello”\n
9
9
9
9
- 1
TokenSource
TokenStream
?
PA R S I N G - T O K E N I Z I N G
Parser
![Page 22: Building a Python IDE with Xtext](https://reader033.vdocument.in/reader033/viewer/2022052301/55abe6c81a28abba678b4594/html5/thumbnails/22.jpg)
R E C A P
Block: OPEN stmt+=Statement+ CLOSE;
IfStatement: ‘if’ cond=Condition ‘:’ then=Block;
![Page 23: Building a Python IDE with Xtext](https://reader033.vdocument.in/reader033/viewer/2022052301/55abe6c81a28abba678b4594/html5/thumbnails/23.jpg)
R E C A P
if name==“Sebastian”: println “Hello”
![Page 24: Building a Python IDE with Xtext](https://reader033.vdocument.in/reader033/viewer/2022052301/55abe6c81a28abba678b4594/html5/thumbnails/24.jpg)
if.name==“Sebastian”:\n....println.“Hello”\n
\n OPEN . . . .
9
9
9
9
- 1
1 3 9
\n CLOSE1 4
PA R S I N G - T O K E N S P L I T T I N G
![Page 25: Building a Python IDE with Xtext](https://reader033.vdocument.in/reader033/viewer/2022052301/55abe6c81a28abba678b4594/html5/thumbnails/25.jpg)
if.name==“Sebastian”:\n....println.“Hello”\n
9
9
9
9
- 1
1 3 9
1 4
PA R S I N G - T O K E N S P L I T T I N G
![Page 26: Building a Python IDE with Xtext](https://reader033.vdocument.in/reader033/viewer/2022052301/55abe6c81a28abba678b4594/html5/thumbnails/26.jpg)
if.name==“Sebastian”:\n....println.“Hello”\n
1 3 9
1 4
PA R S I N G - T O K E N S P L I T T I N G
1 2
9
1 0
7
8
6
9
1 0
9
9
8
- 1
![Page 27: Building a Python IDE with Xtext](https://reader033.vdocument.in/reader033/viewer/2022052301/55abe6c81a28abba678b4594/html5/thumbnails/27.jpg)
if.name==“Sebastian”:\n....println.“Hello”\n
PA R S I N G - T O K E N S P L I T T I N G
1 3 9
1 4
TokenStream
Parser
SplittingTokenSource
TokenSource1 2
9
1 0
7
8
6
9
1 0
9
9
8
- 1
Lexer
![Page 28: Building a Python IDE with Xtext](https://reader033.vdocument.in/reader033/viewer/2022052301/55abe6c81a28abba678b4594/html5/thumbnails/28.jpg)
L E T ’ S D O T H I SI N D E N TA T I O N A W A R E L A N G U A G E S
![Page 29: Building a Python IDE with Xtext](https://reader033.vdocument.in/reader033/viewer/2022052301/55abe6c81a28abba678b4594/html5/thumbnails/29.jpg)
Block: OPEN stmt+=Statement+ CLOSE;
1. D E F I N E I N D E N TAT I O N T O K E N S
![Page 30: Building a Python IDE with Xtext](https://reader033.vdocument.in/reader033/viewer/2022052301/55abe6c81a28abba678b4594/html5/thumbnails/30.jpg)
1. D E F I N E I N D E N TAT I O N T O K E N S
terminal OPEN: ‘synthetic:OPEN’; terminal CLOSE: ‘synthetic:CLOSE’;
![Page 31: Building a Python IDE with Xtext](https://reader033.vdocument.in/reader033/viewer/2022052301/55abe6c81a28abba678b4594/html5/thumbnails/31.jpg)
fragment = parser.antlr.ex.rt.AntlrGeneratorFragment {} .. fragment = parser.antlr.ex.ca.ContentAssistParserGeneratorFragment {}
2. A D J U S T M W E 2 W O R K F L O W
1. GENERATE SPLITTING TOKEN SOURCE 2. DISABLE PARTIAL PARSING
![Page 32: Building a Python IDE with Xtext](https://reader033.vdocument.in/reader033/viewer/2022052301/55abe6c81a28abba678b4594/html5/thumbnails/32.jpg)
public class MyDslTokenSource extends AbstractIndentationTokenSource { public MyDslTokenSource(TokenSource delegate) { super(delegate); } @Override protected boolean shouldSplitTokenImpl(Token token) { return token.getType() == InternalMyDslParser.RULE_WS; }
@Override protected int getBeginTokenType() { return InternalMyDslParser.RULE_OPEN; }
@Override protected int getEndTokenType() { return InternalMyDslParser.RULE_CLOSE; } }
3. M A R K T O K E N S A S S P L I T TA B L E
![Page 33: Building a Python IDE with Xtext](https://reader033.vdocument.in/reader033/viewer/2022052301/55abe6c81a28abba678b4594/html5/thumbnails/33.jpg)
3. M A R K T O K E N S A S S P L I T TA B L E
protected Token createEndToken(int offset) { CommonToken result = new CommonToken(getEndTokenType()); result.setText(""); result.setChannel(Token.DEFAULT_CHANNEL); result.setStartIndex(offset); result.setStopIndex(offset-1); return result; }
public class AbstractIndentationTokenSource extends AbstractSplittingTokenSource { …
… }
![Page 34: Building a Python IDE with Xtext](https://reader033.vdocument.in/reader033/viewer/2022052301/55abe6c81a28abba678b4594/html5/thumbnails/34.jpg)
@Override protected boolean shouldSplitTokenImpl(Token token) { return token.getType() == InternalMyDslParser.RULE_WS; }
@Override protected int getBeginTokenType() { return InternalMyDslParser.RULE_OPEN; }
@Override protected int getEndTokenType() { return InternalMyDslParser.RULE_CLOSE; }
3. M A R K T O K E N S A S S P L I T TA B L E
![Page 35: Building a Python IDE with Xtext](https://reader033.vdocument.in/reader033/viewer/2022052301/55abe6c81a28abba678b4594/html5/thumbnails/35.jpg)
@Override protected boolean shouldSplitTokenImpl(Token token) { return token.getType() == InternalMyDslParser.RULE_WS; }
@Override protected int getBeginTokenType() { return InternalMyDslParser.RULE_OPEN; }
@Override protected int getEndTokenType() { return InternalMyDslParser.RULE_CLOSE; }
3. M A R K T O K E N S A S S P L I T TA B L E
![Page 36: Building a Python IDE with Xtext](https://reader033.vdocument.in/reader033/viewer/2022052301/55abe6c81a28abba678b4594/html5/thumbnails/36.jpg)
@Override protected boolean shouldEmitPendingEndTokens() { return false; }
4. M A R K T O K E N S A S S P L I T TA B L E A G A I N1. Configure Content-Assist Parser 2. Almost the same, but:
![Page 37: Building a Python IDE with Xtext](https://reader033.vdocument.in/reader033/viewer/2022052301/55abe6c81a28abba678b4594/html5/thumbnails/37.jpg)
CharStream
o.a.IntStream TokenSource
Lexer
AntlrStringStream
P U T T I N G E V E R T H I N G T O G E T H E RI N D E N TA T I O N A W A R E L A N G U A G E S
TokenStream Parser
AbstractSplittingTokenSource
AbstractIndentationTokenSource
![Page 38: Building a Python IDE with Xtext](https://reader033.vdocument.in/reader033/viewer/2022052301/55abe6c81a28abba678b4594/html5/thumbnails/38.jpg)
O N E M O R E T H I N GI N D E N TA T I O N A W A R E L A N G U A G E S
![Page 39: Building a Python IDE with Xtext](https://reader033.vdocument.in/reader033/viewer/2022052301/55abe6c81a28abba678b4594/html5/thumbnails/39.jpg)
def void format(Block block, extension IFormattableDocument document) { val open = block.regionForRuleCallTo(OPENRule) open.prepend[ newLineAndIncreaseIndentation ] val close = block.regionForRuleCallTo(CLOSERule) if (block.isLast) close.append[ decreaseIndentation ] else close.append[ newLineAndDecreaseIndentation ] }
def void format(Block block, extension IFormattableDocument document) { val open = block.regionForRuleCallTo(OPENRule) open.prepend[ newLineAndIncreaseIndentation ] val close = block.regionForRuleCallTo(CLOSERule) if (block.isLast) close.append[ decreaseIndentation ] else close.append[ newLineAndDecreaseIndentation ] }
5. F O R M AT T I N G
1. Indent Blocks 2. Adjust NewLines 3. Remove superfluous spaces
![Page 40: Building a Python IDE with Xtext](https://reader033.vdocument.in/reader033/viewer/2022052301/55abe6c81a28abba678b4594/html5/thumbnails/40.jpg)
PA R S I N G C O N T E N T A S S I S T F O R M AT T I N G F O L D I N G O U T L I N E
D O N EI N D E N TA T I O N A W A R E L A N G U A G E S
![Page 41: Building a Python IDE with Xtext](https://reader033.vdocument.in/reader033/viewer/2022052301/55abe6c81a28abba678b4594/html5/thumbnails/41.jpg)
Q & A