serializing emf models with xtext
DESCRIPTION
as presented at http://www.xtextcon.orgTRANSCRIPT
![Page 1: Serializing EMF models with Xtext](https://reader035.vdocument.in/reader035/viewer/2022081718/5564b0d1d8b42a366c8b5711/html5/thumbnails/1.jpg)
Moritz Eysholdt, itemis AG
Serializing EMF models with Xtext
Xtextcon 2014!Kiel, Germany
![Page 2: Serializing EMF models with Xtext](https://reader035.vdocument.in/reader035/viewer/2022081718/5564b0d1d8b42a366c8b5711/html5/thumbnails/2.jpg)
Agenda❖ Parsing vs. Serializing!
❖ Use Cases!
❖ Generating vs. Serializing!
❖ Challenges!
❖ The Contract!
❖ Architecture!
❖ Hooks!
❖ Advice
![Page 3: Serializing EMF models with Xtext](https://reader035.vdocument.in/reader035/viewer/2022081718/5564b0d1d8b42a366c8b5711/html5/thumbnails/3.jpg)
The New and the Old Serializer
org.eclipse.xtext.serializer• The New Serializer. This presentation is about it.
org.eclipse.xtext.parsetree.reconstr• The Old Serializer. Don’t used it.!• Cryptic Error Messages!• Bad Performance for Large Models!• marked @Deprecated
![Page 4: Serializing EMF models with Xtext](https://reader035.vdocument.in/reader035/viewer/2022081718/5564b0d1d8b42a366c8b5711/html5/thumbnails/4.jpg)
Parser vs. SerializerXtextResource
ModelModelTextual!Model
Parser
Serializer
load()
save()
![Page 5: Serializing EMF models with Xtext](https://reader035.vdocument.in/reader035/viewer/2022081718/5564b0d1d8b42a366c8b5711/html5/thumbnails/5.jpg)
Parser vs. SerializerXtextResource
ModelModelTextual!Model
AST
Parser
Serializer
load()
save()
![Page 6: Serializing EMF models with Xtext](https://reader035.vdocument.in/reader035/viewer/2022081718/5564b0d1d8b42a366c8b5711/html5/thumbnails/6.jpg)
Use CasesQuickfix
Refactoring
Non-Textual !Editors
Non-Textual!Persistence
Generators
![Page 7: Serializing EMF models with Xtext](https://reader035.vdocument.in/reader035/viewer/2022081718/5564b0d1d8b42a366c8b5711/html5/thumbnails/7.jpg)
Use CasesQuickfix
Refactoring
Non-Textual !Editors
Non-Textual!Persistence
Generators
![Page 8: Serializing EMF models with Xtext](https://reader035.vdocument.in/reader035/viewer/2022081718/5564b0d1d8b42a366c8b5711/html5/thumbnails/8.jpg)
Use CasesQuickfix
Non-Textual !Editors
Non-Textual!Persistence
Generators
Refactoring
![Page 9: Serializing EMF models with Xtext](https://reader035.vdocument.in/reader035/viewer/2022081718/5564b0d1d8b42a366c8b5711/html5/thumbnails/9.jpg)
Use CasesQuickfix
Non-Textual!Persistence
Generators
Refactoring
Non-Textual !Editors
EMF Store
<?XML?><?XMI?>
![Page 10: Serializing EMF models with Xtext](https://reader035.vdocument.in/reader035/viewer/2022081718/5564b0d1d8b42a366c8b5711/html5/thumbnails/10.jpg)
Use CasesQuickfix
Non-Textual!Persistence
Generators
Refactoring
Non-Textual !Editors
Read
Transform
Serialize
Generate, Transpile, Compile, Migrate…
![Page 11: Serializing EMF models with Xtext](https://reader035.vdocument.in/reader035/viewer/2022081718/5564b0d1d8b42a366c8b5711/html5/thumbnails/11.jpg)
Why Serialize?
❖ object model usually easier to modify than its textual representation!
❖ guaranteed syntactical correctness!
❖ automatic handling of comments!
❖ automatic handling of whitespace
…when I have Xtend, THE language for generators
![Page 12: Serializing EMF models with Xtext](https://reader035.vdocument.in/reader035/viewer/2022081718/5564b0d1d8b42a366c8b5711/html5/thumbnails/12.jpg)
When not to Serialize
❖ Avoid serializing models that are broken due to parse errors.!
❖ Template languages are simpler when writing out large text chunks that never change.
![Page 13: Serializing EMF models with Xtext](https://reader035.vdocument.in/reader035/viewer/2022081718/5564b0d1d8b42a366c8b5711/html5/thumbnails/13.jpg)
The hard Contract
Parsing a serialized Model must result !in a model equal to the original.
Model original = createModel()String serialized = serialize(original)Model parsed = parse(serialized)assertEqual(original, parsed)
…enables textual models as persistence format
![Page 14: Serializing EMF models with Xtext](https://reader035.vdocument.in/reader035/viewer/2022081718/5564b0d1d8b42a366c8b5711/html5/thumbnails/14.jpg)
The soft Contract
Serializing a model that has been modified after parsing!should only change the smallest number of characters necessary.
…keep diffs small
String originalDoc = loadDocument()Model parsed = parse(originalDoc)Model modified = applyModifications(parsed)String newDoc = serialize(modified)int numberOfChars = diff(originalDoc, newDoc).size()
1. Keep textual diffs small.!2. Strictly comply with the semantic model!3. Loosely comply with the node model
![Page 15: Serializing EMF models with Xtext](https://reader035.vdocument.in/reader035/viewer/2022081718/5564b0d1d8b42a366c8b5711/html5/thumbnails/15.jpg)
Challenges
❖ unassigned elements!
❖ ambiguous mapping from EStructuralFeatures to Assignments!
❖ ambiguous mapping from EClasses to ParserRules and Actions
…because the AST is actually abstract, as in “lack of information”
![Page 16: Serializing EMF models with Xtext](https://reader035.vdocument.in/reader035/viewer/2022081718/5564b0d1d8b42a366c8b5711/html5/thumbnails/16.jpg)
API Frontends #1
package org.eclipse.xtext.resource;!public class XtextResource extends ResourceImpl {! (…)! public void save(Map<?, ?> options) throws IOException {}! public final void save(OutputStream outputStream, Map<?, ?> options) throws IOException {}}
![Page 17: Serializing EMF models with Xtext](https://reader035.vdocument.in/reader035/viewer/2022081718/5564b0d1d8b42a366c8b5711/html5/thumbnails/17.jpg)
API Frontends #2package org.eclipse.xtext.serializer;!@ImplementedBy(Serializer.class)public interface ISerializer { public String serialize(EObject obj); public String serialize(EObject obj, SaveOptions options); public void serialize(EObject obj, Writer writer, SaveOptions options) throws IOException; public ReplaceRegion serializeReplacement(EObject obj, SaveOptions options);}
Options:!- format: !
- true: format all!- false: format regions without whitespace information!
- validate: don’t use, it’s an old algorithm.
![Page 18: Serializing EMF models with Xtext](https://reader035.vdocument.in/reader035/viewer/2022081718/5564b0d1d8b42a366c8b5711/html5/thumbnails/18.jpg)
Before SerializationCreate one State Machine per Context per EClass
Root: "optional"? ID children+=(List | Path);!List returns Child: "list" item+=ID ("," item+=ID)*;!Path returns Child: "path" seg=ID ({Segment.parent=current} seg=ID)+;
Context EClassRoot RootList ChildPath SegmentPath_Segment_2_0 ChildPath_Segment_2_0 Segment
![Page 19: Serializing EMF models with Xtext](https://reader035.vdocument.in/reader035/viewer/2022081718/5564b0d1d8b42a366c8b5711/html5/thumbnails/19.jpg)
During Serialization #1Find right state machines and !find right path through them
Root: "optional"? ID children+=(List | Path);!List returns Child: "list" item+=ID ("," item+=ID)*;
Context EClassRoot Root
Context EClassList Child
optional Foo list a, b, cOutput:
![Page 20: Serializing EMF models with Xtext](https://reader035.vdocument.in/reader035/viewer/2022081718/5564b0d1d8b42a366c8b5711/html5/thumbnails/20.jpg)
Architecture #3
Context EClassRoot Root
Context EClassList Child
Context EClassRoot Root
Context EClassList Child
Semantic!assigned grammar elements
Syntactic!unassigned grammar elements
optional Foo list a, b, cOutput:
![Page 21: Serializing EMF models with Xtext](https://reader035.vdocument.in/reader035/viewer/2022081718/5564b0d1d8b42a366c8b5711/html5/thumbnails/21.jpg)
Architecture #4: Observer
Semantic!Sequencer
Syntactic!Sequencer
HiddenToken!Sequencer Formatter Writer
Events
Listens To
assigend!* RuleCalls!* Terminals!* DataTypes!* Keywords!* CrossRefs
unassigned!* RuleCalls!* Terminals!* DataTypes!* Keywords
* whitespace!* comments
modifies!whitespace
writes to !stream
![Page 22: Serializing EMF models with Xtext](https://reader035.vdocument.in/reader035/viewer/2022081718/5564b0d1d8b42a366c8b5711/html5/thumbnails/22.jpg)
Output: /*X*/ Foo list a
Serializer HiddenTokenSequencerSyntacticSequencerSemanticSequencer
createSequence(Root, Root)
enterAssignedParserRuleCall(children=List)
enterAssignedParserRuleCall(children=List)
createSequence(List, Children)
acceptUnassignedRuleCall(ID)
acceptAssignedTerminalRuleCall(itemi=ID)
acceptKeyword("list")
acceptAssignedTerminalRuleCall(itemi=ID)
leaveAssignedParserRuleCall(children=List)
leaveAssignedParserRuleCall(children=List)
Formatter Writer
enterAssignedParserRuleCall()
acceptWhitespace()
acceptWhitespace()
acceptComment(/*X*/)
acceptUnassignedRuleCall()
acceptKeyword()
acceptWhitespace()
acceptAssignedTerminalRuleCall()
acceptWhitespace()
enterAssignedParserRuleCall()
Root: "optional"? ID children+=(List | Path);
List returns Child: "list" item+=ID ("," item+=ID)*;
![Page 23: Serializing EMF models with Xtext](https://reader035.vdocument.in/reader035/viewer/2022081718/5564b0d1d8b42a366c8b5711/html5/thumbnails/23.jpg)
SerializerFragment
fragment = serializer.SerializerFragment auto-inject { generateStub = true // generateDebugData = true}
• SerializerFragment not required to use serializer!!• Only purpose of SerializerFragment is to generate convenience API!• generateDebugData to generate pretty state machine diagrams (graphviz dot)
![Page 24: Serializing EMF models with Xtext](https://reader035.vdocument.in/reader035/viewer/2022081718/5564b0d1d8b42a366c8b5711/html5/thumbnails/24.jpg)
Hooks #1: ITransientValueService• selectively exclude model-objects and -values from serialization!• transient == NOT serialized!• Default setting from Ecore model: EStructuralFeature.isTransient()
public interface ITransientValueService {! enum ListTransient { NO, SOME, YES }! enum ValueTransient { NO, PREFERABLY, YES }! public ListTransient isListTransient(EObject semanticObject, EStructuralFeature feature);! public boolean isValueInListTransient(EObject semanticObject, int index, EStructuralFeature feature);! public ValueTransient isValueTransient(EObject semanticObject, EStructuralFeature feature);}
![Page 25: Serializing EMF models with Xtext](https://reader035.vdocument.in/reader035/viewer/2022081718/5564b0d1d8b42a366c8b5711/html5/thumbnails/25.jpg)
Hooks #2: Token Serializationpublic interface ICrossReferenceSerializer { boolean isValid(EObject context, CrossReference crossref, EObject target, INode node); String serializeCrossRef(EObject context, CrossReference crossref, EObject target, INode node);}
public interface IKeywordSerializer { boolean isValid(EObject context, Keyword keyword, Object value); String serializeAssignedKeyword(EObject context, Keyword keyword, Object value, INode node);}
public interface IValueSerializer { boolean isValid(EObject context, RuleCall ruleCall, Object value); String serializeAssignedValue(EObject context, RuleCall ruleCall, Object value, INode node);}
public interface IEnumLiteralSerializer { boolean isValid(EObject context, RuleCall ruleCall, Object value); String serializeAssignedEnumLiteral(EObject context, RuleCall ruleCall, Object value, INode node);}
![Page 26: Serializing EMF models with Xtext](https://reader035.vdocument.in/reader035/viewer/2022081718/5564b0d1d8b42a366c8b5711/html5/thumbnails/26.jpg)
Hooks #3: SemanticSequencerclass SerSipSemanticSequencer extends AbstractSerSipSemanticSequencer {! @Inject SerSipGrammarAccess grammarAccess;! /** * Constraint: * (children+=Path | children+=List) */ override protected sequence_Root(EObject context, Root semanticObject) { val feeder = createSequencerFeeder(semanticObject) for (child : semanticObject.children) { if (child.seg != null) { feeder.accept(grammarAccess.rootAccess.childrenPathParserRuleCall_2_0_0, child) } else { feeder.accept(grammarAccess.rootAccess.childrenListParserRuleCall_2_0_1, child) } } feeder.finish() }}
Root: "optional"? ID children+=(Path | List);!List returns Child: "list" item+=ID ("," item+=ID)*;!Path returns Child: "path" seg=ID ({Segment.parent=current} seg=ID)+;
![Page 27: Serializing EMF models with Xtext](https://reader035.vdocument.in/reader035/viewer/2022081718/5564b0d1d8b42a366c8b5711/html5/thumbnails/27.jpg)
Hooks #4: SyntacticSequencerclass SerSipSyntacticSequencer extends AbstractSerSipSyntacticSequencer {! /** * terminal ID : '^'?('a'..'z'|'A'..'Z'|'_') ('a'..'z'|'A'..'Z'|'_'|'0'..'9')*; */ override getIDToken(EObject semanticObject, RuleCall ruleCall, INode node) { if (node != null) return getTokenText(node); return ""; }! /** * Syntax: * 'optional'? */ override emit_Root_OptionalKeyword_0_q(EObject semanticObject, ISynNavigable transition, List<INode> nodes) { acceptNodes(transition, nodes); }}
Root: "optional"? ID children+=(Path | List);
![Page 28: Serializing EMF models with Xtext](https://reader035.vdocument.in/reader035/viewer/2022081718/5564b0d1d8b42a366c8b5711/html5/thumbnails/28.jpg)
Understanding Ambiguities• A grammar is ambiguous for the serializer if there can be more
than one textual syntax for a given model.!!
• A grammar is ambiguous for the parser if there can be more than one model for a given textual syntax1.
R1: foo=ID | bar=ID;
R2: (“foo” val=ID) | (“bar” val=ID);
Does “a” parse
to or ?foo=a bar=b
Does serialize to
“foo b” or “bar b”?
val=b
pars
ese
rializ
e
1. that is not completely correct. A grammar is ambiguous when there is more than one path to consume a given input. That, however, usually leads to different models.
![Page 29: Serializing EMF models with Xtext](https://reader035.vdocument.in/reader035/viewer/2022081718/5564b0d1d8b42a366c8b5711/html5/thumbnails/29.jpg)
Understanding Ambiguities
Root: (foos=Foo | bars=Bar)*;!Foo: “foo” foo=ID;!Bar: “bar” bar=ID;
(1) .
(2) bar b foo a(3) foo a bar b
Rootfoo=a bar=b
Does (1) serialize to (2) or (3)?
![Page 30: Serializing EMF models with Xtext](https://reader035.vdocument.in/reader035/viewer/2022081718/5564b0d1d8b42a366c8b5711/html5/thumbnails/30.jpg)
Avoiding AmbiguitiesAnalysis: Grammars can be ambiguous for the serializer.!!Example:
Root: (foos=Foo | bars=Bar)*;!Foo: “foo” foo=ID;!Bar: “bar” bar=ID;
Root: members+=Member;!Member: Foo | Bar;!Foo: “foo” foo=ID;!Bar: “bar” bar=ID;
• Single “member” list to maintain order!• “Foo” and “Bar” extend “Member”!• implement getFoo() and getBar() as filters on getMembers()
![Page 31: Serializing EMF models with Xtext](https://reader035.vdocument.in/reader035/viewer/2022081718/5564b0d1d8b42a366c8b5711/html5/thumbnails/31.jpg)
Avoid Unserializeable ModelsThe Xtext grammar can imply constraints on your model. !!Serialization is only possible if the model complies with theses constraints because the grammar does not define syntax for models that don’t comply. Examples:
GRAMMAR CONSTRAINT
R1: name=ID; name != null
R2: (name=ID | title=STING); (name != null) ^ (title != null)
R3: items+=ID+; items.size() >= 1
R4: (a+=ID b+=ID)*; a.size() == b.size()
![Page 32: Serializing EMF models with Xtext](https://reader035.vdocument.in/reader035/viewer/2022081718/5564b0d1d8b42a366c8b5711/html5/thumbnails/32.jpg)
Avoid Unserializeable ModelsSolution:!!a) Ensure your grammar does not imply constraints. !b) Ensure your TransientValueService prevents constraint violations.!Implement Xtext Validation to enforce constraints.!!This will also improve your error messages but may make content assist too chatty.
RESTRICTIVE SAFE
R1: name=ID; R1: name=ID?;R1: (name=ID | “?”);
R2: (name=ID | title=STING); R2: name=ID? title=STING?;R3: items+=ID+; R3: items+=ID*;
R4: (a+=ID b+=ID)*;R4: (a+=ID b+=ID)* (“spareA={“ a+=ID+ “}”)? (“spareB={“ b+=ID+ “}”)?;
![Page 33: Serializing EMF models with Xtext](https://reader035.vdocument.in/reader035/viewer/2022081718/5564b0d1d8b42a366c8b5711/html5/thumbnails/33.jpg)
Configure Your Scope RightYou can (and need) to configure global scoping for a ResourceSet!
XtextLiveScopeResourceSetProviderXtextResourceSetProvider
ResourceSet
Dirty Editor State
Index
(default) (what you need)
<shadows>
<shadows>
Dirty Editor State
Index
<shadows>
![Page 34: Serializing EMF models with Xtext](https://reader035.vdocument.in/reader035/viewer/2022081718/5564b0d1d8b42a366c8b5711/html5/thumbnails/34.jpg)
Happy Serializing!