![Page 1: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/1.jpg)
Who needs Pandoc when you have Sphinx?An exploration of the parsers and builders of the Sphinx documentation tool
FOSDEM 2019
@stephenfin
![Page 2: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/2.jpg)
reStructuredText, Docutils &Sphinx
1
![Page 3: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/3.jpg)
A little reStructuredText=========================
This document demonstrates some basic features of |rst|. You can use **bold** and *italics*, along with ``literals``. It’s quite similar to `Markdown`_ but much more extensible. CommonMark may one day approach this [1]_, but today is not that day. `Docutils`__ does all this for us.
.. |rst| replace:: **reStructuredText**
.. _Markdown: https://daringfireball.net/projects/markdown/
.. [1] https://talk.commonmark.org/t/444__ http://docutils.sourceforge.net/
💾 intro.rst
![Page 4: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/4.jpg)
A little reStructuredText=========================
This document demonstrates some basic features of |rst|. You can use **bold** and *italics*, along with ``literals``. It’s quite similar to `Markdown`_ but much more extensible. CommonMark may one day approach this [1]_, but today is not that day. `Docutils`__ does all this for us.
.. |rst| replace:: **reStructuredText**
.. _Markdown: https://daringfireball.net/projects/markdown/
.. [1] https://talk.commonmark.org/t/444__ http://docutils.sourceforge.net/
💾 intro.rst
![Page 5: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/5.jpg)
A little reStructuredText
This document demonstrates some basic features of reStructuredText. You can use bold and italics, along with literals. It’s quite similar to Markdown but much more extensible. CommonMark may one day approach this [1], but today is not that day.
Docutils does all this for us.
[1] https://talk.commonmark.org/t/444/
💾 intro.html
![Page 6: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/6.jpg)
A little more reStructuredText==============================The extensibility really comes into play with directives androles. We can do things like link to RFCs (:RFC:`2324`, anyone?)or generate some more advanced formatting (I do love me someH\ :sub:`2`\ O).
.. warning::
The power can be intoxicating.
Of course, all the stuff we showed previously *still works!* The only limit is your imagination/interest.
💾 more.rst
![Page 7: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/7.jpg)
A little more reStructuredText==============================The extensibility really comes into play with directives androles. We can do things like link to RFCs (:RFC:`2324`, anyone?)or generate some more advanced formatting (I do love me someH\ :sub:`2`\ O).
.. warning::
The power can be intoxicating.
Of course, all the stuff we showed previously *still works!* The only limit is your imagination/interest.
💾 more.rst
![Page 8: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/8.jpg)
A little more reStructuredText
The extensibility really comes into play with directives and roles. We can do things
like link to RFCs (RFC 2324, anyone?) or generate some more advanced formatting(I do love me some H
2O).
WarningThe power can be intoxicating.
Of course, all the stuff we showed previously still works! The only limit is your imagination/interest.
💾 more.html
![Page 9: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/9.jpg)
reStructuredText provides the syntax
Docutils provides the parsing and file generation
![Page 10: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/10.jpg)
reStructuredText provides the syntax
Docutils provides the parsing and file generation
Sphinx provides the cross-referencing
![Page 11: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/11.jpg)
Docutils use readers, parsers, transforms, and writers
Docutils works with individual files
![Page 12: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/12.jpg)
Docutils use readers, parsers, transforms, and writers
Docutils works with individual files
Sphinx uses readers, parsers, transforms, writers and builders
Sphinx works with multiple, cross-referenced files
![Page 13: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/13.jpg)
How Does Docutils Work?
2
![Page 14: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/14.jpg)
About me========
Hello, world. I am **bold** and *maybe* I am brave.
💾 index.rst
![Page 15: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/15.jpg)
$ rst2html index.rst
![Page 16: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/16.jpg)
About me
Hello, world. I am bold and maybe I am brave.
💾 index.html
![Page 17: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/17.jpg)
index.rst index.html
![Page 18: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/18.jpg)
$ rst2pseudoxml index.rst
![Page 19: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/19.jpg)
<document ids="about-me" names="about\ me" source="index.rst" title="About me">
<title>About me
<paragraph>Hello, world. I am<strong>
boldand<emphasis>
maybeI am brave.
💾 index.xml
![Page 20: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/20.jpg)
$ ./docutils/tools/quicktest.py index.rst
![Page 21: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/21.jpg)
<document source="index.rst"><section ids="about-me" names="about\ me">
<title>About me
<paragraph>Hello, world. I am<strong>
boldand<emphasis>
maybeI am brave.
💾 index.xml
![Page 22: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/22.jpg)
Readers (reads from source and passes to the parser)
Parsers (creates a doctree model from the read file)
Transforms (add to, prune, or otherwise change the doctree model)
Writers (converts the doctree model to a file)
![Page 23: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/23.jpg)
Readers (reads from source and passes to the parser)
Parsers (creates a doctree model from the read file)
Transforms (add to, prune, or otherwise change the doctree model)
Writers (converts the doctree model to a file)
![Page 24: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/24.jpg)
What About Sphinx?
3
![Page 25: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/25.jpg)
About me========
Hello, world. I am **bold** and *maybe* I am brave.
💾 index.rst
![Page 26: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/26.jpg)
master_doc = 'index'
💾 conf.py
![Page 27: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/27.jpg)
$ sphinx-build -b html . _build
![Page 28: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/28.jpg)
About me
Hello, world. I am bold and maybe I am brave.
💾 index.html
![Page 29: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/29.jpg)
Readers (reads from source and passes to the parser)
Parsers (creates a doctree model from the read file)
Transforms (add to, prune, or otherwise change the doctree model)
Writers (converts the doctree model to a file)
![Page 30: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/30.jpg)
Builders (call the readers, parsers, transformers, writers)
Application (calls the builder(s))
Environment (store information for future builds)
![Page 31: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/31.jpg)
Builders (call the readers, parsers, transformers, writers)
Application (calls the builder(s))
Environment (store information for future builds)
![Page 32: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/32.jpg)
...updating environment: 1 added, 0 changed, 0 removedreading sources... [100%] indexlooking for now-outdated files... none foundpickling environment... donechecking consistency... donepreparing documents... donegenerating indices... donewriting additional pages... donecopying static files... donecopying extra files... donedumping search index in English (code: en) ... donedumping object inventory... donebuild succeeded.
![Page 33: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/33.jpg)
Docutils provides almost 100 node types
documentsectiontitle
subtitleparagraph
block_quotebullet_list
note...
(the root element of the document tree)(the main unit of hierarchy for documents)(stores the title of a document, section, ...)(stores the subtitle of a document)(contains the text and inline elements of a single paragraph)(used for quotations set off from the main text)(contains list_item elements marked with bullets)(an admonition, a distinctive and self-contained notice)...
![Page 34: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/34.jpg)
Sphinx provides its own custom node types
translatablenot_smartquotable
toctreeversionmodified
seealsoproductionlist
manpagepending_xref
...
(indicates content which supports translation)(indicates content which does not support smart-quotes)(node for inserting a "TOC tree")(version change entry)(custom "see also" admonition)(grammar production lists)(reference to a man page)(cross-reference that cannot be resolved yet)...
![Page 35: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/35.jpg)
Docutils provides dozens of transforms
DocTitleDocInfoSectNumContentsFootnotesMessages
SmartQuotesAdmonitions
...
(promote title elements to the document level)(transform initial field lists to docinfo elements)(assign numbers to the titles of document sections)(generate a table of contents from a document or sub-node)(resolve links to footnotes, citations and their references)(place system messages into the document)(replace ASCII quotation marks with typographic form)(transform specific admonitions to generic ones)...
![Page 36: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/36.jpg)
Sphinx also provides additional transforms
MoveModuleTargetsAutoNumbering
CitationReferencesSphinxSmartQuotesDoctreeReadEvent
ManpageLinkSphinxDomains
Locale...
(promote initial module targets to the section title)(register IDs of tables, figures and literal blocks to assign numbers)(replace citation references with pending_xref nodes)(custom SmartQuotes to avoid transform for some extra node types)(emit doctree-read event)(find manpage section numbers and names)(collect objects to Sphinx domains for cross referencing)(replace translatable nodes with their translated doctree)...
![Page 37: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/37.jpg)
Using Additional Parsers
4
![Page 38: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/38.jpg)
There are a number of parsers available
reStructuredText (part of docutils)
Markdown (part of recommonmark)
Jupyter Notebooks (part of nbsphinx)
![Page 39: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/39.jpg)
# About me
Hello, world. I am **bold** and *maybe* I am brave.
💾 index.md
![Page 40: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/40.jpg)
$ cm2html index.md
![Page 41: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/41.jpg)
About me
Hello, world. I am bold and maybe I am brave.
💾 index.html
![Page 42: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/42.jpg)
$ cm2pseudoxml index.md
![Page 43: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/43.jpg)
<document ids="about-me" names="about\ me" source="index.md" title="About me">
<title>About me
<paragraph>Hello, world. I am<strong>
boldand<emphasis>
maybeI am brave.
💾 index.xml
![Page 44: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/44.jpg)
# About me
Hello, world. I am **bold** and *maybe* I am brave.
💾 index.md
![Page 45: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/45.jpg)
from recommonmark.parser import CommonMarkParser
master_doc = 'index'
source_parsers = {'.md': CommonMarkParser}source_suffix = '.md'
💾 conf.py
![Page 46: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/46.jpg)
from recommonmark.parser import CommonMarkParser
master_doc = 'index'
source_parsers = {'.md': CommonMarkParser}source_suffix = '.md'
💾 conf.py
![Page 47: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/47.jpg)
$ sphinx-build -b html . _build
![Page 48: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/48.jpg)
About me
Hello, world. I am bold and maybe I am brave.
💾 index.html
![Page 49: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/49.jpg)
Using Additional Writers, Builders
5
![Page 50: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/50.jpg)
Docutils provides a number of in-tree writers
docutils_xmlhtml4css1latex2emanpage
nullodf_odtpep_htmlpseudoxml
...
(simple XML document tree Writer)(simple HTML document tree Writer)(LaTeX2e document tree Writer)(simple man page Writer)(a do-nothing Writer)(ODF Writer)(PEP HTML Writer)(simple internal document tree Writer)...
![Page 51: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/51.jpg)
$ rst2html5 index.rst
![Page 52: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/52.jpg)
from docutils.core import publish_filefrom docutils.writers import html5_polyglot
with open('README.rst', 'r') as source:publish_file(source=source,
writer=html5_polyglot.Writer())
![Page 53: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/53.jpg)
$ pip install rst2txt
![Page 54: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/54.jpg)
$ rst2txt index.rst
![Page 55: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/55.jpg)
from docutils.core import publish_filefrom rst2txt
with open('README.rst', 'r') as source:publish_file(source=source,
writer=rst2txt.Writer())
![Page 56: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/56.jpg)
htmlqthelpepublatextextman
texinfoxml...
(generates output in HTML format)(like html but also generates Qt help collection support files)(like html but also generates an epub file for eBook readers)(generates output in LaTeX format)(generates text files with most rST markup removed)(generates manual pages in the groff format)(generates textinfo files for use with makeinfo)(generates Docutils-native XML files)...
Sphinx provides its own in-tree builders
![Page 57: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/57.jpg)
$ sphinx-build -b html . _build
![Page 58: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/58.jpg)
$ pip install sphinx-asciidoc
![Page 59: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/59.jpg)
$ sphinx-build -b asciidoc . _build
![Page 60: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/60.jpg)
Writing Your Own Parsers, Writers
6
![Page 61: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/61.jpg)
Reading (reads from source and passes to the parser)
Parsing (creates a doctree model from the read file)
Transforming (applies transforms to the doctree model)
Writing (converts the doctree model to a file)
![Page 62: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/62.jpg)
from docutils import parsers
class Parser(parsers.Parser): supported = ('null',) config_section = 'null parser' config_section_dependencies = ('parsers',)
def parse(self, inputstring, document): pass
💾 docutils/parsers/null.py
![Page 63: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/63.jpg)
We’re not covering Compilers 101
![Page 64: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/64.jpg)
We’re not covering Compilers 101
We’re going to cheat 😄
![Page 65: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/65.jpg)
<?xml version="1.0" encoding="utf-8"?><document source="index.rst">
<section ids="about-me" names="about\ me"><title>About me</title><paragraph>Hello, world. I am <strong>bold</strong> and <emphasis>maybe</emphasis> I am brave.</paragraph>
</section></document>
💾 index.xml
![Page 66: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/66.jpg)
from docutils import parsersimport xml.etree.ElementTree as ET
class Parser(parsers.Parser): supported = ('xml',) config_section = 'XML parser' config_section_dependencies = ('parsers',)
def parse(self, inputstring, document): xml = ET.fromstring(inputstring) self._parse(document, xml)
...
💾 xml_parser.py
![Page 67: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/67.jpg)
...
def _parse(self, node, xml): for attrib, value in xml.attrib.items():
# NOTE(stephenfin): this isn't complete! setattr(node, attrib, value)
for child in xml: child_node = getattr(nodes, child.tag)(text=child.text) node += self._parse(child_node, child)
if xml.tail: return node, nodes.Text(xml.tail) return node
💾 xml_parser.py
![Page 68: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/68.jpg)
Reading (reads from source and passes to the parser)
Parsing (creates a doctree model from the read file)
Transforming (applies transforms to the doctree model)
Writing (converts the doctree model to a file)
![Page 69: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/69.jpg)
from docutils import writers
class Writer(writers.Writer): supported = ('pprint', 'pformat', 'pseudoxml') config_section = 'pseudoxml writer' config_section_dependencies = ('writers',) output = None
def translate(self): self.output = self.document.pformat()
💾 docutils/writers/pseudoxml.py
![Page 70: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/70.jpg)
from docutils import writers
class Writer(writers.Writer): supported = ('pprint', 'pformat', 'pseudoxml') config_section = 'pseudoxml writer' config_section_dependencies = ('writers',) output = None
def translate(self): self.output = self.document.pformat()
💾 docutils/writers/pseudoxml.py
![Page 71: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/71.jpg)
from docutils import nodes, writers
class TextWriter(writers.Writer): supported = ('text',) config_section = 'text writer' config_section_dependencies = ('writers',) output = None
def translate(self): visitor = TextTranslator(self.document) self.document.walkabout(visitor) self.output = visitor.body
💾 rst2txt/writer.py
![Page 72: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/72.jpg)
from docutils import nodes, writers
class TextWriter(writers.Writer): supported = ('text',) config_section = 'text writer' config_section_dependencies = ('writers',) output = None
def translate(self): visitor = TextTranslator(self.document) self.document.walkabout(visitor) self.output = visitor.body
💾 rst2txt/writer.py
![Page 73: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/73.jpg)
...
class TextTranslator(nodes.NodeVisitor): ...
def visit_document(self, node): pass
def depart_document(self, node): pass
def visit_section(self, node): pass
💾 rst2txt/writer.py
![Page 74: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/74.jpg)
from sphinx.builders import Builder
class TextBuilder(Builder): name = 'text'
def __init__(self): pass
def get_outdated_docs(self): pass
def get_target_uri(self): pass
💾 sphinx/builders/text.py
![Page 75: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/75.jpg)
...
def prepare_writing(self, docnames): pass
def write_doc(self, docnames, doctree): pass
def finish(self): pass
💾 sphinx/builders/text.py
![Page 76: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/76.jpg)
Wrap Up
6
![Page 77: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/77.jpg)
Sphinx and Docutils share most of the same architecture…
Readers
Parsers
Transforms
Writers
![Page 78: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/78.jpg)
…but Sphinx builds upon and extends Docutils’ core functionality
Builders
Application
Environment
![Page 79: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/79.jpg)
There are multiple writers/builders provided by both…
HTML
Manpage
LaTeX
XML
texinfo (Sphinx only)
ODF (Docutils only)
...
![Page 80: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/80.jpg)
...and many more writers/builders available along with readers
Markdown (reader and builder)
Text (writer)
ODF (builder)
AsciiDoc (builder)
EPUB2 (builder)
reStructuredText (builder)
...
![Page 81: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/81.jpg)
It’s possible to write your own
![Page 82: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/82.jpg)
It’s possible to write your own
![Page 83: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/83.jpg)
Fin
🎉
![Page 84: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/84.jpg)
Who needs Pandoc when you have Sphinx?An exploration of the parsers and builders of the Sphinx documentation tool
FOSDEM 2019
@stephenfin
![Page 85: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/85.jpg)
Useful Packages and Tools● recommonmark (provides a Markdown reader)● sphinx-markdown-builder (provides a Markdown builder)● sphinx-asciidoc (provides an AsciiDoc builder)● rst2txt (provides a plain text writer)● asciidoclive.com (online AsciiDoc Editor)● rst.ninjs.org (online rST Editor)
![Page 86: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/86.jpg)
References● Quick reStructuredText● Docutils Reference Guide
○ reStructuredText Markup Specification
○ reStructuredText Directives
○ reStructuredText Interpreted Text Roles
● Docutils Hacker’s Guide● PEP-258: Docutils Design Specification
![Page 87: Who needs Pandoc when you have - FOSDEM · Who needs Pandoc when you have Sphinx? An exploration of the parsers and builders of the Sphinx documentation tool FOSDEM 2019 @stephenfin](https://reader035.vdocument.in/reader035/viewer/2022063013/5fcdef6f8e5f9f28974e1d7f/html5/thumbnails/87.jpg)
References● A brief tutorial on parsing reStructuredText (reST) -- Eli Bendersky● A lion, a head, and a dash of YAML -- Stephen Finucane (🌟)● OpenStack + Sphinx In A Tree -- Stephen Finucane (🌟)● Read the Docs & Sphinx now support Commonmark -- Read the Docs Blog