texmlbus - tug.org

<TeX>

texmlbus

H. Stamerjohanns

TUG, August 2021

H. Stamerjohanns texmlbus TUG, August 2021 1 / 35

<TeX>

Outline

1 MotivationLATEX and XMLWhy convert to XML?Previous projectstexmlbus: change of focus

2 WebsiteBuild system

3 Summary


<TeX>

Just use LATEXLATEX is the format to write math

I millions of scientific publications have beenwritten using LATEX

I best way to produce high quality mathtypesetting

drawbacksI mixes form and contentI no real semanticsI style files change over timeI no formal validationI long term preservation?


<TeX>

Conversion to XML

XMLI not something you want to directly editI document can be validated

⇒ possible archive formatI JATS Journal Article Text Suite

I MathML, XHTML⇒ render document directly in web browser

I easier for searching and indexing tools,screenreaders


<TeX>

Project based on...

arxivml build systemI written at Jacobs University BremenI use LaTeXML to create XMLI mass conversion to XHTML

≈ 500.000 documents converted

I create real-world MathML⇒ improve LaTeXML1

1B. Miller and D. Ginev, https://dlmf.nist.gov/LaTeXML/H. Stamerjohanns texmlbus TUG, August 2021 5 / 35

https://dlmf.nist.gov/LaTeXML/

<TeX>

Build system

I open source (MIT licence)I implemented in scripting language (here php)I uses SQL database to store state

I distributes jobs on several hostsI sets timeout for each jobI analyzes conversion process

checks filesparses the result files (stderr.log)classifies results

I stores information in DB


<TeX>

texmlbus: change of focusI easy installation

use Docker images

I more interactivityupload files via browserupload files via browserupload files via browser

I other targets than XHTMLresult table for each target

I create same target using different systemsintroduce stages (target combined with image)needs subdirectories for each stage


<TeX>


⇒ use Docker images

I more interactivityupload files via browserupload files via browserupload files via browser




<TeX>



I more interactivity⇒ upload files via browser⇒ import files directly from Overleafupload files via browser




<TeX>



I more interactivity⇒ upload files via browser⇒ import files directly from Overleaf⇒ schedule jobs via browser




<TeX>




I other targets than XHTML⇒ result table for each target



<TeX>




I other targets than XHTML⇒ result table for each target

I create same target using different systems⇒ introduce stages (target combined with image)⇒ needs subdirectories for each stage)


<TeX>

texmlbus build system

1

Interactive usage via browser

- add and remove documents- queue conversion jobs - manages document files

SQL database

- workqueue - statistics

Document files(each article

in subdirectory)

Worker containers

- handle conversion - call latexml and latexmlpost or conversion commands

Build Manager

- operates on workqueue- schedules make jobs on worker containers- analyzes log files

manages entries in workqueue - state and priority

reads workqueue

invokes make stores analyzed

log data

default: create result files

and log files

reads log files


<TeX>


3



SQL database



in subdirectory)

Worker containers


Build Manager



reads workqueue


log data


and log files

reads log files


<TeX>


4



SQL database



in subdirectory)

Worker containers


Build Manager



reads workqueue


log data


and log files

reads log files


<TeX>


5



SQL database



in subdirectory)

Worker containers


Build Manager



reads workqueue


log data


and log files

reads log files


<TeX>


6



SQL database



in subdirectory)

Worker containers


Build Manager



reads workqueue


log data


and log files

reads log files


<TeX>


7



SQL database



in subdirectory)

Worker containers


Build Manager



reads workqueue


log data


and log files

reads log files


<TeX>


8



SQL database



in subdirectory)

Worker containers


Build Manager



reads workqueue


log data


and log files

reads log files


<TeX>


9



SQL database



in subdirectory)

Worker containers


Build Manager



reads workqueue


log data


and log files

reads log files


<TeX>


10



SQL database



in subdirectory)

Worker containers


Build Manager



reads workqueue


log data

create result files

and log files

reads log files


<TeX>


11



SQL database



in subdirectory)

Worker containers


Build Manager



reads workqueue


log data


and log files

reads log files


<TeX>


12



SQL database



in subdirectory)

Worker containers


Build Manager



reads workqueue

invokes make

stores analyzed log data


and log files

reads log files


<TeX>


13



SQL database



in subdirectory)

Worker containers


Build Manager



reads workqueue


log data


and log files

reads log files

Webserver / PHP

MySQL DB LaTeXML

Shared Volume

Document repository


<TeX>


MySQL DB Build SystemPHP

LaTeXMLworker

LaTeXMLworker

LaTeXMLworker

docker-compose

SharedVolume


<TeX>



<TeX>

The web interfaceResult statistics


<TeX>

Summary

I texmlbus allows to convert documents andgather statistics about conversions

I especially useful to detect regressions withreal-world documents

I stages allow to have same targets usingdifferent systems

I supports any converter


<TeX>

Outlook

Things to be done

I add converters more easily

I help to improve LaTeXML


<TeX>

texmlbus

https://github.com/stamer/texmlbus

Thanks to Overleaf for their support!


https://github.com/stamer/texmlbus

texmlbus - tug.org

Documents