Content-based Interoperability Beyond Technical Specifica-tions of InterfacesTobias Schweizer Lukas Rosenthaler Peter Fornaro Digital Humanities Lab University of Basel Basel Switzerland

AbstractOn the technical level the International Image Interoper-

ability Framework (IIIF) offers a common standard for referringto images or parts of images using a well-defined URL-based syn-tax This allows transformations like scaling rotation mirroringand format conversions to be applied However for the full inte-gration of digital objects in other environments interoperabilityhas to go beyond mere technical specifications for access Non-technical metadata such as image descriptions annotations andreferences to other objects (eg texts transcriptions and othervisual representations) must be available as well Using the ex-ample of digital facsimiles of the scientific notebook of Jacob IBernoulli and its transcriptions we show how XML together withthe IIIF Image API allows for a seamless integration and mergingof image and text sources in the same environment

IntroductionAccessibility and interoperability of digital content is be-

coming increasingly important The Image API of the Interna-tional Image Interoperability Framework (IIIF) is an innovativedevelopment in this context It solves the problem of differentnon-standardized APIs for accessing images

Interoperability is crucial for bringing together various dig-ital sources from different repositories into an infrastructure thatallows researchers to annotate digital objects and supports themin gaining new knowledge

However the definition of a standardized access format isonly one important step on the way to achieving this aim Besidesthe definition of technical specifications to access resources prop-erly the ability to merge or compare digital objects themselves isneeded We call this next step ldquocontent-based interoperabilityrdquoWe will demonstrate the importance of this approach with a con-crete use case from the Bernoulli-Euler Online (BEOL) project1

BEOL is an interdisciplinary collaboration between the Bernoulli-Euler Centre and the Digital Humanities Lab (both University ofBasel) focusing on the mathematics influenced by the Bernoullidynasty and Leonhard Euler

Combining a VRE and an Image RepositoryThe Digital Humanities Lab is developing a flexible Virtual

Research Environment (VRE) called KnoraSALSAH2 Knorarefers to the server software SALSAH to the web-based graphi-cal user interface Knora is based on Semantic Web technologies(an RDF triplestore and OWL ontologies) and offers a RESTfulAPI for performing all necessary operations3 The Knora base on-

1httpp3snfchproject-1660722httpsgithubcomdhlab-baselKnora3See documentation on httpwwwknoraorg

tology provides generic abstract definitions that can be used andfurther extended in project-specific ontologies SALSAH runs ina web browser and communicates with Knora using the API Kno-raSALSAH is part of the strategy of the Swiss Data and ServiceCenter for the Humanities for making research data available onthe long term using a lsquokeep-aliversquo archiving strategy [3 8] Re-sources and annotations are stored in the VRErsquos RDF triplestoreor are loaded from an external repository A resource that can beworked with in KnoraSALSAH can be represented by digital im-ages and moving images and marked up texts are also supported

Image-based content however is not stored or maintainedby the VRErsquos triplestore itself Instead it is stored in an im-age repository called Simple Image Processing Interface (Sipi)4

that puts a strong emphasis on preserving the imagesrsquo metadataas required in image archives [4] To make the VRE and the im-age repository interoperable we rely on the IIIF Image API WithIIIF the image archiving function of SIPI can be combined withthe direct usability of the images by the VRE

Use case The MeditationesKnoraSALSAH is being used to create a digital edition of

the Meditationes the scientific notebook of Jacob I Bernoulli(1654ndash1705) which is part of the Bernoulli-Euler Online project5

The edition is based on the definition of segments on themanuscriptrsquos pages that are transcribed individually Each seg-ment is represented by a region of interest (ROI) that is stored inKnoraSALSAH

Figure 1 shows page 184 of the Meditationes with ROIs de-fined for Meditatio 151 The different colors indicate differentcategories of image segments title text lsquoTTrsquo and text lsquoTrsquo (green)mathematical formulae lsquoMrsquo (red) and mathematical figures lsquoFrsquo(blue) The transcriptions are encoded in an XML-based formatand refer to a ROI by its IRI Listing 2 shows the XML transcrip-tions for three regions shown in Figure 1 (the two green regionson the top and the red region on the upper left as well as the blueregion for the mathematical figure) Per entry ndash a Meditatio ndashone XML file is created that contains the transcriptions for all itsregions In the XML each ROI is represented by a region el-ement indicating its IRI and category as well as its name TheXML markup represents layout information authorial changes ofthe text expansions of abbreviations and editorial interventionsand LATEX is also used to encode the mathematical notation in themanuscript Entities are used to encode special characters The

4httpsgithubcomdhlab-baselSipi5The original manuscript with the full title Meditationes Annota-

tiones Animadversiones Theologicae amp Philosophicae is kept at theManuscript Department of the Basel University Library (shelfmark L Ia3 367 pages) It is accessible (without transcriptions or commentaries) athttpdxdoiorg107891e-manuscripta-43865

Figure 1 Regions of Interest defined on page 184 of the Meditationes

(Meditatio 151)

XML does not conform to the TEIXML guidelines but TEI def-initions are used whenever possible6 The region element is sim-ilar to the facsimile element defined by the TEI [9 section 111]Since our approach is based on regions the paragraph structurehas to be encoded using milestone tags (lsquoltparabgtrsquo) Conven-tionally paragraphs would constitute the primary hierarchy of aTEIXML transcription

With the XML transcriptions referring to Knora regions itis now possible to merge text and image content automaticallyby converting Knora ROIs to IIIF URLs Listing 1 depicts thetitle text region (green ROI at the very top ldquoM151-01-TTrdquo) fromFigure 1

lineColor33ff33lineWidth2points[

x 02478y 02804

x 09621y 03888

]typerectangle

Listing 1 JSON representation of a Knora ROI

Each Knora ROI is a rectangular shape defined by the po-sitions of the top left and lower right corners as relative values

6We intend to offer an XSL transformation that converts our XMLformat to TEIXML that may be used as an interchange format

ranging from 0 to 1 ROIs are rendered in the web-based SAL-SAH GUI as shown in Figure 1 by drawing on an HTML5 canvaselement that is overlaid on the actual image (please note that theupper left corner of the canvas has the coordinates 0 0) Relativevalues have the advantage that they are easy to apply to a scaledimage

When these segments have been defined and transcribedwe can associate the transcription of each segment not onlywith the corresponding manuscript page but also with a par-ticular section of the image Given the rectangular shape ofa ROI a IIIF URL can easily be constructed From the twopositions defining the ROI depicted in Listing 1 the regionparameter of a IIIF-URL can be created using percent val-ues httpsipihostportprefixMed151jpxpct2478280471431084full0defaultjpg Please notethat the IIIF region parameter requires the regionrsquos width andheight [1 section 41] which can be determined by calculatingthe difference between the two given points

The technical specification of a IIIF URL is thus combinedwith the domain knowledge we have from Knora to present a re-gionrsquos transcription aligned with its image section as shown inFigure 2 In some cases it may also be helpful to rotate an imagesection when displaying it to the user Sometimes Jacob Bernoulliadded text in the margin of the page rotated 90 When we iden-tify these ROIs in Knora the rotation parameter can be used inIIIF to make the imagersquos content easier to read ROIs are catego-rized by their content (text mathematical formulae and figures)Knora can generate all IIIF URLs for ROIs of a certain categoryto facilitate image processing (eg automatic recognition of fig-ures)

The IIIF Presentation APIA promising way to merge images with additional digital

content is the IIIF Presentation API [2] It makes use of the SharedCanvas Data Model [7] which allows multiple images of the samephysical object to be mapped to a common virtual canvas Thusdigital facsimiles of the same physical objects can be aligned Ause case for this model is the medieval manuscript domain whereonly fragments of the original physical page may exist These canbe combined to reconstruct the page as a whole The authors ofthe Shared Canvas Data Model had also in mind the issue that thefacsimiles may come from different repositories [6]

The IIIF Presentation API aims to make digital images avail-able with additional digital content eg the pages of a manuscriptwith their transcriptions The specification thus standardizes thepresentation of digital image collections with metadata Thesecould then be displayed by any viewer complying with that stan-dard eg the Universal Viewer7

The use case introduced in the previous section could surelybenefit from these standardization efforts However the IIIF Pre-sentation API seems to focus on image-based resources and addsmetadata to them In the case of the digital edition of the Medita-tiones both image segments and transcriptions are equally impor-tant The transcriptions shown in Figure 2 are the result of apply-ing XSL transformations to the XML source file shown in Listing2 XML-based transcriptions are complex objects in themselvesand may need to be processed before the resulting texts can be

7httpuniversalviewerio

Figure 2 Meditatio 151 Image Segments and their Transcriptions (par-

tially)

displayed The IIIF Presentation API briefly discusses the inclu-sion of XPath expressions (XPointer) to extract a certain elementfrom an XML source file [2 section 61]

For the use case of the Meditationes not only XSL trans-formations haven to be applied to the XML source file beforedisplaying it on the web as HTML but also the mathematicalformulae written in LATEX have to be rendered In order to doso we rely on MathJax8 which requires additional configurationoptions The XSL transformations are applied using the SaxonXSLT Processor9

Towards a generic API for querying textual In-formation

Content-based interoperability can be achieved by combin-ing digital representations of objects such as images and videowith text-based metadata and annotations We have demonstratedthe importance of this kind of functional interoperability using anexample involving the IIIF Image API and XML in combinationwith the KnoraSALSAH VRE While XML is a well-accepted

8httpswwwmathjaxorg9httpsaxonsourceforgenet

technology for the transcription of manuscripts in the humanitiesand this use has been standardized by the TEI community the IIIFImage API offers a standardized way to access visual representa-tions They can be combined to integrate these two categories ofsources and this has already been standardized to some extentby the IIIF Presentation API However for textual informationstandards for simplified access (without knowledge of the under-lying technical implementation) are still lacking We propose thatfor this kind of request a standardized interface (API) should beestablished by analogy with the IIIF Image API The diverse na-ture of such information from images makes the creation of sucha standard more difficult

An API for textual data could allow for the request of spec-ified ranges of a text (indicated by character positions by anal-ogy to an image region) or elements identified by an ID Suchmethods are also discussed in the context of the Web AnnotationData Model to refer to parts of external web resources [5 sec-tions 323 and 425] The user should also be able to specifythe desired output format However the user should not need toknow about the underlying storage format since the proposed APIshould equally work for various text formats (eg XML Mark-down and LATEX) Therefore an initial version of such an APIdesigned to support the lowest common denominator of variousmarkup languages could support only simple requests If the un-derlying text format is known to be XML queries involving XPathor even complex XSL transformations could be allowed We pro-pose that for any given text a preliminary request could be madeto obtain information about the available request options as wellas predefined IDs (eg elements with an ID in an XML or Mark-down file or elements with a label in LATEX)

ConclusionOur experience with the KnoraSALSAH framework shows

that a generic interface standard like proposed above is feasibleWe have already implemented a generic platform dealing with thecreation the querying and the manipulation of RDF resources andtheir properties including versioning and permissions SemanticWeb technologies such as RDF RDFS OWL and SPARQL allowfor complex querying of data and their semantics But masteringSPARQL in such an environment is difficult In KnoraSALSAHwe therefore implemented a much simpler interface based on aRESTful API

We believe that it would be valuable to add an API syntax forquerying texts to KnoraSALSAH complying with the proposedstandard that has yet to be elaborated and defined We have al-ready added basic functionality for representing XML documentsin an RDF triplestore and we could add similar functionality forother markup languages as well This would allow us to add atext-format-agnostic layer to KnoraSALSAH and still providetext-format-specific querying methods such as XPath The mainadvantage of integrating textual information into KnoraSALSAHis the combination of complex marked up texts with other sorts ofdata using the basic value types defined in the Knora base ontol-ogy KnoraSALSAH offers ways to represent the semantic con-nections between resources of different types and this makes iteasier to integrate images and text-based resources By using on-tologies these semantic connections can be adequately describedand formalized

In the case of the Meditationes KnoraSALSAH could au-

tomatically create the required information needed for a viewercomplying with the IIIF Presentation API by connecting imagesROIs and parts of the transcription This would involve the IIIFImage API as well as requests for textual information using theproposed standard API Like the IIIF Image API this API couldalso be used by itself outside the context of the IIIF PresentationAPI

References[1] Michael Appleby Tom Crane Robert Sanderson Jon Stroop Simeon

Warner IIIF Image API 21 httpiiifioapiimage21iiif-image-api-2-1

[2] Michael Appleby Tom Crane Robert Sanderson Jon StroopSimeon Warner IIIF Presentation API 21 httpiiifioapipresentation21

[3] Lukas Rosenthaler et al Final report for the pilot projectldquoData and Service Center for the Humanitiesrdquo Tech repSwiss Academy of Humanities and Social Sciences (2015)httpwwwsagwchdmssagwlaufende_projekte

DaSCHFinalReport-DaSCH_print[4] Lukas Rosenthaler Peter Fornaro The ldquoInternational Image Interop-

erability Frameworkrdquo and its Implication to Preservation in Proceed-ings of ISampT Archiving 2016 pages 95ndash99 (2016)

[5] Robert Sanderson Paolo Ciccarese Benjamin Young WebAnnotation Data Model httpswwww3orgTR2017

REC-annotation-model-20170223[6] Robert Sanderson Benjamin Albritton Rafael Schwemmer Her-

bert Van de Sompel SharedCanvas a collaborative model for me-dieval manuscript layout dissemination in Proceedings of the 11thAnnual International ACMIEEE Joint Conference on Digital Li-braries pages 175ndash184 (2011) httpdoiacmorg10114519980761998111

[7] Robert Sanderson Benjamin Albritton Shared Canvas Data Modelhttpiiifiomodelshared-canvas10

[8] Tobias Schweizer Andreas Wassmer Ivan Subotic Lukas Rosen-thaler Long-term Access to Research Data as a Challenge to Migra-tion in Proceedings of ISampT Archiving 2014 pages 28ndash33 (2014)

[9] TEI Consortium (eds) Guidelines for Electronic Text Encoding andInterchange (2016-12-15) httpwwwtei-corgP5

Author BiographyTobias Schweizer is a research associate at the Digital Humanities

Lab Since July 2016 he works on the project Bernoulli-Euler Online thataims at integrating the works of the Bernoullis and Leonhard Euler in oneplatform He also takes part in the development of Knora

Lukas Rosenthaler is the head of the Digital Humanities Lab of theUniversity of Basel He also leads the Swiss national data center for pro-viding long-term access to digital research data in the humanities an ini-tiative of the Swiss Academy of Humanities and Social Sciences He isan expert for image analysis image processing data base systems linkedopen data virtual research environments and digital archiving and he issupporting open access initiatives

Peter Fornaro is deputy head of the Digital Humanities Lab of theUniversity of Basel He is doing research in the field of digital archivingimaging cultural heritage preservation and computational photographyFornaro is teaching at the University of Basel Besides research and lec-turing he is giving consulting to companies archives and museums

ltxml version=10 encoding=utf-8gtltDOCTYPE meditatioSYSTEM entitiesdtdgt

ltmeditatio xmlnsxsi=httpwwww3org2001XMLSchema-instance id=151xsinoNamespaceSchemaLocation=meditatioxsdgt

ltregion id=httpdataknoraorgbeolMed151-01-TT name=M151-01-TT regiontype=TT pagenumber=184gtlthi rend=centergtlthi rend=underlinegtCLIlthigtlthigtltlbgtlthi rend=underlinegtExamen modi Renaldiniani inscribendi ampquaevis polygona regularia in circuloltlbgt

depromti ex ltexpangtLibltamgtltamgtltexgtroltexgtltexpangt II de ltexpangtResolltamgtltamgtltexgtutioneltexgtltexpangtampamp ltexpangtComposiltamgtltamgtltexgttioneltexgtltexpangt ltexpangtMathemltamgtltamgtltexgtaticaltexgtltexpangt p367lthigt (lthi rend=underlinegtltexpangtvidltamgtltamgtltexgteltexgtltexpangt Sturmii ltlbgt

Mathesin enucleatam p 38lthigt) ltparabgtltregiongt

ltregion id=httpdataknoraorgbeolMed151-02-T name=M151-02-T regiontype=T pagenumber=184gtModus hic est Fiat ltexpangttriangltamgtltamgtltexgtulumltexgtltexpangt

ltexpangtampaequilltamgtltamgtltexgtaterumltexgtltexpangt ltformula notation=texgtABDltformulagtdivisampahatampque diametro ltformula notation=texgtABltformulagt in tot partes ampaequales ltlbgt

quot laterum est figura inscribenda duabusampque earum amppraetermissis ltdelgtetltdelgt ampagrav ltformulanotation=texgtBltformulagt versampugravs ltformula notation=texgtAltformulagt ducaamptur ltlbgt

per initium tertiampae recta ltformula notation=texgtDFltformulagt ampamp hinc recta ltformulanotation=texgtFBltformulagt quam putat esse latus polygoni optatiltreggtltreggt ltparabgt

ltregion id=httpdataknoraorgMed151-03-F name=M151-03-F regiontype=F pagenumber=184gtlt-- Figure 151-03 as SVG here --gt

ltregiongtlthi rend=underlinegtltexpangtAnalltamgtltamgtltexgtysisltexgtltexpangtlthigtltreggtltreggt Sit secta diameter

utcunampque in ltformula notation=texgtEltformulagt ltformula notation=texgteltformulagtductampaeampqueltformula notation=texgtDEFltformulagt ampamp

ltaddgtltformula notation=texgtFAltformulagtltaddgt ltformula notation=texgtFBltformulagt ampamp demissa indiametrum peramphyphenpendicularis ltformula notation=texgtFGltformulagt fiat ltformulanotation=texgtCB=altformulagt ltformula notation=texgtCEltformulagt vel ltformulanotation=texgtCe=bltformulagt ltformula notation=texgtFB=xltformulagt unde ltformulanotation=texgtAF=sqrtmathstrutoverline4aa-xxltformulagt ltparabgt

ltregiongt

ltregion id=httpdataknoraorgMed151-04-M name=M151-04-M regiontype=M pagenumber=184border=borderrightgt

ltformula notation=texgtbeginarraycccccccAB ampamp ampamp BF ampamp ampamp BF ampamp ampamp BG 2a ampamp ampamp x ampamp ampamp x ampamp ampamp dfracmathstrut xx2a

endarrayqquad

ltformulagt ltlbgthincltformula notation=texgtbeginarraylcccampamp pm GB ampamp mp CB ampamp -CE GE = ampamp pm dfracmathstrut xx2a ampamp mp a ampamp -bendarray

ltformulagt ltparabgtltregiongt

ltmeditatiogt

Listing 2 XML transcription of Meditatio 151 (partially)

• Abstract
• Introduction
• Combining a VRE and an Image Repository
• Use case The Meditationes
• The IIIF Presentation API
• Towards a generic API for querying textual Information
• Conclusion
• Author Biography