the mint mapping tool the more aggregator vassilis tzouvaras, dimitris gavrilis national technical...
TRANSCRIPT
![Page 1: The Mint Mapping tool The MoRe aggregator Vassilis Tzouvaras, Dimitris Gavrilis National Technical University of Athens Digital Curation Unit - IMIS, Athena](https://reader036.vdocument.in/reader036/viewer/2022070415/5697bf921a28abf838c8ec53/html5/thumbnails/1.jpg)
The Mint Mapping toolThe MoRe aggregator
Vassilis Tzouvaras, Dimitris Gavrilis
National Technical University of AthensDigital Curation Unit - IMIS, Athena Research Center
LoCloud is funded by the European Commission's ICT Policy Support Programme
![Page 2: The Mint Mapping tool The MoRe aggregator Vassilis Tzouvaras, Dimitris Gavrilis National Technical University of Athens Digital Curation Unit - IMIS, Athena](https://reader036.vdocument.in/reader036/viewer/2022070415/5697bf921a28abf838c8ec53/html5/thumbnails/2.jpg)
Cultural Heritage Content
• Diversity of cultural heritage content– Numerous metadata schemas to annotate content
(LIDO, CIDOC-CRM, EAD, METS ) • Massive digitization and annotation activities are in
progress• Need for interoperability
![Page 3: The Mint Mapping tool The MoRe aggregator Vassilis Tzouvaras, Dimitris Gavrilis National Technical University of Athens Digital Curation Unit - IMIS, Athena](https://reader036.vdocument.in/reader036/viewer/2022070415/5697bf921a28abf838c8ec53/html5/thumbnails/3.jpg)
MINT Mapping Tool
• Provides users the ability to perform a mapping of their own metadata schemas to reference domain models
• Follows a typical web based architecture• It was developed for ATHENA, but it is currently used
for EUScreen, CARARE, Judaica, ECLAP, DCA and Linked Heritage
![Page 4: The Mint Mapping tool The MoRe aggregator Vassilis Tzouvaras, Dimitris Gavrilis National Technical University of Athens Digital Curation Unit - IMIS, Athena](https://reader036.vdocument.in/reader036/viewer/2022070415/5697bf921a28abf838c8ec53/html5/thumbnails/4.jpg)
MINT 2 – What’s new?
• The backend was reconstructed for better performance– File size for imports is extended
• The frontend was updated– New interface– Workflow is integrated in UI– Facilitated browsing of input and target schema
![Page 5: The Mint Mapping tool The MoRe aggregator Vassilis Tzouvaras, Dimitris Gavrilis National Technical University of Athens Digital Curation Unit - IMIS, Athena](https://reader036.vdocument.in/reader036/viewer/2022070415/5697bf921a28abf838c8ec53/html5/thumbnails/5.jpg)
![Page 6: The Mint Mapping tool The MoRe aggregator Vassilis Tzouvaras, Dimitris Gavrilis National Technical University of Athens Digital Curation Unit - IMIS, Athena](https://reader036.vdocument.in/reader036/viewer/2022070415/5697bf921a28abf838c8ec53/html5/thumbnails/6.jpg)
![Page 7: The Mint Mapping tool The MoRe aggregator Vassilis Tzouvaras, Dimitris Gavrilis National Technical University of Athens Digital Curation Unit - IMIS, Athena](https://reader036.vdocument.in/reader036/viewer/2022070415/5697bf921a28abf838c8ec53/html5/thumbnails/7.jpg)
![Page 8: The Mint Mapping tool The MoRe aggregator Vassilis Tzouvaras, Dimitris Gavrilis National Technical University of Athens Digital Curation Unit - IMIS, Athena](https://reader036.vdocument.in/reader036/viewer/2022070415/5697bf921a28abf838c8ec53/html5/thumbnails/8.jpg)
![Page 9: The Mint Mapping tool The MoRe aggregator Vassilis Tzouvaras, Dimitris Gavrilis National Technical University of Athens Digital Curation Unit - IMIS, Athena](https://reader036.vdocument.in/reader036/viewer/2022070415/5697bf921a28abf838c8ec53/html5/thumbnails/9.jpg)
![Page 10: The Mint Mapping tool The MoRe aggregator Vassilis Tzouvaras, Dimitris Gavrilis National Technical University of Athens Digital Curation Unit - IMIS, Athena](https://reader036.vdocument.in/reader036/viewer/2022070415/5697bf921a28abf838c8ec53/html5/thumbnails/10.jpg)
![Page 11: The Mint Mapping tool The MoRe aggregator Vassilis Tzouvaras, Dimitris Gavrilis National Technical University of Athens Digital Curation Unit - IMIS, Athena](https://reader036.vdocument.in/reader036/viewer/2022070415/5697bf921a28abf838c8ec53/html5/thumbnails/11.jpg)
![Page 12: The Mint Mapping tool The MoRe aggregator Vassilis Tzouvaras, Dimitris Gavrilis National Technical University of Athens Digital Curation Unit - IMIS, Athena](https://reader036.vdocument.in/reader036/viewer/2022070415/5697bf921a28abf838c8ec53/html5/thumbnails/12.jpg)
MORe Overall Architecture
Registry
Apache Cassandra cluster
Fedora-commons
Temporary storage
Vocabulary services
Storage
JMS logging
Messaging
Core services
Enrichment service management
Entity matching / NLP
Geocoding / Historic Place names
REST
External enrichment services
Publish service management OAI-PMH
RDF Store
Elastic Search
Archive
![Page 13: The Mint Mapping tool The MoRe aggregator Vassilis Tzouvaras, Dimitris Gavrilis National Technical University of Athens Digital Curation Unit - IMIS, Athena](https://reader036.vdocument.in/reader036/viewer/2022070415/5697bf921a28abf838c8ec53/html5/thumbnails/13.jpg)
Cloud architecture
• De-centralized• Scalable• Four cloud environmets– Storage– Monitoring & logging– Core services deployment– Enrichment services deployment
![Page 14: The Mint Mapping tool The MoRe aggregator Vassilis Tzouvaras, Dimitris Gavrilis National Technical University of Athens Digital Curation Unit - IMIS, Athena](https://reader036.vdocument.in/reader036/viewer/2022070415/5697bf921a28abf838c8ec53/html5/thumbnails/14.jpg)
Distributed
• Enrichment services run on:– Austria– Spain– Greece– Lithuania– Slovenia– Norway
• Scalability can be facilitated through a virtualization infrastructure
![Page 15: The Mint Mapping tool The MoRe aggregator Vassilis Tzouvaras, Dimitris Gavrilis National Technical University of Athens Digital Curation Unit - IMIS, Athena](https://reader036.vdocument.in/reader036/viewer/2022070415/5697bf921a28abf838c8ec53/html5/thumbnails/15.jpg)
Workflow
OAI-PMH
LoCloud Collections
Wikimedia
MINT
Harvest
Ingest
Transform Enrich
Publish
OAI-PMH
Archive
RDF Store
SolR
Validate Index
Delete Reject
Omeka
![Page 16: The Mint Mapping tool The MoRe aggregator Vassilis Tzouvaras, Dimitris Gavrilis National Technical University of Athens Digital Curation Unit - IMIS, Athena](https://reader036.vdocument.in/reader036/viewer/2022070415/5697bf921a28abf838c8ec53/html5/thumbnails/16.jpg)
Intermediate Schemas
Dublin Core
LIDO
CARARE
EAD
ESE
EDM
Dublin Core
LIDO
CARARE
EAD
ESE
EDM
OMEKA-XML
OGD
![Page 17: The Mint Mapping tool The MoRe aggregator Vassilis Tzouvaras, Dimitris Gavrilis National Technical University of Athens Digital Curation Unit - IMIS, Athena](https://reader036.vdocument.in/reader036/viewer/2022070415/5697bf921a28abf838c8ec53/html5/thumbnails/17.jpg)
• Harvesting• Validation• Ingestion• Transformation• Enrichment• Previewing• Publishing
Core services
Harvests content from metadata sourcesOAI-PMH repositoryMINTLoCloud CollectionsWikimedia
Multiple schemas are supportedOAI_DCCARARECARARE 2.0LIDOEADEDMESE
![Page 18: The Mint Mapping tool The MoRe aggregator Vassilis Tzouvaras, Dimitris Gavrilis National Technical University of Athens Digital Curation Unit - IMIS, Athena](https://reader036.vdocument.in/reader036/viewer/2022070415/5697bf921a28abf838c8ec53/html5/thumbnails/18.jpg)
• Harvesting• Validation• Ingestion• Transformation• Enrichment• Previewing• Publishing
Core services
Validates incoming information packagesExecutes validation schemesValidation micro-services
StructureSchemaLinkingSchematron rules
Flexible
How it is used in MoRe:Pre-validation Post-validation
![Page 19: The Mint Mapping tool The MoRe aggregator Vassilis Tzouvaras, Dimitris Gavrilis National Technical University of Athens Digital Curation Unit - IMIS, Athena](https://reader036.vdocument.in/reader036/viewer/2022070415/5697bf921a28abf838c8ec53/html5/thumbnails/19.jpg)
• Harvesting• Validation• Ingestion• Transformation• Enrichment• Previewing• Publishing
Core services
Ingest content into storageUses storage layer APIPluggable drivers for attaching different technologies / repositories
Apache CassandraFilesystem-basedFedora-commons
Versioning supportComplex digital object support
![Page 20: The Mint Mapping tool The MoRe aggregator Vassilis Tzouvaras, Dimitris Gavrilis National Technical University of Athens Digital Curation Unit - IMIS, Athena](https://reader036.vdocument.in/reader036/viewer/2022070415/5697bf921a28abf838c8ec53/html5/thumbnails/20.jpg)
• Harvesting• Validation• Ingestion• Transformation• Enrichment• Previewing• Publishing
Core services
Content Model
Digital objects comprise data streams
Each data stream can hold any kind of information• XML/RDF, Image, Video, Documents, etc.
Each different representation of an information object is stored as a different data stream
Each curation action generates a new version• Transformation, Enrichment
![Page 21: The Mint Mapping tool The MoRe aggregator Vassilis Tzouvaras, Dimitris Gavrilis National Technical University of Athens Digital Curation Unit - IMIS, Athena](https://reader036.vdocument.in/reader036/viewer/2022070415/5697bf921a28abf838c8ec53/html5/thumbnails/21.jpg)
• Harvesting• Validation• Ingestion• Transformation• Enrichment• Previewing• Publishing
Core services
Transforms entire information packages into the Europeana Data Model (EDM), or any other schema
Multiple transformation routinesPer schemaPer projectPer provider
User can attach rights statement
![Page 22: The Mint Mapping tool The MoRe aggregator Vassilis Tzouvaras, Dimitris Gavrilis National Technical University of Athens Digital Curation Unit - IMIS, Athena](https://reader036.vdocument.in/reader036/viewer/2022070415/5697bf921a28abf838c8ec53/html5/thumbnails/22.jpg)
• Harvesting• Validation• Ingestion• Transformation• Enrichment• Previewing• Publishing
Core services
The generic enrichment service facilitates the execution of the enrichment micro-services
• Hides the complexity from the user by using enrichment plans
• Provides seamless integration with the UI of MORE
Virtual Enrichment driver• Allows developers/creative industries to create
their own enrichment services and declare/use them within MoRe
![Page 23: The Mint Mapping tool The MoRe aggregator Vassilis Tzouvaras, Dimitris Gavrilis National Technical University of Athens Digital Curation Unit - IMIS, Athena](https://reader036.vdocument.in/reader036/viewer/2022070415/5697bf921a28abf838c8ec53/html5/thumbnails/23.jpg)
• Harvesting• Validation• Ingestion• Transformation• Enrichment• Previewing• Publishing
Core services
Preview the XML record information for all datastreams
Preview the record in HTML (using the Europeana style sheet)
![Page 24: The Mint Mapping tool The MoRe aggregator Vassilis Tzouvaras, Dimitris Gavrilis National Technical University of Athens Digital Curation Unit - IMIS, Athena](https://reader036.vdocument.in/reader036/viewer/2022070415/5697bf921a28abf838c8ec53/html5/thumbnails/24.jpg)
• Harvesting• Validation• Ingestion• Transformation• Enrichment• Previewing• Publishing
Core services
Publish transformed / enriched information• Internal OAI-PMH provider• XML export • Publish directly to RDF repositories
• Sesame• Virtuoso
• SolR index server
![Page 25: The Mint Mapping tool The MoRe aggregator Vassilis Tzouvaras, Dimitris Gavrilis National Technical University of Athens Digital Curation Unit - IMIS, Athena](https://reader036.vdocument.in/reader036/viewer/2022070415/5697bf921a28abf838c8ec53/html5/thumbnails/25.jpg)
• Thematic– Thesauri collections– Vocabulary matching– Background links
• Spatial– Geo normalization– Geo coding– Reverse geo-coding– Historic place names
• Other– Language identification
Enrichment micro-services
SKOS Thesauri
Geo-Names
DBPedia
Wikipedia
![Page 26: The Mint Mapping tool The MoRe aggregator Vassilis Tzouvaras, Dimitris Gavrilis National Technical University of Athens Digital Curation Unit - IMIS, Athena](https://reader036.vdocument.in/reader036/viewer/2022070415/5697bf921a28abf838c8ec53/html5/thumbnails/26.jpg)
Enrichment Plan
• Enrichment micro-services are used within enrichment workflows: – Enrichment plans
• Each enrichment plan applies to a specific schema
• Each enrichment plan executes enrichment micro-services in a specific order
Enrichment plans
Language identification
Vocabulary matching
Geo-normalization
Geo-coding
![Page 27: The Mint Mapping tool The MoRe aggregator Vassilis Tzouvaras, Dimitris Gavrilis National Technical University of Athens Digital Curation Unit - IMIS, Athena](https://reader036.vdocument.in/reader036/viewer/2022070415/5697bf921a28abf838c8ec53/html5/thumbnails/27.jpg)
Enrichment Plan
• Each enrichment plan defines run-time parameters for specific services– Content based
Enrichment plans
Language identification
Vocabulary matching
Geo-normalization
Geo-coding
Add subject collection A only if term X or Y
are matched
![Page 28: The Mint Mapping tool The MoRe aggregator Vassilis Tzouvaras, Dimitris Gavrilis National Technical University of Athens Digital Curation Unit - IMIS, Athena](https://reader036.vdocument.in/reader036/viewer/2022070415/5697bf921a28abf838c8ec53/html5/thumbnails/28.jpg)
Dashboard
![Page 29: The Mint Mapping tool The MoRe aggregator Vassilis Tzouvaras, Dimitris Gavrilis National Technical University of Athens Digital Curation Unit - IMIS, Athena](https://reader036.vdocument.in/reader036/viewer/2022070415/5697bf921a28abf838c8ec53/html5/thumbnails/29.jpg)
Packages organization
![Page 30: The Mint Mapping tool The MoRe aggregator Vassilis Tzouvaras, Dimitris Gavrilis National Technical University of Athens Digital Curation Unit - IMIS, Athena](https://reader036.vdocument.in/reader036/viewer/2022070415/5697bf921a28abf838c8ec53/html5/thumbnails/30.jpg)
Package overview
![Page 31: The Mint Mapping tool The MoRe aggregator Vassilis Tzouvaras, Dimitris Gavrilis National Technical University of Athens Digital Curation Unit - IMIS, Athena](https://reader036.vdocument.in/reader036/viewer/2022070415/5697bf921a28abf838c8ec53/html5/thumbnails/31.jpg)
Package lifecycle overview
![Page 32: The Mint Mapping tool The MoRe aggregator Vassilis Tzouvaras, Dimitris Gavrilis National Technical University of Athens Digital Curation Unit - IMIS, Athena](https://reader036.vdocument.in/reader036/viewer/2022070415/5697bf921a28abf838c8ec53/html5/thumbnails/32.jpg)
Preview
![Page 33: The Mint Mapping tool The MoRe aggregator Vassilis Tzouvaras, Dimitris Gavrilis National Technical University of Athens Digital Curation Unit - IMIS, Athena](https://reader036.vdocument.in/reader036/viewer/2022070415/5697bf921a28abf838c8ec53/html5/thumbnails/33.jpg)
Metadata completeness & statistics
![Page 34: The Mint Mapping tool The MoRe aggregator Vassilis Tzouvaras, Dimitris Gavrilis National Technical University of Athens Digital Curation Unit - IMIS, Athena](https://reader036.vdocument.in/reader036/viewer/2022070415/5697bf921a28abf838c8ec53/html5/thumbnails/34.jpg)
Enrichment services overview
![Page 35: The Mint Mapping tool The MoRe aggregator Vassilis Tzouvaras, Dimitris Gavrilis National Technical University of Athens Digital Curation Unit - IMIS, Athena](https://reader036.vdocument.in/reader036/viewer/2022070415/5697bf921a28abf838c8ec53/html5/thumbnails/35.jpg)
Direct access to 27 thesauriCreate & (re)use subject collections