ist- 2001-320015 open archives initiative in europe and germany uwe müller humboldt university...

Open Archives Initiative in Europe and Germany

Uwe MüllerHumboldt University Berlin, Germany

Electronic Publishing GroupUniversity Library / Computer and Media Service

[email protected]

Uwe Müller, 11.08.2003: Information Technology and DCMI - "Open Archives Initiative in Europe and Germany"

Agenda

1. Open Archives

2. The Open Archives Initiative

3. OAForum: European Activities

4. DINI: German Activities


Open Archives

Archive “repository” of digital information

text documents, images, revords, audio/video sequences, ...

Open Archive

provides open machine interface for making content externally available

provides open machine interface for making content externally available

not necessarily: open (= free) usage of metadata (and digital objects)

not necessarily: open (= free) usage of metadata (and digital objects)

mostly: usage of open standards as exchange protocols

mostly: usage of open standards as exchange protocols


Value of Open Archives

“open” archive A“open” archive A “open” archive B“open” archive B “open” archive C“open” archive C

external serviceexternal service

InterfaceInterface InterfaceInterface InterfaceInterface

InterfaceInterface


Open Archives

Different types of “archives”scholarly publication server (pre-prints, e-prints)libraries (OPAC, e-journals)museum databases (object metadata)archives (historical documents) and cultural heritageeducation

Origin self archivingunclosing existing databasesestablishing new databases

“open archives” approach gains popularitycross archive accesslow cost dissemination of previously “hidden” resourcesbuilding of new service provision


Open Archives: Problems

lacking interoperabilitydifferent metadata standards

formats (DC, MAB, Marc ...)

interpretations (Creator: Author vs. Artist vs. Photographer)

different terminology

different languages

different access strategies

different interfaces / transfer protocols

different copyright regulations

Difficulty to establish joint services based on open archives


General Search Methods

Cross Search Approach (e.g. Z39.50)

Harvest Approach (e.g. DIENST protocol)

serviceserviceRequest

Answerarchivearchive

1

4

2

3

serviceserviceRequest

Answerarchivearchive

4

6

1

databasedatabase

35

2


Agenda

1. Open Archives





The Open Archives Initiative (OAI)

Main ideasworld-wide consolidation of scholarly archives

free access on the archives (at least: metadata)

consistent interfaces for archives and service provider

effortless implementation

based on existing standards (e.g. HTTP, XML, DC)

Basic functioning

Harvester Repository

Requests (based on HTTP)

Metadata (encoded in XML)

Metadata(Documents)

Metadata

Service Provider Data Provider

„Service”


OAI: General Assumptions

exchange metadata, not digital objects themselves based on harvest approach (asynchronous) two groups of participants Data Providers (Open Archives, Repositories)

free access of metadatanot necessarily: free access and usage of resourceseasy to implement, low barriers

(useable for small institutions) Service Providers

use OAI interfaces of the Data Providers harvest and store metadatamay select certain subsets from Data Providers

(set hierarchy, date stamp)may enrich metadataoffer (value-added) service on the basis of the metadata


OAI: Technical Model

Se

rvic

e P

rovi

der

e-print

Da

ta

Pro

vid

er e-prints

e-print

Da

ta

Pro

vid

er Images

e-print

Da

ta

Pro

vid

er

OPAC

e-print

Da

ta

Pro

vid

er Museum

e-print

Da

ta

Pro

vid

er Archive

Requests:

Identify

ListMetadataformats

ListSets

ListIdentifiers

ListRecords

GetRecord

Responses:

General information

Metadata formats

Set structure

Record identifier

Metadata

Da

ta

Pro

vid

er Harvester

Repository

Repository

Repository

Repository

Repository


OAI-Protocol for Metadata Harvesting

Basics of OAI-PMHprotocol based on HTTP

request arguments as GET or POST parameters

six request types

e.g. http://archive.org?verb=ListRecords&from=2003-08-01

responses are encoded in XML syntax

supports any metadata format (at least: Dublin Core)

Details of OAI-PMHlogical set hierarchy (definition: data providers)

date stamps (last change of metadata set)

error messages

flow control


OAI-Protocol for Metadata Harvesting

Metadata sets (Records)1. header

unique identifier (key for further archive requests),e.g. oai:HUBerlin.de:30000231

datestamp, e.g. 2003-08-11

logical sets in which the record is contained

2. metadata

metadata prefix (identifier for metadata format)

metadata set (at least: Dublin Core, but arbitrary other metadata formats can be transmitted)


Example: http://edoc.hu-berlin.de/OAI-2.0?verb=ListIdentifiers&from=2002-01-03&until=2002-01-08&metadataPrefix=oai_dc&set=doctypes:dissertations

<?xml version="1.0" encoding="UTF-8"?> <OAI-PMH xmlns="http://www.openarchives.org/OAI/2.0/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/ http://www.openarchives.org/OAI/2.0/OAI-PMH.xsd"> <responseDate>2002-10-22T17:49:49+01:00</responseDate> <request verb="ListIdentifiers" from="2002-01-03" until="2002-01-08" metadataPrefix="oai_dc" set="doctypes:dissertations">http://edoc.hu-berlin.de/OAI-2.0</request> <ListIdentifiers> <header> <identifier>oai:HUBerlin.de:3000819</identifier> <datestamp>2002-01-08</datestamp> <setSpec>doctypes</setSpec> <setSpec>doctypes:dissertations</setSpec> <setSpec>dnb</setSpec> <setSpec>dnb:dnb33</setSpec> </header> <header> <identifier>oai:HUBerlin.de:3000831</identifier> <datestamp>2002-01-07</datestamp> <setSpec>doctypes</setSpec> <setSpec>doctypes:dissertations</setSpec> <setSpec>dnb</setSpec> <setSpec>dnb:dnb27</setSpec> </header> </ListIdentifiers> </OAI-PMH>


OAI: Data Provider Architecture

SQL-Database

OAI Data Provider

Web server (e.g. Apache, IIS)

OAI request(HTTP request) Programming

extension (e.g. PHP, Perl)

SQL request

DB response

OAI response(XML instance)


OAI: Service Provider Architecture

Data Provider Data Provider Data Provider

Scheduler

Flow controlXML Parser

Normaliser

Database

Service module

User Harvester User

OAI Service Provider

Dublication checker

Update mechanism


Problems beyond the OAI-PMH

agreement on metadata usage (except DC)semantics

XML schema

agreement on set definitionsselective harvesting

e.g. subject gateways

definition of rights statementsagreement on different right states

machine readable information


OAI: Examples / Participants

Data Provider / Repositoriessee http://www.openarchives.org/Register/BrowseSites.pl

Service ProviderRepository Explorer: http://oai.dlib.vt.edu/cgi-bin/Explorer/oai2.0/testoai/

Cross Archive Searching Service: http://arc.cs.odu.edu/

MyOAI: http://www.myoai.org/

DINI: http://edoc.hu-berlin.de/e_suche/oai.php

Physnet: http://physnet.uni-oldenburg.de/oai/query.php

…


Agenda

1. Open Archives





OAForum: Background

Project Open Archives ForumEuropean Union Information Society Technologies (IST) Programmeaccompanying measureproject start: October 2001 (duration: 2 years)project partners:

UKOLN, University of Bath, United Kingdom I.E.I.-CNR, Pisa, Italy Humboldt University Berlin, Germany

http://www.oaforum.org/ Motivation

increasing discussion about open archives approachsetting up a framework for the approach in generalpromotion of the open archives approachEuropean view ...


OAForum: Project Partners

UKOLN, Bathvarious projects contexts of metadata and interoperability, cross searching

Renardus project, Schemas, DESIRE

I.E.I.-CNR, PisaCYCLADES project, DELOS

development of services on top of OAI specification

Humboldt University, BerlinDissertation Online, NDLTD

DINI workshops on OAI


OAForum: Objectives

raise and sharpen awareness on the open archives approach

promote “low barrier” interoperability

opening cultural resources

detecting potentials for new services

encourage collaborative development of solutionsdiscussion of problems

exchange of experiences and information

support European liaison with OAI

… build community of interest


OAForum: Objectives (2)

Establish an information repository activities related to the open archives approachexperiences, developed services (e.g. document delivery, searching, browsing, summarisation, linking)share the database with other organisations (e.g. OAI)

Validation of European experience concerningimplementing and using the OAI-PMH and other similar approachesrequirements from implementers and users

Organisational review and analysis possible business modelsIntellectual Property Rights

Dissemination of the open archives approachworkshops


OAForum: Ask and Answer Questions

Not: yet another OAI implementation project … supporting activities

European initiatives with open archives based approach clustering activities

existing and new communitiesIST projectsnational initiatives

dissemination activitiesshare experiences on Open Archivesinvestigation of usage: different paradigmsglobal availabilityshare developments


OAForum: Benefits

tool to reach communities bring together what is happening in Europe raise awareness on and discuss main issues

common terminology on digital repositories

metadata / full text harvesting models

needs of users and communities

advanced services

make European projects ready for actiondevelop possible solutions

establish business models


Communitiesinstitutions of cultural heritage

museums

European digitising projects

scholarly institutions

public libraries

special user groups

publishers

commercial sector

education

Service Providere-print archives

subject gateways with aggregating functions

value-added services

Data Providerexisting metadata repositories

new metadata collections

OAForum: Participants

Carriers and Users of Open Archives


OAForum: Topics

1. Workshops, Distribution distribute information and experiences on technologies, etc.produce interest for topics connected with open archivesarticles, talks, etc.

2. Organisational Evaluation analyse business modelstackle issues of copyrights (IPR …)

3. Technical Evaluationapply the technical framework of OAIdevelop an information portal with

information on projects, repositories, service providermetadata schemas software and implementations

discuss problems of interoperability


OAForum: Workshops, Distribution

Contact: I.E.I.-CNR, Pisa – Donatella Castelli May 2002, Pisa, Italy

experiences from the European e-prints community

establish the forum

December 2002, Lisbon, Portugalarchives and libraries

open access to hidden archives

March 2003, Berlin, Germanymetadata schemas

networking multimedia resources

September 2003, Bath, United Kingdom“In Practice – Best Practice”

final workshop, EU review


OAForum: Organisational Evaluation

Contact: UKOLN, Bath – Philip Hunter Business models

co-operations between data providers to establish service networks

metadata exchange between archives and services

provision of value-added services (e.g. enrichment of metadata, automatic classification, OpenURL)

Copyright issuesIPR and copyright (influence on producers and distributors)

property rights on metadata (collective use of metadata, metadata exchange, agreements with publishers etc.)

long-term availability of digital resources

Discussion group Dennis Nicholson (University Glasgow)


OAForum: Technical Evaluation

Contact: Humboldt University, Berlin – Susanne Dobratz Interoperability

integration of new (open archive) approaches with existing technologiesIs unqualified Dublin Core sufficient?

Issues on database management concurrency and update problemsscalabilityde-dublication

Software and toolscollect and share experiences and existing solutionsestimation of necessary expenditure

establish data provider / service …skills, manpower, time

Tutorials on OAI technologiesonline tutorial – will be presented at Bath Workshop


OAI Activities in Europe

6053

4 2

0

10

20

30

40

50

60

Overview of OAI activity (continents)

Europe

America

Australia

Asia

15

12

7 76

32 2

1 1 1 1 1 1 1 1

0

2

4

6

8

10

12

14

16

Overview of european countries engaged in OAI implementationUK

Germany

France

Sweden

Italy

Netherlands

Austria

Finland

Belorussia

Belgium

Denmark

Ireland

Norway

Portugal

Russia

Switzerland

(numbers from November 2002)


http://www.oaforum.org/resources/tecvalq2.php


http://www.oaforum.org/resources/glossary.php


http://www.oaforum.org/oaf_db/register/


www.oaforum.org/oaf_db/list_db/list_services.php


Agenda

1. Open Archives





DINI recommendations for usage of OAI-PMH

created by DINI-OAI working group http://www.dini.de/ target: agreement on syntax and semantics of OAI set

definitions for German data and service providers enhance retrieval quality and support subject gateways (e.g.

Physnet, Dissertation search engine, ...) definition of three classification types

subjects (according to DNB)

formal publication types (e.g. dissertation)

formal document types (e.g. text, audio)

example service provider based on recommended sets: http://edoc.hu-berlin.de/e_suche/oai.php


Classification according to subjects

SetSpec SetName

dnb:01 Knowledge and Culture in Generaldnb:02 Books and Libraries, Information and Documentationdnb:03 Reference Books, Bibliographiesdnb:04 Directories and Phone Booksdnb:05 Calendarsdnb:06 Journalismdnb:07 Children's and Youth Literaturednb:08 Comics, Cartoons, Caricatures Miscellaneadnb:09 Esoterica Manuscripts, Book Artdnb:10 Philosophydnb:11 Psychologydnb:12 Christianitydnb:13 General and Comparative Theology, Non-Christian

Religiondnb:14 Sociology, Sociography... ...dnb:65 Economic History


Classification according to formal publication types

SetSpec SetName

pub-type:monograph Books, Monographspub-type:article Journal Articlespub-type:dissertation Dissertations and Professional Dissertationspub-type:masterthesis Diploma Thesespub-type:report Reportpub-type:paper Paperpub-type:conf-proceeding Conference Proceedingspub-type:lecture Lecturespub-type:music Musicpub-type:program ProgramsPub-type:play PlayPub-type:news NewsPub-type:standards Standards


Classification according to formal document types

SetSpec SetName

doc-type:text Textdoc-type:notes Notesdoc-type:image Imagedoc-type:audio Audiodoc-type:video Videodoc-type:multimedia Multimediadoc-type:data Datadoc-type-binary Binary data, (executable) program


Multiple Data and Service Providers

Data providers

Service providers

Harvestingbased onOAI-PMH


Aggregators – Example: HBZ Köln

Data providers

Service providers

Aggregator


Hybrid Search – Example: Metalib at HU

Data providers

Service providers

Harvestingbased onOAI-PMH

Searchingbased onZ39.50 orSRW


A German OAI Example: ProPrint

project of SUB Göttingen and CMS of HU Berlin duration: 2000 – 2003 target: integrate heterogenous document servers in order to

provide PoD service components:

search engine for documents

PDF documents preview

generation of compound PDF file with front page and table of contents

production and delivery by selected print service provider

underlying technology: extension of Dublin Core and OAI-PMH (with an extension for document exchange)

http://www.proprint-service.de/


Thank you …

Questions?

Uwe Müller

Humboldt University Berlin, Germany

[email protected]


Additional Information

Open Archives Initiativehttp://www.openarchives.org/ http://www.openarchives.org/OAI/openarchivesprotocol.html (OAI-PMH)http://www.openarchives.org/service/listproviders.html (Service Providers)http://www.openarchives.org/Register/BrowseSites.pl (Data Providers)http://www.openarchives.org/tools/index.html (Tools, ...)

Open Archives Forumhttp://www.oaforum.org/ http://www.oaforum.org/workshops/bath_invitation.php (workshop, Bath, September 2003)http://www.oaforum.org/resources/tecvalq2.php (Technical Validation Questionnaire)

DINIhttp://www.dini.de/http://edoc.hu-berlin.de/e_suche/oai.php (OAI search engine)

ist- 2001-320015 open archives initiative in europe and germany uwe müller humboldt university...

Documents

metadata slide

information technology

usage of open standards

open machine interface

http metadata

change of metadata

store metadata

free usage of metadata