Search Interoperability, OAI, and Metadata

Sarah ShreevesUniversity of Illinois at Urbana-Champaign

Basics and BeyondGrainger Engineering LibraryApril 18, 2005

April 18, 2005 Basics and BeyondGrainger Engineering Library

Scenario:A teacher is putting together a lesson plan comparing immigration in the early 20th century to immigration and wants to include a variety of primary sources

April 18, 2005 Basics and BeyondGrainger Engineering Library

IMLS funded digital collections with relevant content

April 18, 2005 Basics and BeyondGrainger Engineering Library

Search interoperability

“the ability to perform a search over diverse sets of metadata records and obtain meaningful results.”

– Priscilla Caplan

Metadata Fundamentals for All Librarians

April 18, 2005 Basics and BeyondGrainger Engineering Library

Keys to Search Interoperability

Communication protocol (Z39.50, OAI, etc.)

Organizational commitment

Standards

Standards

And More Standards

April 18, 2005 Basics and BeyondGrainger Engineering Library

Sharing metadata : Federated search

The distributed databases are searched directly.

Mill?<title>My resource</title><date>04

<title>My resource</title><date>04

<title>My resource</title><date>04

For Example:

Z39.50, SRU/SRW

April 18, 2005 Basics and BeyondGrainger Engineering Library

Sharing metadata : Data aggregationThe user searches a pre-aggregated

database of metadata from diverse sources.

Mill?

<title>My resource</title><date>04

For Example:

Search engines, union catalogs, OAI

April 18, 2005 Basics and BeyondGrainger Engineering Library

Open Archives Initiative Protocol for Metadata Harvesting

The OAI-PMH is a tool

Moves metadata (not content for the most part yet) from a data provider to a service provider (or harvester)

A set of rules that defines the communication between two systems (like FTP and HTTP)

Facilitates the aggregation of metadata(like a union catalog)

April 18, 2005 Basics and BeyondGrainger Engineering Library

Basic OAI-PMH Concepts “Aggregated search” rather than “Federated search”

Data providers – support OAI PMH as a means to expose metadata

Service providers – ‘harvests’ metadata from data providers via the OAI-PMH

OAI-PMH based upon HTTP and XML

OAI-PMH requires use of simple Dublin Core BUT supports and encourages use of other metadata schemas

Unique and Persistent Identifiers and a Datestamp for each OAI record

April 18, 2005 Basics and BeyondGrainger Engineering Library

OAIster: http://www.oaister.org/o/oaister/CIC Metadata Portal: http://nergal.grainger.uiuc.edu/cgi/b/bib/oaister

April 18, 2005 Basics and BeyondGrainger Engineering Library

How OAI Works (Technically)

6 distinct ‘verbs’ or requests

OAI requests are sent via HTTP

Responses are sent in valid XML

Dig.

Mngt.

Sys.

OAI

HARVESTER

OAIData

PROVIDER

Service Provider Data Provider

HTTP Request

(OAI Verb)

HTTP Response

(Valid XML)

AGGREGATED

METADATA

April 18, 2005 Basics and BeyondGrainger Engineering Library

Examples of OAI Service Providers

OAIster: http://oaister.umdl.umich.edu/o/oaister/

Engineering, Computer Science, and Physics: http://g118.grainger.uiuc.edu/engroai/

Open Language Archives Community:http://www.language-archives.org/

April 18, 2005 Basics and BeyondGrainger Engineering Library

OAI “VERBS”

Identify

ListMetadataFormats

ListSets

ListIdentifiers

GetRecord

ListRecords

April 18, 2005 Basics and BeyondGrainger Engineering Library

Challenges for the OAI Community

No best practices (yet) ‘Shareablity of metadata’

Heterogeneity of items described Loss of Context / Information loss Knowledge structures differ so….

Native metadata schemas differ Controlled vocabularies differ Use and presentation of items differ

April 18, 2005 Basics and BeyondGrainger Engineering Library

OAI ≠ Dublin Core

DC is OAI’s lowest common denominator

BUT

OAI supports & encourages use of other community-driven metadata schemas

April 18, 2005 Basics and BeyondGrainger Engineering Library

Metadata Interoperability

Semantics What is the metadata format used? Mapping from one format to another

Content rules How are values for the metadata elements selected and

represented?

Syntax How are the metadata elements encoded in machine

readable form?

Documentation

April 18, 2005 Basics and BeyondGrainger Engineering Library

April 18, 2005 Basics and BeyondGrainger Engineering Library

Metadata for different communities

April 18, 2005 Basics and BeyondGrainger Engineering Library

Metadata for different communities

April 18, 2005 Basics and BeyondGrainger Engineering Library

Loss of Context:

Record in OAI aggregation

April 18, 2005 Basics and BeyondGrainger Engineering Library

Context:

Record in native database

April 18, 2005 Basics and BeyondGrainger Engineering Library

Loss of context / data

April 18, 2005 Basics and BeyondGrainger Engineering Library

Loss of context / data

April 18, 2005 Basics and BeyondGrainger Engineering Library

Granularity of Description: Excerpt of Metadata Record Describing “American Woven Coverlet”

April 18, 2005 Basics and BeyondGrainger Engineering Library

Granularity of Description: Excerpt of Metadata Record Describing "Cotton coverlet with embroidered butterfly design"

April 18, 2005 Basics and BeyondGrainger Engineering Library

What does this record represent?

identifier:http://images.umdl.umich.edu/cgi/i/image/image-idx?view=entry;subview=detail;cc=fish3ic;entryid=X-0802;viewid=1004_112

publisher: UMMZ Fish Division format: jpeg type: image subject: 1926-05-18;1926;0812;18;Trib. to Sixteen Cr. Trib. Pine

River, Manistee R.;R10W;S26; S27;JAM26-460;05;T21N;1926/05/18

language: UND description: Flora and Fauna of the Great Lakes Region;

April 18, 2005 Basics and BeyondGrainger Engineering Library

April 18, 2005 Basics and BeyondGrainger Engineering Library

Data providers can:

Create metadata for interoperability

Reusable metadata - think beyond your local users and environment

Use well structured and defined schemas; move beyond simple DC

Use and identify controlled vocabularies

Document, document, document

April 18, 2005 Basics and BeyondGrainger Engineering Library

Service Providers can…

Analyze metadata and cluster and normalize some aspects

Provide contextual information (such as collection descriptions)

Custom interfaces and selective views for target audiences / domains

April 18, 2005 Basics and BeyondGrainger Engineering Library

Contact Information

Sarah Shreeves

Project Coordinator,

IMLS Digital Collections and Content Project

University of Illinois at Urbana-Champaign

sshreeve@uiuc.edu

217-244-7809

Presentation available:http://imlsdcc.grainger.uiuc.edu/basicsbeyondMar2005.ppt