Harvesting Process

Isovera Consulting

Feb. 2008

Internet consulting for non-profits2

Quarterly Harvesting Process

Internet consulting for non-profits3

Harvesting Overview

• Goal: To collect decentralized metadata for learning objects, located at various collections, in a central “portal” database.

• Requirements:– Portal and Collections must agree upon the metadata format, i.e., the

BEN Metadata Specification

– Collections must provide metadata which adheres to Portal’s specifications, i.e., is of sufficient quality, includes all required fields, etc.

– Portal and Collections must agree upon a protocol (communication procedure), i.e., the Protocol for Metadata Harvesting

– Portal must implement an OAI “Harvester” program which requests data from Collections

– Collection must implement a OAI “Provider” program which responds to thresher requests.

Internet consulting for non-profits4

Quarterly Harvesting Process, Part 1

• BEN Project Manager asks each collaborator to determine whether a significant body of resources has been added or modified in the past quarter, and have been properly peer reviewed.

• Collaborator responds, indicating readiness or lack of readiness for harvesting.

• BEN Technical Staff use OAI Harvester administrative tools to harvest resources from Collaborator to the BEN staging site.

• BEN Technical Staff prepare a report of all resources harvested from the Collaborator.

Internet consulting for non-profits5

Quarterly Harvesting Process, Part 2

• BEN Project Manager and Collaborator Project Manager review the resources on the staging site.

• If some resources have technical problems, BEN Technical Staff re-harvest to the staging site, and another round of review begins.

• If resources appear to be technically sound, BEN Technical staff use OAI Harvester administrative tools to harvest resources from Collaborator to the BEN production site.

• Once resources have been harvested to the production site, BEN Portal end users may view them.

Internet consulting for non-profits6

Requirements for Harvested Resources

• Metadata includes required metadata fields (see Metadata presentation.)

• Resource is peer-reviewed for scientific accuracy and educational value.

• URL is well-formed and resolves to a proper digital resource.

• Metadata is accurate, and lacks spelling and grammatical errors.

Internet consulting for non-profits7

Process Technical Overview

1. BEN OAI Harvester issues “Identify” request.

2. Collaborator OAI Provider responds with identifying information.

3. BEN OAI Harvester issues “ListIdentifiers” request.

4. Collaborator OAI Provider responds with a list of identifiers for all resources created or modified since the last harvest.

– Usually, an identifier is just a number automatically generated by the database as a primary key.

5. BEN OAI Harvester issues a “GetRecord” request for each resource listed in Step 4.

6. Collaborator OAI Provider responds with a BEN-LOM XML document for the requested resource.

Internet consulting for non-profits8

Process Technical Overview

BEN OAI Harvester

CollaboratorOAI Provider

Internet consulting for non-profits9

How the Harvester/Reaper Works

• BEN OAI Harvester issues HTTP Request– e.g., http://www.collaborator.org/OAI-provider?verb=GetRecord&identifier=123

&metadataPrefix=oai_BEN

• Collaborator web server executes OAI Provider program (Perl script or PHP module)

• Collaborator OAI Provider parses HTTP request

• OAI Provider requests BEN LOM metadata records from library system.

• Library system returns BEN-LOM XML.

• OAI Provider wraps BEN-LOM document in OAI-PMH envelope.

• BEN OAI Harvester opens OAI-PMH envelope, reads BEN-LOM document, and inserts metadata into BEN Portal database.

Internet consulting for non-profits10

References

• Open Archives Initiative Protocol for Metadata Harvesting - http://www.openarchives.org/OAI/openarchivesprotocol.html

• BEN Collaborators Peer Review Policies - http://www.biosciednet.org/project_site/PeerReviewProcessOfBENPartners.pdf