introduction to institutional data repositories workshop

Purdue UniversityPurdue e-Pubs

Libraries Research Publications

3-4-2008

Introduction to Institutional Data RepositoriesWorkshopMichael WittPurdue University, [email protected]

Melissa CraginUniversity of Illinois at Urbana-Champaign, [email protected]

Follow this and additional works at: http://docs.lib.purdue.edu/lib_research

This document has been made available through Purdue e-Pubs, a service of the Purdue University Libraries. Please contact [email protected] foradditional information.

Witt, Michael and Cragin, Melissa, "Introduction to Institutional Data Repositories Workshop" (2008). Libraries Research Publications.Paper 83.http://docs.lib.purdue.edu/lib_research/83

http://docs.lib.purdue.edu?utm_source=docs.lib.purdue.edu%2Flib_research%2F83&utm_medium=PDF&utm_campaign=PDFCoverPages

http://docs.lib.purdue.edu/lib_research?utm_source=docs.lib.purdue.edu%2Flib_research%2F83&utm_medium=PDF&utm_campaign=PDFCoverPages

http://docs.lib.purdue.edu/lib_research?utm_source=docs.lib.purdue.edu%2Flib_research%2F83&utm_medium=PDF&utm_campaign=PDFCoverPages

Introduction to Institutional Data Repositories

Metropolitan Library System Chicago, IL

March 4, 2008 9:30 am – 3:30 pm

Michael Witt Purdue University

Melissa Cragin

University of Illinois at Urbana-Champaign

Today’s Agenda

9:30 - 11:00 Introductions: ourselves, data, repositories Rationale

11:00 - 12:00 Lab activity #1

12:00 -1:00 Lunch

1:00 - 1:30 Data curation

Data collections

1:30 - 2:15 Issues to discuss

2:15 - 3:00 Lab activity #2

3:00 - 3:30 Roles, resources, and approaches Closing discussion

A I t d ti t I tit ti lAn Introduction to Institutional Data RepositoriesData Repositories

Metropolitan Library SystemMarch 4, 2008 9:30 am - 3:30 pm, p

Chicago, IL

Michael Witt Melissa Cragin Purdue University University of Illinois


IntroductionsIntroductions• Melissa Cragin – [email protected]

Vi iti P j t C di tVisiting Project CoordinatorData Curation Education Program (DCEP)Graduate School of Library and Information ScienceyUniversity of Illinois at Urbana-Champaign

• Michael Witt – mwitt@purdue edu• Michael Witt – [email protected] Research Librarianassistant professor of library scienceDi t ib t d D t C ti C t (D2C2)Distributed Data Curation Center (D2C2)Purdue University Libraries



Overview of the dayOverview of the day9:30 - 11:00 Introductions: ourselves, data, repositories

R ti lRationale11:00 - 12:00 Lab activity #112:00 1:00 Lunch12:00 -1:00 Lunch1:00 - 1:30 Data curation

Data collections1:30 - 2:15 Issues to discuss2:15 - 3:00 Lab activity #2y3:00 - 3:30 Roles, resources, and approaches

Closing discussion



What is data?What is data?DATA

INFORMATION

KNOWLEDGEKNOWLEDGEZins, C. (2007). Conceptual approaches for defining data, information, and knowledge. Journal of the American

Society for Information Science and Technology. 58(4), 479-493.



What is an institutional repository?What is an institutional repository?

“A university-based institutional repository is a set of services that a university offers to theof services that a university offers to the members of its community for the management and dissemination of digital materials created by the institution and its community members It isthe institution and its community members. It is most essentially an organizational commitment to the stewardship of these digital materials, i l di l t ti hincluding long-term preservation where appropriate, as well as organization and access or distribution."

Lynch, C. (2003). Institutional repositories: essential infrastructure for scholarship in the digital age. ARL Bimonthly Report No. 226.



Purposes of Repositories• Enable digital asset management • Offer preservation services

Purposes of Repositories

• Provide institutional visibility through access to collective intellectual work • Support learning, teaching, and research • Facilitate discovery of contentFacilitate discovery of content • Enable re-use and re-purposing of content • Support archival business requirements • Offer alternative channels in support of scholarly communication• Offer alternative channels in support of scholarly communication • Organize information to allow effective content management and access • Provide access to outcomes of publicly funded research initiatives

St th t hi b t t t t / id d t t• Strengthen partnership between content creators/providers and content managers

Rieger, Oya Y. (2007). Select for Success: Key Principles in Assessing Repository Models. D-Lib Magazine, 13(7/8).



Organization of RepositoriesOrganization of RepositoriesWho runs them?

– domain (e.g. chemistry – NIST Chemistry WebBook)( g y y )– discipline (e.g. crystallography – National Crystallography Service)– institutional (most often university based)

Content– metadata– e-prints– cultural history materials– natural history records– audio-visual (multimedia)

i tifi d t– scientific data– digital humanities and social science scholarship– social science data (quantitative)



What is IDEALS?What is IDEALS?

Institutional repository for the scholarship and research inInstitutional repository for the scholarship and research in digital form of the faculty, students, and staff of the

University of Illinois at Urbana-Champaign.

• Completed pilot phase• In “quiet” production as of Fall 2007• A joint initiative between the University Library and CITES

with support from the Office of the Provost

http://ideals.uiuc.edu/



The Issues ( h h U i i f Illi i i i i i(or why the University of Illinois is investing in

IDEALS)

• What research can you access?• Who can access your research?

Will h k th bi t ibl i t?• Will your research make the biggest possible impact?

• What can you do with your published research?y y p

• Will your research be available 10, 50, 100+ years into the future?

• What about all those technical reports, data sets, and other grey literature?



IDEALS goalsIDEALS goals….• Help increase access to published and unpublished

hresearch

• Help increase the impact to published and p p punpublished research

• Provide a persistent, permanent URL for yourProvide a persistent, permanent URL for your research

• Preserve research for long term access and usePreserve research for long term access and use



Other services we offerOther services we offer…

• Consultation on copyright issues• Access restricted items or collections in

IDEALS• Statistics on number of downloads (regular

weekly or monthly reports are coming soon…)



(live demo)(live demo)http://docs.lib.purdue.edu



Why share/archive data?Why share/archive data?

To get all of your organization's data in one placeplace.

• described properly• preserved properly• accessible using the same tools and g

services




It’s good science.( lt b d d)(your results can be reproduced)




Some funding agencies and organizations are beginning to require data archivingare beginning to require data archiving and sharing.( NIH ti l ti )(e.g., NIH, open access stipulations)




Data can be re-used.$$ b t ti th d t• save $$ by not recreating the same data

• can be used to advance similar research• can be used to advance research in

another disciplinep




Data can be re-purposed – the Long Tail.( l i bj t)(e.g., as a learning object)




Datasets are assets with value.l• legacy

• credibility• annual reporting




Datasets can be the building blocks of a virtual organization or help build an onlinevirtual organization or help build an online community of scholarship and advance standardsstandards.




Th I f tiThe Information Bottleneck

Hacker, T. (2007).



Why librarians?Why librarians?• technical services - description & classificationp• access services – how people use information• archival science & digital libraries• 200-year view• institutional commitment

&• trust & neutrality• awareness of research taking place• interdisciplinary collaboration (strengthen grant and• interdisciplinary collaboration (strengthen grant and

project proposals)



RolesRoles• Librarians

E ti / t l l d i i t t• Executives / top-level administrators• IT department• Legal counsel• Legal counsel• Research administration• And most importantly, your researchersp y, y

• Communities• SustainabilitySustainability• Who does what will be different in every organization.



Lab Activity #1 (45 minutes)Lab Activity #1 (45 minutes)Scenario:The Water Quality Field Station at Purdue’s ACRE research farm comprises plots of landThe Water Quality Field Station at Purdue s ACRE research farm comprises plots of land

with different soil compositions and treatments. Pipes running under the ground collect water and channel it into sensors that measure the amount of flow by counting the number of times that a calibrated bucket fills up and “tips” in an hour. These files are the output of a data logger that is attached to one of these sensors.

Instructions:1. Create an account in either Many Eyes [1] (use dataset-many.txt) or Swivel [2] (use

dataset-swivel.csv).2. Describe and then upload the data.p3. Once it has been uploaded, analyze the results.4. Compare your dataset and results with others in the repository.

We’ll share our results, observations, and questions before breaking for lunch at noon., , q g

[1] Many Eyes, http://www.many-eyes.com/[2] Swivel, http://swivel.com



ReflectionReflection

Take 10 minutes to write down notes in yourTake 10 minutes to write down notes in your “data repository prospectus” and relate the lab activity and discussion to how youthe lab activity and discussion to how you would plan and provision data repository services for your institutionservices for your institution.

We will share these at the end of theWe will share these at the end of the workshop for our closing discussion.



LunchLunch



Intro to Data CurationIntro to Data Curation• Curation: managing and promoting the use of data from its point of

creation to ensure it is fit for contemporary purpose and available forcreation, to ensure it is fit for contemporary purpose, and available for discovery and re-use. … Higher levels of curation will also involve maintaining links with annotation and with other published materials. Curation activities include:

– Archiving: A curation activity which ensures that data is properly selected, stored, can be accessed and that its logical and physical integrity is maintained over time, including security and authenticity.

– Preservation: An activity within archiving in which specific items of data are maintained over time so that they can still be accessed and understood through changes in technology.

http://www.jisc.ac.uk/media/documents/programmes/preservation/e-sciencereportfinal.pdf



Intro to Digital Data CollectionsThree levels of collections

R h

g

Research“small science,” local, limited to no

funding, not often standardResource

driven by a research community, some funding, emerging standards

ReferenceReferenceLarge scale, generally homogeneous

and standardized data, for use by multiple communities for various purposes

http://www.nsf.gov/pubs/2005/nsb0540/



Small Science or Research Collectionsdata and web site often managed “in house” – generally no plans for long-term access or preservation



Resource and Reference Collections



Issues to discussIssues to discuss…• Access• Persistence• Provenance

Ingest and scale• Ingest and scale• Intellectual property and permissions• PoliciesPolicies• Selection and appraisal• Metadata• Preservation



AccessAccess

• user interface• interoperabilityy



PersistencePersistence

• Provides the ability to "cite" data: a uniqueProvides the ability to cite data: a unique identifier and persistent, resolvable link

• Example: Handle system (digital object• Example: Handle system (digital object identifiers)



ProvenanceProvenance“Data provenance is the description of the origins p p g

of a piece of data and the process by which it arrived in a database.” (Buneman et. al., 2001)

– in the context of databases

Data provenance is “information that helps determine the derivation history of a data product starting from its original sources ”product, starting from its original sources. (Simmhan et. al., 2005)



Provenance is used forProvenance is used for…• Reliability and quality• Justification and audit (e.g. audit trails)• Reusability, reproducibility and repeatability

– “replication recipes”• Change and evolution• Ownership, security, credit and copyright

– attribution• Migration and storage• Aggregation• VersioningC. Goble, "Position Statement: Musings on Provenance, Workflow and (Semantic Web) Annotations for Bioinformatics," in Workshop on Data Derivation and Provenance, Chicago, 2002.

Y. Simmhan, B. Plale, and D. Gannon (2005). A survey of data provenance in e-science. SIGMOD Record, 34, 31-36.



Ingest & ScaleIngest & Scale

t t h ibl• automate processes as much as possible• batch and scheduled ingest• scale is a problem with large datasets• don’t mistake size for valuedon t mistake size for value



Intellectual Property & A C lAccess Control

• Who owns the data?• Who is allowed to access the data?

– open access / IP / local user group / directory (LDAP, Active Directory, NDS) / Shibboleth / ( , y, )dark archive, embargo

– proprietary, clinical, privacy (FERPA)



PoliciesPolicies

• selection policyselection policy• submission policy

ti li• preservation policy• usage policy



Selection and appraisalSelection and appraisal• Scientific criteria

– relevance; size; scope• Technical

internal format; size; transfer (media)– internal format; size; transfer (media)• Administrative

– documentation; privacy; ownershipp y p• Financial

– costs (data preparation and management over time)



MetadataMetadata• Descriptive • Administrative

– rights– provenance

• Technical – file formats– components and relationships

• Preservation



PreservationPreservation

• When is an IR part of a preservation plan?When is an IR part of a preservation plan?

N d i l t d t li i• Need in place… metadata, policies, • commitment from institution

– resources



Lab Activity #2 (20 minutes)Lab Activity #2 (20 minutes)Select a data collection from the provided list of repositories that is in your area

of interest. Explore the repository or datasets and try to answer these ti Sh d di fi di ith thquestions. Share and discuss your findings with the group.

• Who administers the repository?p y• How is the data organized?• What metadata format is used for description? What is being described?• How do they acquire data?• Who owns the data?• Who owns the data?• Is there a data selection policy?• Is there a data use policy?• Is it clear how to use the data?• What services are offered by the repository (e.g., search, browse,

preservation)?



ReflectionReflection

Take 10 minutes to revisit your “data repositoryTake 10 minutes to revisit your data repository prospectus” and relate the activities and discussion to your local environment. Write down any new ideas or observations and try to fill in any remaining blanks.

We will share these at the end of the workshop for l i di iour closing discussion.



How is it that data collections are coming to be part of the UIUC IR?

• Currently working with:25 years of vole demographics– 25 years of vole demographics

– Polar temperature data100 f d t– 100 years of corn data

– XRay Crystallography data



Crystallography:Extending Collaborations and Services

Raw Data

Extending Collaborations and Services

UIUCCIFs

OA

I Dat

aP

rovi

der

OAI Request

OAI Request

OAI Response

End User

ter

earc

h

Aggregated Metadata from CIFs

Cambridge U. CIFs

OA

I Dat

a P

rovi

der

OAI Request

OAI Response

OAI RequestO

AI

Har

vest

Sof

twar

e

SPEC

TRa

Se

Raw Data

Raw Data

Other CIFs

OA

I Dat

aP

rovi

der

OAI Response

OAI Request

Other CIFsOther

CIFsCIFsOther CIFsOther

CIFs



Supporting interdisciplinary research ith d t ti t P dwith data curation at Purdue…

d f lib i t t i l• new dean of libraries, new strategic plan• creation of the Distributed Data Curation Center

(D2C2) http://d2c2 lib purdue edu(D2C2), http://d2c2.lib.purdue.edu• Interdisciplinary Research Librarian: help

integrate library science and librarians into g yinterdisciplinary research; Discovery Park

• experimentation > projects > towards production i f t t d iinfrastructure and services

• Purdue e-Data



Project plans

&&

closing discussionclosing discussion



Data Repository Prospectus

What is the value of a data repository to your specific organization and how does it fit into your mission?

Who will use your repository and what will their needs be?

What services or features will be offered by your repository?

What different kinds of data will you collect in your repository? Will you include other kinds of research materials?

Introduction to Institutional Data Repositories – March 4, 2008 – Michael Witt & Melissa Cragin

What formats might you use to describe data collections and datasets?

What is your plan for acquiring content and building data collections? (How will data be submitted or ingested into the repository? What might you include in a selection policy?)

How might you market the data repository?

What are some ways you could measure and evaluate the success of the data repository?

How will you address intellectual property and access control concerns?


What human resources will be required to launch and then maintain the repository? (For example, what resources will be needed to manage the ingest process? Who will prepare and upload the data? What kinds of metadata and description will you require?)

What different roles and responsibilities will be needed and how do these map to your organization and personnel?

What information resources are available for you to learn more about repository software to facilitate the selection process for your organization? (Who else in your organization might have valuable information that would be helpful to you in making technical decisions?)

Just for fun… What are some possible names for your data repository?

List of Data Repositories

Organization Web address Notes

1.

NOAA’s National Environmental Satellite, Data, and Information Service http://www.nesdis.noaa.gov/datainfo.html

Entry-way for several data centers and collections.

2. CIA World Factbook https://www.cia.gov/library/publications/the-world-factbook/index.html

geographic, economic and population data

3. US Census Bureau http://factfinder.census.gov/home/saff/main.html?_lang=en census data (various) 4. USGS Earthquake hazards program http://earthquake.usgs.gov/eqcenter/catalogs/ 5. National Snow and Ice Data Center http://nsidc.org/

6. North Temperate Lakes Long Term Ecological Research http://lterquery.limnology.wisc.edu/index_new.jsp

7. British Atmospheric Data Center http://badc.nerc.ac.uk/home/index.html

8.

ICPSR - The Interuniversity Consortium for Political and Social Research http://www.icpsr.umich.edu/ICPSR/access/index.html

9. Roper Center Public Opinion Archives http://www.ropercenter.uconn.edu/data_access.html 10. UK Data Archive http://www.data-archive.ac.uk/findingData/data.asp

11. GBIF - Global Biodiversity Information Facility http://data.gbif.org/welcome.htm Also see: http://ge.gbif.net/

12. Harvard’s Freebase http://freebase.com/view/allDomains “open, shared database of the world’s knowledge”

13. Data360 http://www.data360.org/index.aspx "clarify the current condition" 14. StatCrunch http://www.statcrunch.com/ “data analysis on the web”

15. Long Term Vole Demographic Data and Selected Publications http://www.ideals.uiuc.edu/handle/2142/161

“…two primary sets of long-term vole demographic data files collected and maintained by Professor Lowell L. Getz.”

16. Evolutionary Infrastructure: Boston's Back Bay Fens http://www3.iath.virginia.edu/backbay/

Small data collection for a humanities project

17. Allen Institute For Brain Science - Allen Brain Atlas http://www.brain-map.org/welcome.do

Maps of brain “geography” at many scales

18. fMRI Dcata Center http://www.fmridc.org/f/fmridc cognitive neuroscience collection



SELECTED BIBLIOGRAPHY

1. NSF National Science Board. Long‐lived digital data collections enabling research and education in the 21st Century. September 2005. http://www.nsf.gov/pubs/2005/nsb0540/.

2. ARL/NSF Workshop on Long‐Term Stewardship of Digital Data Collections. To stand the test of time: long‐term stewardship of digital data sets in science and engineering. September 2006. http://www.arl.org/pp/access/nsfworkshop.shtml.

3. Anna Gold. Cyberinfrastructure, data, and libraries, part 1: A cyberinfrastructure primer for librarians. D‐Lib Magazine, 13(9/10), September/October 2007. http://www.dlib.org/dlib/september07/gold/09gold‐pt1.html.

4. Anna Gold. Cyberinfrastructure, data, and libraries, part 2: Libraries and the data challenge: Roles and actions for libraries. D‐Lib Magazine, 13(9/10), September/October 2007. http://www.dlib.org/dlib/september07/gold/09gold‐pt2.html.

5. Microsoft Research. Towards 2020 science. March 2006. http://research.microsoft.com/towards2020science/downloads/T2020S_ReportA4.pdf

6. NSF Cyberinfrastructure Council. Cyberinfrastructure vision for 21st century discovery. March 2007. http://www.nsf.gov/od/oci/CI_Vision_March07.pdf.

7. Philip Lord, Alison Macdonald, Liz Lyon, David Giaretta. From data deluge to data curation. Proceedings of the UK e‐Science All Hands Meeting, September 2004. http://www.ukoln.ac.uk/ukoln/staff/e.j.lyon/150.pdf.

8. Oya Y. Rieger. Select for success: Key principles in assessing repository models. D‐Lib Magazine, 13(7/8), July/August 2007. http://www.dlib.org/dlib/july07/rieger/07rieger.html.

9. NSF Blue‐Ribbon Advisory Panel on Cyberinfrastructure. Revolutionizing science and engineering through cyberinfrastructure. January 2003. http://www.nsf.gov/od/oci/reports/atkins.pdf.

10. Inter‐University Consortium for Political and Social Research (ICPSR). Guide to social science data preparation and archiving. 2005. http://www.icpsr.umich.edu/access/dataprep.pdf.

11. Ross Harvey. Appraisal and selection. Data curation manual, Digital Curation Centre. 2007. http://www.dcc.ac.uk/resource/curation‐manual/chapters/appraisal‐and‐selection/appraisal‐and‐selection.pdf.

12. Neil Beagrie. Digital curation for science, digital libraries, and Individuals. The International Journal of Digital Curation, 1(1), 3‐16. 2006. http://www.ijdc.net/ijdc/article/view/6/5.

13. Chris Greer. A vision for the digital data universe. January 2007. http://www.nanohub.org/resources/2291/.

14. NISO Framework Advisory Group. A framework of guidance for building good digital collections. Second edition, 2004. http://www.niso.org/framework/Framework2.pdf.

15. Richard L. Moore, Jim D’Aoust, Robert H. McDonald, David Minor. Disk and tape storage cost models. Proceedings of the 2007 IS&T Archiving Conference. http://www.sdsc.edu/~mcdonald/content/papers/dt_cost.pdf.

introduction to institutional data repositories workshop

Documents