Supporting Cyberscholarship in American Social History: The DLF

Aquifer Story

Katherine KottWith content contributed by Susan Harum and

Chick MarkleyCNI Task Force Meeting

December 11, 2007

Focus on American Social History Online

• Mellon funded project within DLF Aquifer initiative

• Outcome to make digital collections in American culture and life easier for scholars to find and use– Aggregate content to enable multiple

collections to be used as one– Deliver content through multiple channels

Who’s who

• American Social History Online relies on Aquifer working groups, collaborative infrastructure in place

• Core team focused on development and assessment, coordination with working groups– Susan Harum (UIUC) business analyst/assessment

expert– Kat Hagedorn (Michigan) data analyst– Chick Markley (Freewheeling contractor) systems

architect/developer– Tom Habing (UIUC) developer

DLF Aquifer participant libraries

• Bibliotheca Alexandrina

• California Digital Library

• Cornell University

• Emory University

• Indiana University

• Johns Hopkins University

• Library of Congress

• New York University

• Stanford University

• University of Illinois, Urbana-Champaign

• University of Michigan

• University of Minnesota

• University of Southern California

• University of Tennessee

• University of Virginia

• Yale University

Overview of accomplishments

• Architecture• Portal with Zotero—initial SEO analysis• MODS

– Guidelines– Levels of adoption– MARC to MODS transform

• Harvesting workflow• Agile development process

Website with Zotero in Firefox

RESERVED

How we got here

• Initially unsure we would build a website– SEO required for commercial search service scenario

– Zotero implementation requires website

– Website seemed like best way to present services (SRU) for Sakai and federated search scenarios

– Website could be used as cohesive way to present the project

– In June 2007, team set goal to have website ready for DLF Fall Forum in November 2007

Website development

• Started with baseline functionality on Michigan MODS portal

• Added and cross checked common business functions developed by SWG (integrated with DLF Services Framework)

• Architecture chosen to support– Website functionality– Sakai– Federated search

TWG architectural framework

ToJSONjava

MODSmysql

JSONmysql

to SOLRjava

Railsruby

SOLR/Lucenejava

HarvesterDLXS

eXistjava

to JSONjava

browsers

to eXistjava

services

bots

Architectural implementation

A plug for Ruby on Rails from our systems architect

• It's the framework• DRY (don't repeat yourself)• Tightly coupled testing harness• Exuberant and dedicated community

Enter, Agile development• Small development team, distributed

environment• Short timeline—no time for “traditional”

FRD process• Desire within digital library community for

lightweight process model• Emphasis on user driven development• Not to be confused with final assessment

activities for American Social History Online

Model

• Timeframe: August 29 – November 2, 2007 (8 weeks)

• Business Analyst identified 8 participants– 1 faculty member– 3 graduate students– 4 user services librarians

• Services Working Group and Business Analyst drafted questions

Method

• Initial conference call on August 27th, 2007– all participants– business analyst – systems architect– developer

• Questions regarding – search and find– identify– obtain– manage and use– user workflow

Initial Brainstorming call

• Pros– developers present to hear what participants say– developers able to prioritize next tasks

immediately after the call

• Cons– Level of participant feedback was lower than

anticipated– Too much developer involvement?

Participant feedback and development action

• Need to see size and scale of collection to determine how much time to spend searching and browsing– Implementation of collection registry and “browse collections” feature.

• Need ability to limit a search (once you have results) in many ways– Implementation of additional ways to limit after initial search– Implementation of date range limiting

• Importance of ability to browse– Implementation of subject clouds for browsing – still working on faceting for

additional browsing

• Importance of visual aids, especially for undergraduate students – Implementation of Simile timeline view for results– Thumbnail previews for item selection

Integration with collection registry at UIUC

Adaptively re-used technology

• CDL date normalization utility• UIUC thumbgrabber with asset

actions• Simile timeline from MIT• Collectus tool from UVA• Just say no to NIH syndrome

Implementation of “asset actions” using Collectus from UVa

Simile Timeline Integration

Participant wish lists for future development

• Limiting Search Results– users want to “drill down” in the search

• continue to narrow using different limiters, e.g. rights for use

• Social Networking and Tagging– Implementation of StumbleIt for full records– Monitoring Zotero server development– Penn tags

• Sorting and Narrowing– Plan to implement faceted browsing based on NINES

Collex flare development (NIH)

Tell me something I don’t

know

Lessons learned

• Agile Assessment effective in prioritizing tasks

• Multiple phone interviews and email updates resulted in buy in for participants

• Brainstorming session for distributed group not as effective as personal interviews

• Shorter more frequent 1:1 interviews yielded the best results

• Graduate students were most engaged

What’s next

• Add collections• Workflow, including

asset actions• Support for static

repositories• Bug fixing• Tagging? • Faceted browse• Map rights statement

to CC license?

• Evaluate ORE model (March meeting)

• Develop “local implementations”– Sakai @ Indiana– Federated search @

UIUC• More SEO• Assessment planning• Sustainability

planning

Actively seeking collections

• Advantage to contributor– SEO drives traffic to your collection(s)– Benefit from date normalization

• Seeking collections– Pertaining to American culture and life– All formats– MODS—can be mapped from MARC– Static repository setup for contributors without

OAI data providers

The DLF Aquifer engine

• From DLF for development to ??? for sustaining– Consistent attention

– Administrative support

• The working groups– Organizational links

– Community links

• Resources– From collaborators (directly and through investment in DLF)

– Grant funding

– Fees???

Sustaining Aquifer

• Part it out– Community standards– Software and services– Collections

• Take it to the next level?• Affiliate with another (service provider)

organization• Move to participant library host

Resources for ongoing support

“If enough people who are viewed as important want it, we will find a way to pay.” Paul Courant

• Assessment during development period is critical

• Revenue streams– Subscription

– Libraries pay for services• Increased visibility for collections

• Metadata enhancement

Emerging organizations sustaining inititiatives

• LOCKSS/CLOCKSS

• Fedora Commons

• DSpace Foundation

• Educopia Institute

Questions? Comments?

Website: www.dlfaquifer.org

kkott@diglib.org