data-informed decision making for libraries - athenaeum21
Post on 15-Apr-2017
373 Views
Preview:
TRANSCRIPT
Data-Informed Decision Making
for Digital ResourcesChristine Madsen + Megan Hurst
Athenaeum21
Electronic Resource & Libraries ConferenceAustin, TX | April 2016
Library Assessment & Evaluation ProgramsPast, Present, Future● 3 case studies of assessment and evaluation programs in libraries● 3 different modes of data gathering and analysis● show the power of understanding user needs and how well your
organization is meeting them.
Agenda
1. Introduction (5 min)
2. Harvard University Library, Open Collections Program (10 min)
3. Resource Discovery @ The University of Oxford (10 min)
4. University of California, Davis, Open Library Assessment Dashboard & Toolkit (10 min)
5. Questions & Discussion (10 min)
Who We AreMegan HurstMLIS, MFA integrated media@MHzUX
Athenaeum21
Christine MadsenMLIS, DPhil (PhD)@mccarthymadsen
Athenaeum21
Expertise
● User experience research and design● Library assessment ● Publishing workflows and models
Work
Harvard, EBSCO, Athenaeum21
Expertise
● Digital library systems ● Digitization (doing it right)● Methods for measuring impact
Work
Harvard, Oxford, Bodleian, Athenaeum21
Setting Goals Is Easy, but Achieving Them Isn't65% of organizations have an agreed-upon strategy.
14% of employees understand the organization’s strategy.
Less than 10% of all organizations successfully execute the strategy.
(source Forbes.com)
Users
Timeline
2002 20172007 2012
2004Harvard University LibraryOpen Collections Program“Outreach and Evaluation” Program initiated
2015 University of Oxford Resource Discovery User Needs Analysis
2008UK JISC TIDSR Toolkit,Developed by Christine Madsen for the Oxford Internet Institute, based on Michalak, Madsen and Hurst’s work for Open Collections Program
2016- 2017 Library Assessment Dashboard & Toolkit, UC Davis w/ The Bodleian Libraries, Goettingen State and University Library
2007iPhone Released
2006 Google Analytics Released
2006 First “Assessment Librarian” position advertised
2007First “UX Librarian” position advertised
2015 Use and Usability Assessment of Digital Libraries Survey, Digital Library Federation
2006 Facebook opened to general public 13+ years old
2017NISO Link Origin
Tracking Best Practices
2002-2007
Harvard University LibraryOpen Collections Program
Open Collections Program, 2002-2007
● Digitization program aimed at making a
selection of Harvard’s primary sources
freely available to high school and
community college students
Open Collections Program, 2002-2007● Two initial goals of the program:
○ Increase the availability and use of historical resources from
Harvard's libraries, archives and museums for teaching, learning,
and research
○ Offer a new model for digital collections that will benefit students
and teachers around the world
Open Collections Program, 2002-2007Three main principles of quality:
● Selection Standards - Create comprehensive, topic-based digital collections by carefully selecting topics, and materials;
● Production Standards - Create digital surrogates that are both faithful to the original publications and of such high quality that there will be no need for re-digitization by other institutions;
● Access Standards - Provide easy online access to digital collections within the Harvard community and around the world.
○ A commitment not just to making the materials freely available online, but to making sure people knew these collections existed.
Open Collections Program, 2002-2007● Context / Climate
○ No “Assessment Librarian” or “User Experience Librarian” positions in libraries
○ Google Analytics did not yet exist
○ Online marketing and conversion analysis was still very young
○ Libraries dominated by a ‘build it and they will come’ approach to digitization
● Motivation for Assessment ○ To measure and increase usage
○ To demonstrate and report progress and impact to funders
Understanding & Prioritizing User Types
Students General Public
Open Web
LibrariansEducators
Target Users
Library Websites
Researchers
Open Collections Website/s
OCP: Tools & Methods for Assessment
1. Web Analytics (pre-Google Analytics)○ Early inlink and referral analysis
○ Homegrown “digital circulation” tracking
2. User Research
○ Task-completion studies
○ Ethnographic and heuristic studies
○ Surveys (undergraduates, educators)
○ Focus Groups
○ Interviews
OCP Tools: Highlight of Quantitative Methods
1. Homegrown “Digital Circulation” Trackinga. Time in book
b. # of pages viewed per book
c. Provided insights into user behaviors inside
digital objects (including multi-page items), and
across collection
OCP Tools: Highlight of “Outreach” Methods1. Early Search Engine Optimization (SEO) efforts
a. Added keywords to collection websites that reflected
open web searches, rather than library subject terms
b. Created web content that would attract open web
searchers in our target audience (before Google
Knowledge Graph existed)
i. Historical events
ii. Historical figures
iii. Historical organizations
c. Enhanced Wikipedia articles with OCP content
Lessons LearnedUse multiple methods, iterate, and triangulate!
● Match assessment methods to research questions● Analyze target audience and usage types to set expectations ● Assessment questions should then flow from your understanding of that audience
Understand the Data
● Repeat visitors (“Converts”) is more important than visits or hits● Inlinks (links to your site from other sites) are important and worth working for● Quality of use is important, because it can tell stories of impact
Outreach &
Evaluation
Program
Begins
Email & Listserv Outreach Campaign
Faithful Converts: Year-over-YearUnique
visitors who visited
2004 2005
1x 21,292 88,643
2x 2,414 8,015
3x 766 2,449
4x 392 1,107
5x 201 631
>5x 849 3,932
TOTAL 25,914 104,777
Lessons Learned, part 2Assessment helped to:
● Document and illustrate a clear connection between outreach efforts and collection usage
● Guide enhancements to digital services● Tell stories of impact and value
Lessons Learned, part 2
● Outreach & Evaluation Program ○ Cost 2% of project budget
(approx $30,000USD)○ Increased visits by 400% ○ Increased # and % of “ Converts”
(Visitors who visited >5x)
Lessons Learned, part 2
‘Well-marketed’ collections are still more heavily used, many years later, than those that have not been ‘marketed’
2008-2010
Toolkit for the Impact of Digitised Scholarly Resources
jisc - UK / Oxford Internet Institute
2015-
Resource Discovery@ The University of Oxford
● Context / Climate○ 92 Libraries (30 Bodleian; 41 College; 21 Department) 7 museums, 2 gardens○ Some have less than 50% of their collections ‘catalogued’ with electronic
metadata○ No single discovery solution (nor is there likely to be in the near future)
● Motivation for Assessment ○ To understand how Oxford should prioritise resources to maximise users’
discovery of resources across libraries and museums○ To scope new approaches to finding information and collections of relevance
to research and teaching at Oxford
Resource Discovery @ The University of Oxford
Peer Institutions
Users Metadata Experts
Vendors & Publishers
Literature Review
Roadmap
5 Strands of Research
Resource Discovery @ The University of Oxford
Synthesis
Tools & Methods:
● Interviews (in person)● Observational/
ethnographic research ● Think-aloud protocols
Users targeted:
● Undergraduate students● Graduate students● Faculty● Researchers● Staff● Alumni
Resource Discovery @ The University of Oxford
The Future of Finding @ OxfordOutcomes:
● Detailed roadmap for investment in tools and infrastructure
Lessons Learned:
● In depth, ethnographic (qualitative) methods are useful for assessing long term goals and outcomes
● Pattern recognition (that is, being able to ‘code’ your data) is an important part of making using of qualitative methods -- you need to be able to see the patterns
2016-
Open Library Assessment Dashboard & ToolkitUniversity of California, Davis
Setting Goals Is Easy, but Achieving Them Isn't65% of organizations have an agreed-upon strategy.
14% of employees understand the organization’s strategy.
Less than 10% of all organizations successfully execute the strategy.
(source Forbes.com)
Users
Open Library Assessment Dashboard & Toolkit
Context / Climate○ Lots of Assessment Librarians using lots of different tools
○ As many metrics and formulae as there are questions
○ Libraries are getting better at using data
○ Standards for digital content have evolved
Open Library Assessment Dashboard & Toolkit
Motivation for Assessment: ○ How do you give someone the ‘big picture’?
○ How do you deal with data overload?
■ Quality of use and kinds uses■ Online and off■ How much does it really cost?■ What is the impact? - before you can understand impact, you
have to have good data and understand how to use it.○ How do you make assessment useful throughout the organization?
Open Library Assessment Dashboard & Toolkit
The Project1. Research and scoping the work required to develop an open Library
Assessment Dashboard and Toolkit to give library leaders and managers a
single-page overview into all of their library data to better facilitate data-
driven decision making
2. Developing resource and cost estimates for a 2017 kickoff
Open Library Assessment Dashboard & Toolkit
Tools & Methods:
● Create an open, customizable framework of tools, data sources and standard metrics / key performance indicators (KPIs)
Users targeted:
● Senior library managers and directors
Dashboards from Other Industries
Sources: http://www.informationbuilders.com/products/intelligencehttps://www.johndaniel.com/index.php/solutions/industry-segment/healthcare/
Open Library Assessment Dashboard & Toolkit
Strategic(Quarterly to Annual)
Managerial(Daily to Monthly)
Metrics / KPIs Data Sources Metrics / KPIs Data Sources
Service Area 1 Key Management Questions >>Key formulas
Top 3-5 Data Sources Key Management Questions >>Key formulas
Top 3-5 Data Sources
Service Area 2 Key Management Questions >>Key formulas
Top 3-5 Data Sources Key Management Questions >>Key formulas
Top 3-5 Data Sources
etc Key Management Questions >>Key formulas
Top 3-5 Data Sources Key Management Questions >>Key formulas
Top 3-5 Data Sources
… Key Management Questions >>Key formulas
Top 3-5 Data Sources Key Management Questions >>Key formulas
Top 3-5 Data Sources
● Opening hours compared to demand● Availability of required titles● Percentage of rejected sessions● Ratio of requests received to requests sent out
in interlibrary lending● Staff per capita● User satisfaction● Library visits per capita● Seat occupancy rate● Number content units downloaded per capita● Collection use (turnover)
ISO 11620: Library Performance Indicators● Percentage of stock not used● Loans per capita● Percentage of loans to external users● Reference questions per capita● Ratio of acquisitions costs to staff costs● Acquisition speed● Lending speed● Interlibrary loan speed● Percentage of acquisitions expenditure
spent on the electronic collection● And more....
https://www.iso.org/obp/ui/#iso:std:iso:11620:ed-3:v1:en
● Currently engaged in user needs research with multiple libraries at varying stages of assessment
● Discussion!○ What are you measuring in your library today? ○ What would you like to be measuring?
Open Library Assessment Dashboard & Toolkit
More Information & Assessment Resources● Resource Discovery @ The University of Oxford Final Report, December 2015
● University of California, Davis, Open Library Assessment Dashboard & Toolkit Scoping Project
● Toolkit for the Impact of Digitised Scholarly Resources (TIDSR)
● Surveying the Landscape: Use and Usability Assessment of Digital Libraries. Digital Library Federation (DLF) Assessment Interest Group (AIG) User Studies Working Group White Paper, November 2015
● NISO Link Origin Tracking initiative
Questions? Comments? Please be in touch!
Athenaeum21www.athenaeum21.com
Christine Madsen (UK) madsen@athenaeum21.com
@mccarthymadsen
Megan Hurst (US)hurst@athenaeum21.com
@MHzUX
top related