discovery or displacement?: a large scale longitudinal study of the effect of discovery systems on...
DESCRIPTION
Plenary session for Charleston Conference 2013. Authors: Michael Levine-Clark, John McDonald, Jason Price. In this first large scale study of the effect of discovery systems on electronic resource usage, the authors present initial findings on how these systems alter online journal usage by academic library researchers. The study examines usage of content hosted by four major academic journal publishers at 24 libraries that have implemented one of the major discovery systems, EBSCO's EDS, Ex Libris' Primo, OCLC's Worldcat Local, or SerialsSolutions’ Summon. A statistically rigorous comparison of COUNTER-compliant journal usage at each library from the 12 months before and after implementation will determine the degree to which usage rises or falls after discovery tool implementation and address rumors that discovery tools differ in their impact on electronic resource usage.TRANSCRIPT
![Page 1: Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage](https://reader034.vdocument.in/reader034/viewer/2022051514/5484044db47959e20c8b4a92/html5/thumbnails/1.jpg)
Discovery or Displacement? A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online
Journal Usage
Charleston Conference November 7, 2013
Michael Levine-‐Clark, University of Denver John McDonald, University of Southern California
Jason Price, SCELC ConsorJum
![Page 2: Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage](https://reader034.vdocument.in/reader034/viewer/2022051514/5484044db47959e20c8b4a92/html5/thumbnails/2.jpg)
“…a steep increase in full text downloads and link resolver click‐throughs suggests Summon had a dramatic impact on user behavior and the use of library collections during this time period.” The Impact of Web-scale Discovery on the Use of a Library Collection Doug Way (2010) http://scholarworks.gvsu.edu/library_sp/9/
![Page 3: Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage](https://reader034.vdocument.in/reader034/viewer/2022051514/5484044db47959e20c8b4a92/html5/thumbnails/3.jpg)
h"p://www.oclc.org/partnerships/econtent/solu4ons.en.html
Vendor marke5ng
![Page 4: Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage](https://reader034.vdocument.in/reader034/viewer/2022051514/5484044db47959e20c8b4a92/html5/thumbnails/4.jpg)
Does implementation of a discovery service impact journal usage?
![Page 5: Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage](https://reader034.vdocument.in/reader034/viewer/2022051514/5484044db47959e20c8b4a92/html5/thumbnails/5.jpg)
![Page 6: Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage](https://reader034.vdocument.in/reader034/viewer/2022051514/5484044db47959e20c8b4a92/html5/thumbnails/6.jpg)
Web-‐scale discovery services
• Single source for finding informaJon – Books – ArJcles – Local content
• Metadata and/or full text
• Content is pre-‐indexed and/or pre-‐harvested
• Single fast search
ILS
HathiTrust
MLA Bibliography
InsJtuJonal Repository
Publisher Metadata
Discovery Service
![Page 7: Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage](https://reader034.vdocument.in/reader034/viewer/2022051514/5484044db47959e20c8b4a92/html5/thumbnails/7.jpg)
An assump5on
• At any given insJtuJon, given a relaJvely stable user base, the total search effort will remain roughly the same. – X students will have Y assignments and Z hours per day to search
– X faculty will publish Y papers and have Z hours per day to search
![Page 8: Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage](https://reader034.vdocument.in/reader034/viewer/2022051514/5484044db47959e20c8b4a92/html5/thumbnails/8.jpg)
Discovery services § Will take up an increasing amount of a finite Jme for searching
§ Will draw users from other (more or less efficient) search tools
§ Will alter the overall producJvity of searches (users will find more or less)
§ Will alter the overall efficiency of users (users will access more or less full-‐text)
![Page 9: Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage](https://reader034.vdocument.in/reader034/viewer/2022051514/5484044db47959e20c8b4a92/html5/thumbnails/9.jpg)
Prior studies
• Some studies have indicated substanJal increases in usage a]er Discovery implementaJon – DescripJve staJsJcs only – Single insJtuJon studies only
• Some publishers report decreased usage of content – Anecdotal, may affect some and not others
![Page 10: Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage](https://reader034.vdocument.in/reader034/viewer/2022051514/5484044db47959e20c8b4a92/html5/thumbnails/10.jpg)
Data collec5on • List of libraries with discovery services
> Searched on lib-‐web-‐cats • Surveyed Libraries
> Discovery service Implemented > ImplementaJon Date (month/year) > Search box locaJon > MarkeJng effort
• 149 Libraries Gave Approval > 24 libraries selected for this phase > 6 for each of the 4 major discovery services
![Page 11: Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage](https://reader034.vdocument.in/reader034/viewer/2022051514/5484044db47959e20c8b4a92/html5/thumbnails/11.jpg)
Library demographics • 20 US, 1 each from UK, AUS, NZ, CA • 10 ARL Libraries included • WorldCat book holdings
> Average: 1,114,193 > Median: 1,044,153 > High: 2,665,796 > Low: 298,365
• ImplementaJon dates: > 2010 (3), 2011 (19), 2012 (2)
![Page 12: Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage](https://reader034.vdocument.in/reader034/viewer/2022051514/5484044db47959e20c8b4a92/html5/thumbnails/12.jpg)
![Page 13: Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage](https://reader034.vdocument.in/reader034/viewer/2022051514/5484044db47959e20c8b4a92/html5/thumbnails/13.jpg)
Dataset
• 24 Libraries
• 4 Discovery services
• 6 Publishers
• 9,206 Journals
• 159,278 ObservaJons
• 141,048 Usable ObservaJons
![Page 14: Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage](https://reader034.vdocument.in/reader034/viewer/2022051514/5484044db47959e20c8b4a92/html5/thumbnails/14.jpg)
Methodology Compared COUNTER JR1 total full text arJcle views for the 12 months before vs 12 months a]er implementaJon date
June
2010
Start
Implem
entaJo
n May 201
1
May 201
2 End
Year 1 Year 2
Included implementaJon month in Year 1 to ensure that both periods included an enJre academic year
![Page 15: Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage](https://reader034.vdocument.in/reader034/viewer/2022051514/5484044db47959e20c8b4a92/html5/thumbnails/15.jpg)
Collec5ons notes
o Excluded journals that did not have 24 months of COUNTER reporJng
o Limited ability to control for changes in aggregator, backfile access, or expanded holdings
o Outliers removed from analysis
![Page 16: Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage](https://reader034.vdocument.in/reader034/viewer/2022051514/5484044db47959e20c8b4a92/html5/thumbnails/16.jpg)
![Page 17: Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage](https://reader034.vdocument.in/reader034/viewer/2022051514/5484044db47959e20c8b4a92/html5/thumbnails/17.jpg)
![Page 18: Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage](https://reader034.vdocument.in/reader034/viewer/2022051514/5484044db47959e20c8b4a92/html5/thumbnails/18.jpg)
![Page 19: Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage](https://reader034.vdocument.in/reader034/viewer/2022051514/5484044db47959e20c8b4a92/html5/thumbnails/19.jpg)
![Page 20: Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage](https://reader034.vdocument.in/reader034/viewer/2022051514/5484044db47959e20c8b4a92/html5/thumbnails/20.jpg)
![Page 21: Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage](https://reader034.vdocument.in/reader034/viewer/2022051514/5484044db47959e20c8b4a92/html5/thumbnails/21.jpg)
![Page 22: Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage](https://reader034.vdocument.in/reader034/viewer/2022051514/5484044db47959e20c8b4a92/html5/thumbnails/22.jpg)
General trends
• VariaJon by insJtuJon within each discovery service
• VariaJon by publisher within each discovery service
• Some publishers saw overall net increase, while some experienced a decrease in usage
![Page 23: Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage](https://reader034.vdocument.in/reader034/viewer/2022051514/5484044db47959e20c8b4a92/html5/thumbnails/23.jpg)
![Page 24: Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage](https://reader034.vdocument.in/reader034/viewer/2022051514/5484044db47959e20c8b4a92/html5/thumbnails/24.jpg)
Goals of our inferential statistics Determine whether observed differences are significant or resulted from chance effects
Determine which of the three factors (i.e. library, publisher, discovery service) contribute to determining differences in usage change at the journal level
Start with an exploratory analysis and end with a comprehensive model
![Page 25: Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage](https://reader034.vdocument.in/reader034/viewer/2022051514/5484044db47959e20c8b4a92/html5/thumbnails/25.jpg)
ANOVA - Analyzing the data
Observation = Fit + Residual Change In = Library x + Publisher y + Disc Svc z + Residual Err usage +17 = (+2) + (-3) + (+10) + (+8)
After Cobb 2003 Introduction to design and analysis of experiments. Fig 3.1
![Page 26: Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage](https://reader034.vdocument.in/reader034/viewer/2022051514/5484044db47959e20c8b4a92/html5/thumbnails/26.jpg)
ANOVA – F Ratio Tests whether the means for levels within a factor are distinguishable from each other
Average variability due to the factor F-ratio = --------------------------------------------------- Average variability due to chance error So, when F ≈ 1, means are not distinguishable, when F is > 1, there are real differences among some means
![Page 27: Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage](https://reader034.vdocument.in/reader034/viewer/2022051514/5484044db47959e20c8b4a92/html5/thumbnails/27.jpg)
Does usage change vary across libraries?
Institution (sorted by Mean Change)
Overall Average = 8.5
![Page 28: Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage](https://reader034.vdocument.in/reader034/viewer/2022051514/5484044db47959e20c8b4a92/html5/thumbnails/28.jpg)
Does usage change vary across libraries?
Institution (sorted by Mean Change)
Overall Average = 8.5
![Page 29: Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage](https://reader034.vdocument.in/reader034/viewer/2022051514/5484044db47959e20c8b4a92/html5/thumbnails/29.jpg)
Does usage change vary across publishers?
Publisher (sorted by Mean Change)
Overall Average = 8.9
![Page 30: Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage](https://reader034.vdocument.in/reader034/viewer/2022051514/5484044db47959e20c8b4a92/html5/thumbnails/30.jpg)
Does usage change vary across discovery services?
Overall Average = 8.9
![Page 31: Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage](https://reader034.vdocument.in/reader034/viewer/2022051514/5484044db47959e20c8b4a92/html5/thumbnails/31.jpg)
Does the affect of discovery service differ across publishers?
![Page 32: Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage](https://reader034.vdocument.in/reader034/viewer/2022051514/5484044db47959e20c8b4a92/html5/thumbnails/32.jpg)
Does the affect of discovery service differ across publishers?
Publishers (distinguished by color)
![Page 33: Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage](https://reader034.vdocument.in/reader034/viewer/2022051514/5484044db47959e20c8b4a92/html5/thumbnails/33.jpg)
Do the discovery service means differ in the 2 way model?
Publishers (distinguished by color)
4.5
12.3
15.0
3.7
![Page 34: Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage](https://reader034.vdocument.in/reader034/viewer/2022051514/5484044db47959e20c8b4a92/html5/thumbnails/34.jpg)
Publishers (distinguished by color)
4.5
12.3
15.0
3.7
![Page 35: Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage](https://reader034.vdocument.in/reader034/viewer/2022051514/5484044db47959e20c8b4a92/html5/thumbnails/35.jpg)
Publishers (distinguished by color)
Do publisher means differ significantly in the two way model?
23.8
-3.9
6.8-9.5
![Page 36: Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage](https://reader034.vdocument.in/reader034/viewer/2022051514/5484044db47959e20c8b4a92/html5/thumbnails/36.jpg)
Publishers (distinguished by color)
Do publisher means differ significantly in the two way model?
23.8
-3.9
6.8-9.5
![Page 37: Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage](https://reader034.vdocument.in/reader034/viewer/2022051514/5484044db47959e20c8b4a92/html5/thumbnails/37.jpg)
Does the affect of discovery service differ across publishers?
Publishers (distinguished by color)
![Page 38: Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage](https://reader034.vdocument.in/reader034/viewer/2022051514/5484044db47959e20c8b4a92/html5/thumbnails/38.jpg)
Publishers (distinguished by color)
![Page 39: Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage](https://reader034.vdocument.in/reader034/viewer/2022051514/5484044db47959e20c8b4a92/html5/thumbnails/39.jpg)
Full Model – including disc srvc, publisher, and library
![Page 40: Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage](https://reader034.vdocument.in/reader034/viewer/2022051514/5484044db47959e20c8b4a92/html5/thumbnails/40.jpg)
ANOVA Model including all three factors
![Page 41: Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage](https://reader034.vdocument.in/reader034/viewer/2022051514/5484044db47959e20c8b4a92/html5/thumbnails/41.jpg)
Results - Can we detect differences between Discovery Services, Publishers, and/or Libraries and/or their interactions?
Discovery Service – Yes Publisher – No Library – Yes Differential discovery service effect by publisher – Yes
Differential library effect by publisher -- Yes
![Page 42: Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage](https://reader034.vdocument.in/reader034/viewer/2022051514/5484044db47959e20c8b4a92/html5/thumbnails/42.jpg)
Interpreta5ons & Conclusions > Analyzing usage is a complex task > No discovery service increased or decreased usage across all libraries and/or all publishers
> Discovery service and publisher as variables on their own were significant predictors of usage change
> InteracJon of Discovery service & Publisher was significant
> Some control needed for no discovery service and for size of insJtuJon.
![Page 43: Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage](https://reader034.vdocument.in/reader034/viewer/2022051514/5484044db47959e20c8b4a92/html5/thumbnails/43.jpg)
A plethora of pending possible pursuits • Design & test for effects of:
– Aggregator full text availability – InsJtuJon Size / Enrollment Profile – Publisher Size – Journal Subject – Overall usage trends (Requires Disc Srvc ‘control’) – ConfiguraJon opJons in Discovery services
• Follow-‐up presenta5on at UKSG (April 2014) – Including Control group & AddiJonal libraries – Add AddiJonal variables & further analysis