METRICS

· Keeping Track of it all: The Challenge

of Measuring Digital Resource Usageby Christopher StewartAvailable online 18 February 2011

A s online content continues to expand at a rapid pace, ourability to measure usage of digital resources is one of thefundamental management challenges facing academic librar-

ies today. These collections have grown to include an array ofelectronic resources and consume larger portions of the librarybudget, far outweighing print budgets in most libraries. Justifyingspending for these resources is an important, critical, and ongoingexercise that is predicated on our ability to measure use and demandfor these resources. It requires sophisticated technical tools and staffexpertise. Full text, online content has been available in academiclibraries for over two decades. As we enter another, it is a good timeto review our capacity to make data-driven decisions in a time ofcontinuing change for the academic library. In this column, I willoutline the evolution and current state of some key methods used formeasuring use of electronic resources.

While there is a myriad of information one can endeavor tocapture when investigating e-resource use (though the tools maynot be available), there are, in general, two types of inquiry thatdescribe how most academic libraries currently go about measuringuse of subscription-based, online material. The first method ofinquiry is quantitative and straightforward. This method, conductedthrough counting tools, measures how often a resource is used. Thesecond type of inquiry seeks more descriptive information abouthow online resources (and services) are used. This methodtypically involves a survey instrument and seeks information notabout many times a resource is selected, but also by whom and forwhat purpose. We will begin with the first method, starting with abrief background.

Collecting data on e-resource usage is a central part of decisionmaking for collection development. Cost, use, and demand all factorinto the equation, and data are reported internally as well as,increasingly, externally for national surveys, peer analysis, accred-itation, and other purposes. In recent years, methods for measuringuse of digital resources have evolved, though the process can still betime consuming and complex for limited data return. A coretechnical underpinning that enables basic usage information to begathered is the OpenURL framework, developed in the late 1990sand adopted as NISO standard Z39.88 in 2004.1 OpenURL enables“interoperability between a resource and a service component,”connecting users to appropriate resources.2 User context is mediatedby a link resolver such as Ex Libris' SFX, which works in tandem withOpenURL to present information to the user after the user movesthrough a gateway authentication system such as a proxy server,

Christopher Stewart is Assistant Professor, Graduate School of Libraryand Information Science, Dominican University, 7900 W. Division St,River Forest, IL 60305, USA<cstewart@dom.edu>.

174 The Journal of Academic Librarianship, Volume 37, Number 2

, pages

which generally assigns an IP address “identity” to the user(although more sophisticated identity schemes such as Shibbolethare now being used).

Link resolvers such as SFX are important components of the toolmatrix for measuring use of online resources. SFX and an OpenURLframework provide basic data that can be generated into reports onuse of resources that are OpenURL compliant and routed through thelink resolver. Before new standards such as SUSHI (which will bediscussed shortly) became more widely adopted, SFX provided one ofthe few methods for garnering data on e-resource usage other thanweb and proxy server log files. SFX “click through” reports providedata on howmany times a journal is accessed. Equally important, SFXdata provide information on user choice for full text, online resourcesnot available through the home institution. For many years, however,a number of major database providers were not OpenURL compliant,which limited the ability of libraries to gain comprehensive data on e-resource use. As most major vendor platforms are now OpenURLcompliant, most academic research libraries use some type linkresolver and drive access to digital resources through the resolver fora number of reasons. Today, even non-subscription based resourcessuch as Google Scholar are OpenURL compliant.

While generating usage data using log files, SFX reports, and webanalyzers is still common, different tools are now available formeasuring access and use of digital resources. In recent years,standards developed earlier in the decade have become more widelyadopted by publishers and other content providers. These standardsand protocols have changed theway libraries deal with usage data andhave, for themost part, shiftedmost data collection to the vendor via ahosted service. The first of these protocol standards is SUSHI.

SUSHI, the Standardized Usage Statistics Harvesting Initiativeprotocol was adopted by the National Information StandardsOrganization (NISO) in 2007 and is intended to “replace timeconsuming, user-mediated collection of data usage reports.”3 SUSHIwas designed to be both “generalized and extensible,” enabling it tobe used for a “variety of usage reports.”4 SUSHI is designed to workwith another standard, COUNTER (Counting Online Usage ofNetworked Electronic Resources). COUNTER is an international effortaimed at setting standards that enable reporting of usage statistics ina “consistent, credible, and compatible way.”5 The COUNTERinitiative meets an obvious need for libraries to “better understandhow the information they buy from a variety of sources is used,” andthe needs of publishers to know how the “products they disseminateare being accessed.”6 The most recent COUNTER Code of Practice wasissued in 2008.

Over the past few years, most journal database providers havebecome COUNTER compliant. Usage reports for these resources aretypically generated via vendor provided interfaces, usually web-based. COUNTER reports provide session data that log file analysis and

174–176

other methods for generating usage reports cannot produce. Forexample, in a recent report I ran for Illinois Institute of Technology's(IIT) IEEE (Institute of Electrical and Electronic Engineers), subscrip-tion, one of the library's most expensive full text databases, theCOUNTER report provided information on the number of sessions andarticle requests by journal, per month. Another query of IIT's GaleVirtual Reference subscription provided a breakout of number ofaccesses for 95 e-book titles.

COUNTER reports are useful, and to a certain degree address basicreporting needs—but they are by no means a comprehensive metric.Vendor supplied usage datamust bemanually generated and exported,usually as a delimited text file that must then be imported into aspreadsheet. Librarians gather information from each informationprovider and aggregate data as necessary. At IIT, for example, staffdownload data fromover 60 vendor sites in order extract usage data formost of the library's subscription-baseddatabases and journals. This canbe a very time consuming process, and one that must be repeated atdifferent times of the year to support decision making in an often fluidbudgetary environment. For many libraries manually extracting thesedata is the only available method for generating usage reports.

This is not to say, however, that there are not other tools available forcompiling and aggregating usage reports. Third party providers offertools for integrating COUNTER and other e-resource usage data.SwetsWise Selection Support, for example, is a subscription-based servicethat provides a single point for usage statistics for approximately 60vendor platforms. SwetsWise Selection Support's value propositionincludes the ability for the customer to “utilize a wealth of standardizedreports to uncover the underlying usage trends of your collection,including low-usage journals and price per use built on saving theuser.”7 Dashboard reports eliminate “manually collating usage statisticsand cost information.”8 While SwetsWise and other providers provide asingle point of service for aggregating and organizing usage statisticsfrom a variety of content providers, these services are not inexpensiveformost libraries. Librarians seeking to subscribe to oneof these servicesmust balance the cost against funds that could be spent on collections,staff, and other mission-critical resources.

Integrated library systems also offer tools for measuring use ofdigital resources. Ex Libris' Verde e-resource management toolprovides “extensive usage and cost statistics,” that enable librarystaff to make “data-driven collection development decisions regard-ing serials and e-books.”9 Verde is SUSHI/COUNTER compliant and caningest data on a range of e-resources including databases, journals, e-books, and other digital collections. Verde also serves as a resourceand licensemanagement tool. Formany libraries, however, even thosewho currently use an ExLibris system, adding Verde may present costchallenges in today's economic climate.

Vendor-supplied data are by far the most popular source forgarnering e-resource usage information for academic libraries.10

While the SUSHI harvesting protocol and COUNTER standards areengines for generating reports built on basic criteria, they are notappropriate tools for generating deeper data on user demographicsand user preference for particular resources. User behavior anddemographics are important elements in a comprehensive analysis ofe-resource usage. For these types of data, the web-based survey isgenerally the accepted method of inquiry. Franklin and Plum11 listseveral reasons why web-based surveys are generally not preferredover vendor-supplied data. Foremost among these reasons is thatvendor-supplied data are based on true counts, whereas web-basedsurveys are based on sampling. In addition to the problem of responsebias, web-based surveys are not based on real usage, but onremembered or intended activity. Finally, web-based surveys oftenfocus on users and sessions, but not actual usage. All of these factorsresult in a not-surprising preference among academic librarians torely on vendor-provided usage reports.12

Vendor-supplied data, while useful, nonetheless leave a gap inconnecting e-resource usage with user demographics. The web-basedsurvey is a tool with enormous potential to enhance the librarian'smetric toolkit. A good recent example of a web-based surveyimplementation is the Association of Research Libraries MINES(Measuring the Impact of Networked Electronic Services) project.MINES is an “online, transaction-based survey that collects data on thepurpose and use of electronic resources and on the demographics ofusers.”13 MINES has been used successfully in recent years, mostnotably in two implementations at the Ontario Council on UniversityLibraries (OCUL) – a consortium of 21 research university libraries inthe province of Ontario – in 2004 and again in 2010.14 Themost recentiteration of MINES at OCUL illustrates the survey's potential foracademic and research libraries.

Unlike web-based surveys that query users before or afterinformation seeking activity, MINES “intercepts a user as they attemptto access an electronic resource.”15 In the 2010 OCUL MINES survey,user participation was either voluntary or mandatory based on theparticipating institution's policy on conducting studies on faculty andstudent populations. The survey request appears as the user isrequesting a resource. Once the user completes the brief survey, heor she is passed through to the resource requested. The 2010–2011OCUL survey queried five facets: patron status (graduate, undergrad-uate, faculty); departmental affiliation; patron location (off campus,on campus); purpose of use (including course work and, if for originalresearch, whether it was funded/unfunded); and why the patronchose the particular resource. To ensure systematic sampling, thesurvey was delivered at random every 250 instances within thetwelve-month period between February 2010 and February 2011.16

TheMINES survey instrument spans multiple services including, ofcourse, electronic databases and journals, but also institutionalrepositories, e-books, and library catalogs.17 The preliminary 2010results of the MINES survey are interesting, and are of course best leftto the survey's architects and implementation team to discuss. Whatis important to note in the immediate is MINES' potential as acomplimentary tool to COUNTER-based metrics. Real time, targetedtools such as MINES can provide the necessary links between overallusage data provided by COUNTER compliant publishers and deeper,equally meaningful information on the “who, what, where, and why”of library e-resource usage.

As survey instruments such as MINES allow for deeper analysis ofuser choices, new tools designed both to measure usage and enableuser-centered collection development will evolve in the comingyears. These types of systems will, according to one leadingdeveloper, “shed light on scholarly information based on whatusers are telling each other.”18 These recommender services havethe potential to create a kind of usage measurement loop for e-resources and other materials. Articles recommended by the systemto the searcher are based on similar searches by other searchers.Usage can be measured directly at the article level. While these toolsare currently being used for scholarly resources in the “commercialspace,”19 they will undoubtedly evolve beyond subscription-based e-resources to include digital repositories, archival systems, and othertypes of native digital content.

One of the ironies we face as information professionals is thechallenge of managing data about our own enterprise, particularlyusage data for digital resources. The tools available for this task areevolving and improving, but they are far from ideal. The evolutionand widespread adoption of SUSHI and COUNTER provide us with away to garner data in relatively simple form directly from contentproviders as well as through intermediary products offered bylibrary systems vendors and others. Survey-based tools such asMINES have the potential to provide a more complete framework foranalyzing e-resource usage and our ability to make more informed

March 2011 175

collection development decisions. To be sure, we have come a longway from the days of log file parsing, SFX reports, and proxy serverlogins. The work of developing and integrating tools in affordable,usable, and scalable ways for the mutual benefit of libraries andcontent providers will continue. For most academic researchlibraries, collection budgets for digital resources long ago surpassedprint collections. Subscription and purchasing decisions for theseresources must be data-driven. I have no doubt that contentproviders, integrated library systems architects, and our librariancolleagues working on these issues will keep up the good workunderway in developing and improving the metrical tools we use tomeasure e-resource use. We have come a long way in recent years,but there is still much work to be done.

NOTES AND REFERENCES

1.NISO Standards - National Information Standards Organization.http://www.niso.org/kst/reports/standards?step=2&gid=None&project_key=d5320409c5160be4697dc046613f71b9a773cd9e(December 10, 2010).

2. Ex Libris the bridge to knowledge, OpenURL. http://www.exlibrisgroup.com/category/sfxopenurl (December 8, 2010).

3. Standardized Usage Statistics Harvesting Initiative (SUSHI) -National Information Standards Organization. http://www.niso.org/workrooms/sushi (December 8, 2010).

4. Ibid.5. Ibid.6. COUNTER - Online Usage of Electronic Resources. http://www.projectcounter.org/ (December 10, 2010).

176 The Journal of Academic Librarianship

7. Swets - SwetsWise Selection Support.http://www.swetswise.com/web/show/id=1933301/langid=42 (December 14, 2010).

8. Ibid.9. Ex Libris the bridge to knowledge, Overview. http://www.

exlibrisgroup.com/category/VerdeOverview (December 14, 2010).10. Franklin B. & Terry P. “Successful Web Survey Methodologies for

Measuring the Impact of Networked Electronic Resources (MINESfor Libraries),” IFLA Journal 32, no. 1 (2006): 28.

11. Ibid., p.29.12. Ibid.13. Association of Research Libraries: MINES for Libraries®: Measur-

ing the Impact of Networked Electronic Services. http://www.arl.org/stats/initiatives/mines/index.shtml (December 15, 2010).

14. Davidson, C., and Kyrillidou, M. 2010. The Value of ElectronicResources: Measuring the Impact of Networked Electronic Services(MINES for Libraries®) at the Ontario Council of University Libraries.Research Library Issues: A Bimonthly Report from ARL, CNI, andSPARC. August.

15. Davidson, C., Plum, T., Kyrillidou, M., and Thomas, D. 2010.Measuring Use of Licensed Electronic Resources: A SecondIteration of the MINES for Libraries® Survey on Scholars Portaland Other Resources for the Ontario Council of University Librariespresented at the Library Assessment Conference, October 26,Baltimore, MD. http://libraryassessment.org/bm~doc/davidson_catherine.pdf (December 10, 2010).

16. Ibid.17. Ibid.18. Oren Belt-Arie, Chief Strategy Officer, Ex Libris, LTD., in discussion

with the author, December 2010.