beyond roi: transitioning assessment into action

8/12/2019 Beyond ROI: Transitioning Assessment into Action

1/22

University of Colorado

Boulder | Denver | Anschutz Medical Campus

Beyond ROITransitioning Assessment into Action

The Acquisitions Instituteat Timberline LodgeMonday May 21, 2012

Denise Pan, University of Colorado DenverLeslie Williams, University of Colorado, Health Sciences Library

Gabrielle Wiersma, University of Colorado BoulderYem Fong, University of Colorado Boulder


2/22



Outline

Colorado Context

Defining Value

Pilot methodology, participants & results

Considerations & recommendations

UCDenver.edu
http://www.ucdenver.edu/admissions/tours/Pages/index.aspxhttp://www.ucdenver.edu/admissions/tours/Pages/index.aspx


3/22



University of Colorado Context

University ofColorado System Three institutions

Four campuses Five libraries

Tradition of cooperative purchases to

leverage budgets and negotiate prices facilitate shared and commonly held resources

Streamline procurement and legal review

CU.edu
https://www.cu.edu/https://www.cu.edu/https://www.cu.edu/content/campuses


4/22



University of Colorado Context

Downturn in economy and Boulder campusbudget cuts

Needed to communicate Library collaborativepurchasing to administrators

Conducted Return on Investment (ROI) andCost-benefit Analysis (CBA)

Demonstrated the value of jointly leveraginglibrary budgets to university administrators

Pan, D. and Fong, Y. (2010) ROI of Collaborative Collection Development:Evaluating the Cost-Benefit of Consortia Purchasing. Collaborative Librarianship,2 (4), 183-192. http://collaborativelibrarianship.org/index.php/jocl/article/view/108/72

Colorado.edu
http://collaborativelibrarianship.org/index.php/jocl/article/view/108/72http://ucblibraries.colorado.edu/research/index.htmhttp://ucblibraries.colorado.edu/research/index.htmhttp://collaborativelibrarianship.org/index.php/jocl/article/view/108/72http://collaborativelibrarianship.org/index.php/jocl/article/view/108/72


5/22



Denver Campus Context

Auraria Library Program Review

Analyze the contribution of LibraryCollections to faculty teaching andresearch outcomes How is the relationship between library

resources and instructional and scholarlyoutcomes measured?

How will the results obtained from

measuring the value of Library Collectionsbe used to revise and strengthen theprogram(s)?

UCDenver.edu
http://www.ucdenver.edu/student-services/Pages/Library.aspxhttp://www.ucdenver.edu/student-services/Pages/Library.aspx


6/22



Defining Value

Implicit Value Assumes that libraries are valued because they are used Demonstrated via usage stats (e.g. circulation, gate counts, etc.) Does not demonstrate outcomes

Explicit Value Asks users directly Measured by qualitative techniques (e.g. interviews) Requires asking users about their outcomes

Derived ValuesROI Use multiple types of data collected on both the returns

(benefits) and the library costs (investment) to explain value inmonetary terms

- Tenopir, Beyond Usage, 2011


7/22



Pilot Methodology

Implicit values Curriculum vita

analysis

Publication citation

analysis

Explicit values

Interviews

Derived Value Contingent Valuation

Office.com
http://office.microsoft.com/en-us/images/results.aspx?qu=meeting&ctt=1http://office.microsoft.com/en-us/images/results.aspx?qu=meeting&ctt=1


8/22



Contingent Valuation Methods

The willingness of the market to payfor non-market resources or services. Monetary value of having access to

library collections Pilot based market value on the price

to download journal articles frompublisher

Contingent valuation compares the time orother costs of not having a service with the

time or cost of the service. This method hasbeen used to estimate the value of electronic

resources, especially journals.- Oakleaf, Value of Academic Libraries, 2010

Office.com
http://office.microsoft.com/en-us/images/results.aspx?qu=stackhttp://office.microsoft.com/en-us/images/results.aspx?qu=stack


9/22



Pilot Implementation

Denver - focused on the Chemistry departmentrecommended by Dean and support fromDepartment Chair

Boulder - recruited

faculty from severaldepartments; usedfaculty library

committee AMCfocused on

physical therapy department

Office.com
http://office.microsoft.com/en-us/images/results.aspx?qu=stairway&ctt=1http://office.microsoft.com/en-us/images/results.aspx?qu=stairway&ctt=1


10/22



The Colorado Model

Based on Cost Benefit Analysis (CBA) andReturn on Investment (ROI) CBAis the ratio showing dollar value of benefits

gained for dollar value of costs.Benefits Costs = CBA

ROI is the percentage showing the return or increasein value on dollars spent to achieve a benefit.((BENEFITS COSTS) COSTS) x 100

Benefits= Estimated cost for faculty to buy articles

directly from publisher with pay-per-view Costs= Library costs for online journals/databasescited

10


11/22



Variable Data Source

A # of published articles in 1 year CV

B # of all cited references Citation Analysis

C # (%) of cited references from journals Citation Analysis

D# (%) of cited references from online

subscriptionsCitation Analysis

E # of additional articles read (but not cited) Interview Q1

F# of additional articles read from journals (but not

cited)E x C%

G# of additional articles read from online journals

(but not cited)F x D%

H# of citations and articles read (but not cited) from

online resourcesD + G

IAverage cost per article (random sampling of

publisher pay-per-view prices)Cost Estimates

J Estimated cost to buy articles directly frompublisher with pay-per-view

H X I

KLibrary costs - online journal subscriptions for

cited referencesCost Estimates

Pilot StudyVariables

based on

interviewswith faculty onlibrary resourceusage

&citationanalysisof publishedarticles/chapters

Interview Q1 how manyadditional articlesdid you read butdid not cite?

FacultyBenefits

LibraryCosts


12/22



Phase 1: Denver & Boulder Results

Value of purchased

articles

( J = Benefits to Faculty)

Journal

subscriptions

(K = Library costs)

CBA

(J / K)

ROI

(J K /K x 100)

Denver $99,057.94 $68,042.83 $1.46 46%

Boulder $147,004.85 $67,701.00 $2.17 117%

Cost Benefit Analysis (CBA) = $Benefits Costs

Return on Investment (ROI) = %((BENEFITS COSTS) COSTS) x 100


13/22



Phase 1: AMC Results

Value of purchased

articles

( J = Benefits to Faculty)

Journal

subscriptions

( K = Library costs)

CBA

( J / K )

ROI

(J K / K x 100)

Denver $99,057.94 $68,042.83 $1.46 46%

Boulder $147,004.85 $67,701.00 $2.17 117%

AMC $94,342.37 $127,558.76 $0.74 -26%

Cost Benefit Analysis (CBA) =$Benefits Costs

Return on Investment (ROI) = %((BENEFITS COSTS) COSTS) x 100

Negative CBA / ROI


14/22



Calculating Contingent Valuation

Phase 1 Representative sampling of

journals to purchase articlesfrom publisher

Calculate faculty benefit withaverage cost per article

Phase 2

Random sample of journals

Calculate Standard Deviationto determine low and highestimates Office.com
http://office.microsoft.com/en-us/images/results.aspx?qu=stackhttp://office.microsoft.com/en-us/images/results.aspx?qu=stack


15/22



Phase 2: Results with RandomSampling & Standard Deviation

Phase 1 AverageRepresentative

Sampling

Phase 2 Avg w/Random Sampling

Phase 2 LowStd Deviation

Phase 2 HighStd Deviation

Denver

cost per article$ 30.00 $ 34.22 $ 31.94 $ 36.51

CBA$ 1.46 $ 1.66 $ 1.55 $ 1.77

ROI 46% 66% 55% 77%

Boulder

cost per article$ 30.00 $ 32.53 $ 29.66 $ 35.41

CBA $ 2.07 $ 2.44 $ 2.05 $ 2.44

ROI 107% 144% 105% 144%

AMC

cost per article$ 33.00 $ 36.36 $ 30.44 $ 42.27

CBA$ 0.74 $ 0.81 $ 0.68 $ 0.95

ROI -26% -19% -32% -5%


16/22


17/22



Collaborative Assessment Framework

What are the benefits ofa ROI study?

Provides a framework for

implementing and evaluatingcollection development

In-depth analysis of a singleacademic unit

Qualitative liaison activity Quantitative data-driven

analysisOffice.com
http://office.microsoft.com/en-us/images/results.aspx?qu=books&ex=1http://office.microsoft.com/en-us/images/results.aspx?qu=books&ex=1


18/22



Collection Development Strategies

Outreach opportunity Identify gaps in collections

Citation analysis andinterviews

Evidence to supportpurchase-on-demandacquisition models

Informing cancellations Alternative to cost-per-use analysis

Could identify low use but highcitations

Office.com
http://office.microsoft.com/en-us/images/results.aspx?qu=spiral&ctt=1http://office.microsoft.com/en-us/images/results.aspx?qu=spiral&ctt=1


19/22



FAQs

Is the model scalable?

Are the results generalizable?

Are the results comparable across institutions?

Can the model accommodate disciplinarydifferences?

How best can the data be used? In conjunction

with other metrics?


20/22



Future Possibilities & Blue Sky

Potentially scalable with tools that extractcitation data from CVs and cost data in anautomated way (e.g. APIs)

Could be generalizable depending onsample size and scope of the target group(e.g. single department versus entirecampus or institution)

Expanding and adapting the model tomeasure ROI for non-journal resources(e.g. monographs, primary resources)

Office.com
http://office.microsoft.com/en-us/images/results.aspx?ex=2&qu=directionshttp://office.microsoft.com/en-us/images/results.aspx?ex=2&qu=directions


21/22



Resources Association of College and Research Libraries. (2010). Value of Academic Libraries: A

Comprehensive Research Review and Report. Researched by Megan Oakleaf.Chicago: Association of College and Research Libraries.Available online at: www.acrl.ala.org/value

Barnes, S. & Blake, M. (2010). Measuring Your Impact: Using Evaluation for LibraryAdvocacy, A library evaluation workshop. National Network of Libraries of Medicine,http://nnlm.gov/evaluation/workshops/measuring_your_impact/.

Kaufman, P. (2008). The library as strategic investment: Results of the Illinois return oninvestment study.Liber Quarterly, 18(3/4), 425.

Luther, J. (2008). University investment in the library: Whats the return?: A case study at the

university of Illinois at UrbanaChampaign. Available online at:http://libraryconnect.elsevier.com/whitepapers/0108/lcwp0101.pdf.

Pan, D. and Fong, Y. (2010) ROI of Collaborative Collection Development: Evaluating the Cost- Benefit of Consortia Purchasing. Collaborative Librarianship, 2 (4), 183-192.

http://collaborativelibrarianship.org/index.php/jocl/article/view/108/72

Pan, D., Wiersma, G., & Fong, Y. (2011). Towards demonstrating value: Measuring thecontributions of library collections to university research and teaching goals.Proceedings of the 2011 ACRL Conference Proceedings. Available onlineat:http://www.ala.org/acrl/files/conferences/confsandpreconfs/national/2011/

papers/towards_demonstratin.pdf Tenopir, C. (2012). Beyond usage: measuring library outcomes and value. Library

Management, 33(1/2), 5-13.

21
http://www.acrl.ala.org/valuehttp://nnlm.gov/evaluation/workshops/measuring_your_impact/http://libraryconnect.elsevier.com/whitepapers/0108/lcwp0101.pdfhttp://collaborativelibrarianship.org/index.php/jocl/article/view/108/72http://www.ala.org/acrl/files/conferences/confsandpreconfs/national/2011/papers/towards_demonstratin.pdfhttp://www.ala.org/acrl/files/conferences/confsandpreconfs/national/2011/papers/towards_demonstratin.pdfhttp://www.ala.org/acrl/files/conferences/confsandpreconfs/national/2011/papers/towards_demonstratin.pdfhttp://www.ala.org/acrl/files/conferences/confsandpreconfs/national/2011/papers/towards_demonstratin.pdfhttp://www.ala.org/acrl/files/conferences/confsandpreconfs/national/2011/papers/towards_demonstratin.pdfhttp://www.ala.org/acrl/files/conferences/confsandpreconfs/national/2011/papers/towards_demonstratin.pdfhttp://www.ala.org/acrl/files/conferences/confsandpreconfs/national/2011/papers/towards_demonstratin.pdfhttp://www.ala.org/acrl/files/conferences/confsandpreconfs/national/2011/papers/towards_demonstratin.pdfhttp://collaborativelibrarianship.org/index.php/jocl/article/view/108/72http://collaborativelibrarianship.org/index.php/jocl/article/view/108/72http://libraryconnect.elsevier.com/whitepapers/0108/lcwp0101.pdfhttp://libraryconnect.elsevier.com/whitepapers/0108/lcwp0101.pdfhttp://nnlm.gov/evaluation/workshops/measuring_your_impact/http://nnlm.gov/evaluation/workshops/measuring_your_impact/http://www.acrl.ala.org/value


22/22



Questions? Thank You!

Denise Pan

[email protected]

Gabrielle Wiersma

[email protected]

Leslie Williams

[email protected]

Yem Fong

[email protected]
mailto:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]

beyond roi: transitioning assessment into action

Documents