Management Metrics That Work

Karen White USAID

kwhite@usaid.gov

Kris Vajs

Federal Reserve Board Kris.M.Vajs@frb.gov

Karen Krugman

Export-Import Bank of the US Karen.krugman@exim.gov

April 9, 2013

Agenda

Advantages of Management Reporting

Collecting, Storing, & Analyzing Metrics

Presenting & Packaging Metrics

Management Reporting at Our Organizations

Questions & Answers

Why Report to Your Management?

3

To communicate the library’s value and demonstrate return-on-investment Reason 1

To generate support for continued or even increased staff and resources Reason 2

To demonstrate how the library supports the organization’s mission and strategic goals Reason 3

To demonstrate transparency and accountability in how resources are used Reason 4

To show how library contributions have changed over time Reason 5

Metrics: What Categories To Measure?

4

Collect This Data: • Library Size and Patron Base • Transactions: Circulations and Reference • Budget, Resource Allocation, and Value of Print &

Digital Materials • Facilities and Physical Foot Traffic • Virtual Foot Traffic • Education and Outreach • Operations

To Measure These Areas:

• Customer Satisfaction • Return on Investment • Effectiveness of Patron Outreach & Education • Impact

• Comparability to Like Libraries

Sample Metrics

5

Library Size and Patron Base

• Percentage of actual patrons to potential patrons

• Percentage of patrons seeking assistance in person, via e-mail, via phone, or via web

• Percentage of patrons by patron type

Transactions: Circulations &

Reference

• Total number of checkouts per year

• Percentage of checkouts by patron type

• Percentage inbound vs. outbound ILL requests

Budget, Resources Allocation, & Value

of Materials

• Percentage increase or decrease in overall budget over time

• Change in print vs. digital resource budget allocation over time

• Average cost per full-text article downloaded

See handout for more examples.

Sample Metrics (continued)

6

Facilities & Physical Foot

Traffic

• Number of patrons entering library, year over year

• Average duration of a library visit

• Average duration of use of various physical spaces

Virtual Foot Traffic

• Average number of library website visits per day

• Most used pages on library website

• Increase in database users over time

Education & Outreach

• Number of database trainings and attendees per year

• Increase in database users after database trainings

• Number of new library users after attending orientation

Operations

• Average minutes to process, catalog, and shelve an item

• Average daily library staff time spent on core work vs. non-core work (troubleshoot-ing computers, looking for passwords, etc.)

See handout for more examples.

Storing Your Metrics

7

Type Pros Cons

Microsoft Excel

• No cost, easy to setup • Can produce pie charts & graphs • Can export to Access and create

a database

• Inflexible • Not a true database

Relational Database (in-house or cloud-based)

• No cost, somewhat easy to setup • Consistent data that can be

searched, manipulated and reported

• Automatic calculations and reports

• Higher data storage capacity • Data exportable to multiple tools

• Steeper learning curve than Excel.

• Needs care in planning and setup

• Sometimes created for a different purpose

• If in-house, needs to interact with the network on an ongoing basis.

Virtual reference software (i.e., Altarama RefTracker, Springshare LibAnalytics)

• Collects & analyzes data for you • Reports & graphics on demand • Consolidates requests from chat,

email, phone & SMS text • Supports a searchable

knowledge base • Ability to tag individual metrics

• More expensive • Security concerns if hosted by

vendor • Possibility of vendor going out

of business

8

Decide what you’re trying to measure before you begin to collect metrics

Collect data proactively so you have them when needed

Collect quantitative and qualitative metrics. Collect data, stories and testimonials

Identify metrics from other libraries and organizations or industry standards that can be used to contextualize your metrics

Look to library vendor-supplied usage data as a resource

Limit the number of metrics to provide focus and to concentrate on the metrics with greatest impact

Collecting Your Metrics

9

Ways To Collect Metrics Data

Tally sheets/Forms

Surveys/Customer Feedback/Observations

Focus Groups/Interviews

Benchmarking

10

Interpreting Your Metrics

Number of library patrons Take simple measures

Percentage of actual patrons to potential patrons

Use ratios to compare two measures

Year-over-year increase or decrease in actual patrons to potential patrons

Use trends to see changes over time

Percentage of actual patrons to potential patrons vs. like libraries

Use benchmarks to compare to like libraries

Year-over-year increase or decrease in actual patrons to potential patrons vs. like libraries

Combine benchmarks and trends to compare to like libraries over time

See handout for more examples.

11

Presenting Your Metrics: Best Practices

Report on a regular basis: weekly, monthly, quarterly or annually

Use creative, single-use metrics to supplement your ongoing metrics program and illustrate unique points

Tailor your reporting to different audiences (executives, your supervisor, the public, your customers)

Compare metrics to past years, a small sample of similar libraries, or industry averages to create a frame of reference

Use a mixture of quantitative & qualitative data; if possible, have your report tell a story with words, pictures, and numbers.

Understand how your metrics relate to each other and be prepared to answer questions about “what it all means”

Share metrics and reports with your staff to foster an understanding of how they contribute to the library’s success

12

Presenting Your Metrics: Packaging Methods

Written Reports

Verbal Presentations

Dashboards

Infographics

Impact stories

13

Our Management Metrics: USAID

About the USAID Library

Official Department Name Knowledge Services Center

What the

Organization Does

Promotes economic prosperity; strengthens

democracy; improves global health, food

security, environmental sustainability &

education; provides humanitarian assistance.

Staff Size 11,000 Agency staff

Subject Matter International development

Our Customers USAID staff in Washington & overseas, U.S.

government agencies, the public

14

Our Management Metrics: USAID

Our Metrics Experience

How We

Collect the Data

• From our shared mailbox into SharePoint

• At the reference desk, saved in a Google Doc

• Customer feedback from emails into SharePoint

• Database usage from vendors saved in Excel

What Data

We Collect

• Type of request

• Type of client (public, partner, USAID bureau or mission)

• Customized research by sector and topic

• Door count, circulation, PC & WiFi use

• E-resource usage

• Results of outreach activities (new employee orientations,

office presentations, open houses) in terms of requests

• Use examples from customers

How We Interpret &

Report on the Data

• Weekly, monthly & annual reports

• Mixture of quantitative & qualitative information

• Trends analysis

• How we contribute to the “USAID Forward” goals

Our Management Metrics: Ex-Im Bank

About the Ex-Im Bank Library

Official Department Name The Research Library & Archives

What the

Organization Does

It backs the financing of US exports, mainly to

emerging markets.

Exports can be products (Boeing jets, oil rigs,

nail files, candy) or services (engineering,

architects)

Staff Size 400 FTE in agency; 7 staff in Research Library

Subject Matter International Business and Economics

Our Customers Chair & Board, Economists, Bankers, Lawyers;

Staff from Policy, Communications, Business

Development

15

Our Management Metrics: Ex-Im Bank

Our Metrics Experience

How We

Collect the Data

• Collected through Quickbase, a cloud-based relational

database.

• Library staff report data via dropdowns, checkboxes, and

written overviews of requests (mostly copied from e-mail)

• Anecdotal evidence collected through written and verbal

follow-up

What Data

We Collect

• Research, research, research:

• Who makes requests; purpose of requests; sources

used; time spent; % actual vs. potential users; highest

volume users

• To collect going forward: ROI of databases, gap analysis

How We Interpret &

Report on the Data

• When we report: weekly reports detailing research

requests; annual reports analyzing the year.

• Annual report contains different formats of information:

charts showing trends, pie charts showing concentrations,

tables showing impact, and bulleted lists detailing past

accomplishments & future goals.

16

17

Our Management Metrics: Federal Reserve Board

About the Board Research Library

Official Department Name Board Research Library

What the

Organization Does Provide the nation with a safer, more flexible,

and more stable monetary and financial system

Staff Size 1800 professional staff members; 13 staff in

Research Library

Subject Matter Economics, banking, and finance

Our Customers Economists, financial analysts, research

assistants, and other professionals. The Library

is open to all Board staff.

18

Our Management Metrics: Federal Reserve Board

Our Metrics Experience

How We

Collect the Data

• Service Interaction Access Database – used to collect

and generate data on customer use of library services

• Ongoing metrics are supplemented by “snapshot” data

demonstrating impacts. A snapshot looks at sample data

to make a specific point

• SharePoint site – money saved on data purchases

• SurveyMonkey – just beginning to use this to get

feedback from customers on projects and services

• Vendor statistics

What Data

We Collect

• Access database – customer name, division, library staff

member, type of request, length of time to craft response,

notes

• Our favorite metrics – market penetration and percentage

of repeat customers. We use repeat customers as an

indication of customer satisfaction.

• ROI – money saved and cost avoidance calculated for

data acquisitions work.

How We Interpret &

Report on the Data

• Annual budget presentation

19

More Resources To Get Started

• “5 Best Practices for Creating Effective Dashboards,” Tableau White Paper, August 2011. http://www.tableausoftware.com/learn/whitepapers/5-best-practices-for-effective-dashboards

• Ard, Constance. “Beyond Metrics: The Value of the Information Center,” Information Outlook (September 2012).

• Ard, Constance. Adding Value to Corporate Libraries and Information Services. London: Ark Group, 2012.

• Davis, Hilary. “Not Just Another Pretty Picture,” 2009. http://www.inthelibrarywiththeleadpipe.org/2009/not-just-another-pretty-picture/

• Dugan, Robert E., Peter Hernon, and Danuta A. Nitecki. Viewing library metrics from different perspectives: inputs, outputs, and outcomes. Santa Barbara, Calif.: Libraries Unlimited, 2009.

• Hales, Stuart. “Metrics for Special Libraries,” Information Outlook (September 2012).

• Hiller, Steve. “Performance Measurement in Libraries,” paper presented at the NISO Webinar on Measure, Assess, Improve, Repeat: Using Library Performance Metrics, September 8, 2010. http://www.niso.org/apps/group_public/download.php/4872/performance10web.pdf

20

More Resources To Get Started • Hiller, Steve. “What Are We Measuring, and Does it Matter?” Information

Outlook (September 2012). • Hiller, Steve. “Performance Measurement in Libraries,” paper presented at the

NISO Webinar on Measure, Assess, Improve, Repeat: Using Library Performance Metrics, September 8, 2010. http://www.niso.org/apps/group_public/download.php/4872/performance10web.pdf

• Kyrillidou, Martha. “Looking Ahead: The Future of Performance Metrics,” paper presented at the NISO Webinar on Measure, Assess, Improve, Repeat: Using Library Performance Metrics, September 8, 2010. http://www.niso.org/apps/group_public/download.php/4872/performance10web.pdf

• “More Alike than We Think,” Library Journal vol. 136 no. 8 (2011).

• Stevens, Kerrie, “Introduction to Library Metrics: Statistics, Evaluation, and Assesment, Australian and New Zealand Theological Library Association, unknown date. http://www.anztla.org/Conference10/postconf10/Stevens_1.pdf

• Strouse, Roger. Information management under fire: measuring ROI for

enterprise libraries. Outsell, Inc., Nov. 9, 2007. 11 p.

• Strouse, Roger. Using ROI to support the IM value proposition. Outsell, Inc., May 27, 2010. 18 p.