Recognize Progress: Measuring Outcomes Not Outputs

Yvonne Attard, Director, Customer Development, Oakville Public Library

Rebecca Jones, Partner, Dysart Jones Associates

Agenda

• Basis for discussion• Measurement System: On Overview• Critical success factors• Options• Can you really measure impact?• Measurement Framework – Theory and Reality:

Oakville example• Gaps and hurdles• “Libraries change lives”

Basis for discussion

• There is no one magic measure• There IS a strong correlation between an

organization’s success and:• Its clarity of purpose• Its ability to understand its culture and its stakeholders’• A well-designed performance measurement system

that fits that culture.

• “Value” is client or stakeholder defined• Definition is dynamic• It is two-dimensional: economic & psychological• It is relative to alternatives

Basis for discussion

• Measuring for Results: The Dimensions of Public Library Effectiveness by Joe Matthews, 2004

• Asserts that few public libraries have a “culture of assessment”– Difficult and complex– Most measures indicate past performance– No cause-and-effect relationship between measures– Performance measures measure quantitatively, but library

outcomes are largely qualitative

Measurement System: an overview

Outcomes

OutputInput

Impact

Measurement System - Input

• Inputs are basically resources or capabilities:• Financial resources*******

– Content – Staff– Technology– Facilities

• Ultimately, who enables you to obtain these inputs?

Measurement System - Output

The service or program created by using the Input.

Measurement System - Outcomes

What the client is able to do with the service or product; how the client uses the Output.

Measurement System - Impact

The effect or influence of the service or program.

What changed for the client and/or the client organization.

How the library changed a life or a group of lives.

Measurement System - Example

Input $100K collection budget

Output 2500 new resources in collection

Outcomes 75% of students in the community schools used collection resources

Impact % of students graduating rose 5% in the past 2 years; EQAO results improved in 3 of the 5 community schools

Measurement System - Oakville

Input $20,000 spent on an outreach program to students budget

Output 2,500 students reached

Outcomes 1,500 new student library cards issued during outreach program and increased use in online resources

Impact Students assignment grades increase, ease of research, teachers view marked improvement in studies

Systems View

Library Clients

Stakeholders

Feedback

Outputs

Inputs

Impact

Critical Success Factors

For defining, measure & communicating progress & impact?

• Alignment of stakeholder goals and your goals– Balance of government demands vs. business plan vs. library

demands

• Your definitions, measures & communication style match those of your stakeholders, other service organizations, non-profits and others in the local community

• “Doable” capture and analysis of measures• Measures communicated in a meaningful and

understandable way for stakeholders• Conversations outside the library

Some Options

• Balanced Scorecard• LibQual• Cost-benefit

Balanced Scorecard

• Aligns measures with strategies to track progress, reinforce accountability and prioritize improvement opportunities

• A “system” of measures based on 4 perspectives:– customer – internal– financial– Innovation

• Limits measures to those most critical

Customer PerspectiveHow do we look to our clients?

Goals Measures

Financial PerspectiveHow do we look to

our funders or stakeholders?

Goals Measures

Innovation PerspectiveHow can we improve

& create value?Goals Measures

Internal PerspectiveWhat must we

excel at?Goals Measures

Kaplan & Norton

Balanced Scorecard

Balanced Scorecard: Example

METRIC U.1.a. OVERALL RATING IN STUDENT AND FACULTY SURVEYS.

Target1: A score of at least 4.00 (out of 5.00) from each of the major constituencies: undergraduate students, graduate students, humanities faculty, social science faculty, science faculty.

Target2: A score of at least 3.90 from each of the major constituencies.

Method: The University Library conducts extensive surveys of our clientele approximately every other year. A final question in each survey asks the respondent to "rate your overall satisfaction…" with the Library using a 1 to 5 scale. This metric will consider the two latest surveys on record.

Benefits of Scorecard

“…a clear understanding of what drives value within your area and what doesn’t, greater insight into senior management’s strategic plans, and a better knowledge not only of the strategic role you play within the organization but how you can enhance that role and sit at the decision-making table”

Joseph DeFeo, Measuring What Matters.

Industrial Management, v.42,n.3, May 2000

LibQual+(TM)

• Based on ServQual • Collaboration of Association of Research

Libraries (ARL) & Texas A&M University Libraries

• Diagnostic tool for measuring library users' perceptions of service quality

• Identifies gaps between desired, perceived, & minimum service expectations

• Defined survey questions, dimensions, & data gathering processes for academic libraries

http://www.libqual.org/

Benefits of LibQual+(TM)

• Provides better understanding of patrons’ perceptions & desires, & how services are meeting these

• Libraries using this have learned that they must keep narrowing categories & questions

Cost-benefit

• Does the benefit exceed the cost?• St. Louis Public Library

• Measures the value people place on the consumption of a service in excess of what they pay to receive that service

• Found benefits received >$10 for each dollar of tax support

• Other public libraries place a fair market value on their output measures

• San Diego Library and Miami-Date Public Libraries; benefits exceed costs by 6:1

Benefits of cost-benefit

• Stakeholders may easily understand

• Difficult, however, to determine valid market prices

Understandthe

context

Alignobjectives

Definemeasures

Managecollection

Interpretdata

Communicateresults

Critical Success Factor:

Measurement Framework

Adapted from Measuring What Matters: A Library/LRC Outcomes Assessment Manual by Lindauer

Context for your framework

• Who are your key stakeholders?• Who do you really need to prove your value to?

• How do they prove themselves?• What measures/indicators do they use?

• Ask yourself, “in our environment, what is it critical for us to we measure?”

Page 1 of Framing Template

Framework: Align

• This is where you invest the most time• Requires conversations with councillors, or

local government “superiors” • Profile critical stakeholders

• What are their goals & objectives?• What makes them ‘look good’ to their

superiors or constituents?

Page 2 of Framing Template

Align: Profile Stakeholders

• Stakeholders are those who can put a stake of support under your organization or a stake of destruction through your organization

• Superiors• Funders• Constituents or market• Clients & potential clients

Align: Know Your Stakeholders

• Who are your stakeholders• How do they determine value?• What comprises value to them?• How do they express it?• What are the most important questions they have?• What are the most important decisions they make?• What are they communicating to their clients?• Which of their goals & objectives do you contribute

towards?

Framework: Define

• Clearly define this alignment by articulating & documenting your goals & objectives

• Clarify how these contribute towards your stakeholders’ desired outcomes

• Test these goals & objectives with your stakeholders

• Test them with staff to ensure they are in keeping with your purpose & will aid with planning & decisions

Page 2 of Framing Template

Framework: Identify

• How will you track your success/progress towards meeting these goals & objectives?

• What indicators or measures will you use?– Qualitative? Quantitative?– What data needs to be collected? How?– Keep it key (KISS principle)– Don’t get mired

Page 3 of Framing Template

Framework: Collect

• Do it• Determine a collection schedule• Are you currently collecting data or indicators that

are no longer relevant?• How long do you need to keep data?• Who is responsible?

Page 3 of Framing Template

Framework: Analyze & Interpret

• So…..what?– What does the data say? What doesn’t it say?– Examine it from various angles– What is the progress towards the goals &

objectives?

Page 3 of Framing Template

Framework: Communicate

• Actually begins back at the ‘define’ stage• If your goals & objectives are meaningful for

stakeholders, your measures will be too• Your message to them• If your goals & objectives are meaningful for your

planning & decision-making, your measures will be indispensable

• Your message to you & to staff: “we have to start, we have to stop & we have to continue…”

Page 3 of Framing Template

Oakville Public Library

• Background• Oakville Library Board operates on 3 year

business planning model• Arms-length relationship with Town of

Oakville Community Services Commission, operates on a rolling 3 year Integrated Business Planning model

• Brand focused business plan goals and KPIs

Context

• Key stakeholders: Municipality, board, donors, public, staff, other libraries

• Measures that are important– efficiency measures (ROI) and – effectiveness measures (usage, market

penetration etc.)

ALIGN and DEFINE - Oakville Public Library Business Plan 2005 At A Glance

“The Centre for Learning – Your Gateway to Knowledge”

To provide the ideal environment for the customer’s learning experience.

To provide the ideal environment for the customer’s learning experience.

To support, educate, motivate and recognize staff and volunteers in providing the ideal

environment for learning.

To support, educate, motivate and recognize staff and volunteers in providing the ideal

environment for learning.

• Increase awareness• Increase value in the community• Increase usage• Align organizational culture with external position in the community

• Increase awareness• Increase value in the community• Increase usage• Align organizational culture with external position in the community

VisionThe ultimate goal, a

future state, a picture of what we want to be.

MissionThe purpose of the organization, for all stakeholder groups.

ObjectivesMeasures how well we achieve our mission.

StrategiesThe means used to achieve the mission.

Customer Service Priorities• Brand Integration• Collections Management• Outreach Services• Early Childhood Literacy

Corporate Priorities• Strategic Planning• Board Orientation & Advocacy• Human Resources Strategies• Town Integration• Infrastructure Management• E-services Development

IDENTIFY – KPIs used at Oakville

Value Maintain satisfaction level in Citizen’s Survey

Maintain Cardholders per capita 70%

Usage Number of service transactions per capita 47.56

Awareness Overall awareness of breadth of programs & services improved

Align culture Satisfaction measure through informal customer satisfaction & staff awareness survey

Number of service

transactions

> 1.0 Annual Expenditures

COLLECT

• Tools used for collection include:– Citizen’s survey – measures value and

awareness every 3 years– Cardholders per capita (Dynix/Horizon)– Informal satisfaction surveys and feedback forms

– focus/advisory groups, web surveys, print surveys

– Collection satisfaction survey

ANALYZE AND INTERPRET

• Service transactions = circulation + people entering + program attendance + outreach + information queries answered + e-transactions

• Trends analysis• Comparative analysis with other areas

(libraries, town departments)• Variance reporting

COMMUNICATION

• Look at frequency, depth and message and customize for specific stakeholder

• Yearly communication to municipality through budget process

• Yearly communication to Library board, with quarterly updates throughout the year

• Annual Report to the Community to donors and public

• Semi-annual updates to staff through semi-annual reports, intranet and staff newsletter

Gaps and Hurdles

• Consistency in measurement over the years• Ability to measure• Staff engagement and education• Detail of measurement required for different

stakeholders• Meaningful communication• Extrapolation errors – it’s part of the equation• Cause and effect – direct or indirect?• Advocacy plan• Identify comparators across libraries, local

community groups and other non profits etc.

Can you really measure impact?

• Are there times you cannot?

Can we measure impact?

Sometimes what counts can’t be counted, and what can be counted doesn’t count.

– Albert Einstein

We should be a bit wary of the “little library” …For when it is good, it is very, very good

and when it is bad, it’s a “pretty good library for a town this size.”- E. Rodger, New Zealand Libraries, March 1990

Measuring Impact - “Libraries change lives”

• “Knowledge of alphabet letters at entry into kindergarten is a strong predictor of reading ability in 10th grade” – Every Child Ready to Read, West Bloomfield Township

• “…it helped be improve my ability to help students in their essay and civic topics; all useful” says a teacher-librarian of the Youth Online! program