demonstrating the library's impact through assessment and evaluation

V M G

Do you know your

organization's VMG?

Do you know your library’s

VMG?

Do you know the role of your

library in your organization? In

the society?

Do you know your library’s

purpose?

DEMONSTRATING THE LIBRARY’S

IMPACT THROUGH ASSESSMENT AND

EVALUATIONChristine M. Abrigo

De La Salle University Libraries

Managing Today’s Learning Commons: Re-skilling Seminar for information professionals

Philippine Association of Academic/Research Librarians, Inc.-20-22 September 2016, Baguio City

Explain the importance of purpose and evidence

Explain why evaluate Explain the measures for evaluation Explain about impact and impact

evaluation (looking beyond the numbers)

Workshop

OBJ

ECTI

VES

OF

THIS

LEC

TURE

Importance of Purpose and

Evidence

Purpose the reason why something is done or

used (Merriam-Webster, 2015)

something you do to achieve something the intention answers the questions: “Why”, “What for”

and “Who for”? Aim the main and overarching stated

purposes of the library service (Markless & Streatfield, 2013)H

AVIN

G A

PU

RPO

SE

Why is having a purpose important?

Library’s purpose = borne out of your organization’s VMG

Creating a purpose sets the direction to what you want to achieve and how you will achieve it

HAV

ING

A P

URP

OSE

So what do we have in mind?

USERS We want to satisfy their needs.

PROGRAMS/PROJECTS We want to improve our services.

RESOURCES We want to provide the best/top of

the line.

HAV

ING

A P

URP

OSE

So what are we inclined to do?

We come up with new services, programs, projects, build resources (aside from the traditional operational services). Why?

We come up with activities under these programs and services

HAV

ING

A P

URP

OSE

BOTTOMLINE:

Are these working based on your purpose?

Are these effective? Are we creating impact?

HAV

ING

A P

URP

OSE

We need evidence… to show that change has occurred to show effectiveness (that the

programs, services, projects that we do is making a difference)

to prove that what we are doing is meaningful

to demonstrate our value to our parent organization, learners and communityTHE

NEE

D F

OR

EVID

ENCE

Why evaluate?

Assessment Evaluationthe systematic collection of data to monitor the success of a program or course in achieving intended outcomes

the process of determining worth, merit, or value of something; assessment of “goodness”

judging people’s knowledge or skills; getting a baseline of what your stakeholders know and what they need

a judgment about whether the program has met its intended outcomes

Mostly descriptive and quantitative

Mostly qualitative

Why evaluate? Because… it tells you how well the library is performing a library rarely does it (comes up with a

service/program, implements, and leaves it be)

a library focuses on inputs and outputs (no impact)

a library would likely be inclined to calculate efficiency (i.e., making the most out of resources/service to produce a desired result) using traditional performance indicators (e.g., usage stats, entrants, assessment studies, ROI, et al.), and not effectiveness

Measures for evaluation

Levels of analysis

Individual (i.e., user experience)

Service (i.e., user group’s experience)

Organizational (i.e., all library users)

Societal (i.e., impact on local community)

MEA

SURE

S FO

R EV

ALUA

TIO

N

Inputs (resources; What

are needed)

Process (activities; What

to do)

Outputs (products; What the

results are)

Outcomes (effect/turnout/benefits; What the consequences

are)

MEA

SURE

S FO

R EV

ALUA

TIO

NBasic model (Richard Orr, 1973)

MEA

SURE

S FO

R EV

ALUA

TIO

NInputs Process Outputs Outcome

IL training courses

IL reference materials

Program Resource

persons Students

Trainings/Workshops for librarians

Development of IL program materials

modules IL sessions quizzes and

games self-paced IL

learning tools instructional IL

videos and guides

information products (e.g., pathfinders, guides, etc.)

Librarians know what relevant information are for appropriate users

Users become confident in finding authoritative sources

Users become independent learners and practice information ethics

Sample: Information Literacy Program

On impact and impact

evaluation (looking beyond the

numbers)

Let’s review: You identify your VMG Create your library’s purpose aligned to your

organization’s VMG Demonstrate evidence of your value to your

community Come up with initiatives (services, programs), noting

your purpose and aims Evaluate these initiatives

Are you creating impact?ON

IMPA

CT A

ND

IMPA

CT

EVAL

UATI

ON

Impact is... seeing change or difference in what you

do concerned about effectiveness any effect (maybe positive or negative)

of the service on either an individual, group or both

long-term; requires time a series/assessment of outcomes

ON

IMPA

CT A

ND

IMPA

CT

EVAL

UATI

ON

Measuring impactImpact evaluation (outcomes-based evaluation) largely focused on monitoring service

efficiency systematic causation (cause-effect) requires clear and consistent use of

terms

ON

IMPA

CT A

ND

IMPA

CT

EVAL

UATI

ON

Measuring impactImpact evaluation (outcomes-based evaluation)

Seeks to answer:

“How much better off are your patrons (in terms of service) after introducing a service/program?”

“Does the program/service/resource have a different impact on different groups?”

"Did the program/service/resource cause the impact?” “What would have happened if the program/service/resource had

not taken place?”ON

IMPA

CT A

ND

IMPA

CT

EVAL

UATI

ON

MET

HO

DS

FOR

IMPA

CT

EVAL

UATI

ON

QUANTITATIVEmeasuring changes in competences or behavior or to find correlations between library use and a person’s academic or professional success

QUALITATIVEevaluating users’ experiences and opinions

user skills tests before and after a training or the use of a library service

surveys (print, telephone, or online)

performance monitoring/data mining

Interviews

unobtrusive observation focus groups, discussion groups

analysis of citations in course work or research publications

users’ self-assessment of skills and competences gained

comparison of success data with use data

MET

HO

DS

FOR

IMPA

CT

EVAL

UATI

ON

Inputs Process Outputs OutcomeImpact(changes in confidence,

competencies/skills, behavior)

IL training courses

IL reference materials

Program Resource

persons Students

Trainings/Workshops for librarians

Development of IL program materials

modules IL sessions quizzes and

games self-paced IL

learning tools instructional IL

videos and guides

information products (e.g., pathfinders, guides, etc.)

Librarians know what relevant information are for appropriate users

Users become confident in finding authoritative sources

Users become independent learners and practice information ethics

• Librarians become library research consultants and collaborators

• Users become independent learners and manage information use on their own

• Users succeed in certification exams and in finding employment

Sample: Information Literacy Program

IMPA

CT

EVAL

UATI

ON

PR

OCE

SSStep 1: Impact ObjectivesStep 2: Impact IndicatorsStep 3: Evidence to collectStep 4:

Use the evidence to improve services and secure service sustainability

Focus on Step 1Impact areas Impact objectives

Identify key areas in your service/program that you want to make an impact (e.g., in IL, finding information)

Use specific terms Time-limited (e.g., achievable in 3 years) IM

PACT

EV

ALUA

TIO

N

PRO

CESS

Focus on Step 1Impact areas Impact objectives

Objectives are often confused with activities.

Objectives – what you want to achieve Activities – what you will do to make the

objectives happen

IMPA

CT

EVAL

UATI

ON

PR

OCE

SS

QuizObjective or Activity?

(Markless & Streatfield, 2013)

1. To support managers in carrying out their work effectively and efficiently.

2. To provide ICT-based learning in IL sessions3. To enable students to operate effectively in

an e-environment4. To provide materials appropriate to the

needs of graduate students

IMPA

CT

EVAL

UATI

ON

PR

OCE

SS

QuizObjective or Activity?

(Markless & Streatfield, 2013)

1. To support managers in carrying out their work effectively and efficiently.

2. To provide ICT-based learning in IL sessions

3. To enable students to operate effectively in an e-environment

4. To provide materials appropriate to the needs of graduate students

IMPA

CT

EVAL

UATI

ON

PR

OCE

SS

QuizObjective or Activity?

(Markless & Streatfield, 2013)

5. To increase the confidence and competence of socially excluded people in using library resources

6. To develop children as readers7. To collaborate with faculty in order to integrate

the library into their curriculum8. To develop the skills of the library staff in

delivering effective user-education

IMPA

CT

EVAL

UATI

ON

PR

OCE

SS

QuizObjective or Activity?

(Markless & Streatfield, 2013)

5. To increase the confidence and competence of socially excluded people in using library resources

6. To develop children as readers7. To collaborate with faculty in order to

integrate the library into their curriculum8. To develop the skills of the library staff in

delivering effective user-education

IMPA

CT

EVAL

UATI

ON

PR

OCE

SS

Before you even think about coming

out with a program/project/ser

vice for your users...

Think about the purpose for your initiatives first, and then make sure that it supports your organization's VMG.

Think about impact in planning your initiatives. Always integrate evaluation.

Be convinced that your library has value to your user community and organization. Think lifelong learning effect.

DEMONSTRATING THE LIBRARY’S IMPACT THROUGH ASSESSMENT AND EVALUATION 21 September 2016

Christine M. Abrigochristine.abrigo@dlsu.edu.ph

REFE

REN

CES

Bawden, D., Calvert, A., Robinson, L., Urquhart, C., Bray, C., & Amosford, J. (2009). Understanding our value; assessing the nature of the impact of library services. Library and Information Research, 33(105).

Blixrud, J. C. (n.d.). Evaluating library service quality: use of LibQUAL+. Retrieved from http://libqual.org/documents/admin/blixrud.pdf

Broady-Preston, J., & Lobo, A. (n.d.). Measuring the quality, value and impact of academic libraries: the role of external standards.

Markless, S., & Streatfield, D. (2013). Evaluating the impact of your library. London: Facet Publishing.

Matthews, J. R. (2007). The evaluation and measurement of library services. Westport, Connecticut: Libraries Unlimited.

Poll, R. (2012). Can we quantify the library’s influence? Creating an ISO standard for impact assessment. Performance Measurement and Metrics, 13(2), 121130. http://doi.org/10.18352/lq.7746

Poll, R., & Payne, P. P. (2006). Impact measures for libraries and information services. Library Hi Tech, 24(4), 547–562. http://doi.org/10.1108/07378830610715419

Streatfield, D., & Markless, S. (2009). What is impact assessment and why is it important? Performance Measurement and Metrics, 10(2), 134141.

Workshop