Lessons Learned: How to Survive the Development of
Outcome Measures
Cherie McCraw, M.L.S., Library Consultant, Division of Library and Information Services
Linda B. Schrader, Ph.D., Florida State University
Overview
Overview of the four-year implementation process
Development of an outcome evaluation system for the State Library of Florida
Examination of lessons learned and challenges encountered
History 1993 Government Performance Results
Act – Requires federal agencies to establish
specific objective, quantifiable, and measurable performance goals for each program
1996 reauthorization of Library Services and Technology Act (LSTA)
LSTA Grant Background State-based program Focuses on information access through
technology and information empowerment through special services
Emphasizes public libraries, but encourages all types of libraries to apply
Emphasizes accountability and evaluation
Institute of Museum and Library Services (IMLS)
Turned to outcome-based evaluation for LSTA grants– United Way model– Looks at the effect of an institution's
activities and services on the people it serves
– Powerful tool for identifying successes and communicating their value
IMLS Plan Five states were selected to implement
outcomes evaluation of LSTA grants early and develop model programs– Florida– Minnesota– North Carolina– Maryland– Washington
Florida’s Four Year Implementation Plan
Development of outcome model and standardized outcomes
Marketing of outcomes evaluation process
Planning for staff training and technical assistance
Conducting outcome evaluations
Implementation Activities
Administrative and management tools Senior level buy in State Library and field training
1998 IMLS selects states for pilot projects1998 Consultant hired to help develop model1998 State Library staff receive United Way Outcomes Evaluation trainingOct 1998 Outcomes Evaluation concept presented at annual Public Library Directors’ Conference1999 Standardized outcomes for 6 LSTA categories developed1999 OPPAGA Review favorable toward outcome conceptOct 1999 Outcomes Evaluation program presented at annual Public Library Directors’ ConferenceNov 1999 Born to Read Capacity Building Workshop (outcome plans developed)July 1999 Florida’s “LSTA Outcome Evaluation Plan” publishedFall 1999 Workshops on outcome evaluation presentedJan 2000 LSTA Grants workshop featured new guidelinesMar 2000 LSTA applications required outcome plan (not scored)Sep 2000 “Workbook: Outcome Measurement of Library Programs” publishedOct 2000 Panel discussion of outcomes at annual Public Library Director’s ConferenceNov 2000 LSTA Capacity Building Workshop (outcome plans refined)Oct 2000 Outcomes based LSTA grants began2000 LSTA categories changed from 6 to 2/new standardized outcomes createdMar 2001 LSTA applications required outcome plan (scored)Jul 2001 Outcomes based evaluation workshops conductedNov 2001 LSTA Capacity Building Workshop (evaluation strategies and tools)Nov 2001 LSTA annual reports reflected outcomes evaluationDec 2001 LSTA Grants workshop included session on outcomes
State Library of Florida’s Outcomes Timeline
Development of Outcome Model
Hired consultant Adapted United Way Outcome Model Identified standardized state outcomes Revised rules, guidelines and forms
Outcome Model Components
Inputs Activities Outputs Outcomes Evaluation Indicators Evaluation Sources/Methods
Florida’s LSTA OUTCOMES PLANProject Name ______________________________ Library______________________
Project Summary/Program Purpose:
Inputs Activities Outputs Outcomes Indicators Sources/Methods
Standardized State Outcomes
Characteristics– Broad scope, relevant– Easily understandable– Focus on skills/behaviors– Can include quantitative and qualitative
indicators– Tailored to local programs
Standardized State Outcomes
Access for persons having difficulty using libraries– Persons having
difficulty using library services use services or information that were not previously available
Library Technology Connectivity and Services– Public uses technology
to get information
– Public learns to use technology
Access for persons having difficulty using libraries
Golden Gateways Library Family
Learning Centers Small Business
Information Resource Center
REACH: REmote ACcess to the Homebound
Parents Plus Born to Read Juniors to Seniors:
Hillsborough Remembers
Library Elderly Outreach (LEO) Project
Library Technology Connectivity and Services
Development of the Everglades Information Center
Electronic Library: Community Training and Outreach
Seniors Connect @ Jacksonville Public Library
Mobile Training Lab Osceola Internet
Improvement Library Automation:
Moving Legal Information Toward the Public
Mi Servicio de Biblioteca
Florida’s LSTA OUTCOMES PLANProject Name __________Family Literacy for Lifelong Learning____ Library_______________
Project Summary/Program Purpose: With support from schools and community partners, the library will continue withthe family literacy program. Services will be expanded at our four branches where the program is currently offered.Additionally, services will be initiated at a fifth branch. Elements to be offered: (1)One-on-one tutoring for children andsupport for parents in meeting the child’s reading needs, (2)literacy- or basic-skills support for adults through the libraryor its partners, (3)increased services for Spanish-speaking families.Inputs
Establishedlibrary systemwith full timestaff
45 literacytutors
Literacyresources(books, 75videos, 13Brainchilds, 44LearningPAKs)
16 partners
Grant funds
Local match
Activities
Hire literacycoordinator
Designmarketingcampaign
Visit otherliteracyprograms toexchange ideas
Attend literacyworkshops
Order literacymaterials
Recruit families
Recruit tutors
Collaborate withpartners
Outputs
# familiesserved
# childrentutored
# hourschildrentutored
# adultstutored
# hoursadults tutored
# parent/childinteractionmeetings
# PAKsloaned
# librarycards issued
Outcomes
1. Persons havingdifficulty using libraryservices useservices orinformation that werenot previouslyavailable
Indicators
1a. # and %children whosereading skillsimprove
1b. # and %increase in basicadult literacylevels.
1c. # and %parents who meettheir personalgoals.
1d. # and %parents who readto their children15 min/day (3times/week).
Sources/Methods
1a. Pre-and post-testing
Teacher’s reports
1b. Pre-and post-testing
1c. Interviews
1d. Interview orlog
Florida’s LSTA OUTCOMES PLANProject Name _Electronic Library: Community Training & Outreach Program____ Library_____
Project Summary/Program Purpose: To provide consistent training presence at seven electronic libraries operatedin the county. The personnel funded through this program will provide a regular training presence and will conducttraining classes for electronic library users. The classes will meet user demand for assistance in learning how touse the resources available in the electronic libraries. The project personnel will survey users and evaluate theresources and effectiveness of the e-libraries.Inputs
LibrariansLibrary
TechnicalAssistant
E-libraries
Computers
Software
Partneragency staff
Grant funds
Local match
Activities
Hire personnel
Conduct classes
Evaluate classes
Evaluateresources
Survey e-libraryusers
Write/send pressreleases
Visit communityschools/areacommunitycenters
Outputs
# classes
# trainingprograms
# attendees
# evaluationscollected
# surveyscollected
# schools &communitycenterscontacted
# pressreleases
Outcomes
Public learns to usetechnology
Public usestechnology to getinformation
Indicators
# of peopleindicating theyhave learned howto use theelectronicresources
# of people whoaccessinformation at thee-libraries
Sources/Methods
Surveys
Program records
Evaluation forms
Sign-in sheets
Program records
Surveys
Marketing the Outcome Process
State leadership vanguard for change– Paradigm shift in what gets measured
Senior level buy in of new evaluation process
Continuous communication throughout the four year period
Ongoing support
Staff Training Developmental
approach to training Technical assistance
provided by regional library consortiums and state library consultants
Statewide conferences
Conducting Outcome Evaluations
Phased in requirements for outcome evaluations in grant process
Continued training and support to field staff
Initial review of annual reports
Lessons Learned
A State Library assumed leadership roleB Implementation phased in over timeC Standardized the outcomesD Measurement tools could be the sameE Better communication between libraries
and grant partnersF Multiple training opportunities
ChallengesA Resistance from State and field staffB Cannot be implemented quicklyC Adapting local programs to the standardized
outcomesD Tailoring existing data collection tools E Libraries and grant partners interpret
outcomes evaluation differentlyF Different levels of knowledge about outcomes
Where Do We Go From Here?
Continue training field librarians Promote/pilot Outcomes Model for other
library programs Develop new Long Range Plan using
outcome measurement