Two birds with one evaluation

How to assess your services and publish too?

Suzana Sukovic, Jane Van Balen,Ashley England & Janet Chelliah

Research for LIS Practitioners, February 2011

Overview

• Part 1. Evaluation (with a touch of mystery)

• Part 2. Information literacy evaluation: Bachelor of Nursing Self Discovery Learning 2010-

• Part 3. Discussion

Part 1: Evaluation with a touch of mystery

Presented by

Suzana Sukovic

Evaluation

Assessment Librarian

The University of Tennessee Libraries

6 January 2011 

The University of Tennessee, Knoxville (UTK) Libraries invites applications for the newly created position of Assessment Librarian. The Assessment Librarian fosters a culture of assessment within the University Libraries and leads the design and implementation of a successful, sustainable assessment program. Engaging staff throughout the Libraries, the Assessment Librarian analyzes systems and services for continuous improvement.

Evaluation• Critical part of evidence based practice

and action research

Cycles of action research (State of NSW, Department of Education and Training, Professional Learning and Leadership Development Directorate, 2010, Action research in education: guidelines, 2nd ed, p. 4)

Evaluation: approach

Where to start? • Strategic direction• Particular purpose (e.g., solving a

problem, grant application, part of a larger project)

• Opportunity (e.g., additional resource, a particularly suitable/innovative project)

Evaluation: approach

PurposeWhy evaluate?

Evaluation focusWhich aspect to evaluate?

EvidenceWhat is the evidence?

Data-gathering methods How to obtain evidence?

Analysis & interpretationHow to interpret evidence?

•Service improvement•Demonstrating value etc.

•Development of students’ IL skills•Cost benefit•Improvement of clients’ satisfactionetc.

Clients’ ability to•describe A•do B•apply A&B in new situationsetc.

•Tests•Experiments•Observation•Interviews with teachers & studentsetc.

Consider•Criteria•Benchmarks•Interpretation framework & methods•Software

Example: Evaluation of an information literacy program

Evaluation: approach

• Brainstorm• Focus on– the best fit for the purpose–most effective & efficient approach– ‘maximizing potential’

• ‘Cutting salami’ approach

Maximising evaluation projects

Make a snow ball effect!

http://blog.balancedspirituality.com/wp-content/uploads/2009/12/monkey-snowball.jpg

Maximising evaluation projects

• Choose a general theme – relates to strategic direction, a large or long-lasting project, a significant problem or opportunity

• Keep research on mind • LINK

– several small projects– with projects in other departments/organisations– with your skills, knowledge and experiences– with literature

There is nothing so practical as a good theory (Kurt Lewin, 1951)

• Publish & present – to your department, at conferences, in professional and academic journals

Wire Guys Chillin at the Baltic, New Castle UK / Marc Hartley

http://vi.sualize.us/view/invertebra/86f69425bccb521a251cb3eb8053c53d/

Part 2: Information literacy evaluation Bachelor of Nursing Self Discovery Learning 2010-

Presented by

Jane Van Balen

Background

• Directive/Need for library to better evaluate Information Literacy programs generally

• Newly revised Bachelor of Nursing course - a worthy pilot because:– New course – fresh opportunity– Follow-up possible over next years of course– InfoLit training embedded by faculty– Self Discovery approach suggested by

faculty

Background

• New course has a whole day allocated for Library information literacy as part of the subject “Workshops in Practice Readiness”.

• The Faculty suggested a self discovery approach for one of the components of the training which the library was already developing ideas for.

• Several factors came together fortuitously which allowed us to maximise this opportunity.

• We were allowed freedom to incorporate elements to better enable us to use for research. Eg Pre-Test addition.

Project objectives/directives

• To determine whether Information Literacy programs offered by the library are effective in enabling students to become independently information literate.

• To determine whether they continue developing over time after having received information literacy training.

Library Program – First year BNurs

• Online Pre-test : students could do this before they arrived or in a classroom at the start of the day. Not assessed by their tutor, but formed some of our data.

• Self discovery questionnaire: students grouped in pairs could answer a range of questions about library services. Token prizes were offered to the first pair who finished, and informal feedback was given to many as they returned their completed answers.

• Databases session: presented by their tutors using library’s lesson plan and handouts. Tutors attended a “Train the trainer” session with library staff.

• Online Post-test: This was compiled by the subject co-ordinator and was assessed. Library had access to overall results.

Data collected to date

• Results from the pre-test. Available online.

• Answers from the Self Discovery questionnaire. Written form only.

• Results from the post-test. Available online

What does the data tell us?

• If the overall scores from the post test were higher than the pre test, can we conclude that learning took place?

• Until we follow up with the next phase of training, it will be difficult to determine our original objective.

Issues – how to deal with

• Inconsistencies:– Various quality of db session presenters– Timetabling– Not everyone did the pre-test first– Self discovery questionnaire feedback

varied considerably.

• Compiling data and interpretation

Next steps

• We will meet the same groups of students when they do an Evidence Based Practise subject in 2011

• Associated with more in-depth Databases skills session. (PICO)

• Another opportunity to assess information searching skills and determine lasting benefits of training.

Next steps

• We have a number of elements we can use to proceed:– Another online pre-test (stimulus for learning)– online tutorials on Evidence Based Practice

approach to finding specific types of articles. – Library workshop with specialist subject

Librarians– Another online post-test AND/OR obtain

access to the results of their related assessment task.

Gift from Two birds...

Diigo group Research for practitioners

• Explore• Add• Comment

Do we have your email?