gabrielle wong reference librarian hong kong university of science and technology
DESCRIPTION
Enhancing Quality of Library Instruction Programs through Delayed Assessment 6th Northumbria International Conference on Performance Measures in Libraries and Information Services, Durham, UK. August 22-25, 2005. Gabrielle Wong Reference Librarian Hong Kong University of Science and Technology. - PowerPoint PPT PresentationTRANSCRIPT
Enhancing Quality of Library Instruction Programs through Delayed Assessment
6th Northumbria International Conference on Performance Measures in Libraries and Information Services, Durham, UK. August 22-25, 2005 Gabrielle Wong
Reference LibrarianHong Kong University of Science and Technology
PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 2
Presentation Outline
1. Introduction2. Background of the Survey3. Survey Design and the
Questionnaire4. Sampling5. Key Findings6. Future Assessments
PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 3
Introduction
A survey was conducted by HKUST Library to assess the enduring impact of its library instruction program
Questionnaires were sent 4-8 weeks after library classes in Fall 2004
The survey provides data that demonstrates the value of the program, and helps librarians to better understand the characteristics and needs of users
It was a valuable learning process for HKUST librarians; this experience serves as the basis for future assessment endeavour
PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 4
HKUST
Founded in 1991 One of the three
research universities in Hong Kong
Undergraduates: 5,500; Postgraduates: 3,000; Faculty: 440
4 Schools: Science, Engineering, Business, Humanities & Social Sciences
PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 5
Library Instruction at HKUST Library Serves the whole university community:
students, staff and faculty Over 200 library sessions every year,
reaching 4,000 attendees One-shot sessions, 50-80 minutes Wide variety: database workshops, course-
specific library sessions, topic/subject specific instruction (e.g. job hunting, microforms, web searching, research workshops for postgraduates)
PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 6
Background of the SurveyEvaluation Practice since 1996 Evaluation forms were distributed at the
end of sessions Measured users' satisfaction Instructors' summary for future
improvement Generally, over 80% attendees agreed
that the class was worth recommending to their peers
Voluntary evaluation Results were not analyzed systematically
PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 7
Background of the SurveyThe Needs for a Formal Assessment Do the library sessions help the
attendees in their research and studies?
What do they think after a period of time, during which they are expected to apply the skills for practical purposes?
How can we do better?
PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 8
Background of the Survey The University Librarian, Dr. Samson
Soong, recommended the Reference Department to launch a study to answer these questions
The survey aimed to measure the enduring impact of the library program, and produce figures that he may use to demonstrate such value to the university administration
It should also expose the strengths and weaknesses of the program, thus facilitating improvements
PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 9
Survey Design: Major Considerations first attempt of such kind simple, straightforward and effective cover as many classes as possible
(programmatic) librarians' workload demand on users' side assess the enduring/long term valueMethodology adopted: a delayed
perception survey
PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 10
Survey Design: Delayed Assessment The survey was administered 4-8
weeks after the sessions The purpose of the delay period was to
allow attendees to apply the skills on their assignments and projects
Skills may be reinforced, proved insufficient or simply forgotten
A mean to assess the enduring value of the classes
PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 11
Survey Design: The Questionnaire 2 pages, 14 questions, should take less
than 5 minutes to complete composed mostly of structured
questions rather than open questions to encourage feedback
provided options for open comments whenever possible
PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 12
Survey Design: The QuestionnairePerception Questions: Q.2, Assessing impact: how the class
helped/changed the attendees Q.3, Specific skills learned Q.4, Skill retention Q.5, Overall class rating Q.6. Overall instructor rating Q.7 Suggestions for improvement
PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 13
Survey Design: The QuestionnaireOther Questions: Library use habit
information search confidence no. of library classes attended previously research frequency
Demographic status place of permanent residence gender
PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 14
Sampling
Selection criteria: class size > 15 if it was a course-specific session, there
should have been assignment or project associated with it
No intention to mimic the university population
25 sessions selected: 15 Course-Specific, 10 Open Workshops
PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 15
About Class Types
Course-Specific (CS) Open Workshop (OW) library session serves the
information need of the associated credit course, e.g. FINA221
library session was "compulsory"
all students enrolled in the course were to be surveyed
well-defined group, we could use print and anonymous questionnaire distributed by the professors
high response rate
library session opens for all interested users to register by email
attendees were naturally more motivated
composed of a higher proportion of PGs and staff, therefore a more mature group
we have no choice but to survey them with emails
lost anonymity much lower response rate
PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 16
Responses
688 questionnaires sent out Received (Print): 388 (90.7%) Received (E-mail): 78 (30.0%) Total: 466 (67.7%) 89% of respondents attended the
library sessions 11% had not attended the sessions, but
answered the demographic and library use habit questions
PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 17
Findings: Impact
Question 2, rate on a 4-point scale ("1" Strongly Disagree to "4" Strongly Agree) :
As a result of the class, Ia. learned about sources to find needed information.b. learned about search methods to find needed
information.c. learned how to find needed information more
quickly.d. was more confident in researching information.e. have increased my interest in using the Library.
PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 18
Findings: Impact
Mean Score
Agree % Disagree %
a. sources 3.15 95.6 4.4
b. search method
3.12 93.0 7.0
c. efficiency 3.04 85.3 14.7
d. confidence
2.85 74.1 25.9
e. interest 2.68 61.2 38.8
PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 19
Findings: Impact
The differences of these mean scores are statistically significant (p<.05).
a sources
b search method
c efficiency
d confidence
e interest
CS – mean
3.09 3.06 2.98 2.80 2.59
Agree % 95.2 91.8 84.7 72.2 57.9
Disagree %
4.8 8.2 15.3 27.8 42.1
OW – mean
3.52 3.50 3.38 3.16 3.21
Agree % 98.3 100 98.1 86.0 81.0
Disagree %
1.7 0 10.9 14.0 19.0
PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 20
Findings: Skills LearntQuestion 3:Please indicate the skills in which this class has helped you (you may check more than one item):
a. Choose the best databases for my information needb. Form better search strategiesc. Select more appropriate keywords to searchd. Export or save my results more effectivelye. Find more relevant internet resourcesf. Use printed resources more effectivelyg. Others
PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 21
Findings: Skills Learnt Average no. of choices per respondent:
2.06 Top 3 choices:
Form better search strategies Find more relevant internet resources Select more appropriate keywords to
search
PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 22
Findings: Skills Retention
CS OW All
Yes 233 39 272
No 107 18 125
Valid % 68.5% 68.4% 68.5%
Question 4: Have you continued to use the skills that you learned in this class?
PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 23
Findings: Skills RetentionWritten Comments – "Yes"
Continued to use the skills learned to find information - general remarks 17 - specified "finding more relevant information" 5 - specified efficiency 5 - specified particular information sources 10 37
Used the skills to find better Internet resources or use search engines more effectively 11
Used the library catalog 5
Used databases 40
Applied the search skills learned 27
Used the skills learned to achieve particular goals (projects, assignments, etc.) 64
Used the subject guide learned in class 1
PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 24
Findings: Skills RetentionWritten Comments – "No"
No chance or need to apply the skills 22
Forgot what they learned in the classes 8
No time 7
Preferred or were satisfied with using the Internet 3
Low interest in using the Library resources 4
Considered the class not useful 2
PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 25
Findings: Overall RatingsQuestion 5 and 6:
5.What is your OVERALL rating of the class:6.What is your OVERALL rating of the instructor:
Very goodGoodAbove averageAverageBelow averageBadVery bad
PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 26
Findings: Overall Ratings
Overall Class Rating
0.6% 0.6%
5.1%
25.2%
36.0%
27.8%
4.8%
0.0% 0.0% 0.0%
3.5%
15.8%
52.6%
28.1%
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
Very Bad Bad Below Average Average Above Average Good Very Good
CS OW
PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 27
Findings: Overall Ratings Overall Instructor Rating
0.0%0.9%
3.5%
25.6%
29.1%
31.7%
9.3%
0.0% 0.0% 0.0% 0.0%
21.1%
45.6%
33.3%
0.0%
5.0%
10.0%
15.0%
20.0%
25.0%
30.0%
35.0%
40.0%
45.0%
50.0%
Very Bad Bad Below Average Average Above Average Good Very Good
CS OW
PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 28
Findings: Overall Ratings
To benchmark with the grading scale in the University's credit course evaluation:
Very bad=0, Bad=16.7 Below average=33.3, Average=50, Above average=66.7 Good=83.3, Very good=100
Mean Scores CS OW All University average scores in the same semester
Class Score 66.34 84.21 68.82 70.2-75.1
Instructor Score
69.19 85.38 71.49 72.4-78.5
PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 29
Findings: Suggestions for ImprovementQuestion 7:How could this class be improved? (you may check more than
one item) a. I already find the class satisfactory b. Shorter time c. Longer time d. Cover fewer topics, please specify e. Cover more topics, please specify f. Cover some topics in more detail, please specify g. Cover some topics in less detail, please specify h. Instructor should speak more slowly i. Instructor should speak more clearly j. Offer students more hands-on practice k. Have a smaller class l. Others, please specify
PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 30
Findings: Suggestions for Improvement Average no. of choices per respondent: 1.48 Top Three Choices:
CS OW
Shorter time(31%)
Satisfactory(55%)
Satisfactory(27%)
Shorter time(20%)
Hands-on(22%)
Hands-on(18%)
PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 31
Findings: Suggestions for ImprovementWritten Comments: Fewer topics: 2 commented on the flood of
information in the class More topics: 7 comments, 4 called for more
coverage on databases More details: 14 comments on various
issues: 4 about search skills, 2 called for more examples, 2 for databases
Others: 15 various comments, 6 asked for more and better handouts
PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 32
Other Observations
Class size: Bigger classes gave lower class ratings,
instructor ratings and lower skill retention Information search confidence:
Respondents with higher confidence in information search rated higher in these three key indicators
PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 33
What the Findings show Us It demonstrates positive value of the Library's
bibliographic program by yielding these figures: After a period of time, over 85% of users remain
affirmative that the class was useful Almost 70% of users retain the skills learnt The overall ratings are comparable to credit courses
in the University It tells us what users want:
Smaller class More hands-on Better class handouts
PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 34
What we Learnt from the Survey There are significant differences between
the Open Workshop Group and the Course-Specific Group
Why users did not use the skills: did not see the need/did not have the chance forgot what they learnt
Users with higher confidence in information search are more receptive and appreciative of library instruction
Class size does matter
PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 35
Where do We Go from Here? Nurture a culture of assessment at HKUST
Library Next assessment in 2006 Administer a comparable immediate
evaluation attempting to quantify the "delay" effect
Two separate schemes for CS and OW? Focus on the outcomes of one particular
class instead of a programmatic assessment? Consider testing attendees' library skills
instead of measuring perception?
Thank you!