Practice Makes Perfect: applying and adapting best
practices in information literacy
Sheril Hook Esther Atkinson Andrew NicholsonInstruction Coordinator Liaison Librarian GIS/Data Librarian
University of Toronto Mississauga
WILU Conference, May 18, 2007
Agenda
IL Program Development (Sheril) Category 5: articulation with the curriculum
Examples of BP Category 5 (Andrew) research-based learning IL learning outcomes
IL Program Development (Sheril) Category 10: Assessment/Evaluation
Examples of BP Category 10 (Esther) data and its impact on instruction and planning
ALA/ACRL Characteristics of Programs of Information Literacy that Illustrate Best Practices
Category 5: Articulation with the Curriculum
Articulation with the curriculum for an information literacy program:
is formalized and widely disseminated;
emphasizes student-centered learning;
uses local governance structures to ensure institution-wide integration into academic or vocational programs;
identifies the scope (i.e., depth and complexity) of competencies to be acquired on a disciplinary level as well as at the course level;
sequences and integrates competencies throughout a student’s academic career, progressing in sophistication; and
specifies programs and courses charged with implementation.
http://www.ala.org/ala/acrl/acrlstandards/characteristics.htm
IL Program Development: PlanningPart 1
ACRL Best Practices Document environmental scan
internal scan & internal development external scan & external development
current state & next steps
Shared Philosophical Framework training & development
informing our pedagogical practices developing expertise as shared responsibility
use of IL Standards and terminology
Environmental Scan
Core curricula (horizontal/vertical integration in Part 2)
Departmental goals Required courses for baseline expectations
Representation on curriculum committees
Movements in teaching/learning student engagement
Environmental Scan
Student Engagement NSSE http://nsse.iub.edu/
Peer learning, aka peer assisted learning, supplemental instruction http://www.peerlearning.ac.uk/ http://www.umkc.edu/cad/SI/index.htm
Re-invention Center http://www.sunysb.edu/Reinventioncenter/ Inquiry-based, discovery, problem-based, or research-
based learning
http://www.reinventioncenter.miami.edu/BoyerSurvey/index.html
http://www.reinventioncenter.miami.edu/pdfs/2001BoyerSurvey.pdf
Student Engagement
research-based learning problem-based learning inquiry-based learning discovery learning
knowledge building
Scardamalia, M., & Bereiter, C. (2003).
Shared Philosophical Framework
information literacy as concept tool-based vs. concept-based teaching other literacies, e.g., technology, media, spatial, data
inventory of current practices and outreach activities
articles & workshops that help develop framework
Learning theory Bloom’s taxonomy SOLO Taxonomy (Biggs)
development & use of assessment tools
What is embedded IL?
Embedded Assignment(s) collaboratively developed with instructor. IL stated
learning outcomes in instructor's course materials. Session by librarian may or may not have been delivered during class time (e.g., series of walk-in workshops)
Integrated Session content tailored to course assignment in consultation with
instructor. Session may or may not have been delivered during class time (e.g., series of open workshops available to students). Session may or may not have been optional.
Supplemental Generic information literacy instruction; is not tied directly to course
outcomes or an assignment. Session may or may not have been optional for students. Session may or may not have been delivered during class time.
ANZILL, p6 ANZIL Framework, 2004ACRL, 2007Learning Commons, University of Guelph, n.d.
IL StandardsStandard OneThe information literate student determines the nature and extent of the information
Performance Indicator 2. The information literate student identifies a variety of types and formats of potential
sources for information.
Outcomes include Knows how information is formally and informally produced, organized, and
disseminated Recognizes that knowledge can be organized into disciplines that influence the way
information is accessed
Identifies the value and differences of potential resources in a variety of formats (e.g., multimedia, database, website, data set, audio/visual, book)
Differentiates between primary and secondary sources, recognizing how their use and importance vary with each discipline
Realizes that information may need to be constructed with raw data from primary sources
"Information Literacy Competency Standards for Higher Education." American Library Association. 2006.http://www.ala.org/acrl/ilcomstan.html (Accessed 15 May, 2007)
Examples of IL Standards tailored and embedded into course curricula
U of T Mississauga LibraryWhen we collaborate with our instructors on designing a class assignment, we emphasize
the Library Vision -“Leading for Learning”
the availability of thousands of Research and Information Resources through the U of T Libraries: as of May 15, 2007:
395,184 e-holdings including e-books, journals, newspapers, etc.
the key role of these resources in enhancing student engagement with their learning.
U of T Mississauga LibraryWe also stress to instructors that our electronic resources can be utilized
to enhance their instructional content.
to foster an active learning environment in the course. Students will begin to think both conceptually and critically about the material.
to develop information literacy competencies among the students, such as retrieving and critically evaluating information in any format.
More details about information literacy can be found at the Association of College & Research Libraries (ACRL) website.http://www.ala.org/ala/acrl/acrlstandards/informationliteracycompetency.htmMany disciplines are now releasing their own information literacy standards, based on the ACRL model.
Examples fromSocial Sciences
Sciences
Humanities
Assignment: Changes in Canadian Society
Outcomes identify and locate statistics needed evaluate statistics for use (do they cover the
correct geography?, time period?, etc) analyze statistics communicate the results in term paper and
presentation acknowledge the use of information
Social Sciences 1.
Research Question By examining census data related to
occupation, how have women’s working lives changed in a 100 year period?
2.Social Sciences
Social Sciences
• Students recognize that the Census collects statistics on occupation
3.
Outcomes•identify and locate statistics needed.
Social Sciences 4.
Outcomes•evaluate statistics for use.
• Students differentiate between census years and census geographies available.• Students identify value and differences of resources in a variety of formats.
Social Sciences5.
Outcomes•analyze statistics
• Students recognize the occupation categories being used
5.
Social Sciences6.
Outcomes•analyze statistics
• Students create a cross tabulation table between Occupation and Sex1901 Census of Canada: Occupation by Sex
Social Sciences
•Students next identify and locate the2001 Census Variables relating tooccupation and sex.
On the next slide:•A 2001 Census cross tabulation isthen compared with 1901 Censuscross tabulation.
•Students will recognize thatoccupation categories will havechanged in the 100 year time span.
•Students realize that the data canbe extrapolated into multiplecategories
7.
Outcomes•analyze statistics
2001 Census of Canada: Occupation by Sex
1901 Census of Canada: Occupation by Sex 2001 Census of Canada: Occupation by Sex
8.Outcomes
• analyze statistics
Social Sciences 9.
Outcomes
•communicate the results in term paper and presentation
•Students add tables to term paper and also to a class slideshow presentation.
•acknowledge the use of information
1901 Census of Canada Bibliographic EntryCanada. Statistics Canada. Census of Canada, 1901: public use microdata file– individuals file [computer file]. Victoria, B.C.: University of Victoria; Canadian Families Project [producer][distributor]. January 2002. <http://myaccess.library.utoronto.ca/login?url=http://r1.chass.utoronto.ca/sdaweb/html/canpumf.htm>
2001 Census of Canada Bibliographic EntryCanada. Statistics Canada. Census of Canada, 2001: public use microdata file - individuals file [computer file].Revision 2. Ottawa, Ont.: Statistics Canada [producer]; Statistics Canada. Data Liberation Initiative [distributor],2006/04/26. (STC 95M0016XCB) <http://myaccess.library.utoronto.ca/login?url=http://r1.chass.utoronto.ca/sdaweb/html/canpumf.htm>
Examples fromSocial Sciences
Sciences
Humanities
Assignment: Cited Reference Searching in the Sciences
Outcomes evaluate available resources to see if their
scope will include citation tracking statistics and journal impact factor
locate and interpret the citation information
Sciences 1.
Research QuestionWYTTENBACH, R. and HOY, R. “DEMONSTRATION OF THE PRECEDENCE EFFECT IN AN INSECT” JOURNAL OF THE ACOUSTICAL SOCIETY OF AMERICA; 94 (2): 777-784 AUG 1993.
• Before including this reference in a paper, check to see how “reputable” both the article and the journal is in the discipline. Should it be included?
2.Sciences
Outcomes•Evaluate available resources to see if their scope includes citation tracking.
• Students recognize that journal articles have value in a particular discipline and that they can be measured in a variety of ways, including specialized citation indexes.
Sciences 3.
Sciences
•Students recognize the abilityto perform cited reference searchingin a variety of ways.
4.
Outcomes•Evaluate available resources.
Sciences 5.
Outcomes•locate and interpret the citation information.
• Students locate the citation and realizes that the authors consulted a varietyof sources (“Cited References”); and more importantly this citation has beencited frequently (“Times Cited”) in the years since publication.
Sciences 6.
Outcomes•interpret the citation information.
•Students can review the cited references from the article and examinethe origins of the research
Sciences 7.
Outcomes•interpret the citation information.
•By checking the “Times Cited”, students gain insight into the impact of thearticle in the discipline.
Sciences 8.
Outcomes•interpret the citation information.
•Students also access the JCR to check the “Impact Factor”
Sciences
•Students can also rank other journals in the discipline by impact factor.
9.
Outcomes•interpret the citation information.
Examples fromSocial Sciences
Sciences
Humanities
Assignment:Myth over Time
Outcomes Explore the dynamism of myth by comparing
and contrasting a selection of ancient and modern primary sources of a myth (at least one literary, one material)
Identify the most significant changes from ancient to modern source and discuss those changes in light of the context in which each source was created
Interpret those changes in terms of how they affect the meaning of the myth and how they came about in the first place
Humanities 1.
Research Question
How have myths changed over time?
2.Humanities
Humanities
•Students begin by finding primary sources--art works, music, scripts, opera and background information on artists
3.
Outcomes•compare and contrast a selectionof primary sources (art)
Google has images, but no provenance information
Camio has images, plus provenanceand usage rights information
Humanities
Students build on the learning acquired by finding background information on a time period/place
4.
Outcomes•identify the most significant changes...in light of the context in which each source was created.
Humanities
Students place a myth in the cultural context in which it’s being used or re-told
5.
Outcomes•identify the most significant changes...in light of the context in which each source was created.
Students listen to a symphony to identify the dynamism of the myth and interpret its significance
Humanities 6.
Outcomes•compare and contrast a selection of primary sources (music)
Summary The U of T Mississauga Library provides access
to thousands of digital and interactive resources for a variety of active and conceptual based learning activities.
These resources can be utilized to promote both student engagement and the embedding of IL standards and outcomes.
ALA/ACRL Characteristics of Programs of Information Literacy that Illustrate Best Practices
Category 10: Assessment/Evaluation
Assessment/evaluation of information literacy includes program performance and student outcomes and:
for program evaluation:
establishes the process of ongoing planning/improvement of the program;
measures directly progress toward meeting the goals and objectives of the program;
integrates with course and curriculum assessment as well as institutional evaluations and regional/professional accreditation initiatives; and
assumes multiple methods and purposes for assessment/evaluation-- formative and summative-- short term and longitudinal;
http://www.ala.org/ala/acrl/acrlstandards/characteristics.htm
ALA/ACRL Characteristics of Programs of Information Literacy that Illustrate Best Practices
Category 10: Assessment/Evaluation (cont’d)
Assessment/evaluation of information literacy includes program performance and student outcomes and:
for student outcomes:
acknowledges differences in learning and teaching styles by using a variety of appropriate outcome measures, such as portfolio assessment, oral defense, quizzes, essays, direct observation, anecdotal, peer and self review, and experience;
focuses on student performance, knowledge acquisition, and attitude appraisal;
assesses both process and product;
includes student-, peer-, and self-evaluation;
http://www.ala.org/ala/acrl/acrlstandards/characteristics.htm
How are we teaching/Who are we reaching? Reflective teaching practices
Teaching portfolios Sharing with colleagues and course instructors Evaluation and assessment Student focus groups
Inventory of outreach & teaching How are you reaching students? How many?
Who are current campus partners? Who are potential campus partners? Who will keep these relationships going?
As a group where are you teaching? Horizontally and vertically
IL Program Development: PlanningPart 2
Assessment standardized assessments (ETS, SAILS,
JMU) creation, use and reflection of assessments
(background knowledge probe, muddiest point, observation, dialogue)
instruction database
National standardized tools
iSkills™ (aka Information and Communication Technology (ICT) Literacy Assessment) developed by the Educational Testing Service. $35.00 US per studenthttp://www.ets.org/
Measures all 5 ACRL Standards. Two test options: Core and Advanced. Computerized, task-based assessment in which student complete several tasks of varying length, i.e., not multiple choice. Intended for individual and cohort testing. 75 minutes to complete
Standardized Assessment of Information Literacy Skills (SAILS) developed by Kent State University Library and Office of Assessment. It is also endorsed by the Association of Research Libraries. $3.00 US per student (capped at $2,000), but we can also administer ourselves for free.https://www.projectsails.org/
Measures ACRL Standards 1,2,3,5. Paper or Computerized, multiple-choice. Intended for cohort testing only. 45 questions, 35 minutes to complete.
Information Literacy Test (ITL) developed by James Madison University (developed by JMU Libraries and Center for Assessment and Research Studies)http://www.jmu.edu/icba/prodserv/instruments_ilt.htm
Measures ACRL Standards 1,2,3,5. Computerized, multiple-choice. Intended for cohort and individual testing. 60 questions, 50 minutes to complete.
NPEC Sourcebook on Assessment: http://nces.ed.gov/pubs2005/2005832.pdf
ETS: Advanced Level – Access
http://www.ets.org/Media/Products/ICT_Literacy/demo2/index.html
ETS: Core Level - Manage
http://www.ets.org/Media/Products/ICT_Literacy/demo2/index.html
ETS: sample score reportAccess: Find and retrieve information from a variety of sources.
What was I asked to do? Search a store’s database in response to a customer’s inquiry
How did I do? You chose the correct store database on your first search. You selected the most appropriate category for searching. You chose the best search term for the database you selected. You selected one inappropriate item for the customer in addition to
appropriate ones.
http://www.ets.org
ETS Pilot at UTM
Evaluating the Results The relationship between the Core and Advanced score ranges is not clear. Are the
two tests on a continuous scale (e.g., with Core representing 100 – 300 and Advanced 400 – 700)?
The University of Toronto Mississauga norms seem to be consistent with the norms from other institutions, and they all seem to be clustering in the middle.
Though students received written feedback on their performance within each category, it is unclear how this feedback relates to their aggregate score and how it is derived from the students’ performance on the test (e.g., time taken to perform each task, number of clicks).
It is unclear if students are being tested on the same variables within each category across all different versions of the test (e.g., the student reports suggest that some students were evaluated on different criteria in certain categories).
The institution does not receive any granular statistical data (e.g., by performance within each category or by question), and only has access to individual student reports and the aggregate score for each student.
Learning Outcomes Assessment
classroom assessment techniques (CATs)
self-awareness inventories
in-class pre-/post-assessments
class assignments
Instruction Database
Instruction Database
U of T Mississauga LibraryInformation Literacy Program Data Records various characteristics of the
instruction sessions May 2005 to April 2007 Early data reflects what is being done and
what needs to be addressed
U of T Mississauga LibraryAssessing Our Program Market penetration Reflective of current teaching practices
U of T Mississauga1. Market Penetration Number of students reached Departmental contact Number of instruction sessions given Level of vertical integration
U of T Mississauga Library
2005 2006 2007
No. of Courses
Summer Fall Winter Summer Fall Winter TOTALS
One 77 2279 1640 285 2718 1393 8392
Two 0 1118 151 8 850 232 2359
Three 0 248 8 0 113 18 387
Four 0 17 1 0 12 1 31
Five 0 0 0 0 0 0 0
Table 1: Number of students reached per course
Fig. 1 Number of students reached per department
Fig. 2 Number of unique instruction sessions taught per department
Fig. 3 Number of instruction sessions per course level
Fig. 4 Number of instruction sessions per course level per department
U of T Mississauga LibraryWhat next? How do we gain further access to underserved
departments? How do we add new departments to our IL
program? Would we abandon classes with little impact on
student experience? Developing stronger vertical integration by
including more upper year courses
U of T Mississauga Library2. Reflective of current teaching practices Type of session Which ACRL Standards are addressed What tools are covered in the sessions Building a class profile
U of T Mississauga Library
Fig. 5 Number of unique instruction sessions given by type
U of T Mississauga Library Year
2005 2006 2007
Scope 20 30 8 Standard 1
Topic 1 7 1
Search 14 17 5 Standard 2
Tools 29 24 23
Standard 3 Evaluation 21 50 23 Le
arn
ing
Go
als
Standard 5 Legal 35 38 17
Table 2 Number of instruction sessions with stated ACRL Standards
U of T Mississauga Library Year
2005 2006 2007
Catalogue 36 40 17
Databases 2 17 17
GIS 17 13 9
Scholars Portal 6 29 15
Library Website 46 51 21
Other Tools 8 11 8
To
ols
Co
ve
red
No Tools Covered 2 4 2
Table 3 Number of instructions sessions teaching specific tools
U of T Mississauga Library 2005 Summer to 2007 Winter
An
thro
po
log
y
Bio
log
y
Ch
em
ica
l & P
hy
sic
al
Sc
ien
ce
s
Ec
on
om
ics
En
glis
h &
Dra
ma
Ge
og
rap
hy
His
tori
ca
l Stu
die
s
Ma
na
ge
me
nt
Ma
the
ma
tic
s &
C
om
pu
tati
on
al
Sc
ien
ce
s
Ph
ilos
op
hy
Po
litic
al S
cie
nc
e
Ps
yc
ho
log
y
So
cio
log
y
ICC
Eri
nd
ale
Co
urs
es
Catalogue 17 6 2 1 15 13 16 1 2 1 0 3 6 10 0
Databases 5 4 2 0 4 3 8 1 1 0 1 1 4 2 0
GIS 16 2 0 0 0 18 0 0 1 0 0 0 2 0 0
Scholars Portal 11 0 0 1 6 9 5 1 2 0 1 3 5 6 0
Library Website 17 13 4 1 12 25 15 2 3 1 2 5 10 8 0
Other Tools 5 3 0 1 1 6 3 0 0 0 0 0 3 5 0
To
ols
Co
ve
red
No Tools Covered 0 4 0 0 0 2 0 0 0 0 0 0 0 1 1
Table 4 Number of instruction sessions teaching specific tools by department
U of T Mississauga Library Anthropology
AN
T101H
5
AN
T102H
5
AN
T200Y
5
AN
T204Y
5
AN
T205H
5
AN
T306H
5
AN
T310H
5
AN
T312H
5
AN
T338H
5
AN
T432H
5
FS
C239Y
5
Catalogue 1 1 0 1 1 0 1 0 3 1 8
Databases 1 0 1 0 0 0 0 0 2 1 0
GIS 0 0 0 0 0 8 0 2 6 0 0
Scholars Portal 1 0 0 0 0 0 0 0 2 1 7
Library Website 1 1 0 1 1 0 1 0 3 1 8
Other Tools 1 0 0 0 0 1 0 0 3 0 0
To
ols
Co
vere
d
No Tools Covered 0 0 0 0 0 0 0 0 0 0 0
Table 5 Tools taught in instruction sessions: Department of Anthropology2005-2007
U of T Mississauga LibraryReflective of teaching practices Identify strengths and weaknesses Gain an understanding of current
teaching sessions Develop strategies to address the goals
of an embedded program across the curriculum
U of T Mississauga LibraryBuilding a class profile First year Classics course 55 students enrolled Summer session
U of T Mississauga Library
Unique Students
Once 18
Twice 4
3 Times 3
4 Times 2
5 Times 1
Total 28
Therefore 51% of student enrolled have had previous instruction
Table 6 Number of students enrolled in a first year Classics course with previous instruction sessions
U of T Mississauga Library
CCT100H5 PSY100Y5
ENG110Y5 ENG110Y5
ANT101H5 PSY100Y5
ANT101H5 HIS101H5
ENG110Y5 ENG110Y5 ENG120Y5
GGR117Y5 HIS101H5 PSY100Y5
CCT100H5 GGR117Y5 HIS101H5
FSC239Y5 PSY311H5 SOC209H5 SOC307H5
ANT101H5 ENG110Y5 ENG110Y5 PSY100Y5
ANT102H5 ANT200Y5 FSC239Y5 GGR117Y5 PSY100Y5
Courses Taken with instruction
Table 7 Courses with previous instruction takes by students enrolled in a Classics course
U of T Mississauga LibraryCourse profile 50% have already had at least one
instruction session 10 students have had two or moreQuestions What were our assumptions? How do we approach this class?
U of T Mississauga LibraryCourse profile continued No easy answer The data allows us to look closely at our
sessions: Is there repetition across classes? Year after year? What were the learning outcomes? What type of session was it?
We are now in the process of reflection and learning to build in time to work towards an embedded program
Thank you!
Questions?
References
ACRL, Information Literacy Glossary. last updated March 2007. Online at http://www.ala.org/ala/acrl/acrlissues/acrlinfolit/infolitoverview/infolitglossary/infolitglossary.htm
Anderson & Krathwohl, 2001
ANZILL, Australia and New Zealand Information Literacy Framework. 2nd edition. Adelaide, AU, 2004. http://www.anziil.org/resources/Info%20lit%202nd%20edition.pd
Biggs, J. (1999). Teaching for quality learning at university. Buckingham, U.K.: Society for Research into Higher Education (SRHE) & Open University Press.
Learning Commons, University of Guelph, (n.d.). Framework for the design and delivery of learning commons programs and services.
Scardamalia, M., & Bereiter, C. (2003). Knowledge Building. In J. W. Guthrie (Ed.), Encyclopedia of Education, Second Edition (pp.). New York: Macmillan Reference, USA. Retrieved from http://ikit.org/fulltext/2003_knowledge_building.pdf
Bereiter, C., & Scardamalia, M. (2003). Learning to work creatively with knowledge. In E. De Corte, L. Verschaffel, N. Entwistle, & J. van Merriënboer (Eds.), Unravelling basic components and dimensions of powerful learning environments. EARLI Advances in Learning and Instruction Series; Retrieved from http://ikit.org/fulltext/inresslearning.pdf
Reinvention Center. http://www.reinventioncenter.miami.edu/pdfs/2001BoyerSurvey.pdf