Download - Phase I: Overview
![Page 1: Phase I: Overview](https://reader035.vdocument.in/reader035/viewer/2022081603/55900c2a1a28abd53d8b463a/html5/thumbnails/1.jpg)
A Standardized Approach to Assessment:A Standardized Approach to Assessment:
Results from a Pilot Study of Library Instruction Sessions at San José State University Results from a Pilot Study of Library Instruction Sessions at San José State University
Shannon M. StaleySocial Sciences LibrarianShannon M. StaleySocial Sciences Librarian
![Page 2: Phase I: Overview](https://reader035.vdocument.in/reader035/viewer/2022081603/55900c2a1a28abd53d8b463a/html5/thumbnails/2.jpg)
Purpose of Study
To Develop and Test an assessment tool that:
• Facilitates the development of pre and post achievement tests
• Links survey questions to learning outcomes and ACRL standards
• Delivers real time reports and statistical analysis
![Page 3: Phase I: Overview](https://reader035.vdocument.in/reader035/viewer/2022081603/55900c2a1a28abd53d8b463a/html5/thumbnails/3.jpg)
Timeline:An Overview
• Review of the Literature – Summer 2008
• Create assessment tool – Fall 2008
• Garner input from library faculty / Refine tool - 12/08
• Develop multiple choice questions – 12/08
• Standardize questions – 1/09
• Pilot test assessment tool – 1/09
• Enact actual study – Spring 2009
![Page 4: Phase I: Overview](https://reader035.vdocument.in/reader035/viewer/2022081603/55900c2a1a28abd53d8b463a/html5/thumbnails/4.jpg)
Methodology:Develop Multiple-Choice Questions
• Investigated questions from already existing tutorials / survey instruments for ideas
• Created 20 multiple-choice questions based on social sciences 100W learning outcomes
• Each question presented four answers, with only one correct answer
• Each question had an added option for “not sure”
![Page 5: Phase I: Overview](https://reader035.vdocument.in/reader035/viewer/2022081603/55900c2a1a28abd53d8b463a/html5/thumbnails/5.jpg)
Methodology:Standardization of Survey Multiple Choice Questions
• Multiple choice questions were emphasized to allow for immediate computation of results
• The co-occurrence of LibQual during Spring 2009 provides for complimentary qualitative data
• Questions were reviewed by library faculty to ensure they reflected learning outcomes / ACRL standards
• Questions and learning outcomes were emailed to all teaching faculty in the Psychology department for review
• Questions were reviewed by experts at the Center for Assessment under the Office of Institutional Research
• Cognitive interviews with 5 students further addressed the clarity of survey questions
![Page 6: Phase I: Overview](https://reader035.vdocument.in/reader035/viewer/2022081603/55900c2a1a28abd53d8b463a/html5/thumbnails/6.jpg)
Methodology:Pilot Study
Conducted with students in a PSYC 139 course to
determine:
• Average length of time to complete surveys
• Overall impact on what could be covered in a single instruction session
• Any technical glitches with survey interface, login, usability etc.
• If survey questions were clear and comprehensible to students
![Page 7: Phase I: Overview](https://reader035.vdocument.in/reader035/viewer/2022081603/55900c2a1a28abd53d8b463a/html5/thumbnails/7.jpg)
Methodology:Pilot Study
Conducted with students in PSYC 139 to
determine:
• Average length of time to complete surveys - 5 minutes (10 minutes total for pre and post survey completion)
• Overall impact on what could be covered in a single instruction session
• Any technical glitches with survey interface, login, usability etc.
• If survey questions were clear and comprehensible to students
![Page 8: Phase I: Overview](https://reader035.vdocument.in/reader035/viewer/2022081603/55900c2a1a28abd53d8b463a/html5/thumbnails/8.jpg)
Methodology:Pilot Study
Conducted with students in PSYC 139 to
determine:
• Average length of time to complete surveys - 5 minutes (10 minutes total for pre and post survey completion)
• Overall impact on what could be covered in a single instruction session - no major impact with tighter planning
• Any technical glitches with survey interface, login, usability etc.
• If survey questions were clear and comprehensible to students
![Page 9: Phase I: Overview](https://reader035.vdocument.in/reader035/viewer/2022081603/55900c2a1a28abd53d8b463a/html5/thumbnails/9.jpg)
Methodology:Pilot Study
Conducted with students in PSYC 139 to
determine:
• Average length of time to complete surveys - 5 minutes (10 minutes total for pre and post survey completion)
• Overall impact on what could be covered in a single instruction session - no major impact with tighter planning
• Any technical glitches with survey interface, login, usability etc. - no major problems. Logins for the post test could not contain typos
• If survey questions were clear and comprehensible to students
![Page 10: Phase I: Overview](https://reader035.vdocument.in/reader035/viewer/2022081603/55900c2a1a28abd53d8b463a/html5/thumbnails/10.jpg)
Methodology:Pilot Study
Conducted with students in PSYC 139 to
determine:
• Average length of time to complete surveys - 5 minutes (10 minutes total for pre and post survey completion)
• Overall impact on what could be covered in a single instruction session - no major impact with tighter planning
• Any technical glitches with survey interface, login, usability etc. - no major problems. Logins for the post test could not contain typos
• If survey questions were clear and comprehensible to students – no confusion was reported
![Page 11: Phase I: Overview](https://reader035.vdocument.in/reader035/viewer/2022081603/55900c2a1a28abd53d8b463a/html5/thumbnails/11.jpg)
Methodology:Planning and Instruction
• Coordinated with instructor beforehand in encouraging students to show up early
• Scripted instructions were read to students prior to taking the pre test survey
• Lecture with embedded exercises; last 15 minutes reserved for getting started on course assignments
• Course outline and handouts were distributed to students after completion of their post test surveys
![Page 12: Phase I: Overview](https://reader035.vdocument.in/reader035/viewer/2022081603/55900c2a1a28abd53d8b463a/html5/thumbnails/12.jpg)
Methodology:Actual Study – Spring 2009
• Conducted with students in 5 PSYC 100W courses (83 students total)
• Survey participation was voluntary
• Participants who completed the pre-test survey but failed to complete the post-test survey were automatically eliminated from the data pool
• 11 multiple-choice survey questions covering 3 ACRL standards and 8 learning objectives
• 6 background questions on the pre-test survey – they were not repeated on the post-test
![Page 13: Phase I: Overview](https://reader035.vdocument.in/reader035/viewer/2022081603/55900c2a1a28abd53d8b463a/html5/thumbnails/13.jpg)
Methodology:Pre-Test Survey Background Questions
![Page 14: Phase I: Overview](https://reader035.vdocument.in/reader035/viewer/2022081603/55900c2a1a28abd53d8b463a/html5/thumbnails/14.jpg)
Methodology:PSYC 100W Multiple-Choice Survey Questions
![Page 15: Phase I: Overview](https://reader035.vdocument.in/reader035/viewer/2022081603/55900c2a1a28abd53d8b463a/html5/thumbnails/15.jpg)
Data AnalysisA Bird’s Eye View
![Page 16: Phase I: Overview](https://reader035.vdocument.in/reader035/viewer/2022081603/55900c2a1a28abd53d8b463a/html5/thumbnails/16.jpg)
Data AnalysisA Bird’s Eye View
![Page 17: Phase I: Overview](https://reader035.vdocument.in/reader035/viewer/2022081603/55900c2a1a28abd53d8b463a/html5/thumbnails/17.jpg)
Data AnalysisA Bird’s Eye View
![Page 18: Phase I: Overview](https://reader035.vdocument.in/reader035/viewer/2022081603/55900c2a1a28abd53d8b463a/html5/thumbnails/18.jpg)
Data AnalysisStatistical Significance
• Paired TTEST of student mean scores
• Excel formula computes statistical significance in the report generation feature of the assessment tool
![Page 19: Phase I: Overview](https://reader035.vdocument.in/reader035/viewer/2022081603/55900c2a1a28abd53d8b463a/html5/thumbnails/19.jpg)
Data AnalysisAssessment Tool Sample Report
![Page 20: Phase I: Overview](https://reader035.vdocument.in/reader035/viewer/2022081603/55900c2a1a28abd53d8b463a/html5/thumbnails/20.jpg)
Data AnalysisFuture Enhancements
• Master spreadsheet that incorporates all reports
• More granular TTEST analysis of individual questions
• Statistical analysis of how background factors (class level, research experience, instruction experience) affect scores
![Page 21: Phase I: Overview](https://reader035.vdocument.in/reader035/viewer/2022081603/55900c2a1a28abd53d8b463a/html5/thumbnails/21.jpg)
Benefits
• Assessment data will inform pedagogical directions and enhance the educational impact on SJSU students
• Data can be shared with department faculty and campus administrators to demonstrate instructional efficacy
• Data can also be used to bolster documentation during the accreditation process
• Librarians can publish findings from their own sessions, particularly with a more rigorous statistical model
• Librarians can include assessment reports in their
dossiers