1

Marquette University Libraries, First Year English (FYE): Information Literacy Modules 10.09.2016 ─

Suvanida Duangudom Owner, A&R Consulting 725 Design Way, San Francisco, CA 94101 (415) 337-4461 sduangudom@ARConsulting.com

2

Executive Summary: Purpose of Evaluation This comprehensive evaluation plan has been created for Marquette University Libraries to measure the effectiveness of the First Year English (FYE) program. Improving the information literacy skills of students have been shown to correlate with higher grades on research assignments, increased academic success, and higher employment ratio for graduates. The strategies outlined in this plan will assist Marquette University in gathering baseline evaluation metrics to compare with future performances. With continued improvement of online modules with more focus on learners, evaluation data and analysis will highlight possible areas for improvement to ensure a better learning experience for all students.

Primary Objectives The primary objective of this evaluation is to measure the effectiveness of online information literacy modules and FYE program in order to improve (increase) the information literacy skills and competencies of undergraduate students at Marquette University.

Summary of Evaluation Plan The evaluation strategies outlined in this plan are based on Kirkpatrick’s Four Level Evaluation model (Kirkpatrick & Kirkpatrick, 2006) to measure the effectiveness of the First Year English (FYE) program and information literacy modules to increase the knowledge, skills, and abilities of students in the area of information literacy. An 18-month project plan and timeline, outlines each step required to evaluate the FYE program to determine learner reactions, learning outcomes, learner behavior and program impact on student learning. Sample evaluation instruments and plans are included in the appendices, along with survey questionnaires, and data statistics.

Key Recommendations With the majority of the FYE program consisting of online modules, the strategies outlined in this plan can be easily repeated each semester with new groups of learners to increase the volume of data, while also adding diversity to the study sample. Learner response rate may be nearly 100%, as the FYE program is a collaborative effort with curriculum areas and student participation is often encouraged by course professors. Analyzing large volumes of data will provide program administrators with evidence on how best to improve the effectiveness of the FYE program and online information literacy modules, as well as, future expansion of this initiative across subject disciplines at Marquette University.

Data Findings Since this is the first phase of evaluation for the FYE program, no training data has yet been collected so there is not any initial data findings to report. Program administrators for this evaluation project can be expected to review various types of data including: overall learner reaction scores: content, relevance, and satisfaction, learning outcome statistics: average scores from pre/post tests and surveys. Data elements will be analyzed and combined to measure the effectiveness of the FYE program and improving the information literacy skills of participants.

3

Evaluation Goals and Scope:

Instructional Product Description - Marquette University Libraries, FYE Information Literacy Modules Marquette University Libraries offers students an integrated approach to introductory library instruction through its partnership with general education courses, such as English 1001 or First Year English (FYE). Initially, the collaboration between the library and FYE began with requiring classes to attend “one shot” sessions that translated to library visits with no time for any discussion on specific course assignments or point of need research skills. Additionally, feedback from students participating in these sessions indicated that while they valued the library sessions; they were only able to recognize library resources, but not how to effectively use them for research assignments. To address these concerns, in Fall 2013, the library and the FYE program collaborated and developed a new approach to information literacy (IL) instruction that would increase student interaction with the librarians. The program emphasized the partnership between librarians and instructors in the classroom, as well as online through a learning management system (D2L). D2L allowed for collaborative planning and facilitated students connecting with a librarian throughout the semester through multiple sessions, in class or virtually. In order, to maximize the class time that students have with librarians, a flipped classroom approach to information literacy instruction was developed. Librarians developed a digital learning object, information literacy tutorials, to upload into D2L course pages to allow students to prepare for the first class session with a librarian in an engaging and motivating way; while also offering a baseline coverage of introductory information (Giles & James, 2015). The online IL tutorials were developed to teach undergraduates information literacy skills such as narrowing a topic, evaluating sources, and citing. Students were required to complete the following tutorials prior to the initial class meeting with the librarian:

Digital Learning Objects Stages Content

Academic Research Instruction Research for expository writing Why use library databases How to use Academic Search Complete

2 instructional videos 1 guided exercise 1 online quiz

Narrowing a Topic How to develop research questions How to narrow down source types

3 scenario-based activities 6 online quizzes

Boolean Bonanza Selecting Boolean search commands Constructing search statements

1 instructional video 3 scenario-based activities 6 online quizzes

Evaluating Sources Analyzing sources to determine validity for research assignment

4 scenario-based activities 12 online quizzes

Anatomy of Citations Citation Formats: MLA and APA Parts of a Citation

2 instructional videos 7 guided exercises 7 online quizzes

Purpose of Evaluation The First Year English (FYE) Program is a collaborative initiative at Marquette University between the libraries and the English department. The program is still in the early stages of

4

implementation with the goal of increasing the competency of information literacy skills for undergraduate students. The goals of the evaluation are to:

• Collect baseline metrics and benchmarks on learning satisfaction, learning outcomes, learning behaviors, and program benefits/impacts

• Recommend areas of improvement for online information literacy modules and in-person sessions, which comprise of the FYE program, to increase learner engagement

• Measure and correlate information literacy competency to academic success and future workforce employability

• Create an ongoing evaluation methodology to track changes in learning satisfaction, learning outcomes, learning behaviors and return of investment (ROI)

Primary Objective During the first year of the FYE program, only 50% of the undergraduates enrolled at Marquette University will be participating in the First Year English (FYE) program, requiring students to complete online information literacy (IL) modules. The primary objective of this evaluation is to measure the effectiveness of the FYE program and IL modules to improve information literacy skills and IL competency of students at Marquette University.

Learner Analysis The mission of Marquette University Libraries is to “support the teaching and research efforts, through the identification, selection, management and preservation of collections and other information resources” (Library Collections, 2016). In an effort to help students acquire essential information literacy (IL) skills; Marquette University Libraries designed several online tutorial modules. The audience will be a sample group of undergraduate students enrolled at Marquette University, who are taking general education courses such as English 1001 or First Year English (FYE). This sample group has been taken from more than 8,300 undergraduates enrolled at the university, representing all states and 70 countries reflecting the cultural diversity of the students at Marquette University. There will be a relatively even mix of males and females from 83 majors, 81 minors, and pre-professional programs in dentistry, law and medicine (About Marquette University, 2014). Marquette University Libraries have designed information literacy (IL) tutorials in collaboration with general education courses such as English 1001 or First Year English to ensure that undergraduate students meet the university’s graduation requirement for IL competency; as this skill is essential for future academic success and workforce development.

Evaluation Process The proposed evaluation strategy is based on Kirkpatrick’s Four Level Evaluation model (Kirkpatrick & Kirkpatrick, 2006), where each level will be examined through an in-depth analysis. This report will seek to measure the effectiveness of the information literacy modules in Marquette University’s First Year English program through a comprehensive evaluation process.

5

Level One Evaluation Measuring Level One, learner reaction is an important component of an overall evaluation strategy for Marquette University Libraries and First Year English (FYE) Program. The information literacy modules are a self-guided learning experience, where undergraduate students, complete information literacy modules on academic research: narrowing topics, Boolean searching, evaluating sources, citation formats, and etc.; which incorporate instructional videos, guided exercises, scenario-based activities, and online quizzes to provide learners with an engaging and motivational learning experience to increase learner satisfaction and memory retention. After the student completes the online instructional modules on their own, they will attend a librarian taught (in-person) instruction session to learn and apply what they have previously learned from the online modules. ● Evaluation Instrument: FYE Information Literacy Feedback Survey ● Evaluation Procedures: The survey can be created online using Google Forms and

delivered electronically to undergraduate students enrolled in general education courses such as English 1001 or First Year English after students have completed all the online tutorials prior to their scheduled in-person library instruction session. The completion of the survey will be mandatory, and serve as a “ticket” to attend the library session; which is required for students to complete the FYE program. (See Appendix A for the complete FYE Information Literacy Feedback Survey). The survey includes questions that address three elements of the instructional content and learning activities: quality, relevance, and overall satisfaction. Feedback questions are grouped into categories to allow for easier identification of areas for improvement. Learner reaction data about online tutorials should be evaluated following each library instruction session to measure learner satisfaction in the program content over time; as results can also be used to demonstrate the value of information literacy to the FYE program.

An online survey tool is highly recommended as the method of evaluation as it provides an efficient and easy way to collect learner reaction data. This type of instrument allows for initial reaction feedback as learners will complete the survey immediately following the completion of the information literacy modules; and results can be utilized immediately for remediation instruction of information literacy skills during the in-person library session.

Level Two Evaluation Measuring Level Two learning outcomes of information literacy modules is essential to the success of the first year English (FYE) program at Marquette University. The foundation of Marquette University’s First Year English program includes interactive modules to teach undergraduate students information literacy skills through traditional face-to-face instruction, flipped classroom instruction, or as stand alone online assignments. Evaluations must be conducted to ensure that students not only have a positive learning experience, but also that students have met the learning objectives for the online modules and library instruction session. ● Evaluation Instruments: Scenario-based activities, guided exercises, online quizzes, and

pre- and post-tests ● Evaluation Procedures: Level Two evaluation data should be collected during and after

the learning process to ensure a positive learning outcome. Scenario-based activities, guided exercises, and online quizzes are features within the information literacy modules, where students are required to complete each during the learning process. The scenario-based activities offer students an opportunity to apply key concepts covered in the instructional tutorials. The guided exercises provide the learner with additional

6

instruction followed by practice opportunities utilizing information literacy skills to allow for student engagement to increase memory retention. The online quizzes provide students with immediate feedback to test information literacy concepts to determine if they have learned these skills through the instructional tutorials and practice activities. Pre-and post-tests will also be administered to measure the undergraduate student’s learning ratio for information literacy skills. The pre-test will be given to students prior to them viewing the information literacy modules to measure what they know about academic research: types of sources, narrowing a topic, Boolean searching, evaluating sources, or citation formats. The exact same test will also be given to students after they have completed the information literacy modules (post-test) to measure the student learning ratio of the teaching content. (See Appendix B for the complete pre- and post-test questions).

By using these four recommended evaluation instruments, different types of data can be collected to provide a more accurate picture of the learning experience. The information literacy modules available for First Year English (FYE) students are presented in a learner-centered format, where students can select from general (academic research) to specific (types of sources, Boolean searching, etc.). Based on their current information literacy skills, students can select which module to view; as well as go back and review any concepts within each instructional module. Scenario-based activities, guided exercises, and online quizzes can be repeated as often as needed without penalty before the student moves onto learning a new information literacy concept. The pre- and post-tests provide a way to assess the skills of the learner to determine the gains in learning that students have achieved after completing the information literacy modules.

Level Three Evaluation Measuring Level Three learning behaviors of information literacy skills, is essential to demonstrating the impact that the First Year English (FYE) program at Marquette University has on a student’s academic success, personal life, and future employability. Students who have acquired information literacy skills are able to locate, access, evaluate and use information effectively (Association of College & Research Libraries, 2016). Evaluators must be able to ensure that participants have a positive learning experience, acquire the intended learning content, as well as provide opportunities for learners to apply what they have learned to real-life situations.

• Evaluation Instruments: Behavior Survey Questionnaires for Faculty (See Appendix C) and Student Behavior Survey Questionnaire (See Appendix D)

• Evaluation Procedure: An online survey questionnaire will be sent out to faculty at the end of each semester (See Appendix C). This survey questionnaire will determine if student research behavior has improved from the beginning of the semester to the end of the semester; as a result of students participating in the first year English (FYE) program and completing the information literacy modules. Faculty will be asked to respond to information literacy competency questions using a 5-point Likert scale, where 5=Excellent/Very High and 1=Poor/None, about student behavior before and after completing the online information literacy modules and how it has impacted their research assignments, papers, and projects. Students will also be given an online survey questionnaire to determine the effectiveness of the information literacy modules; and how it has changed their behavior regarding information need, information literacy skills, and evaluating information. The survey

7

will consist of questions to determine if students spend less time and energy on information literacy competency skills after completing the online information literacy modules and participating in the FYE program as measured by a 5-point Likert scale, where 5=Much more and 1=Much Less. The survey will be administered to students on the following schedule: 30/60/90/365 days after students have completed the online modules and FYE session. In order to get close to 100% response rate, faculty will provide student incentive to complete the survey such as earning extra credit on their coursework. • Rationale: Data collected from survey questionnaires as completed by faculty and

students can be combined with Level One and Level Two data in order to provide a more comprehensive picture of student learning experiences in the First Year English (FYE) program. The goal is to identify if there is a positive change in research and information seeking behavior and determine if learners are able to apply the knowledge, skills, and abilities, learned through information literacy modules and in-person sessions to their course work, personal projects, and future employment performances.

Level Four Evaluation The purpose of level four measures is to determine the effectiveness of online information literacy modules, and the First Year English (FYE) program initiative and the impact that each has on student learning and increase academic success.

• Evaluation Instruments: Level Three behavior data will provide the first two pieces of information to be collected at Level Four. The surveys sent out electronically to faculty and students will determine if the increase in information literacy competency results in students getting better grades on research assignments, and increased employability with marketable information literacy skills: knowledge and ability of being able to identify, locate, and evaluate information.

• Evaluation Procedures: A data analysis from Level One, Level Two, and Level Three measures is needed to determine the effectiveness of the FYE program and information literacy modules. The online survey questionnaires will be administered to faculty collaborating in the FYE program at the end of each semester. Additionally, another instrument will be administered to students 30/60/90/365 days after students have completed the information literacy modules and the First Year English program to determine behavioral changes that these students exhibit over the specified time frame; compared to other undergraduates at Marquette University who have not completed the information literacy modules and/or participated in the FYE program in order to compare these results.

• Rationale: To address the objective of the evaluation plan to expand the First Year program to other subject disciplines, beyond the English curriculum. The participation numbers need to be correlated with data from learner reaction, learning outcomes, and learning behaviors. As the data numbers improve, Marquette University is expected to expand the program to offer participation to other undergraduate students. Measurements from this year will establish the baseline metrics that can be used to benchmark future performances.

8

Data Collection & Analysis Gathering data and feedback from participants, both faculty and students through different evaluation methods provide insight to learner reaction and learner outcomes, respectively. Using a formal evaluation strategy allows Marquette University Libraries to set baseline results and compare data collected on an ongoing basis to strive for continual improvement. It is recommended that evaluators analyze collected data on a semester basis (twice a year) as student diversity may vary between those who begin enrollment at Marquette University in the Fall and Spring semesters, in order to identify areas of improvement to make changes and adapt instructional modules as necessary. After determining the baseline, evaluators should focus on the correlation between the ratio change (gain/loss) in student assessment data and completion rate of information literacy module(s). A control group consisting of undergraduate students at Marquette University who are enrolled in general education courses, but are not participating in First Year English (FYE) this year will also be administered evaluation instruments. The FYE: Information Literacy Feedback Survey (see Appendix A) has 10 questions that correspond to a five-point scale, where 1=Strongly Disagree and 5=Strongly Agree; with a section for additional comments for each question. The collected survey responses will be reported as average scores, as well as total combined reaction score, across each of the three categories: content, relevance, and satisfaction. Through data analysis the positive learning experience can be identified to help administration and management address ineffective content strategies and provide a way to implement new ideas to keep learners engaged, motivated and satisfied about the instructional program. The Information Literacy Pre- and Post- Test (see Appendix B) is a skills assessment, which consists of five questions that correspond to a multiple-choice answer to measure what students know or do not know prior to completing the online information literacy modules. This instrument will be administered to both a sample group and a control group to measure the learning ratio of information literacy concepts. The data will be reported based on the pre- and post- test assessment measure to represent the learner’s experience and growth. The skills assessment will demonstrate the student’s information literacy competency, and how they are able to apply information literacy skills acquired from the online modules and FYE program to research assignments leading to success in their academic work and future career endeavors. The combination of data from the pre/post test assessments will be used with data from other learner activities such as scenario-based activities, guided exercises, and online quizzes from tutorial modules to illustrate a more comprehensive picture of a student’s’ information literacy skills. By measuring the behavior changes of students and whether they are able to apply information literacy skills to actual research assignments, projects, and/or papers; two different questionnaire instruments will be used. Both survey instruments will be administered electronically, one to faculty (see Appendix C) and another to students (see Appendix D). The faculty survey questionnaire will consist of seven questions that allow faculty to observe their students’ information literacy skills before and after completing the information literacy modules; as well as the impact that information literacy skills has on their students’ course work. For each question, responses will be based on a five-point Likert scale with 5=Excellent/Very High and 1=Poor/None. Additionally, students will also have the opportunity to provide feedback, using a five-point Likert scale, about how their time and energy spent after the program has changed from the time and energy spent before the program, where 5=Much more and 1=Much Less. The student survey will consist of questions about the following topics: information needs, information literacy skills, and evaluating information.

9

The evaluation results from Level One, Level Two, and Level Three can be analyzed to produce an analysis for Level Four. For the scope of this evaluation plan, Marquette University will gather baseline metrics at all evaluation levels in order to provide benchmarks to allow this program initiative to be repeated and measured for future performances.

Phase Objective Method Timing Sources Outcome

One Reaction/Satisfaction: Measure undergraduate student satisfaction of information literacy module within the First Year English program

Survey/Questionnaire & Reaction Sheets to training: FYE Information Literacy Feedback Survey

Immediately after training

Undergraduate students enrolled in general education courses such as English 1001 and First Year English

Achieve 80% acceptable satisfaction rate

Two

Learning Measure: 1) Identify source types, differences between Internet and database resources 2) Demonstrate skills & knowledge of conducting academic research, narrowing a topic, Boolean searching, evaluating and citing sources

1) Pre/Post Tests, Skills Assessments 2) Scenario-based activities, guided exercises, and online quizzes provide opportunities to apply information literacy skills

Immediately after training, during training

Undergraduate students enrolled in general education courses such as English 1001 and First Year English

Achieve 90%-100% acceptable satisfaction rate

Three

Behavior Measure: 1) Knowledge, skills, and abilities of undergraduate students in the area of information literacy 2) Observation from faculty regarding student competency in applying information literacy skills

1) Faculty Survey Questionnaire 2) Student Survey Questionnaire

End of semester after training has been received; additionally 30/60/90/365 days after training

Undergraduate students enrolled in the First Year English (FYE) program; faculty collaborating with librarians in the FYE program

Achieve 90-100% response rate due to incentives provided by faculty

Four Evaluating Results: Determining the effectiveness of the FYE program and information literacy modules

Analyzing and combining data results from Levels One, Level Two, and Level Three

30/60/90/365 days after training

Undergraduate students enrolled in the First Year English (FYE) program; faculty collaborating with librarians in the FYE program

Achieve 90-100% effectiveness rating to present to administration for program expansion across disciplines

10

References: Association of College & Research Libraries. (2016, June 25). Information literacy defined.

Retrieved from Information Literacy Competency Standards for Higher Education website: http://www.ala.org/acrl/standards/informationliteracycompetency#ildef

Dow Chemical Company. (1992). Data Analysis and Interpretation. In Evaluation Training. Midland: Author.

Dow Chemical Company. (1992). Disseminating Evaluation Results. In Evaluation Training. Midland: Author.

Dow Chemical Company. (1992). The Dow Process. In Evaluation Training. Midland: Author. Dow Chemical Company. (1992). Methodology. In Evaluation Training. Midland: Author. Dow Chemical Company. (1992). Needs Analysis. In Evaluation Training. Midland: Author. Giles, E. A., & James, H. (2015). Is flipping enough? A mixed approach to introductory

information literacy instruction. College and Research Libraries News, 76(1), 10–13. Retrieved from http://crln.acrl.org/content/76/1/10.full

Kirkpatrick, D. L., & Kirkpatrick, J. D. (2006). Evaluating Learning. In Evaluating Training Programs: The Four Levels (pp. 42–51). San Francisco: Berrett-Koehler Publishers, Inc..

Kirkpatrick, D. L., & Kirkpatrick, J. D. (2006). Evaluating Reaction. In Evaluating Training Programs: The Four Levels (pp. 27–41). San Francisco: Berrett-Koehler Publishers, Inc..

Lowe, M. S., Booth, C., & Savona, M. (2014, March 1). Claremont Colleges Library Faculty Library Survey Summary Report: 2014. Retrieved from http://scholarship.claremont.edu/cgi/viewcontent.cgi?article=1022&context=library_staff

Marquette University. (2014). About Marquette University. Retrieved from http://marquette.edu/about/

Marquette University. (2013). Information Literacy Modules. Retrieved from http://marquetterml.github.io/information-literacy-modules/

Marquette University, & Raynor Memorial Libraries. (2013). Academic Library Research 101. Retrieved from http://www.mu.edu/library/lor/first-year-english/introduction-to-academic-research/

Marquette University, & Raynor Memorial Libraries. (2016). Library Collections. Retrieved from http://www.marquette.edu/library/about/librarycollections.php

Marshall, R. (2006). Information Competency Assessment Instrument. Retrieved from http://jfmueller.faculty.noctrl.edu/icai.pdf

University of Texas at Austin Libraries. (2011). Information Literacy Toolkit: Pre and Post-Test Questions. Retrieved from https://www.lib.utexas.edu/signaturecourses/resources/pre-and-post-test-questions

11

Appendices: ● Appendix A: FYE Information Literacy Feedback Survey ● Appendix B: Information Literacy Pre- and Post-Test ● Appendix C: Faculty Survey Questionnaire ● Appendix D: Student Survey Questionnaire ● Appendix E: Evaluation Plan ● Appendix F: Gantt Chart ● Appendix G: Screenshots from Information Literacy Modules ● Appendix H: Data Results

12

Appendix A: FYE - Information Literacy Feedback Survey

Purpose: Gather reaction and feedback from Marquette University undergraduate students who have completed the library’s information literacy modules. Content: Survey questions listed below:

Quality Relevance Satisfaction

The video tutorials helped me understand the concepts presented in the course.

The skills taught in the information literacy modules are relevant to my personal development.

The interactive components: instructional videos, activities, exercises, and online quizzes motivated me to complete online information literacy modules.

The scenario-based activities helped me validate my understanding of the content covered.

I feel that I am now fully equipped to be able to locate reputable sources for my research assignments.

I would recommend this course to a friend or classmate.

The guided exercises and online quizzes helped me validate my understanding of the content covered.

The level of training was adequate for my beginning skill level (not too easy/hard).

The content was well organized and easy to navigate.

The content covered in the course addressed the research skills I need to successfully locate sources for my assignment.

Responses: Likert scale - select the number that best corresponds to the 5-point scale, where 1=Strongly Disagree and 5=Strongly Agree. Space will be provided for participants to write in any additional comments or feedback that they may have. Student Link to Online Survey (Google Forms): https://goo.gl/forms/uIsZid854dZPhQlo2

13

Appendix B: Information Literacy Pre- and Post-Tests Purpose: Measure student learning of Marquette University undergraduates prior to and after completing the information literacy online modules. The pre/post tests will be administered to both the sample group and control group to better determine the effectiveness of the online modules Content: Survey questions and possible responses listed below:

Questions Potential Responses

Which of the following searches in a library database will give you the MOST results?

cloning OR bioengineering OR genetics cloning AND bioengineering AND genetics cloning AND bioengineering OR genetics I don't know

You know an article is from a peer reviewed journal because:

It was written by someone knowledgeable in the subject area It was reviewed by experts/scholars before being published It contains a general overview of the topic I don't know

You have to write an argumentative paper for your class arguing if exposure to violent media (gaming, movies, music, etc.) leads to violent behavior or not. You decide to take the stance that adolescent exposure to video games leads to violent behavior. You are ready to start searching library databases for supporting evidence to back up your stance. Which of the following search terms will work best for your topic?

media violence, video games, adolescents, effects media violence, video games, adolescents, exposure, behavior, effects video games, adolescents, behavior, effects violence, adolescents, effects, media I don’t know

In what format would you find the information in the following APA citation? Giles, E. A., & James, H. (2015). Is flipping enough? A mixed approach to introductory information literacy instruction. College and Research Libraries News, 76(1), 10–13. Retrieved from http://crln.acrl.org/content/76/1/10.full

An article in a magazine A chapter/article in an edited scholarly book An entry in an encyclopedia An article in a scholarly journal I don't know

What is the difference between Internet search engines and library databases?

Free to anyone with computer access/purchased by the library Information is organized/not organized Content is reviewed and recommended by librarians/no review standards with regard to content Information is not stable; locations and content continually change/information is stable

Student Link to Online Pre/Post Tests (Google Quiz): https://goo.gl/forms/8FQPULi43qOwyCOE2

14

Appendix C: Faculty Survey Questionnaire Purpose: Measures the effectiveness of the information literacy modules in the first year experience program. The survey questionnaire will be administered to faculty online due to convenience and quicker response time. The results of the questionnaire will seek to demonstrate students' competencies on finding, evaluating, and using information for their research assignments and projects. Content: Survey questions and possible responses listed below:

How would you rate your students’ information literacy skills? Potential Responses

Effectively use library databases, catalogs, and information resources to find relevant source materials

Likert scale: Before and After completing information literacy modules

Differentiate between types of information (scholarly vs. popular, literature, fact vs. opinion)

Likert scale: Before and After completing information literacy modules

Evaluate sources to determine if they are authoritative Likert scale: Before and After completing information literacy modules

Able to use sources to further an argument/thesis Likert scale: Before and After completing information literacy modules

Provide proper attribution to source materials in their academic work Likert scale: Before and After completing information literacy modules

Write annotated bibliographies Likert scale: Before and After completing information literacy modules

After completing the information literacy modules, how much has student work improved (achieved better grades on research papers, assignments, and projects)?

Likert scale: ● No improvement, no change in letter grade ● Low improvement ● Moderate improvement, earned one letter grade better ● High improvement ● Very High improvement, increased two or more letter

grade

Faculty Link to Survey Questionnaire (Google Form): https://goo.gl/forms/NmtPNf4mUJBpcgUx1

15

Appendix D: Student Survey Questionnaire

Purpose: This questionnaire will seek to measure the changes in student behavior as a result of completing the information literacy modules. Students will be administered the survey electronically. Content: Survey questions listed below: Information Need Information Literacy Skills Evaluating Information

"Confused" is probably the best term to describe me when starting a project.

I know the difference between "primary" and "secondary" sources.

I am able to figure out for whom the information intended.

When given an assignment for a research paper, I feel confident determining what topic I need to search.

I know how to broaden or narrow a search using Boolean operators (AND, OR, NOT)

I am sure that the information I have answers my question or addresses my topic.

I can take a complex topic and break it down into more useful, simpler items.

I know the difference between an abstract and an article.

A lot of the information I find is irrelevant or unnecessary

I am sometimes unsure of how much information I need for the assignment.

I'm know how to record or cite all my sources.

I am able to learn what processes would be helpful for finding information in the future.

Responses: Likert scale - Time and energy spent after the program compared to time and energy spent before the program; selecting the number that best corresponds to the 5-point scale, where 1=Much Less and 5=Much More Student Link to Online Survey (Google Forms)https://goo.gl/forms/n1XVpFMepHi38tWn2

16

Appendix E: Evaluation Plan Instructional

Design Project Scope Identification Page Owner First Year English (FYE) Program, Information Literacy Modules

Evaluation of the initial collaboration between FYE and the library to identify areas for improvement

See Evaluation Plan Suvanida Duangudom

Goal Purpose Levels Objectives to Measure

To evaluate the effectiveness of the information literacy modules in achieving the objectives described in the needs assessment

Level 1: Reaction Increase information literacy skills and competency of undergraduate students after 1 year of completing the information literacy modules and FYE program

Suvanida Duangudom

Level 2: Learning Suvanida Duangudom Level 3: Behavior Suvanida Duangudom Level 4: Results Suvanida Duangudom

Process Schedule Budget

Step Date Completed by Review/Edit Suggested Evaluation Plan 06/01/2012-07/15/2012 Suvanida Duangudom Create detailed evaluation plan (final) 07/16/2012-08/01/2012 Suvanida Duangudom Conduct FYE sessions with undergraduate students 01/06/2013 - 07/31/2013 Librarians, Students, Faculty Collect data 01/06/2013-12/01/2013 Suvanida Duangudom and Librarians Analyze data 12/09/2013-01/03/2014 Suvanida Duangudom and Librarians Report results 01/04/2014-03/31/2014 Suvanida Duangudom

Data Collection

Data Item When? How Collected? From Whom? By Whom? Reaction of students to information literacy modules Immediately following the completion of

online modules Feedback Survey Students Librarians

Knowledge of information literacy skills and techniques Prior to the FYE program, after students complete modules

Pre/Post Information Literacy Test

Students Librarians

Student application of information literacy skills After students complete information literacy modules

Faculty Survey Faculty Librarians

30/60/90/365 days measure of information literacy competency

30/60/90/365 days following the completion of the FYE program and information literacy modules

Online Student Survey Questionnaire

Students in the program, and those not in the program in order to compare results from both sample and control groups

Suvanida Duangudom

17

Appendix E: Evaluation Plan (cont’d) Analysis Target Measures Methods of Calculation How to Isolate the effects of training Analyst

Student Satisfaction Average from reaction survey Reaction data is not applicable to isolate training Suvanida Duangudom Knowledge of information literacy skills Average from pre/post test Use of a pre-test to create a baseline and compared with post test results Ability to use information literacy skills Average of faculty surveys Surveys will be sent to faculty to measure their students’ ability to use and apply information literacy

skills to assignments Information Literacy Competency following completion of FYE program and graduation

Average of student surveys Surveys will be sent to all students following their completion of information literacy modules, after completion of FYE programs, and after graduation

Report What results reported? To Whom? How Communicated? Student satisfaction and impact of information literacy skills Program administrators, Library director, university administrators Email communication of initial assessment

Oral report of findings on impact of FYE program Demonstrate effectiveness of FYE programs, information literacy modules to increase information literacy skills and competency of undergraduates students

Program administrators, Library director, university administrators Email summarizing 90 day behavior questionnaire assessment; oral presentation of final findings

18

Appendix F: Gantt Chart Evaluation Process Stages Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar

2012 2013 2014

Evaluation Planning

Draft Evaluation Plan 6/1

Review/Edit Evaluation Plan 7/15

Final Evaluation Plan 8/1

Develop Level 1 & 2 Instruments 9/1 10/31

Develop Level 3 Instruments 11/1

Conduct First Year Experience Programs (FYE)

Information Literacy Modules 1/6 7/31

In-Person Session with Librarian 1/6 7/31

Collect Evaluation Data

Reaction Feedback (level 1) 1/6 7/31

Pre/Post Test Information Literacy Skills Test (level 2)

1/6 7/31

Faculty & Student Survey (level 3) 8/1 12/1

Analyze Data

Level 1, 2, & 3 Data Analyzed 12/9

Level 4 Data Analyzed 1/3

Report Results

Email update on Level 1-3 Results 1/4

Final Report & Presentation of Findings

3/31

19

Appendix G: Screenshots from Information Literacy Modules Academic Research Introduction

Narrowing a Topic

Boolean Bonanza

20

Evaluating Resources

Anatomy of Citations - MLA Style

21

Anatomy of Citations - APA Style

Additional Practice Tutorials

22

23

Appendix H: Data Results Reaction Data

Learning Data