What Makes the First-Year Seminar High-Impact?An Exploration of Effective Educational Practices at The University of Tampa

Dr. Edesa Scarborough Director of First-Year Experience The University of Tampa | escarborough@ut.edu

Dr. Chris Maurer Assistant Professor of Information and Technology Management The University of Tampa | cmaurer@ut.edu

1

Institution Description and DemographicsThe University of Tampa is a four-year, private, comprehensive university located in downtown Tampa, Florida, offering 200 programs of study and 65 undergraduate majors across its four degree-granting colleges. In the fall of 2015 the University had an enrollment of 6,820 undergraduate students of whom 2,004 were freshmen. For the purposes of this case, we focus and present results from 1,183 students who were enrolled in and completed assessments as part of a first-year experience course. Of this sample, 63% were female and 37% were male students. Other demographic information is presented in Table 1 and Figure 1 below:

Table 1: Age Demographics of First Year Seminar Students

Age Group Number of Students

Less than 21 1154

21-25 25

26-30 2

31-40 1

40-50 1

Figure 1: Age Demographics of First Year Seminar Students

Percentage of first-generation college students

First generation Not first generation

No answer

Primary language spoken

English Not English

The First-year ProgramThe University’s First-Year Student Seminar I & II (BAC 101/102) is a graded (A-F scale), one year, two credit seminar. During fall 2015 the program began transitioning from an extended orientation seminar to an academic seminar with various topics or themed courses. The course is required for all first-year students. The course meets for 14 weeks in the semester, one hour per week, and the primary focus of the seminar is academic skills development. To accomplish this, the following learning objectives are assessed in the first semester (BAC 101: college transition, time management, and creating an academic Plan. Critical Thinking and Major/Career Exploration are assessed in the second semester (BAC 102).

2

This study focuses specifically on the following three objectives:

I. College Transition: Students should be able to describe the culture of college, including: know the difference between high school and college standards, identify and use higher education resources, identify higher education skills that transfer to the workplace and explain factors that affect degree completion.

II. Time Management: Students should be able to identify effective time management strategies, explain the advantages and disadvantages of various time management tools, apply their understanding of time management planning tools to situations and understand the importance of prioritizing activities.

III. Creating an Academic Plan: Students should be able to describe planning and its value, gather information to build an academic plan, map courses for their academic plan on a term-by-term timeline, including how to adjust an academic plan, and be able to resolve academic planning issues.

Additionally, the course curriculum includes using MyStudentSuccessLab™ and the Conley Readiness Index (CRI) (Conley, 2010). MyStudentSuccessLab is an interactive, online solution that helps students acquire and build a skill set for ongoing personal and professional development that includes time management, academic planning, critical thinking, career exploration, and more. MyStudentSuccessLab personalizes each student’s path to completion and tracks learning gains through pre- and post-module assessments. CRI is a research-based, self-diagnostic tool that measures college and career readiness in order to assess mastery in four areas critical to college success: (a) cognitive strategies (Think), (b) content knowledge (Know), (c) learning skills and techniques (Act), and (d) transition knowledge and skills (Go). Students take the CRI survey during orientation. The results of the survey are then used by the faculty as a tool to identify students’ weak areas and set academic goals early in the first semester, specifically in the areas of critical thinking, time management, academic planning, college transition, and major and career exploration.

There are two first-year courses reserved for specific groups; Pathways to Honors: Exploration of Excellence/Leadership/Service (HON 101/102) is the FYE course for students accepted to the Honors program. The Military Veteran Seminar (BAC 104) is the FYE course for veteran students returning to academia after completing military service. There are also several subsets of the first -year seminar including sections for undeclared students (NTIs), student athletes, international students (BAC 103) and first- generation college students. These sections expand on topics that are specific to the needs of these student groups. For example, topics covered in the veteran seminar include VA benefits, and internships and career strategies for returning military personnel. The international student seminar includes an explanation of the American educational system. The student athlete group is assigned a peer mentor who is also part of the team and therefore can offer support when the team is traveling.

Table 2: Breakdown of First Year Seminar Courses and Formats

Course Total # of Sections % of Total Themed Courses % ThemedBAC 101 69 63.30% 19 17.43%

BAC 102 9 8.26% - -

BAC 103 9 8.26% - -

BAC 104 1 0.92% - -

HON 101 21 19.27% - -

Total 109 100.00% 19 17.43%

3

Several sections of the BAC 101 course were designated as themed sections. Themed sections have the same learning objectives as the traditional sections of BAC 101, however the material is delivered in the context of an interesting contemporary topic. Examples of themed section titles include “Stay on Track: The Health and Fitness Connection to Student Success” and “Understanding Leadership through Social Media.” The purpose of offering themed courses is to get students excited about academic skills development by applying these skills to a topic area they are attracted to. Students self-select into their FYE course by filling out a course selection preference form, which allows them to choose a specific themed section or be placed into a specific subset course (student-athlete, veteran, grouped with students in their major, etc.)

Faculty members apply to teach in the First-Year Experience program and courses are taught in addition to their departmental course load. Faculties are paired with a student peer mentor by academic disciplines who also apply to the program.

Peer mentors must have a 3.0/4.0 GPA and have no institutional violations in order to remain in the program. These are paid positions and require that the mentor be present a minimum of three hours per week including with one of those hours in the classroom during FYE class meetings. Additionally, the peer mentor program is internationally certified by the College of Reading and Learning through their International Mentor Training Program Certification (CRLA-IMTPC) at levels 1 and 2. Mentors must earn their certification by attending workshops on various topics throughout the year.

Educationally Effective Practices:Throughout the semester, students in the BAC 101/102 course take an online pre-test on each of the assessed modules. The faculty receives both class and individual student results, which allows them to determine areas of deficiency. Based on this information the faculty discusses the topic in class. Students must then complete a series of assignments on the topic outside of class.

For example, students spent approximately 20-30 minutes to complete an online pre-test on time management outside of class. The test contains 20 multiple choice questions linked to learning objectives and leveled by Bloom’s Taxonomy. The diagnostic assesses students’ understanding before they complete activities focused on skills development.

The following is a sample assignment used to teach time management:

Use a Syllabus to Plan Your Time: This activity gives students an overview of the importance of using a syllabus as a time management tool. Students are asked to review a “mock” syllabus from two courses and a work schedule. Then, through a drag and drop activity, students calendar one month’s worth of important due dates and work times. Once they’ve finished the calendaring activity, they receive verbal instruction as to how calendaring due dates is only the first step in planning. They need to calendar time to read assignments, study for exams, and complete projects once they know what their deadlines are. As the voice speaks, they see a visual demonstration.

4

This assignment addresses the following skills sets: planning for college using time management tools (a syllabus, a work schedule, and a monthly calendar), identifying important dates, and learning about calendaring sufficient time to meet course responsibilities. Time on Task is about 10 to 12 minutes, depending on time spent reading the syllabus. While this illustrates one particular out of class activity, students were asked to complete two or three activities for each learning objective and spent an average of one hour completing these activities.

Students then took a post-test approximately one week after the activities were completed. This diagnostic assesses students’ understanding after they complete the module and is comprised of 20 multiple choice questions. Students spent approximately 30 to 60 minutes on this test, which is taken online outside of class.

On average a student can spend anywhere between 2.5 to 3 hours per week on a topic in addition to the one hour spent in class. Additional time may include discussion with peer mentors on the topic outside of class or out-of- class experiences (OCEs), which some faculty may incorporate into their individual course experience. Participation in OCEs, however, is not required in any BAC course unless the OCE take place during the class period.

Discussion of the Evaluation of Educationally Effective PracticesCourse learning objectives for the first-year seminar were evaluated in two ways. First, student performance on the post diagnostic tests was evaluated to determine the percentage of students who exceeded, met, and did not meet expectations. Second, student improvement from pre-test to post-test was evaluated to determine whether the pedagogical approach to the first-year seminar was effective.

Figure 2: Assessment of Learning Objectives Based on Post-Test Scores

College Transition

Time Management

Academic Planning

Does Not Meet Expectations Meets Expectations Exceeds Expectations

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

For assessment purposes, scores of 90% or higher exceeded expectations, scores ranging from 75% to less than 90% met expectations and scores below 75% did not meet expectations. As Figure 2 shows, 71%, 94% and 71% of students met or exceeded expectations in the areas of college transition, time management, and academic planning.

While a large majority of students met or exceeded expectations related to each of the three assessed learning objectives, it is important to evaluate the efficacy of educational practices employed in the first-year seminar course. Since the first-year seminar focuses on individualized attention and directed learning, an overall improvement from pre-test to post-test score was anticipated. To statistically evaluate this

5

assertion, a paired-sample t-test was performed for each of the three assessed areas, which allows for an evaluation as to whether the mean pre-test score was different from the mean post-test score.

For all three assessed learning objectives, a significant difference in the means of pre-test scores and post-test scores was observed. The mean pre-test value for college transition was a 72.23% and post-test mean was 77.87% for an average improvement of 5.64% (n = 1136; p-value = 0.000). For time management, the largest improvement was observed with a mean pre-test score of 78.3% and mean post-test score of 87.89%. The difference was 9.59% (n = 1117; p-value = 0.000). In regards to academic planning, the mean pre-test score was 71.69%, mean post-test score was 78.58%, and mean improvement was 6.89% (n = 826; p-value = 0.000). The mean improvement values, along with their 95% confidence intervals are visually depicted in Figure 3.

Figure 3: Improvement from Pre-Test to Post-Test

0%Transition

*Veritical bars represent the 95% confidence interval

Time Management Academic Planning

2%

4%

6%

8%

10%

12%

14%

As previously described, The University of Tampa offers traditional first-year seminar sections as well as themed sections. Themed sections were introduced as a way of enhancing the learning experience of the first-year program and therefore, additional analyses were performed to evaluate whether differences in student performance were observed between traditional and themed sections.

Multivariate analysis of variance (MANOVA) models were constructed comparing student scores on pre-test versus post-test across themed and traditional courses. On the topic of college transition, no significant differences were observed in pre-test scores between traditional and themed courses (71.46% vs. 71.66%). However, a significant difference in post test scores was observed. Students in traditional sections averaged 77.06% whereas students in themed sections averaged 79.42% (n = 1168; p-value = 0.012). Likewise, pre-test scores for time management were nearly identical for traditional and themed courses (77.06% vs. 77.15%). Students in a themed course averaged slightly higher than those in a traditional section (88.37% vs. 87.20%); however this difference was not statistically significant (n = 1158; p-value = 0.219). In regard to academic planning, average pre-test and post-test scores were higher for those students in themed sections compared to those in traditional sections (70.37% vs. 68.51% for pre-test scores and 79.13% vs. 76.88% for post-test scores); however the differences between groups was not statistically significant (n = 888; p-value 0.135). Figure 4 visually represents the comparison of improvement from pre-test to post-test across traditional and themed sections of the first-year seminar.

6

Figure 4: Student Improvement in Themed vs. Traditional Seminars

0%Transition

Transitional Format

*Vertical bars represent the 95% confidence interval

Themed Format

Time Management Academic Planning

2%

4%

6%

8%

10%

12%

14%

ImplicationsThis case describes the results of a formal assessment of student learning objectives related to college transition, time management, and academic planning, all of which are of critical importance to the success of college students. Although only one instance of an educationally effective practice is detailed, the first-year seminar program at the University of Tampa is a robust collection of custom-tailored assignments and activities designed to prepare all students for success in college and beyond. The general approach of measuring student understanding of a particular topic through a pre-diagnostic test, guiding students through in-class activities and discussion, and assigning out of class activities appears to be working effectively to improve student performance in these three areas as demonstrated by scores on post-diagnostic tests.

In an effort to continuously improve the effectiveness of the first-year seminar program, themed courses have been introduced and offer students the opportunity to develop skills that are important for academic success within the context of an interesting and contemporary topic of interest to them. One risk associated with themed courses is that the focus on academic success skills could be overshadowed by the content associated with the theme. This is clearly not the case at the University of Tampa where results of student assessment demonstrate that improvement in two of the three assessed areas was not statistically different across themed and traditional courses and in the case of college transition skills, those students in themed courses outperformed those in a traditional format.

While the first-year seminar program has existed for 20 years at the University of Tampa, formal assessment of student performance and learning was only introduced in the fall 2015 semester. The data that has been collected, in combination with student and faculty feedback, is reviewed and analyzed by a faculty committee dedicated to the continuous improvement of the program. Opportunities to modify the curriculum and implement additional learning objectives in the first-year seminar will arise from the analysis of student performance. The results discussed in this case demonstrate that the practices enacted in this seminar are effective, but ongoing data collection will be helpful in continuing to prepare first-year students for success in their academic and professional careers. In the future, data from the first-year seminar will be combined with academic performance data to track student success in a longitudinal manner. Analyzing data across different areas of study and demographic groups may allow for the identification of personalized and targeted educational practices that best prepare different groups of students for success in their future careers.

ReferencesConley, D. (2010). College and career ready: Helping all students succeed beyond

high school. San Francisco, CA: Jossey-Bass.

Copyright © 2020 Pearson Education, Inc. or its affiliate(s). All rights reserved. HESRV10069-PK-03/20 | Cover illustration by Nathalie Lees.