general education learning outcomes assessment: … · 2011. 3. 14. · = 2.9 2.9 met standard met...

Office of Institutional Research and Effectiveness (IRE)

General Education Learning

Outcomes Assessment

Overall Outcomes

Personal Developm-

ent

Ethics

Quantitative Reasoning

Technology and

Information Literacy

Critical Thinking

Global Awareness

Communicat-ions

GENERAL EDUCATION LEARNING OUTCOMES ASSESSMENT: 2009-2010

Introduction

A. Overall Measures

B. Specific Learning Outcomes

C. General Education Learning Outcomes Targeted for Improvement

D. Development of General Education Learning Outcomes

Appendix

Charts

Introduction

General Education outcomes at the College reflect an integration and synthesis of knowledge and skills acquired through a program of study. College faculty developed and approved seven general education learning outcomes which represent the fundamental aspects of the expected competencies of the general education core. These seven general education learning outcomes are:

Communications: Develop effective reading, writing, speaking, listening, and nonverbal communication skills for a variety of audiences.

Global Awareness: Exhibit a sense of community and be sensitive to cultural and global diversity; exercise civic responsibility; give service to others; and respect and care for our natural environment.

Critical Thinking: Evaluate arguments; separate fact and opinion; recognize points of view; understand implications and consequences, and acknowledge diverse values.

Technology and Information Literacy: Use printed materials, personal communication, observation, and technological applications to find, evaluate organize, and present information in order to achieve educational and professional success.

Quantitative Reasoning: Use college-level mathematical concepts and methods to understand, analyze, and explain issues in quantitative terms.

Ethics: Demonstrate a responsibility for personal, social, professional, educational and natural environments and make informed decisions based on those responsibilities.

Personal Development: Develop an ability to understand and manage self, adapt to change, enhance wellness, learn effectively, establish a framework for aesthetic responsiveness, and set personal and professional goals.

PALM BEACH STATE COLLEGE

GENERAL EDUCATION LEARNING OUTCOMES ASSESSMENT

1

FALL 2009-2010

Part A: General Education Learning Outcomes – Overall Measures

Institutional

Measurement How Measured

When Measured

How data are

collected

Who submits

data

Performance Standard or Benchmark

Results Results

Compared to Standard

Summative Evaluation

Is this Outcome Focused for

Improvement?

1. CCSSE: Community College Survey of Student Engagement

Average score of 1 CCSSE Item (12a)* N=1206

2007 CCSSE IRE Office Greater than or equal to the mean of other extra large colleges = 2.9

2.9 Met Standard

Met Standard Not Applicable

2. Success Rate in general education courses

Extracted from college data: Percentage of students who receive a grade of “C” or better in all Gen Ed Courses

2008-2009 Academic year

IRE Office IRE Office 80% 85% Met Standard

3. Performance of College graduates in their first year in the State University System (%GPA >=2.5)

Accountability Report Annually Spring, 2009

State of Florida

IRE Office Florida System Average = 77%

76% Slightly Below Standard

4. Performance of College graduates in their first year in the State University System (average GPA)

Accountability Report Annually Spring, 2009

State of Florida

IRE Office Florida System Average = 2.95

2.91 Slightly Below Standard

*See appendix



2

Part B: Specific Learning Outcomes

Learning Outcome:

Communications Develop effective reading, writing, speaking, listening, and nonverbal communication skills for a variety of audiences.

Type of


When Measured

How data are collected

Who submits

data


Results

Results Compared

to Standard


Is this Outcome

Focused for Improvement?

1. MAPP: Measure of Academic Proficiency and Progress

Reading Score: MAPP test administered in a random sample of courses where average credits earned = 35. N = 168

Fall 2009 ETS ETS National norms provided by ETS 50

th

Percentile = 118

117 Slightly Below Standard

Met Standard YES


Writing Score: MAPP test administered in a random sample of courses where average credits earned = 35. N = 168


th

Percentile = 114

114 Met Standard

3. Scenario*** Scenario-based Assessment: Administered in a random sample of courses where average credits earned = 35. N = 75. Graded with a 5-point Rubric.

Fall 2009 IRE Office

IRE Office

Students will achieve average score of 2.50 or higher

2.12 Below Standard

4. Success Rate in relevant general education courses

Extracted from college data: Percentage of students who receive a grade of “C” or better in relevant* Gen Ed Courses

2008-2009 Academic Year

IRE Office

IRE Office

80% 86% Met Standard

5. Graduating Student Survey

Students’ self- rating on Gen Ed Learning Outcome: Percentage rating 4 or 5 on a 5-point scale, where 5 is excellent. N= 1268

2009 IRE Office

IRE Office



Average score of 8 CCSSE Items (4a; 4b; 4c; 6a; 6b; 6c; 12c; 12d)** N=1206

2007 CCSSE IRE Office

Greater than or equal to the mean of other extra large colleges = 2.5

2.6 Met Standard



3

Type of


When Measured


Who submits

data


Results

Results Compared

to Standard


Is this Outcome


7. CLAS – College-Level Academic Skills -Essay

Accountability Report Spring 2009

State of Florida

IRE Office

FL System = 99%

99% Met Standard

8. CLAS – College-Level Academic Skills -Reading


State of

Florida

IRE Office

FL System = 99%

99% Met Standard

9. CLAS - College-Level Academic Skills -English


State of

Florida

IRE Office

FL System = 99%

98% Slightly Below Standard

* Courses that contain curriculum relevant to the specific General Education outcome. **See appendix ***The same scenario was used to access both Communication and Quantitative Reasoning; the general education assessment committee determined that by combining the two outcomes into one scenario, students were not able to focus properly on either task. The results therefore do not accurately measure either outcome. Revisions have been made so that each outcome will have its own scenario in future implementations.



4

Learning Outcome:

Global Awareness Exhibit a sense of community and be sensitive to cultural and global diversity; exercise civic responsibility; give service to others; and respect and care for our natural environment.

Type of Measurement How Measured

When Measured

How data are

collected

Who submits

data


Results

Results Compared

to Standard


Is this Outcome



Fall 2009 IRE Office IRE Office


2.19 Below Standard

Met Standard NO




IRE Office IRE Office




Dec. 2009 IRE Office IRE Office



Average score of 10 CCSSE Items (4f; 4g; 4h; 4i; 4s; 4t; 9c; 12h; 12k; 12m)** N=1206



2.1 Met Standard

* Courses that contain curriculum relevant to the specific General Education outcome. **See appendix ***General Education assessment committee determined that this instrument was extremely difficult. The time allotted was insufficient to complete this task, so the results do not accurately reflect student knowledge. The scenario has been revised to correct this problem.



5

Learning Outcome:

Critical Thinking Evaluate arguments; separate fact and opinion; recognize points of view; understand implications and consequences, and acknowledge diverse values.

Type of Measurement

How Measured When

Measured


Who submits

data


Results

Results Compared

to Standard


Is this Outcome



Critical Thinking Score: MAPP test administered in a random sample of courses where average credits earned = 35. N = 168


th

Percentile = 110

111 Met Standard

Met Standard NO

2. Scenario Scenario-based Assessment: Administered in a random sample of courses where average credits earned = 35. N = 80. Graded with a 5-point Rubric.

Dec. 2009 IRE Office

IRE Office


2.95 Met Standard




IRE Office

IRE Office





IRE Office



Average score of 9 CCSSE Items (4d; 4n; 4r; 5b; 5c; 5d; 5e; 5f; 12e)** N=1206



2.7 Met Standard

* Courses that contain curriculum relevant to the specific General Education outcome. **See appendix



6

Learning Outcome:

Technology and Information Literacy Use printed materials, personal communication, observation, and technological applications to find, evaluate organize, and present information in order to achieve educational and professional success.

. Type of Measurement

How Measured When

Measured


Who submits

data


Results

Results Compared

to Standard

Summary Evaluation

Is this Outcome




IRE Office


3.68 Met Standard

Met Standard NO

2. Success Rate in general education courses



IRE Office

IRE Office





IRE Office



Average score of 4 CCSSE Items (4j; 4k; 9g; 12g)** N=1206



2.8 Met Standard




7

Learning Outcome:

Quantitative Reasoning Use college-level mathematical concepts and methods to understand, analyze, and explain issues in quantitative terms.

Type of Measurement

How Measured When

Measured


Who submits

data


Results

Results Compared

to Standard


Is this Outcome



Mathematics Score: MAPP test administered in a random sample of courses where average credits earned = 35. N = 168

Fall 2009 ETS ETS National norms provided by ETS 50th Percentile = 113

114 Met Standard

Met Standard YES



IRE Office


1.71 Below Standard




IRE Office

IRE Office





IRE Office



Average score of 1 CCSSE Item (12f)** N=1206



2.6 Met Standard

6. CLAS - College-Level Academic Skills - Math


State IRE Office

FL System = 97%

98% Met Standard

* Courses that contain curriculum relevant to the specific General Education outcome. **See appendix ***The same scenario was used to access both Communication and Quantitative Reasoning; the general education assessment committee determined that by combining the two outcomes into one scenario, students were not able to focus properly on either task. The results therefore do not accurately measure either outcome. Revisions have been made so that each outcome will have its own scenario in future implementations.



8

Learning Outcome:

Ethics Demonstrate a responsibility for personal, social, professional, educational and natural environments and make informed decisions based on those responsibilities.

Type of Measurement

How Measured When

Measured

How data are

collected

Who submits

data


Results Results


Summary Evaluation

Is this Outcome



Fall 2009 IRE Office IRE Office Students will achieve average score of 2.50 or higher

3.27 Met Standard

Met Standard NO




IRE Office IRE Office 80% 86% Met Standard



Dec. 2009 IRE Office IRE Office 70% 77% Met Standard


Average score of 1 CCSSE Item (12l)** N=1206


2.3 Met Standard




9

Learning Outcome:

Personal Development:

Develop an ability to understand and manage self, adapt to change, enhance wellness, learn effectively, establish a framework for aesthetic responsiveness, and set personal and professional goals.

Type of Measurement How Measured

When Measured


Who submits

data


Results Results


Summary Evaluation

Is this Outcome Focused for

Improvement?



IRE Office Students will achieve average score of 2.50 or higher

3.33 Met Standard

Met Standard NO




IRE Office

IRE Office 80% 86% Met Standard




IRE Office 70% 80% Met Standard


Average score of 7 CCSSE Items (4p; 10c; 12b; 12i; 12j; 12n; 12o)** N=1206


2.2 Met Standard




10

Part C: General Education Learning Outcomes Targeted for Improvement

Targeted Learning Outcome Results compared to

Standard

Improvement Target Use of Data for Improvement Date of Reassessment

Communication: Develop effective reading, writing, speaking, listening, and nonverbal communication skills for a variety of audiences.

1. Writing Symposium. Full day faculty development program on February 5, 2010. The day’s events included a student panel and three speakers who all addressed the issue of how to improve the writing skills of students.

2. Academic Development Day Keynote Address to all College faculty. Dr. Janet Zadina presented information on how to increase student learning by using the latest information from brain research.

3. Academic Development Day Breakout Sessions. Faculty were able to choose between the following sessions:

a. Motivating today’s College Students (focus on engagement and pedagogical techniques).

b. Hidden triggers for stress in the classroom (Increase learning by reducing stress)

c. Promote Active Learning by Warming up the Classroom Atmosphere (focus on increased learning through increased student. engagement).

4. Academic Development Day Cluster



11


Standard


Meetings. All clusters reviewed the general education assessment results and discussed what they meant in terms of their own specific disciplines. Each cluster was asked to submit what has been done (and what is being done this semester) to support the Communication learning outcome.)


1. Developed and implemented Math

Awareness Week. This consisted of

offering a variety of workshops,

presentations and games throughout

the College on subjects related to

mathematics. One of the purposes

of the event was to raise student

awareness as to the value of math

and how it is found throughout the

disciplines and life itself. Another

goal of the event was to reduce

math anxiety among students.

Lastly, another goal was to increase

student interest in taking math

classes.

2. Created Science Foundations Class

for students who are unprepared to

go directly into college-level science

courses. In addition to introducing

scientific concepts, this course is also

designed to strengthen student’s



12


Standard


quantitative reasoning skills. All

students who received a grade of

“D”, “F” or “W” in Biology and

Anatomy were contacted to let them

know about the Science Foundations

course and were encouraged to

enroll.

3. Three additional Math Prep faculty

have been credentialed to teach

MAT1033, the transition course from

Prep to Credit math. Faculty who

have experience with and

understand the mentality of the

typical prep student are better

equipped to help students transition

from dependent to independent

learners.

5. Academic Development Day Keynote Address to all College faculty. Dr. Janet Zadina presented information on how to increase student learning by developing new pathways in the brain.

6. Academic Development Day Breakout Sessions. Faculty were able to choose between the following sessions:

a. Motivating today’s College Students (focus on



13


Standard


engagement and pedagogical techniques).

b. Hidden triggers for stress in the classroom (Increase learning by reducing stress)

c. Promote Active Learning by Warming up the Classroom Atmosphere (focus on increased learning through increased student. engagement).

7. Academic Development Day Cluster

Meetings. All clusters reviewed the general education assessment results and discussed what they meant in terms of their own specific disciplines. Each cluster was asked to submit what has been done (and what is being done this semester) to support the Quantitative Reasoning learning outcome.)



14

GENERAL EDUCATION LEARNING OUTCOMES

5Φ DEVELOPMENT OF GENERAL EDUCATION LEARNING OUTCOMES

The General Education Committee drafted seven General Education learning outcomes in the spring of 2007. The outcomes were sent to all faculty with a request for comments

and suggestions. The final draft was reviewed by the Executive Leadership Council, approved and appeared in the College catalog for the academic year 07 – 08. The seven General

Education Learning Outcomes are:

Communications: Develop effective reading, writing, speaking, listening, and nonverbal communication skills for a variety of audiences.

Global Awareness: Exhibit a sense of community and be sensitive to cultural and global diversity; exercise civic responsibility; give service to others; and respect and care for our natural environment.

Critical Thinking: Evaluate arguments; separate fact and opinion; recognize points of view; understand implications and consequences, and acknowledge diverse values.

Technology and Information Literacy: Use printed materials, personal communication, observation, and technological applications to find, evaluate organize, and present information in order to achieve educational and professional success.


Ethics: Demonstrate a responsibility for personal, social, professional, educational and natural environments and make informed decisions based on those responsibilities.

Personal Development: Develop an ability to understand and manage self, adapt to change, enhance wellness, learn effectively, establish a framework for aesthetic responsiveness, and set personal and professional goals.

DEVELOPMENT OF GENERAL EDUCATION LEARNING OUTCOME ASSESSMENT METHODS

The current assessment plan for General Education outcomes was developed and implemented by the General Education Assessment Committee in the 2009 – 2010 academic year.

All General Education outcomes are assessed every year using a process that has three components.

The first component consists for problem-based assessment, called “Scenarios”. The scenarios were developed by the General Education Assessment Committee and continue to be

refined with each implementation. The scenarios require students to respond in writing to situations that address the College’s learning outcomes. The administration of the

Scenarios assessment involves the selection of a random sample of classes in which the mean number of credits completed by the students in that class exceeds 35. In this way, the

probability is maximized that the students responding to the scenarios have completed most or all of their general education requirements. During the fall, 2009 administration, 23

classes were sampled with 525 students contained in that sample. Following a workshop on holistic grading, the Assessment Committee graded all the scenarios with each scenario

being read by two committee members. In the event that the two readers disagreed on the scoring, a third reader was brought in to break the tie.

The second component of the assessment plan includes the administration of the ETS product, the Measure of Academic Proficiency and Progress (MAPP) exam. At the same time

the random sample was selected for the Scenarios, 13 additional classes utilizing the same criteria were selected to administer the MAPP exam. Over 200 students were included in

the sample for the MAPP exam. The MAPP exam tests three of the College’s general education learning outcomes, specifically, Communications, Critical Thinking and Quantitative

Reasoning.



15

The third component of the general education assessment process is comprised of a variety of institutional data that points to student success and student achievement of the

general education learning outcomes. They are:

PBSC administers annually a Graduating Student Survey which asks students to self-report on how much they feel their classes at the College contributed to the growth in their learning for each general education learning outcome.

The College Administered the Community College Survey of Student Engagement (CCSSE) which asks numerous questions related to the achievement of general education learning outcomes.

Each year the State of Florida provides information on how well PBSC students perform in their first year in the State University System. The performance of Palm Beach State students is compared to the Florida system with regard to grade point average.

The completer success rate in general education courses is examined each year. Specifically, the question that is asked is what percent of Palm Beach State students received a “C” or better for the general education courses that are relevant to each of the general education learning outcomes.

For both quantitative reasoning and communications, the State administers the CLAS exam to those students who have not been exempted through non-exam means.

GENERAL EDUCATION OUTCOME MEASURES TARGETED FOR IMPROVEMENT

During the spring 2010 semester, the General Education Assessment Committee analyzed the assessment data from all the various sources and determined that Palm Beach State

students demonstrated competency in all seven general education learning outcomes. Although the benchmarks were achieved for all of the learning outcomes, the committee

determined that there were two learning outcomes for which they would like to see student learning improved: Communications and Quantitative Reasoning.

PROCESS FOR DETERMINING THE DEVELOPMENT OF AN IMPROVEMENT PLAN

The General Education Assessment Committee discussed a large number of ways that the College might increase student success in communications and quantitative reasoning; the

Committee eventually agreed on a multi-faceted improvement plan based on potential impact and feasibility.

IMPLEMENTATION OF IMPROVEMENT PLAN

The strategies for improvement that were implemented included the following activities:

1) A Writing Symposium which was a day-long workshop designed to provide faculty with the most current information on how to help students improve their writing.

2) The College’s Academic Development Day included a keynote speaker as well as a variety of breakout sessions designed specifically to address issues of how to increase

student learning. This was approached through discussions of the brain and how students learn, how stress reduces the capacity to learn and what to do about it, and how

to increase motivation and engagement in the classroom.

3) Math Awareness Week, which is a week-long series of activities geared toward raising awareness about the importance of math and simultaneously reducing math anxiety,

was held throughout all four campuses of the College.



16

4) A Science Foundations course was created for those students who are not academically prepared to succeed in college-level science courses. One purpose of this course is

to strengthen quantitative reasoning skills among participants.

5) Additional Math Prep faculty were credentialed to teach Intermediate Algebra, the transition course between prep and credit math.

6) On Academic Development Day, all clusters were asked to brainstorm about how each cluster has and will incorporate skills relevant to communications and quantitative

reasoning to help strengthen and reinforce student learning in those two areas.

CURRICULUM AND PROGRAM REVISIONS

None



17

APPENDIX

CCSSE Questions:

4a: Asked questions in class or contributed to class discussions (1=never, 2=sometimes, 3=often, 4=very often) 4b: Made a class presentation (1=never, 2=sometimes, 3=often, 4=very often) 4c: Prepared two or more drafts of a paper or assignment before turning it in. (1=never, 2=sometimes, 3=often, 4=very often) 4d: Worked on a paper/project that required integrating ideas/information from various sources. (1=never, 2=sometimes, 3=often, 4=very often) 4f: Worked with other students on projects during class. (1=never, 2=sometimes, 3= often, 4=very often) 4g: Worked with classmates outside of class to prepare class assignments. (1=never, 2=sometimes, 3= often, 4=very often) 4h: Tutored or taught other students (paid/voluntary). (1=never, 2=sometimes, 3= often, 4=very often) 4i: Participated in a community-based project to work on an assignment. (1=never, 2=sometimes, 3= often, 4=very often) 4j: Used the Internet/instant messaging to work on an assignment. (1=never, 2=sometimes, 3=often, 4=very often). 4k: Used email to communicate with an instructor. (1=never, 2=sometimes, 3=often, 4=very often) 4n: Discussed ideas from your readings or classes with instructors outside of the class. (1=never, 2=sometimes, 3=often, 4=very often) 4r: Discussed ideas from your readings/classes with others outside of the class. (1=never, 2=sometimes, 3=often, 4=very often) 4s: Had serious conversations with students of a different race/ethnicity than your own. (1=never, 2=sometimes, 3= often, 4=very often) 4t: Had serious conversations with students who differ from you in terms of their religious beliefs, political opinions, or personal values. (1=never, 2=sometimes, 3= often, 4=very often) 4p: Worked harder than you thought you could to meet an instructor’s standards/expectations. (1=never, 2=sometimes, 3= often, 4=very often) 5b: Analyzing the basic elements of an idea, experience or theory. (1=very little, 2=some, 3=quite a bit, 4=very much) 5c: Synthesizing and organizing ideas, information, or experiences in new ways, (1=very little, 2=some, 3=quite a bit, 4=very much) 5d: Making judgments about the value/soundness of information, arguments or methods. (1=very little, 2=some, 3=quite a bit, 4=very much) 5e: Applying theories/concepts to practical problems or in new situations. (1=very little, 2=some, 3=quite a bit, 4=very much) 5f: Using information you read or heard to perform a new skill. (1=very little, 2=some, 3=quite a bit, 4=very much) 6a: Number of assigned textbooks, manuals, books or book-length packs of course readings (1=none, 2=between 1 and 4, 3= between 5 and 10, 4=between 11 and 20, 5=more than 20) 6b: Number of books read on your own (not assigned) for personal enjoyment or academic enrichment. (1=none, 2=between 1 and 4, 3= between 5 and 10, 4=between 11 and 20, 5=more than 20) 6c: Number of written papers or reports of any length. (1=none, 2=between 1 and 4, 3= between 5 and 10, 4=between 11 and 20, 5=more than 20) 9c: Encouraging contact among students from different economic, social and racial/ethnic backgrounds. (1=very little, 2=some, 3=quite a bit, 4=very much) 9g: Using computers in academic work. (1=very little, 2= some, 3=quite a bit, 4= very much) 10c: Participating in college-sponsored activities. (1=none, 2=between 1 and 4, 3= between 5 and 10, 4=between 11 and 20, 5=more than 20) 12a: Acquiring a broad general education. (1=very little,2=some, 3=quite a bit, 4=very much) 12b: Acquiring job/work-related knowledge and skills. (1=very little, 2=some, 3=quite a bit, 4=very much) 12c: Writing clearly and effectively. (1-very little, 2=some, 3=quite a bit, 4= very much) 12d: Speaking clearly and effectively. (1-very little, 2=some, 3=quite a bit, 4= very much)



18

12e: Thinking critically and analytically. (1=very little, 2=some, 3=quite a bit, 4=very much) 12f: Solving numerical problems. (1=very little, 2= some, 3=quite a bit, 4= very much) 12g: Using computing and information technology. (1=very little, 2= some, 3=quite a bit, 4= very much) 12h: Working effectively with others. (1=very little, 2=some, 3=quite a bit, 4=very much) 12i: Learning effectively on your own. (1=very little, 2=some, 3=quite a bit, 4=very much) 12j: Understanding yourself. (1=very little, 2=some, 3=quite a bit, 4=very much) 12k: Understanding people of other racial/ethnic backgrounds. .(1=very little, 2=some, 3=quite a bit, 4=very much) 12l: Developing a personal code of values and ethics. (1=very little, 2= some, 3=quite a bit, 4= very much) 12m: Contributing to the welfare of your community. (1=very little, 2=some, 3=quite a bit, 4=very much) 12n: Developing clearer career goals. (1=very little, 2=some, 3=quite a bit, 4=very much) 12o: Gaining information about career opportunities. (1=very little, 2=some, 3=quite a bit, 4=very much)



19

CHARTS



20



21

*Due to a small number of n, BAS was combined with “Other” Students.



22

general education learning outcomes assessment: … · 2011. 3. 14. · = 2.9 2.9 met standard met...

Documents