PREPARING FACULTY FOR PROGRAM AND LEARNING OUTCOMES ASSESSMENT

Terri Manning and Denise WellsCentral Piedmont Community College

Community College Planning and Research Organization

My Opinion (or bias) About Assessment

Learning outcomes assessment is a means to an end – not the end in itself.

It should exist within an overall evaluative process at colleges/universities where assessment is done for improvement of programs and services.

There is no such thing as a “one size fits all” model where we can all measure the same thing.

Institutions must have the freedom to determine their own outcomes based on their mission and the student body they serve.

Opinion, cont.

We must do a good job of establishing appropriate levels of effectiveness or government agencies will do it for us. We don’t need another IPEDS type of structure for learning outcomes.

The focus should be on outcomes, not just learning outcomes. All units and staff impact learning and student success therefore all units should define, measure and use outcome data to improve services and success.

The Problem with Continuous Meaningful

Assessment Institutions get the idea they are doing it for

their accrediting agencies which creates the wrong foundation

Faculty and staff don’t respond well to time-consuming processes being forced on them and being told “SACS says we have to do this.”

The accrediting agencies are requiring it because good institutions do assessment and act/react appropriately.

This is not an end into itself. It is part of a process leading to a much greater

end.

Why Continuation is Hard

Often it is driven from the top down – administration tries to tell the faculty what needs to be done.

Only have support in small pockets. It becomes one more thing we place on the

backs of the faculty. Keep adding more to their plates and taking

nothing away. My philosophy is “do as little assessment as

you can and get meaningful, useful results.” The “if 2 is good then 16 must be better”

approach must be tossed out.

The Trick: Getting Faculty to Commit to Assessment

Assessment tells you what is going on with your students

What works and what doesn’t work What skills are fully developed,

improving or absent Makes sure students have the critical

skills Makes sure students have the building

blocks from which to improve

Good Assessment

Is meaningful to faculty and students Should be authentic Should be imbedded in the classroom Becomes a critical tool for faculty Informs good teaching

What is a Learning Outcome? Learning Outcomes:

What changes in knowledge, skills, attitude, awareness, condition, position (etc.) occur as a result of the learning that takes place in the classroom. These are direct benefits to students.

Examples: general learning skills (e.g. improved writing and speaking abilities), ability to apply learning to the work environment (e.g. demonstrate skills in co-op), program-specific skills developed or enhanced (e.g. take blood pressure.)

What is a Program Outcome? Program Outcomes:

The benefits that results from the completion of an entire program or series of courses. Are there benefits for students who get the entire degree versus those who take a few courses? If so what are they?

Typical examples are: licensure pass rates, employment rates, acceptance into 4-year schools or graduate programs, lifelong learning issues, contributions to society, the profession, etc.)

What is Assessment?

Assessment – the ability to appraise or estimate the attributes of a person, group or program. Tools of assessment can include checklists, inventories, observations, needs assessments, rating scales and all types of tests.

Measurement – the process used to assign numerals to objects or constructs according to rules so that the numbers have quantitative meaning.

Tests – tools used in measurement of knowledge, attitude and behavior. Tests and testing are specific to a defined circumstances, a period of time or set of outcomes.

Direct and Indirect Measures

Direct assessment involves looking at actual samples of student work produced in our programs.

Indirect assessment is gathering information through means other than looking at actual samples of student work, such as surveys, exit interviews, and focus groups.

What is Student Learning Outcomes Assessment?

“The assessment of student learning can be defined as the systematic collection of information about student learning, using the time, knowledge, expertise, and resources available, in order to inform decisions about how to improve learning. (p.2)”

Source: Assessment Clear and Simple: A Practical Guide for Institutions, Departments and General Education by Barbara E. Walvoord, 2004

Resource

National Institute for Learning Outcomes Assessment

http://www.learningoutcomeassessment.org/

New Publication Opening Doors to Faculty Involvement in

Assessment Author: Pat Hutchings

Obstacles to Faculty Engagement

1. The language of assessment as been less than welcoming. The language (accounting, testing, evaluation, measurement, benchmarking, etc.) come from business and education – not necessarily respected disciplines on campus.

Obstacles, cont.

2. Faculty are not trained in assessment. More focus has been placed on scholarly expertise in one’s field in doctoral programs with less attention paid to reflecting on educational purposes, formulating learning goals, developing assessments and exams and using data for improvement. Not enough professional development training offered. When the conversation turns technical, faculty bow out not wanting to appear amateurs. When colleges establish an assessment office with specialized staff to manage it, it can marginalize regular faculty.

Obstacles, cont.

3. The work of assessment is not a match with internal reward systems (promotion, tenure and merit). It is more valued in an environment where teaching is the first and foremost activity. In universities where research is valued more than teaching, assessment is undervalued and not part of the promotion and tenure conversation.

Obstacles, cont.

4. Faculty have not seen evidence that assessment makes a difference. Based on the demands on faculty time and energy – faculty must make a choice – and the rational choice is often “no assessment.” Institutions have a responsibility to use assessment results to improve the educational experiences of their students. The simple presence of data does not lead to improved outcomes.

A few other issues….

In the early days the players in assessment were primarily outside of academe which made it look to faculty like “someone else’s agenda.” When faculty get the idea that this is not their job – you have a problem.

 Much of what has been done with assessment has not involved large numbers of faculty in a significant way.

Faculty need to be significant participants in the assessment process – not just token members of a committee pulled together for an accreditation visit or an after-the-fact audience for assessment results they had no part in shaping.

A few other issues……

Faculty must ask (both individually and collectively) about their students: What purposes and goals are most

important for students? Are these goals being met and what

methods can we use to determine this? How can we shape and share feedback that

can strengthen student learning?

Resource Assessment Clear and Simple: A

Practical Guide for Institutions, Departments, and Genera Education

Jossey-Bass Publishers Authors: Barbara Walvoord and Trudy

Banta $26.51 on Amazon.com

Assessment Characteristics

It is intended to: to inform action not to provide “proof” (too many variable) provide the best available indicators about the

attainment of department learning goals include a variety of tools and not require

standardized test or objective measures. provide evidence on which to base decisions

about curriculum, pedagogy, staffing, advising and student support on the best possible data about student learning and the factors that affect it.

Assessment Characteristics

Faculty regularly: assess complex work in their fields

make informed professional judgments about critical thinking, scientific reasoning, or other qualities in student work

use those judgments to inform departmental and institutional decisions (p. 2)

Source: Assessment Clear and Simple: A Practical Guide for Institutions, Departments and General Education by Barbara E. Walvoord, 2004

Assessment Characteristics A lot of assessment is already going on

in responsible classrooms, departments, and institutions, though we have not always called it that.

We are asking faculty to take something they casually do and make it formal.

Assessment can move beyond the classroom to become program assessment

Assessment Characteristics

Classroom assessment – faculty evaluates her own students’ assignments in the capstone course and uses the information to improve her own teaching the next semester

Program assessment – faculty evaluates her own students’ assignments in the capstone course, outlining the strengths and weaknesses of the students’ work in relationship to departmental learning goals. The department uses the data to inform decisions about curriculum and other factors that affect student learning.

Source: Assessment Clear and Simple: A Practical Guide for Institutions, Departments and General Education by Barbara E. Walvoord, 2004

Methods of Assessment

Course imbedded assessments for learning outcomes which can be (direct): Written works Student journals Speeches Skills-based assessment (demonstrated skills) Observation checklists Tests for specific skills Teamwork assessments

Methods of Assessment

Indirect Assessment Methods Surveys that ask about specific behaviors

indicative of changes in values and attitudes Not self-evaluation of perceptions of learning Surveys that ask about effectiveness and

efficiency Qualitative methods Focus groups, key informant interviews, open ended

surveys, journaling Portfolio analysis

Authentic Assessment

A form of assessment in which students are asked to perform real-world tasks that demonstrate meaningful application of essential knowledge and skills. An authentic assessment usually includes a task for students to perform and a rubric by which their performance on the task will be evaluated.www.oaklandcc.edu/assessment/terminology.htm

Authentic Assessment Examples

Authentic assessment tends to focus on complex tasks, enabling students to demonstrate their competency in a more 'authentic' setting. Examples of authentic assessments include: performance of the skills, or demonstrating

use of a particular knowledge simulations and role plays studio portfolios, strategically selecting items exhibitions and displays

Toughest Concept

Difference between grading and assessing learning outcomes.

Course Grades are a measure of the percentage of all the requirements for a course that a student completes.

Course requirements often include things like attendance, class participation, turning work in on time, following the rules on an assignment, etc.

Grades vs. Outcome Assessment

Grades vs. Outcome Assessment

It is possible for a student to make an A and have learned little or nothing. It is possible for a student to earn an F and have learned more than anyone in the class.

Grades are - "An inadequate report of an inaccurate judgment by a biased and variable judge of the extent to which a student has attained an undefined level of mastery of an unknown proportion of an indefinite material.“ Ohmer Milton, Howard Pollio and James Eison, Making Sense of College Grades, (Jossey-Bass, 1986).

Measurements of Learning Outcomes

Poor example Student writing skills will improve

70% of students will pass the midterm - or 70% of students will pass the final – or 70% of students will make A grade of C or better in ENG 111

Good example Students will demonstrate the ability to locate,

critically evaluate, and present information 70% of student persuasive speeches evaluated will receive at

least a score of 3 or better on a 5-point evaluation rubric designed to measure effective speaking along a continuum on six construct scales.

Methods of Assessment

Surveys that ask about specific behaviors indicative of changes in values and attitudes Not self-evaluation of perceptions of learning

Surveys that ask about effectiveness and efficiency

Qualitative methods Focus groups, key informant interviews, open ended

surveys, journaling Portfolio analysis

Assessments of Behavior Changes

In what behaviors do we observe changes in students? Teamwork Cultural awareness Fear of pubic speaking Ethic issues Math phobias

How might we measure these?

Assessment

Is a developmental, iterative process Like testing swimming pool water

A Seven Step Process

What We Do With Our Faculty

First - Orientation

Instructional administration attends to show support

Review process Review of timelines Preview of program review Introduce players and their skills Have added marketing and recruitment

Second: Provide Training

Nuts and Bolts of Outcome Assessment Defines types of outcomes Establishes reasons for assessment Identifies sources for outcomes Give examples

How to Read Your Data Faculty need to see their student success data (grades, retention,

graduation, etc.) but they are not necessarily good with numbers The assessment just establishes that

students are mastering the learning and program outcomes so they can move forward to the items above

Seven Step Assessment Process

Lots of hand-holding and assistance1. Discovery - During this phase, faculty members are involved in

a working session where they are trained on learning outcomes assessment. They move into an activity on identifying “reasons for being, what we want them to get out of it and overall benefits for students.” Outcome Identification Form

2. Prioritization - During this phase they begin the narrowing of a long list of potential outcomes to the critical ones. This process involves them working through 1) what would be useful for faculty and departments to know, 2) what the faculty/department values, 3) what would inform decision-making and 4) what is most critical for student success. Outcome Prioritization Form

Seven Steps (cont.)

3. Operationalization - During this phase they create operational definitions of each outcome based on the curriculum course or program (e.g. effective writing is defined differently for English faculty than it is for early childhood faculty). Outcome Statements Form

4. Mapping - During this phase, faculty look at the natural fit between 1) courses and outcomes and 2) assessments/projects and outcomes. They create assessment processes where good ones are lacking. Outcome Assessment Form

Seven Steps (cont.)

5. Measurement - This phase involves setting realistic outcome targets, developing/using assessment tools, the process of distribution and collection of data and the reporting chain of command.

6. Analysis and Use - This phase involves analysis among appropriate faculty, determining a plan of action and writing up results.

7. Follow-up one year later on action plans (strongly recommended.)

What’s Needed to Get the Job Done?

Training Time Saving Short-cuts Templates and processes Website support and

assistance Survey/assessment

assistance Coaching and mentoring

from IE Getting faculty buy-in

Bring Faculty On-board

Pat Hutchings suggests six ways to bring the purposes of assessment and the regular work of faculty closer together in the publication: Opening Doors to Faculty Involvement in Assessment

1. Build assessment around the regular, ongoing work of teaching and learning

Build on the process of grading in every course every semester. This can bring forth questions about course design, assignments and exams and feedback to students. This is where faculty talents and interests lie. Embedding assessment in the classroom sets the stage for discussions at the program level and draws on what faculty care most about: their discipline or field. When assessment reflects and respects disciplinary interest, it is more likely to lead to important faculty engagement.

 2. Integrate assessment into the preparation of graduate students;

Finding this is the exception rather than the rule but it is occurring more often.

3. Reframe assessment as scholarship

Student learning should be seen as an important phenomena for investigators. The work should not be seen as service to the college (serving on the assessment committee) but as an important intellectual exercise. Attention should be paid to the use of new forms formulas, and genres for capturing the scholarly work of teaching, learning and assessment.

4. Make a place for assessment in faculty

development  Many institutions of higher education

have created centers for teaching and learning and assessment can be part of these centers. Bringing faculty together and facilitating constructive conversations around assessments, meaning, implications for pedagogy and a commitment to evidence can increase faculty engagement.

5. Create campus spaces & occasions for constructive conversations and

action

Teaching and learning have been traditionally seen and undertaken as private activities. Faculty assume responsibility for their own students and classrooms. It is foreign for them to come together in groups and discuss student learning, difficulties, strategies and possible actions as groups. Institutions have done various things from faculty learning communities, setting aside time in department meetings, multidisciplinary reading groups and inquiry groups to foster these conversations.

6. Involve students in assessment.

Student self-assessment where they monitor and direct their own development is gaining groups. New products exist such as e-portfolios, rubrics to guide student work and involve students in campus conversations about learning and assessment.

Working through the process…

Mentoring Reviewing data, reviewing what research is,

inputting and formatting data Calendars

Reviewing deadlines Putting Learning and Program outcomes

into action Using data for results

How Do We Use What We Learn

You have spent time creating a list of agreed-upon outcomes

You have created an assessment tool You have created a grading process and

established a benchmark for “success” You use the assessment tool for multiple

sections, score it and pull together the results

Your students do not pass at the level you expected

Why didn’t the pass? What do you do?

Use of Results

What processes are in place to make sure you use results to improve programs, services and student learning.

Just because you have data showing that students are having difficulty doesn’t mean you know what to do to fix it.

Can you measure, report and forget it? Then why do it?

Using Results

If faculty can easily figure it out – make changes and move on

Do some research Methods that have been proven to work by

discipline – to improve retention, engage students, help explain content, create authentic assignments, improve content mastery, etc.

May have to make some curricular changes May need more info – to drill down further

Questions We Should Ask Ourselves

1. We’re spending time and resources trying to achieve student learning – is it working?

2. When we claim to be graduating students with qualifications, do they actually have them?

3. We think they have problems with certain concepts or are weak in certain areas, do we have data to back that up?

4. When we identify weaknesses, how can we best address the problem?

5. What are our most cost effective methods of improving learning?

Things to Remember

Outcome measurement must be initiated from the unit/department level (promotes ownership of process).

Measure only what you are teaching or facilitating.

Measure what is “important” to you or your program.

Prioritize your outcomes. Be selective (2-3 outcomes only for a course

or programs). Use what you find.

Support and Contact Info:

http://www.cpcc.edu/planning

http://www.cpcc.edu/IE Presentation Posted in “studies

and reports” Terri Manning terri.manning@cpcc.edu (704) 330-6592