derek herrmann & ryan smith illinois state university university assessment services

IT TAKES TWO:A SYSTEMATIC, EFFECTIVE, & PRACTICAL PROCESS FOR INTEGRATING ASSESSMENT & PROGRAM REVIEW

DEREK HERRMANN & RYAN SMITHIllinois State University

University Assessment Services

Association for Institutional Research, New Orleans, LA, 6-5-2012

ABOUT ISU

• DRU• Total enrollment: 21,080• 35.5% of new students were

transfer students• 82.7% of degrees granted were

baccalaureate degrees

ABOUT ISU

• Retention rate: 85.1%• Graduation rate: 69.8%• Student to Faculty ratio: 19:1• Organizational structure

– UAS is part of Academic Affairs– IR is part of Finance and Planning

ABOUT ISU

• Illinois Board of Higher Education (IBHE)– All academic degree programs

required to have assessment plans on file

– All academic programs required to undergo a periodic and systematic program review

THE PROBLEM

1. Shifting views of higher education

“Higher education is one of the greatest hopes for intellectual and civic progress in this country. Yet, it is seen by many as part of the problem rather than the solution.”

- C. Palomba & T. Banta, Assessment Essentials, 1999

THE PROBLEM

2. Increasing Accountability

Higher Learning Commission

In the 11 pages of Criteria in the proposed pathways model, assessment is mentioned 19 times, with Teaching and Assessing for Learning as its own criteria.

IBHE

- C. Palomba & T. Banta, Assessment Essentials, 1999

THE PROBLEM3. Changing

World

THE PROBLEM

4. Assumptions about evaluation and assessment

“Assessment projects are simply getting in the way of good teaching… We spend far too much time and energy in trying to assess learning, an unattainable goal….I love teaching, however. It is a great pleasure to share my enthusiasm for (my discipline) and I absolutely love the chance to pick the minds of my bright students.”

- B. Moore, Washington State Board of Community & Technical Colleges, Using Assessment to Improve Instruction (video),

http://www.youtube.com/watch?v=BZ3USs16J3Y

THE PROBLEM

HOW COLLEGES ARE ORGANIZED

Board

President

VP

Dean

Director

Staff

VP

Dean

Director

Staff

VP

Dean

Director

Staff

HOW THEY ARE IN REALITY

Random People

Random People

Board

President

Union Leaders

Planning Person

Political Hire

Students

VP

Academic Dean

I.R. Person

Faculty

Provost

Assess. Dean

Director

Adjunct Faculty

BASIC ASSESSMENT MODEL

Establish Learning

Outcomes

Gather Evidence

Analysis

Make a Decision/Change

Mission/Goals

Accreditation, IBHE/ICCB, State of Illinois, Strategic plans, Tech. plans, enrollment mgmt. plans, facilities plans, financial plans, environmental scan data, IR data

Governance, Budgets, Resources, Boards, Politics, Org. culture, good and bad bosses, etc.

Education, training, & experience, life: health, spouses/partners, kids, family, friends, pets, etc.

Establish learning goal

Students will be able to explain the

impact of leadership on their

development

Provide learning opportunities

Mentorship program with

university leaders

Assess learning (gather evidence)Students keep a reflective journal

Coded and analyzed by leadership staff

Use the results (decision)

Students report a lack of contact with

mentors

Mission

Process is the outcome

Failure of decision-making or making it meaningful

FAILURE OF DECISION-MAKING

ADVANTAGES

THE PROBLEM

How does one design processes that…

- Engage faculty and staff?- Can be used for true improvement and

change in today’s society?- Make information meaningful for decision-

making?- Meets accountability standards?- Are genuine, not just transparent?

DISCLAIMER

AT ISU

• 8-Year cycle that includes– Process for the Review of Academic

Assessment Plans (PRAAP)– Self-study– Program Review– Annual Update

• Many across campus involved in these processes

AT ISU

• Year 1 – PRAAP– Chairs/program directors asked to

provide current assessment plans– Assessment Advisory Council

(AAC) reviews them using a rubric and completes a feedback form

THE PRAAP RUBRICUndeveloped Developing Established Exemplary

Program goals and intended student learning outcomes

Systematic assessment of student learning

Feedback from key stakeholders

Use of results/”Closing the Loop”

AT ISU

• Year 1 – PRAAP– Chairs asked to provide current

assessment plans– Assessment Advisory Council

(AAC) reviews them using a rubric and completes a feedback form

– UAS staff meets with chairs/program directors to discuss the feedback

AT ISU

• Year 2 – Self-Study– Faculty and staff complete self-

study for program review– Assessment plan is used to collect

data

AT ISU

• Year 3 – Program Review– Academic Planning Committee

examines Program Review documents

– Often programs submit follow-up reports based on assessment plans

AT ISU

• Years 4-8 – Annual Update– Chairs asked to complete a short

form concerning their assessment activities

– Questions based on Higher Learning Commission’s (HLC) Fundamental Questions for Conversations Student Learning

THE ANNUAL UPDATE FORM• Please provide a brief summary

of the previous year’s data which was collected related to your outcomes.

• What efforts were used during the past year to involve faculty and constituencies in assessment processes?

THE ANNUAL UPDATE FORM• Please outline what action plans

have been developed based upon the assessment process.

• Please provide any additional comments or information.

• Have changes or improvements been made to the Academic Assessment Plan in the previous year?

AT ISU

• Years 4-8 – Annual Update– If applicable, an updated

assessment plan is sent to UAS– Collection of forms provided to

programs to be used in the next cycle

ADVANTAGES

1. Assessment is a separate from program review, but still leads the process.

- A culture of inquiry over a culture of evidence.- One is about performance, the other about

learning.

ADVANTAGES

What does it mean?

Where do we go?

How well are we doing?

- Jane Chapman, Heartland Community College, It’s Eleven O’Clock: Do You Know Where Your Students Are?, 15th Annual Comm. College Assessment Fair, Bloomington, IL, 2011

A focus on what results mean, instead of what they are.

ADVANTAGES

2. Value people’s time, resources, energy, history, thoughts, and opinions.

- Attitudes, beliefs, and engagement with assessment are just as important as processes.

- Faculty have control through the self-study and review process.

- Assessment & program review aren’t used to micro-manage or fix a problem that is not perceived to exist.

- Ideas, people, and dialog lead – technology follows.

ADVANTAGES

ADVANTAGES

3. There is consensus and agreement about the process, and it is used to make decisions.

- It’s more important to be genuine, rather than transparent.

- Assessment and program review are never goals – they’re processes.

CONTINUOUS IMPROVEMENT

Nothing is ever perfect, but it can be healthy.

Incremental changes to rubric.

QUESTIONS?COMMENTS?

FOR MORE INFORMATIONhttp://assessment.illinoisstate.edu/program/resources.shtml

http://provost.illinoisstate.edu/academic/program-review.shtml