using program review to facilitate change mississippi association for institutional research march...
TRANSCRIPT
Using Program Review to Facilitate Change
Mississippi Association for Institutional Research March 31, 2005
Terri M. Manning, EdDCentral Piedmont Community College
Some history….
CPCC is the largest community college in both the Carolinas with approximately 60,000 students.
We have over 100 instructional program areas including for-credit, continuing education and literacy
We have almost 50 administrative and student affairs units
Some history….
When we started this process (1998), the College had done nothing truly “IE related” since the last SACS visit in 1992 when they received a recommendation for every must statement in section III.
The IR office staff had rolled over and all institutional memory was gone.
We had less than 5 years to put every program and unit through a review process.
Under the New Core Requirements and Comprehensive Standards Core Requirement 2.5 states “The institution engages
in ongoing, integrated, and institution-wide research-based planning and evaluation processes that incorporate a systematic review of programs and services that (a) results in continuing improvement, and (b) demonstrates that the institution is effectively accomplishing its mission.”
Comprehensive Standard 3.3.1 states “The institution identifies expected outcomes for its educational programs and its administrative and educational support services; assesses whether it achieves these outcomes; and provides evidence of improvement based on analysis of those results.”
IE at Central Piedmont
CPCC established an IE committee in 1998 and an IE website in 2000 (http://inside.cpcc.edu/IE)
Committee members were from various instructional departments, student services and administrative/business offices
The committee established an IE plan for the college
1: InitiateEnvironmental
Scan
Quality Improvement
5: Assess Outcomes& Measure Goal Achievement
8 Step Institutional Effectiveness Cycle
6: Utilize Resultsto Plan Improvements
2: Establish CollegePriorities & Goals
3: Allocate &Integrate Budget
4: Deliver Programs & Services
Conduct Unit Reviews
Conduct Teaching EvaluationsGeneral Education Assessment
College Surveys
7: Reallocate Budget
8. Make Changes
Visual of the IE Plan
Four Major Overlapping Pieces of the IE Plan
Program/Unit Review
College Assessment
Process
Annual GoalSetting Cycle
General Education
Assessment
Annual goal setting and follow-up required by all units
Program/unit review on a 3-5 year cycle
College assessment plan involving surveys, assessment and data analysis
The evaluation of general education
Developing the Process
We wanted to create a meaningful program/unit review process
We wanted programs to complete the process having learned something valuable (not a document to set on a shelf)
We wanted the process to be “outcome-based” to stand the test of time
Developing the Process
The first review process was created for instructional programs (main focus at most institutions)
During the 1998-99 year, 18 programs were reviewed (approximately 100 programs over five years)
The perception at the beginning was that “this is just another academic exercise”
But the results were very different than any review process previously completed
Organizational Stages of Outcome Evaluation
Disbelief & Denial
Paralysis - Passive resistance
Anger and antagonismResistant & Reactive
Stage 1
Stage 2
Stage 4
Depression - exhaustionCompliance - Passive reactive
Acceptance & adaptationChallenge & competitionCatalyst - Proactive
Stage 5
Stage 3Bargaining - no time/no moneySeek outside sources
Instructional Program Review
The review process contained five sections: I. The Program Profile II. Program Content III. Student Learning Outcomes IV. Need for Change V. Future Issues
Instructional Program Review
Tasks accomplished through review: Defined their program Detailed faculty and staff (credentials, prof. dev,
accomplishments) Unit mission or purpose Link unit mission to the college mission Defined the content of their program Defined the population served Determined student learning outcomes specific to their program Set outcome objectives Performed some means of assessment Analyzed results Determined strengths and weaknesses Created strategies for improvement Determined future needs (including curricular change, needed
resources, staff and space) All units had to complete a one-year follow-up
Identifying Outcomes
We focused on two types of measures:
Instructional Outcomes Program Outcomes Student Learning Outcomes
Administrative Outcomes (outcomes for administrative programs)
Administrative Outcomes
Many units do not directly serve students or they want results within their units that are not truly student outcomes.
They want to improve services or approach an old problem in a new way.
They want to become more efficient and effective.
They will set administrative outcomes.
My Administrative Outcome Objectives
1. 80% of faculty/staff responding to the faculty/staff survey will perceive that Planning and Research responds quickly to their requests for data.
2. 80% of faculty/staff responding to the survey will perceive that Planning and Research makes a significant contribution to the College.
3. 80% of faculty/staff responding to the survey will perceive that Planning and Research contributes to the effectiveness of CPCC.
4. 80% of faculty/staff responding to the survey will indicate that Planning and Research produces enough reports to meet the planning and information needs of faculty and staff.
Administrative Outcomes for Instruction Objectives set for the program (has nothing to
do with students outcomes) Example
to recruit one new faculty member to seek and gain accreditation to increase retention by 10% to send each faculty member to at least once
professional conference per year to gain funding for an innovative program through
a grant proposal
Outcomes
"Outcomes" are benefits for people: changes in knowledge, values, position, skills, behavior or status. More simply stated, outcomes are typically what service providers hope recipients achieve once they complete a program or receive services. This is not the “what” but the “why” of education.
Types of Outcomes
Student learning outcomes are outcomes related to the learning that takes place in the classroom. We measure improvements in writing, speaking, understanding the scientific method, etc.
Program outcomes are the benefits to a student who receives an associate degree in Nursing or completes a certificate in Network Administration? Typical outcomes might be licensure exam scores, job placement, etc.
Outcome objectives are just objectives that relate to the identified outcomes.
Program Outcome Model
Resources Services Products or Results of
Activities
Staff Education (classes) Numbers servedBuildings Services FTE (input next year)Facilities Counseling Classes taughtState funds Student activities Students recruitedFTE
ConstraintsLawsState regulations
INPUTS ACTIVITIES OUTPUTS
Program Outcomes Model
> > >
Benefits for People
*New knowledge *Increased skills
*Changes in values *Modified behavior *Improved condition *Altered status
INPUTSINPUTS ACTIVITIES OUTPUTS OUTCOMES
Instructional Program Review
Rules for the process included: Involvement on the part of all faculty in the
department under review (not just one person). It is recommended that the program begin with a brief faculty retreat to discuss and divide tasks.
All programs must use at least one external committee (advisory groups are fine) to provide feedback to programs.
All programs must utilize feedback from students.
During the first few years…
Planning and Research conducted training on “how to conduct program review” with each unit scheduled to go up for review that year.
A staff member from Planning and Research served as a liaison to each program being reviewed to help them with the process.
What We Learned….
The first group was dragged kicking and screaming through the process
Faculty had very little time for the details Faculty had trouble identifying “student
learning outcomes” The results produced great marketing
materials Once they saw the results, faculty embraced
the process
What Happened
Deans used the results to make a case for resources
The administration became interested in what was learned through the review process
Instruction created a position to deal mainly with program review and IE issues within instruction
Faculty knew their programs were working
IE Committee Decision
The instructional program review process was successful
The review of instructional units was important – but administrative/student services units helped create an environment conducive to learning at the college and supported the learning process
Administrative units should go through a similar process
IE Committee Decision, cont.
Create a committee to draft a similar process for the administrative and students services areas of the college
Administrative units were spread across three VP areas
Two representative from each VP area participated in drafting the new review process
We spent approximately six months creating a workable process
The Administrative Unit Review Design
I. The Unit/Program ProfileA. The Mission/Purpose
1. Role unit plays in the college mission2. Unit/program goals as they relate to the
college’s mission B. The Staff
1. Professional and administrative staff (b-e since the last review)
a. Position description/dutiesb. Credentials (full and part-time, if any)c. Accomplishments (if applicable)d. Service to college, community and
nation
The Administrative Unit Review (continued)
B. The Staff (continued)
e. Professional development activities
2. Classified Staff
a. List of names and positions
b. List of required credentials (if any)
C. The Customer/Client Served
1. Breakdown of students/faculty or staff by type or demographic information (thorough explanation of who is served)
The Administrative Unit Review (continued)
II. Definition of Services or ProgramA. Definition of day-to-day duties of the unit B. Innovations, new projects, new
initiatives, local, state-wide or national effortsC. Required functions of unit (description
and status of compliance)1. SACS requirements2. State mandates3. Federal mandates4. Other
The Administrative Unit Review (continued)
III.Administrative Objectives and Student Outcomes (where appropriate)
A. Administrative objectives (2-3 objectives)B. Outcomes (or status if incomplete) of
innovations, new projects, new initiatives, local, state or national efforts
C. Assessment explanation (what was assessed, who, when, how many)
D. Results of administrative objectives based on assessment
Assessment of Administrative Units
During the year of program review, the “Annual Faculty/Staff Survey” contained questions from those units being reviewed.
Results were given to each unit and broken out by campus and job type
http://www.cpcc.edu/planningClick on “survey results”
The Administrative Unit Review (continued)
IV. Need for Change A. Strengths identified by external sources,
faculty, staff and studentsB. Weaknesses identified by external
sources, faculty, staff and studentsC. Recommendations by faculty, staff,
external sources and students to improve the unit's services and programs
D. Strategies for change (based on input from A, B & C above) - closing the loop
E. A one-year follow-up brief report to the Unit VP reporting on the progress of D above (due April 15, the year following review)
The Administrative Unit Review (continued)
V. Future Issues - Resources needed for future effortsA. Market trends within the broad service unit or
program area (based on best-practices, the literature or training received)
B. Anticipated future changes and needs (based on market trends)
C. Resources, equipment, space, staffing and work load changes needs for future
growth or continuationD. Future plans of unit
Assistance from the Website
The IE website: http://inside.cpcc.edu/IE
Explanation of IE process Templates and forms for review “Solid” examples for clarification Schedule for review
Unit Review – Lessons Learned
To do this well you need several critical pieces:1. Support from the top (President or Chancellor,
Vice Presidents or Vice Chancellors)
2. Buy in from the grassroots level (participation in the development of the process)
3. Across-the-college participation (no one is exempt)
4. Technology to make it easy (web page and review templates)
5. Technical support from institutional research
Administrative Response
Units spend a lot of time working on the program review.
If we want them to take it seriously, we have to take it seriously.
Once reviews are reported at the end of the year, their Dean or Vice President/Chancellor need to: 1. Read them 2. Respond to them
Appropriate Administrative Response
Sit down with the unit (group meeting) Give them your attention Discuss what was learned Discuss their major issues Discuss what they need to make
improvements Actually attempt to direct resources to them to
make those improvements Then they do not see it as an academic
exercise
Overall Benefits
#1 No recommendations in the area of Institutional Effectiveness from SACS during the October 2002 re-accreditation visit
The college became change-oriented Units had to define strategies for change Didn’t have to be perfect but rather making continuous
progress Strategies for change helped identify needed resources for
units Units had to “close the loop” with the one-year follow-up
(couldn’t promise and not deliver) Once they started, we couldn’t get them to stop doing it.
Overall Benefits
Units became empowered to perform their functions in an optimal manner and to ask for what they needed (no one noticed them before, now they do) Created data to support needs Accomplishments were reported to major
administrative groups across the college (President’s Cabinet, Planning Council, IE Committee, etc.)
Overall Benefits
The college community understood the purpose and function of every unit
Senior administration realized the benefits and became strong supporters Data are reported annually from program/unit
review VPs became stronger advocates for their units
making changes to improve services
Administrative Units Reviewed
Security Health and Safety OfficeHuman Resources Professional DevelopmentResource Development The FoundationEEO Office President’s Office Facilities Management Distribution Services Facilities Design/Construction Planning and ResearchFinancial Services PayrollBudgeting Financial Reporting Purchasing Accounts PayableCashiering Audit and ComplianceBasic Skills Reporting Inventory ControlBookstore Campus Printing Vending Information TechnologyAdmin. Computing Services Marketing and Community
Relations
Students Affairs Units Reviewed
Counseling and Advising Financial Aid/Veteran’s Affairs Career Services Disability Services “Campus-based” student services (6 campuses) Admissions and Records Registration Services The Library The Academic Learning Center (tutoring) Testing Center Graduation Office Student Life
Five Units
Some Brief Results
An Instructional Program
Workplace Basic Skills This program is a literacy initiative that goes
directly into the worksite and teaches ESL classes, GED prep and GED classes.
During their review, they surveyed both employers and students.
This was the first time they had ever done this.
What They Learned
Employers said: 43.8% of employers reported increases in
employee performance as a result of participation in the program.
31.3% reported a reduction in absenteeism by participants.
87.5% said classes improved the morale of their employees
37.5% said participants received raises 50% said communication had improved.
What Students Said
70.2% reported being able to fill out job forms better 35.5% said they could now help their children with
their homework 91.1% said they felt better about themselves 44.4% said they had received a raise, promotion or
opportunity as a result of the courses 86.3% said their ability to communicate in the
workplace had improved
What Has Happened Since
Their assessment data has shown up in their marketing brochures to employers.
Their enrollment has grown dramatically. They have received funding and marketing
support from Charlotte Reads (considered a model adult literacy program).
Planning and Research
Weaknesses Identified: Data/information needs to be disseminated to the grass roots
level of the College (not just administration) Data needs to be made more accessible and easier to
understand Strategy for Change:
Make improvements in the department’s webpage Place more information on the webpage
Result: Units are using data to make decisions Planning and Research has a heavily used website with user-
friendly spreadsheets, tables and an online Fact Book. In-person requests for data have dramatically declined because faculty/staff are using the webpage
Professional Development
Weaknesses Identified: Instructors could not attend training sessions due to the
times they were scheduled (top of the hour when classes were scheduled on the half hour)
Strategy for Change: Starting times for all training sessions will be adjusted
to begin on the half hour rather than the hour Results:
More faculty were able to attend trainings and numbers increased in some areas
The Bookstore
Weaknesses Identified: Communication between faculty and staff and the
bookstore management needs improvement Strategies for Change:
A bookstore advisory committee has been formed. Action items identified will be followed up to insure that the bookstore is meeting the needs of faculty/staff
Results: Decided to outsource the bookstore Worked with faculty to ensure an adequate number of
books were available at the beginning of the term Created a process to move books from campus to campus
to accommodate student needs Created more buy-back centers on the campuses to
improve services
Information Technology Services
Weaknesses Identified: Better follow-up is needed on Help Desk requests Requests need to be answered faster Have knowledgeable people at the Help Desk
Strategies for Change: Employ students with pertinent certification to man the Help
Desk Empower Help Desk staff with decision-making ability that
helps provide the required level of service Results:
Help Desk created a tracking system that has radically improved time from request to completion
Major Benefits for IR Office
We created work for ourselves with great payoff. Each unit being reviewed had a liaison from IR to
help them with surveys, objectives, etc. Now, everyone knows us. They think we are very helpful and a great asset to
the institution. They call us for information and then listen to what
we say.
“Best” Results of “Best” Practice
Better use of data across the college We have become more student/customer focused Review gives direction for goals and needed changes Departments are empowered to do their jobs (can’t slip
through the cracks and be unnoticed) Problems must be resolved (there is no hiding and no
excuses) Surveys provide needs assessment data as well as
evaluation which gives departments direction
For a copy of this presentation
Contact Terri Manning [email protected] Download or print presentation: http://www.cpcc.edu/planning
Click on “studies and reports” Listed as MAIR Program Review Presentation