using program/unit review to facilitate change sacs annual meeting december 6, 2004 terri m....
TRANSCRIPT
Using Program/Unit Review to Facilitate Change
SACS Annual MeetingDecember 6, 2004
Terri M. Manning, EdDKathy Drumm, DBA, CPA
Central Piedmont Community CollegeCharlotte, North Carolina
Some history….
CPCC is the largest community college in both the Carolinas with approximately 60,000 enrolled students.
We have over 100 instructional program areas including for-credit, continuing education and literacy
We have almost 50 administrative and student affairs units
Some history….
When we started this process (1998), the College had done nothing truly “IE related” since the last SACS visit in 1992 when they received a recommendation for most must statements in section III.
The IR office staff had rolled over and all institutional memory was gone.
We had less than 5 years to put every program and unit through a review process.
Why this Process?
The evaluation of teaching and educational effectiveness is a top priority for most institutions
However, the evaluation of business, administrative and student services units is often undervalued and overlooked
Evaluating the effectiveness of services (not just customer satisfaction) makes good business sense
Institutional Effectiveness and the Southern Association of College and Schools
At the heart of SACS’ philosophy is the concept of institutional effectiveness (IE)
IE involves a process of planning, evaluation, assessment and use of results
Under the new core requirements and comprehensive standards, there are two “mandates” that specifically address program or unit reviews
Under the New Core Requirements and Comprehensive Standards
Core Requirement 2.5 states “The institution engages in ongoing, integrated, and institution-wide research-based planning and evaluation processes that incorporate a systematic review of programs and services that (a) results in continuing improvement, and (b) demonstrates that the institution is effectively accomplishing its mission.”Comprehensive Standard 3.3.1 States “The institution identifies expected outcomes for its educational programs and its administrative and educational support services; assesses whether it achieves these outcomes; and provides evidence of improvement based on analysis of those results.”
IE at Central Piedmont
CPCC established an IE committee in 1998 and an IE website in 2000 (http://inside.cpcc.edu/IE) (this website gets an average of 61 hits a day – most outside the college)
Committee members were from various instructional departments, student services and administrative/business offices
The committee established an IE plan for the college
1: InitiateEnvironmental
Scan
Quality Improvement
5: Assess Outcomes& Measure Goal Achievement
8 Step Institutional Effectiveness Cycle
6: Utilize Resultsto Plan Improvements
2: Establish CollegePriorities & Goals
3: Allocate &Integrate Budget
4: Deliver Programs & Services
Conduct Unit Reviews
Conduct Teaching EvaluationsGeneral Education Assessment
College Surveys
7: Reallocate Budget
8. Make Changes
Visual of the IE Plan
Four Major Overlapping Pieces of the IE Plan
Program/Unit Review
College Assessment
Process
Annual GoalSetting Cycle
General Education
Assessment
Annual goal setting and follow-up required by all units (Strategic Plan)
Program/unit review on a 3-5 year cycle
College assessment plan involving surveys, assessment and data analysis
The evaluation of general education
Developing the Process
We wanted to create a meaningful program/unit review process
We wanted programs to complete the process having learned something valuable (not a document to set on a shelf)
We wanted the process to be “outcome-based” to stand the test of time
Developing the Process
The first review process was created for instructional programs
During the 1998-99 year, 18 programs were reviewed (approximately 100 programs over five years)
The perception at the beginning was that “this is just another academic exercise”
But the results were very different than any review process previously completed
Organizational Stages of Outcome Evaluation
Disbelief & Denial
Paralysis - Passive resistance
Anger and antagonismResistant & Reactive
Stage 1
Stage 2
Stage 4
Depression - exhaustionCompliance - Passive reactive
Acceptance & adaptationChallenge & competitionCatalyst - Proactive
Stage 5
Stage 3Bargaining - no time/no moneySeek outside sources
Instructional Program Review
The review process contained five sections: I. The Program Profile II. Program Content III. Student Learning Outcomes IV. Need for Change V. Future Issues
Instructional Program Review
Tasks accomplished through review: Defined their program Detailed faculty and staff (credentials, prof. dev,
accomplishments) Unit mission or purpose Link unit mission to the college mission Defined the content of their program Defined the population served Set outcome objectives Performed some means of assessment Analyzed results Determined strengths and weaknesses Created strategies for improvement Determined future needs (including curricular change, needed
resources, staff and space)
Instructional Program Review
Rules for the process included: Involvement on the part of all faculty in
the department under review (not just one person). It is recommended that the program begin with a brief faculty retreat to discuss and divide tasks.
All programs must use at least one external committee (advisory groups are fine) to provide feedback to programs.
All programs must utilize feedback from students.
What We Learned….
The first group was dragged kicking and screaming through the process
Faculty had very little time for the details
Faculty had trouble identifying “student learning outcomes”
The results produced great marketing materials
Once they saw the results, faculty embraced the process
Identifying Outcomes
We focused on two types of measures:
Outcomes Program Outcomes Student Learning Outcomes
Administrative Objectives
Administrative Objectives Many units do not directly serve
students or they want results within their units that are not truly outcomes.
They want to improve services or approach an old problem in a new way.
They want to become more efficient and effective.
They will set administrative objectives.
My Administrative Objectives
1. 80% of faculty/staff responding to the faculty/staff survey will perceive that Planning and Research responds quickly to their requests for data.
2. 80% of faculty/staff responding to the survey will perceive that Planning and Research makes a significant contribution to the College.
3. 80% of faculty/staff responding to the survey will perceive that Planning and Research contributes to the effectiveness of CPCC.
4. 80% of faculty/staff responding to the survey will indicate that Planning and Research produces enough reports to meet the planning and information needs of faculty and staff.
Outcomes
"Outcomes" are benefits for people: changes in knowledge, values, position, skills, behavior or status. More simply stated, outcomes are typically what service providers hope recipients achieve once they complete a program or receive services. This is not the “what” but the “why” of education.
Student learning outcomes are outcomes related to the learning that takes place in the classroom. We measure improvements in writing, speaking, understanding the scientific method, etc.
Program outcomes are the benefits to a student who receives an associate degree in Nursing or completes a certificate in Network Administration? Typical outcomes might be licensure exam scores, job placement, etc.
Outcome objectives are just objectives that relate to the identified outcomes.
Program Outcome Model
Resources Services Products or Results of
Activities
Staff Education (classes) Numbers servedBuildings Services FTE (input next year)Facilities Counseling # Classes taughtState funds Student activities # Students recruitedFTE # Who participated
ConstraintsLawsState regulations
INPUTS ACTIVITIES OUTPUTS
Program Outcomes Model
> > >
Benefits for People
*New knowledge *Increased skills
*Changes in values *Modified behavior *Improved condition *Altered status *New opportunities
INPUTSINPUTS ACTIVITIES OUTPUTS OUTCOMES
We Had to do Training on How to Set Objectives
There’s no magic number e.g. 80% or 90% What is reasonable? What can you afford? What realistically can your staff
accomplish? May need to benchmark (e.g. enrollment
growth) What percent shows you’re not committed
and what percent shows you’re naïve?
How to Set Objectives
Examples: Fifty percent of students will be able to
communicate effectively in writing (complete the writing exam with a grade of 60 [D] or better)
By the end of the spring term, 95% of faculty and staff will have completed 20 contact hours of professional development (workshops, college courses, conferences, onsite trainings, etc.)
More Realistic
Seventy percent of students will be able to communicate effectively in writing (complete the writing exam with a grade of 75 [C+] or better)
By the end of the spring term, the professional development office will increase their offerings for faculty and staff by 10% over what was offered last year (workshops, college courses, conferences, onsite trainings, etc.)
What Happened
Deans used the results to make a case for resources
The administration became interested in what was learned through the review process
Instruction created a position to deal mainly with program review and IE issues within instruction
Faculty knew their programs were working
Unit Review – Lessons Learned
To do this well you need several critical pieces:
1. Support from the top (President or Chancellor, Vice Presidents or Vice Chancellors)
2. Buy in from the grassroots level (participation in the development of the process)
3. Across-the-college participation (no one is exempt)
4. Technology to make it easy (web page and review templates)
5. Technical support from institutional research
IE Committee Decision The instructional program review process was
successful The review of instructional units was important – but
administrative/ESS units helped created an environment conducive to learning at the college and supported the learning process
Administrative units should go through a similar process
Create a committee to draft a similar process for the administrative and students services areas of the college
Administrative units were spread across three VP areas
Two representative from each VP area participated in drafting the new review process
We spent approximately six months creating a workable process
Timeline for Administrative Units During the:
1999-2000 year – six units went through the review
2000-2001 year – eight units went through the
review 2001-2002 year – nine units went
through the review
Vice Presidents set an annual timeline for their units with due dates for each section
The Administrative Unit Review Design
I. The Unit/Program ProfileA. The Mission/Purpose
1. Role unit plays in the college mission2. Unit/program goals as they relate to the college’s mission
B. The Staff 1. Professional and administrative staff
(since the last review)a. Position description/dutiesb. Credentials (full and part-time, if any)c. Accomplishments (if applicable)d. Service to college, community and
nation
The Administrative Unit Review (continued)
B. The Staff (continued)e. Professional development
activities2. Classified Staff
a. List of names and positionsb. List of required credentials (if
any)C. The Customer/Client Served
1. Breakdown of students/faculty or staff by type or demographic information
(thorough explanation of who is served)
The Administrative Unit Review (continued)
II. Definition of Services or ProgramA. Definition of day-to-day duties of the
unit B. Innovations, new projects, new
initiatives, local, state-wide or national efforts
C. Required functions of unit (description and status of compliance)
1. SACS "must" statements2. State mandates3. Federal mandates4. Other
The Administrative Unit Review (continued)
III. Administrative Objectives and Student Outcomes (where appropriate)
A. Administrative Objectives (2-3 objectives)
B. Outcomes (or status if incomplete) of innovations, new projects, new initiatives, local, state or national
effortsC. Assessment explanation (what was
assessed, who, when, how many) D. Results of Administrative Objectives
Based on Assessment
Assessment of Administrative Units
During the year of program review, the “Annual Faculty/Staff Survey” contained questions from those units being reviewed.
Results were given to each unit and broken out by campus and job type
http://inside.cpcc.edu/planning There’s a link to survey results.
The Administrative Unit Review (continued)
IV.Need for Change A. Strengths identified by external
sources, faculty, staff and studentsB. Weaknesses identified by external
sources, faculty, staff and studentsC. Recommendations by faculty, staff,
external sources and students to improve the unit's services and programs
D. Strategies for change (based on input from Sections A, B & C) - closing the loop
E. A one-year follow-up brief report to the Unit VP reporting on the progress of D above
(due at the end of the year following review)
The Administrative Unit Review (continued)
V. Future Issues - Resources needed for future effortsA. Market trends within the broad service
unit or program area (based on best- practices, the literature or training
received)B. Anticipated future changes and needs
(based on market trends) C. Resources, equipment, space, staffing
and work load changes needs for future growth or continuation
D. Future plans of unit
Assistance from the Website
The IE website: http://inside.cpcc.edu/IE
Explanation of IE process Templates and forms for review “Perfect” examples for clarification Schedule for review
Overall Benefits
#1 No recommendations in the area of Institutional Effectiveness from SACS during the October 2002 re-accreditation visit
The college became change-oriented Units had to define strategies for change Didn’t have to be perfect but rather making continuous
progress Strategies for change helped identify needed resources
for units Units had to “close the loop” with the one-year follow-
up (couldn’t promise and not deliver)
Overall Benefits
Units became empowered to perform their functions in an optimal manner and to ask for what they needed (no one noticed them before, now they do) Created data to support needs Accomplishments were reported to major
administrative groups across the college (President’s Cabinet, Planning Council, etc.)
Overall Benefits
The college community understood the purpose and function of every unit
Senior administration realized the benefits and became strong supporters Data are reported annually from
program/unit review VPs became stronger advocates for their
units making changes to improve services
Overall Lessons Learned Through Program/Unit Review
The response rate to on-line surveys was double that of pencil-paper surveys
Faculty and staff did not know what many units did
Faculty and staff were unsure of many policies (how to obtain funds from the Foundation, etc.)
Email was the preferred method of communication for faculty and staff
Overall, faculty and staff were pleased with services
Wednesday afternoons were the best time to hold trainings
Overall Lessons Learned Through Program/Unit Review (cont.) Faculty and staff had good ideas on how to
improve services – if we just asked them Use of P-Cards was saving the college time and
money and faculty/staff preferred using them Faculty and staff wanted to continue to receive
paycheck receipts in the mail (all employees are direct deposit)
More than 80% of faculty and staff were using remote access to email
Faculty and staff wanted changes in the budgeting process – too complicated
Administrative Response
Units spend a lot of time working on the program review.
If we want them to take it seriously, we have to take it seriously.
Once reviews are reported at the end of the year, their Dean or Vice President/Chancellor need to: 1. Read them 2. Respond to them
From Personal Experience
Sit down with the unit (group meeting) Give them my attention Discuss what was learned Discuss their major issues Discuss what they need to make
improvements Actually attempt to direct resources to
them to make those improvements Then they do not see it as an academic
exercise
How It Works for UsHow It Works for Us
Administrative Units Reviewed in 2003 Administrative Technology Services Compliance & Audit, Inventory Control,
Basic Skills Compliance & Reporting Facilities Services Information Technology Services Institutional Advancement/Foundation Planning and Research Resource Development
Faculty/Staff Survey 2003Faculty/Staff Survey 2003
During the Spring 2003, an on-line faculty-staff survey was developed, administered and analyzed.
A total of 222 faculty and staff completed the survey with the following numbers by type:
• Central 156 Classified Staff 51
• North 6 Professional Staff 55
• Northeast 13 Faculty 68
• Levine 20 Administration 48
• Southwest 9 Total 222
• West 15 Full-time 210
• Other 3 Part-time 12
Facilities ServicesFacilities Services
Consists of the following departments: Facilities Design and Construction – Oversee
planning, design, and construction of new facilities and major renovations
Facilities Management – Operates and maintains existing facilities of the College
Security – Provides physical security services at all campuses
Distribution Services – Manages collection and delivery of U.S. and intercampus mail, package deliveries, and centralized shipping and receiving services
Inventory Control – Manages the College’s inventory tracking, reporting, and disposal functions
Facilities ServicesFacilities Services
Survey respondents gave high rankings (in the 80-90% range) for: Newly constructed or newly renovated facilities
being of high quality and meeting needs
Knowing what was planned for the future facilities of their respective campuses
Classrooms in buildings built since 1990 being adequate in size, furnishings and other amenities
Facilities ServicesFacilities Services
Classrooms and common areas being generally clean
Lighting levels being adequate
A safe and secure environment being provided for all members of the College community
Knowing procedures to follow if a medical emergency on campus
Providing mail, shipping and receiving, and inventory control services
Areas Needing Improvement And Actions To Be TakenAreas Needing Improvement And Actions To Be Taken
Only 73% of respondents were pleased with new office spaces.
To address this concern, the Facilities Standards Subcommittee of Facilities Partners will review the adopted standards for offices and furnishings and make recommendations for revisions.
Only 50% faculty and staff were not pleased with the temperatures in classrooms. The Central Energy Plant began operating in the summer of
2003; when connected to the main body of Central Campus, the back-up capabilities of this facility should minimize the disruption of cooling. The College is also developing a plan for the future pairing of boilers to provide back-up heating for Central Campus buildings. At the suburban campuses, central plant approaches are being used to link buildings, thereby providing back-up capability.
Areas Needing Improvement And Actions To Be TakenAreas Needing Improvement And Actions To Be Taken
Only 57% of the respondents expressed satisfaction with the cleanliness of bathrooms. The College entered into a new housekeeping
contract as of July 1, 2003 which employs a new vendor and has enforceable cleanliness standards that must be met. Additionally, Facilities Management is addressing the appearance of some of the older restroom facilities on Central Campus as part of a cyclical rehabilitation program.
Future ConsiderationsFuture Considerations
The College will be adapting its long-term capital planning to new guidelines from the County and seeking the use of alternative funding sources.
Facilities Services will be working with other units of the College to ensure coordinated planning relating to the staffing and specialty services for new facilities.
Facilities Services will be developing new strategies for meeting operational demands in the face of decreasing dollars per square feet in the County budgets.
Planning and ResearchPlanning and Research
Survey respondents gave high rankings (in the 80-90% range) for:
Responding quickly to their need for data/information. Producing enough reports annually to meet the
planning and information needs of faculty and staff. Making a significant contribution to the College. Contributing to the effectiveness of CPCC. Assisting in the area of survey development/design. Assisting with program/unit review. Having a quality Institutional Effectiveness Website. Having a quality Institutional Research Website. Having a quality Fact Book (available online).
Strengths and WeaknessesStrengths and Weaknesses
Strengths Identified Serving the College well Contributing to the College’s effectiveness Doing a good job Assisting the College well with survey design
and program review support Providing helpful web sites and “Fact Book”
How We Can Improve Increase awareness by communicating its
services and results Take a look at the websites for possible
improvement Conduct more studies
Future IssuesFuture Issues
Strategies for Change
The department will consider a name change
The webmasters for the two websites will assess them over the next year to attempt to reduce clutter
The senior research analysts will undertake additional studies for the college (one each over the next year)
Future IssuesFuture Issues
Future Needs/Plans Expect to open the “Center for Community-
based Research”
Will need to increase space (plus parking) and staff for this project
Send staff to training on new SACS criteria
Increased software for survey analysis and qualitative research
Information Technology
Survey respondents gave high rankings (in the 80-90% range) for: Courteous and helpful staff in answering
questions and resolving issues with administrative systems access.
Providing regular scheduled reports accurately and on time.
Reliability of systems and availability especially during heavy registration.
Awareness of the statewide implementation of the Data-tel Colleague system this year.
Strengths and Weaknesses -Administrative Computers Services Strengths and Weaknesses -Administrative Computers Services Strengths Identified
Strong staff retention Strong understanding of policies and procedures Consistency of services Established relationships with other departments
Reliable Mainframe system/applications
What We Can Improve Complex and inconsistent applications Knowledge base
Users unclear on best usage/practices on new CIS Inconsistent knowledge/support of both systems
Communication
Strengths and WeaknessesHelp DeskStrengths and WeaknessesHelp Desk
Strengths Identified Customer Service Progressive Implementation of Technology / New Services Existence of Robust Infrastructure to allow for
implementation of new technology
How We Can Improve Email Concerns Automate account request process/eliminate paper Insufficient Human Resources preventing suitable
marketing / awareness of IT Services Communication Internal/ External Follow Up and customer service
Strategies for Change Strategies for Change
Speed up email migration Communicate scheduled downtime Automate account request process
eliminate paper Develop awareness of IT Services Communication Internal/ External Technology Training Follow Up and customer service
Strategies for Change Strategies for Change
Future Needs (staff, resources) Implement a marketing initiatives
Improve department image to customers Provide Campus Wide Technology Forums Increase customer awareness
Resources Consolidate CPCC Call Center Services to
maximize college resources and provide consistent information increase services provided to all customers.
Future IssuesFuture Issues
Strategies for Change Develop solution to integrate applications
Focus on Application Framework, especially with standards
Portal development Cross-train staff to improve effectiveness and
support Understand best means of communication, plan
for it, and communicate – internally/externally Oral communication highlighting major
events Published communication detailing projects,
issues, and maintenance Published communication on major project
statuses
Future IssuesFuture Issues
Future Needs (staff, resources, etc.) Staff
Architect to properly guide the framework and standards development
Technical Writer/Trainer to assist with communication
Especially focus on patches from Datatel/ACS
Resources Microsoft Project Server
Assist with internal communication of projects
Future Issues Future Issues
Strategies for Change Future Needs (staff, resources, ect) Implement a marketing initiatives
Improve department image to customers Provide Campus Wide Technology Forums Increase customer awareness
Resources Consolidate CPCC Call Center Services(2722)
to maximize college resources and provide consistent information increase services provided to all customers.
An Example from Instruction
Workplace Basic Skills This program is a literacy initiative that
goes directly into the worksite and teaches ESL classes, GED prep and GED classes.
During their review, they surveyed both employers and students.
This was the first time they had even done this.
What They Learned
Employers said: 43.8% of employers reported increases in
employee performance as a result of participation in the program.
31.3% reported a reduction in absenteeism by participants.
87.5% said classes improved the morale of their employees
37.5% said participants received raises 50% said communication had improved.
What Students Said
70.2% reported being able to fill out job forms better
35.5% said they could now help their children with their homework
91.1% said they felt better about themselves
44.4% said they had received a raise, promotion or opportunity as a result of the courses
86.3% said their ability to communicate in the workplace had improved
What Has Happened Since
Their assessment data has shown up in their marketing brochures to employers.
Their enrollment has grown dramatically.
They have received funding and marketing support from Charlotte Reads (considered a model adult literacy program).
“Best” Results of “Best” Practice
Better use of data across the college We have become more client/customer focused Review gives direction for goals and needed
changes Departments are empowered to do their jobs
(can’t slip through the cracks and be unnoticed)
Problems must be resolved (there is no hiding and no excuses)
Surveys provide needs assessment data as well as evaluation which gives departments direction
For a copy of this presentation
Contact Terri Manning or Kathy Drumm [email protected] [email protected] Download or print presentation: http://inside.cpcc.edu/planning
Click on “studies and reports” Listed as SACS Presentation