program quality assurance at algonquin college of applied ...€¦ · web viewprogram quality...

Program Quality Assurance atAlgonquin College of Applied Arts and Technology

Program Quality Review College Certificates

2011-2012

A Guide for Team Leaders

PREFACE......................................................................................................................................4

PROGRAM QUALITY ASSURANCE – AN OVERVIEW...............................................................5

Program Quality Assurance Process Audit (PQAPA)................................................................5

Program Quality Assurance at Algonquin College.....................................................................6

PROGRAM QUALITY REVIEW....................................................................................................7

PQR Goals.................................................................................................................................7

PQR Process.............................................................................................................................7

Schedule....................................................................................................................................7

Roles..........................................................................................................................................9

Orientation................................................................................................................................11

Release Time...........................................................................................................................11

Timeline....................................................................................................................................12

Need Assistance?....................................................................................................................12

THE FINAL REPORT – AN OVERVIEW.....................................................................................13

The Main Document.................................................................................................................13

The Appendices.......................................................................................................................14

DOCUMENT REVIEW.................................................................................................................15

Program Monograph................................................................................................................15

Program Council Meeting Minutes...........................................................................................16

Advisory Committee Meeting Minutes.....................................................................................16

Further Study Pathways...........................................................................................................17

Reports from External Accreditation or Licensing Bodies........................................................17

Program Vocational Learning Outcomes.................................................................................17

Program of Study.....................................................................................................................17

Course Outlines and Course Section Information....................................................................18

DATA REVIEW............................................................................................................................19

CURRICULUM MAPPING...........................................................................................................20

Program Learning Outcomes...................................................................................................20

Purposes of Curriculum Mapping.............................................................................................20

Program Standards..................................................................................................................20

Performing Curriculum Mapping..............................................................................................22

EVALUATION AGAINST THE QUALITY CRITERIA...................................................................25

Evaluation Criteria Listing........................................................................................................25

2

Responding to the Evaluation Criteria.....................................................................................28

Evaluation Criteria Listing – with help text...............................................................................28

CONDUCTING FOCUS GROUPS..............................................................................................45

Memos from the Academic Administrator and Team Leader...................................................45

Setting the Focus Group Agendas...........................................................................................46

Tips for Planning and Facilitating Focus Groups.....................................................................46

Faculty Focus Group................................................................................................................47

External Stakeholders Focus Group........................................................................................48

Student Focus Group...............................................................................................................52

COMPLETING THE FINAL REPORT – AN OVERVIEW............................................................58

ANNUAL FOLLOW UP................................................................................................................59

GLOSSARY AND LIST OF ACRONYMS....................................................................................60

APPENDIX A – THE FINAL REPORT.........................................................................................61

APPENDIX B – THE CREDENTIALS FRAMEWORK.................................................................68

APPENDIX C – TEMPLATES.....................................................................................................76

3

PREFACEThis Program Quality Review (PQR) Team Leader guide is written to aid Program Quality Reviews for College Certificate programs offered at Algonquin College. It is an adaptation of the main Team Leader Guide for Ontario College Credentials, located at http://algonquincollege.com/acad_dev/MinistryApprove.htm.

Many College Certificate programs are offered on a part-time basis, with administration, including admissions, managed through the Centre for Continuing and Online Learning (CCOL). The CCOL website houses information for continuing education programs including those delivered in a fully online learning environment.

4

http://www.algonquincollege.com/ccol/index.htm

http://algonquincollege.com/acad_dev/MinistryApprove.htm

PROGRAM QUALITY ASSURANCE – AN OVERVIEW

Program Quality Assurance Process Audit (PQAPA) One of the prerequisites for granting greater autonomy to the Colleges of Applied Arts and Technology in Ontario to govern their own programs is the implementation of a process to assure program quality. The Ontario Colleges of Applied Arts and Technology Act, 2002 outlined two elements of this expectation:

quality assurance at the individual college level, and a self-regulating process at the system level.

The Program Quality Review process forms a major part of adherence to providing quality assurance at the College level, and is therefore essential to College operations.At the system level, that is, at the level which encompasses all Colleges of Applied Arts and Technology in Ontario, a separate quality assurance process takes place, the Program Quality Assurance Process Audit (PQAPA). This is run through the Ontario College Quality Assurance Service (OCQAS1). OCQAS was established to provide effective and efficient mechanisms that ensure specific program quality and consistency standards are met by the Colleges of Applied Arts and Technology (CAAT) in Ontario.

The following six2 criteria define institutional policies and practices that OCQAS expects a college to have developed and implemented to ensure the quality of their programs:

Criterion 1. Program-level learning outcomes for all programs of instruction are set, are consistent with the college mission and the programs’ intended purpose, and are appropriate for the credential offered upon successful completion of the program.Criterion 2. Admission, credit for prior learning, promotion, graduation, and other related academic policies support program development and student achievement of program learning outcomes.Criterion 3. Programs conform to the Framework for Programs of Instruction3 and the Credentials Framework, are consistent with accepted college system nomenclature / program titling principles, and maintain relevance.Criterion 4. Methods of program delivery and student evaluation are consistent with the program learning outcomes.Criterion 5. Human, physical, financial, and support resources to support student achievement of program learning outcomes are available and accessible.Criterion 6. Regular program quality assessment that involves faculty, students, industry representatives, and others as appropriate for the purpose of continual improvement is in place and happens.

1 http://www.ocqas.org/index-en.html2 A sixth criterion, focusing on program learning outcomes, was introduced in Summer 2011.3 The Minister’s Binding Policy Directive Framework for Programs of Instruction is one of the policy documents issued by the Ministry of Training, Colleges and Universities in April 2003 to guide the development and delivery of programs of instruction offered by the colleges of applied arts and technology in Ontario. For details see http://www.accc.ca/ftp/es-ce/MTCUCollegeFramework.pdf .

5

http://www.accc.ca/ftp/es-ce/MTCUCollegeFramework.pdf

Program Quality Review evaluation criteria are based on the quality criteria above

6

Program Quality Assurance at Algonquin College Program review is annual event at Algonquin College, and currently consists of four processes:

Annual Curriculum Review Annual Program Mix Review4 Cyclical Program Quality Review Cyclical Review of Online Elective General Education courses5

Annual Curriculum Review Each year program faculty members revise and update curriculum to maintain its currency and to ensure that graduates will have the skills necessary to be successful in the workplace.Feedback from faculty, students, and advisory committee members is taken into consideration before updates are made to keep the program current. Revisions to the curriculum and the program narrative information are made in GeneSIS, the student information system, and are subsequently reviewed in the Office of the Vice President Academic. Once the changes are approved, a new program version is created. The information in GeneSIS becomes the foundation for course loading, fees loading, scheduling, registration and College publications.

Program Mix Review Staff in the Academic Operations and Planning department compile data that reflect the Quality Measures for a program. Admissions and registration data, employment rates, capstone questions from Key Performance Indicators (KPI) and course assessment surveys, and program costing data are reviewed. Programs not meeting college benchmarks are reviewed and strategies are developed by the Academic Administrator and Dean to improve the quality of the program and/or the financial contribution.

Program Quality ReviewProgram Quality Review (PQR) ensures that each program goes through a comprehensive review every five years. The PQR process builds on annual review activities by including analysis of three to five years of data compiled for the program mix review, and soliciting direct feedback from stakeholders and students into the program review. Each PQR results in an implementation plan containing resourced actions to maintain and improve the program over the next five years. This manual guides Team Leaders when conducting a Program Quality Review.

Cyclical Review of Online Elective General Education courses Online General Education courses form an integral part of many programs offered at the College. In 2009, the review process for these courses was formalized. Each of the online General Elective courses in the College pool is reviewed according to a published cycle, first by the General Education committee for overall adherence of the course to General Education objectives, and then in detail by individual faculty and student reviewers.

4 Review conducted more informally for CCOL-based programs. Additionally, there is less data available for College Certificate programs.5 Review not applicable for College Certificate programs

7

http://www.algonquincollege.com/acad_dev/documents/ProgramQualityAssuranceAlgonquinCollege-legalNov207.doc

PROGRAM QUALITY REVIEW

PQR Goals To assess the program against established quality criteria in order to determine whether the

program meets or exceeds College and Ontario College Quality Assurance Service (OCQAS) standards (for College Certificate programs and Ontario College Credentials) or Postsecondary Education Quality Assessment Board (PEQAB) standards (for Bachelors’ Degrees).

To establish and implement a quality improvement plan6 for the program, in response to the findings of the review.

PQR Process Program Quality Review follows a systematic process, and is outlined in Figure 1. After the Team Leader has been identified to conduct the review, information pertinent to the program is collected and analyzed, allowing an initial evaluation of the program against established quality criteria. The program information reviewed includes published and internal documentation, program data, current curriculum maps, and input collected from key players in the program – faculty, students, and external stakeholders. Once the review is complete, the evaluation against the criteria is finalized, and all findings and recommendations for improvement are captured in a report. An implementation plan for the recommendations is drawn up, with an update on the actions provided to the PQAA at least annually until completion.

The PQR Team Leader coordinates the PQR, with assistance from their Academic Administrator7.

Note that this guide is written to provide assistance with each of the main components of PQR: Document Review Data Review Curriculum Mapping Evaluation Against the Quality Criteria Conducting Focus Groups Completing the Final PQR Report

Schedule Programs are reviewed according to a defined schedule, normally every five years. The schedule is revised annually, in consultation with the Academic Departments, and aligned with other Quality Assurance requirements, for example, accreditations or consent renewals for bachelor’s degrees. The PQR schedule is published at http://www.algonquincollege.com/acad_dev/services_program_quality.htm.

6 Also referred to as “the implementation plan”7 The term Academic Administrator refers to the Academic Chair or Academic Manager

8

http://www.algonquincollege.com/acad_dev/services_program_quality.htm

Cyclical Program Quality Review

College Certificates

(Figure 1)

9

Confirm Programs to Review

Assign Team Leader

Team Leader and Academic Administrator Review PQR data

summary and conduct initial analysis against criteria

Review of Findings Formulation of

Recommendations Development of Implementation

Team Leader Completes Final Report

Implementation of Recommendations

Annual Follow-up of Implementation Plan

PQAA: Provides Team Leader with Orientation to Process, Resources and Templates

Faculty Review of Curriculum Data Student Focus Group External Stakeholder and/or

Advisory Committee Focus Groups

PQAA: Provides Academic Administrator with 5 Years of Central Data (Registration & Revenues, Course/Program Quality)

Academic Administrator analyses data and summarizes for use in PQR.

Roles

Program Quality Assurance AdministratorProgram Quality Reviews are coordinated by the Program Quality Assurance Administrator (PQAA), who reports to the Dean, Academic Development, who in turn reports to the Vice President Academic.The responsibilities of the Program Quality Assurance Administrator include

ensuring the College is compliant with Ministry requirements regarding Program Quality Assurance.

consulting with major academic constituencies College-wide regarding Program Quality Review.

preparing documents related to Program Quality Review. planning and organizing orientation/training workshops for Team Leaders. supporting the Program Quality Review process in programs. meeting regularly with PQR Team Leaders during the review process to monitor

progress and provide guidance. facilitating evaluation of the Program Quality Review process. arranging printing of the final report; forward copies to the TL, Academic Administrator,

Dean and Vice President Academic.

Program Quality Review Team LeaderThe PQR Team Leader is normally a faculty member who is assigned to conduct Program Quality Review for the program in which s/he teaches.The responsibilities of the Team Leader include

attending all meetings scheduled by the Program Quality Assurance Administrator. attending all orientation/training workshops related to the Team Leader role. reporting to the Academic Administrator weekly to monthly on the progress of the

Program Quality Review. incorporating the results of the data analysis8 in the final PQR report. convening meetings of the program faculty team. convening a Student Focus Group. convening an External Stakeholders / Advisory Committee Focus Group. facilitating Student and External Stakeholder Focus Groups for a program from another

School and submitting reports for these meetings. evaluating the Program Quality Review process. submitting the final report to the Academic Administrator and PQAA, who will review and

sign-off. (The Academic Administrator then forwards the report to the Dean for final review/sign-off, and notifies the PQAA who will advise the Vice President Academic of its completion.)

8 Data are limited for College Certificates, and not always able to be presented anonymously. The Academic Administrator conducts the data analysis, and reviews this analysis with the Team Leader. A summary of the data analysis is also provided and included in Appendix E of the final report.

10

DeanThe responsibilities of the Dean include

identifying programs for Program Quality Review. supporting the Program Quality Review process. reviewing and signing-off on the Program Quality Review report. following up regarding results and recommendations included in the report. reporting to the Vice President Academic on the progress of Program Quality Review

and regarding the follow-up on the recommendations.

Academic AdministratorThe responsibilities of the Academic Administrator include

acting as a facilitator for Focus Groups in the event of illness of the TL. meeting with each TL to discuss the upcoming process, and to encourage and facilitate

attendance at the Aligning and Building Curriculum Conference in the upcoming Fall and following Spring as space allows.

attending information meetings for Academic Administrators. providing release time for the TL as outlined in this guide, or for arranging the contract

as applicable reviewing and summarizing the program data early in the process, meeting with the TL

to conduct the initial assessment of the quality criteria. assisting with agenda setting for focus groups and helping to create the invitation list providing the relevant budget code for all expenses. sending an email to all program faculty inviting them to participate in the PQR process

and to fill out the credentials and experience sheet. providing a summary report of faculty credentials and experience to the Team Leader, to

be included in the final report. arranging secretarial support as outlined in the PQR process meeting regularly with the TL and monitoring progress. attending focus group meetings and introducing the group members to the process. ensuring feedback is provided to all those who participated in the focus groups. providing feedback regarding the PQR process. collaborating in the development of the final recommendations and implementation plan,

ensuring adequate budgetary support for those recommendations that are to be implemented immediately (i.e. within one year) and consideration of those recommendations for which funds are not currently available.

including PQR recommendations in their Performance Contract. facilitating the implementation of the final recommendations. tracking progress on the implementation plan, and preparing the submission of the PQR

Annual Follow Up Report until all recommendations have been implemented.

Orientation Orientation for Team Leaders is scheduled outside of classroom contact weeks to minimize the negative impact of training within teaching weeks.

11

The orientation process for Team Leaders involves a series of workshops, each three hours in length. Upon completing the Team Leader orientation process, Team Leaders are able to

evaluate a program against identified criteria; identify quality in a program when measured against identified standards and determine how to reinforce/maintain this quality.

lead focus groups for internal and external stakeholders. map curriculum to validate that the program of study meets current Ministry

requirements. analyze program data and make inferences based on trends. analyze curriculum maps to identify gaps and redundancies. formulate realistic and attainable recommendations to improve quality where needed.

Release Time Where the Team Leader is a full-time faculty member, s/he is entitled to receive the following release time for the duties and responsibilities associated with Program Quality Review:

Fall 2011 or Winter 2012: Release from teaching hours for the equivalent of one three-hour course including evaluation and preparation time which, for consistency purposes, has been equated to six hours of assigned workload. Faculty members who are reviewing a three-year program, a four-year program, or the equivalent of three or more years of curriculum, will be assigned a nine hour workload. All authorized travel expenses incurred by Team Leaders to facilitate Focus Groups at another campus will be reimbursed.

Where an internal Team Leader is not available to conduct the PQR, an external Team Leader may be assigned to the role. Before the PQR commences, the Academic Administrator, in consultation with the PQAA, identifies a Team Leader and negotiates a contract, with remuneration within the PQR budget allocation for the program under review. Note that the budget allotted may be less for smaller programs, for example, for short College Certificate programs. The PQAA provides details on the funds available for review of each program.

In addition to the Team Leaders’ release time or equivalent, additional financial support is provided to departments as follows:

Support Staff: up to 48 hours per PQR at $15 per hour Part-time Faculty: up to 16 hours per PQR at $30 per hour Hospitality9: up to $100 per PQR to contribute towards light refreshments at focus

groups.PQR expenses are charged to the program budget code with the exception of travel:

Departments are compensated annually (Q1) by Academic Development for the specified release time.

Travel expense forms are submitted directly to the PQAA. 9 Expenditure must follow guidelines as per College Policy AD12 “Travel, Meals and Hospitality”

12

http://www2.algonquincollege.com/directives/policy/ad03-travel-meals-and-hospitality-policy/

Timeline Fall Course Release

April 2011 Confirmation of Programs to undergo PQR, Team Leaders assigned:

June or August 2011 Team Leader orientationSeptember 2011, or as required Refresher orientation (for those who have been TLs

before)October 2011 Touch base meeting with PQAANovember 25, 2011 Draft report due

(submit to Academic Administrator and PQAA):December 16, 2011 Submit final report

Winter Course Release April 2011 Confirmation of Programs to undergo PQR, Team

Leaders assigned:June or August 2011 Team Leader orientationJanuary 2012, or as required Refresher orientation (for those who have been TLs

before)February 2012 Touch base meeting with PQAAMarch 31, 2012 Draft report due

(submit to Academic Administrator and PQAA):April 21, 2012 Submit final report

Need Assistance? Contact the Program Quality Assurance Administrator (PQAA) at any time with questions about Program Quality Review:

Wilma McCormack, Room C-532, 613-727-4723 ext. 7681 [email protected]

13

mailto:[email protected]

THE FINAL REPORT – AN OVERVIEWFor more details on the contents of each section in the main document, see Appendix A.

The final output of the PQR is a report which is submitted to the Office of the Vice President Academic. This report follows a standard structure:

The Main Document

Executive SummaryWritten last, the Executive Summary prefaces the entire report. It summarizes the highlights of the quality review of the program, and includes the overall status of the program, its strengths, weaknesses, as well as challenges and recommendations.

IntroductionWritten first, the Introduction paints a picture of the program at the outset of the review – it does not contain any information gained as part of the review process.

Evaluation of the Program against the CriteriaThis section is the working part of the document. Information is gathered as each stage of PQR - document review, data analysis, and the outcomes of the focus groups – is carried out, and notes are captured against each of the elements of the quality criteria as the review continues. By the end of the review process, each element of the quality criteria will have been responded to (yes/no) and a justification of the response will have been provided.

ConclusionsThe conclusions wrap up the findings of the review. The outcomes of the document and data review are provided, along with the highlights from each of the focus groups. The recommendations are stated, and any observations or non-implementable recommendations are noted.

Recommendations and Implementation PlanThe recommendations that are to be implemented are captured in a table, and are partitioned, as far as possible, under the pillars of the College Strategic Plan. Each recommendation will have an associated list of action items that must be completed in order to meet the goal of the recommendation. Each action item must be resourced, and have a realistic completion date.

The implementation plan is a living document – it is carried forward year over year, and updated at least annually until action items are completed.

The Appendices The appendices carry the information collected during the review process. This information will be referenced within the main document.

Appendix AThis contains program and curriculum information. It consists of the

Program Monograph Program Vocational Learning Outcomes Program Delivery Information Curriculum Mapping Information10:

Vocational Learning Outcomes curriculum map (Overall and by Level) Analysis of the Vocational Learning Outcomes curriculum map Course Outline Review Summary

Appendix BThis contains the information collected from the program faculty, namely

Faculty focus group report(s) Summary of the Faculty Program Self-Audit Summary of Faculty Credentials and Experience

Appendix CThis contains the information collected from the External Stakeholders, normally the

External Stakeholders focus group report

Appendix DThis contains the information collected from the current students, normally the

Students’ focus group report

Appendix EThis contains summary information only from the review of the statistical and survey data for the program. Note that credentials offered through the Centre for Continued and Online Learning have limited statistics available. Please contact the PQAA for guidance as required.

10 Will be adapted if no VLOs are yet defined.

15

DOCUMENT REVIEWThe following documents must be reviewed by the Team Leader, in cooperation with the Academic Administrator and department faculty, to ensure that the information contained therein is complete, accurate, current, that it conforms to Ministry and College Standards, and that pertinent information is readily available for applicants and students to allow them to make informed decisions about program choice.

Program Monograph Program Council Meeting Minutes (or equivalent, as available) Advisory Committee Meeting Minutes (or equivalent, as available) Further Study Pathways Reports from External Accreditation or Licensing bodies Program Vocational Learning Outcomes Program of Study Course Outlines and Course Section Information

These documents are referred to when evaluating the program against the quality criteria.

Program Monograph The program monograph contains several key pieces of information used by applicants when determining which program to choose. It is reviewed annually as part of annual curriculum review. Although the whole monograph should be reviewed during PQR, an in-depth review of the following sections may be necessary in order to respond to some elements of the quality criteria.

Admission Requirements should ensure that students are adequately prepared to enter the program (i.e. that they have the foundational knowledge needed on which to base their learning) without unnecessarily limiting access to a program.

College Eligibility is Ministry mandated and is usually standard for postsecondary programs - Ontario Secondary School Diploma, or at least 19 years of age.

Program Eligibility is determined by the department and should reflect any necessary knowledge and skills that the students must have prior to entering the program, for example a specific level of mathematics or science. Mature students may be required to meet the program eligibility even though they are 19 or older and do not have a high school diploma. Proven competency in English is a College-wide requirement.

Note that the current monograph publishes admission requirements for the subsequent academic year.

16

Fees and Expenses should list all relevant program expenses to allow students to plan their financial commitment. Fees and Application Information are supplied by the Registrar’s Office.

The Program of Study is published in the monograph and is the foundation of the College’s contract with an applicant. It outlines the learning opportunities that student can expect in return for paying tuition.

The program monograph is included in the final PQR report within Appendix A.

Program Council Meeting Minutes Program Council meetings should be held for all full-time programs, regardless of credential offered. Note that most College Certificate programs are offered on a part-time basis, and do not hold program council meetings. However, items such as minutes of any student meetings and program survey data should be kept on file and reviewed as part of PQR.

Minutes of Program Council meetings are reviewed to ensure that Program Council meetings occur on a regular basis, and in accordance with Policy AA03

– Program Council. (Program Council meetings must happen once a term at a minimum.)

issues raised at Program Council meetings are captured, and have an action plan to address.

there is a feedback mechanism in place to ensure that the students are informed of what action, if any, will take place and the proposed timeline for any resolution.

The review of the minutes may also help identify agenda items for the upcoming focus groups.

The Program Council meeting minutes are not included in the final PQR report.

Advisory Committee Meeting Minutes Advisory Committees should be established for all programs resulting in an Ontario College Credential or a Degree. Although most College Certificate programs do not have an independent advisory committee, some do, or partner with similar/relevant full-time programs. Feedback from such meetings should be reviewed as part of PQR.

Minutes of Advisory Committee meetings are reviewed to ensure that Advisory Committee meetings occur on a regular basis, and in accordance with Policy

AA01 – Advisory Committees. (Advisory Committees must meet at least once a term.) issues raised at Advisory Committee meetings are captured, and have an action plan to

address these issues. there is a feedback mechanism in place to ensure that students are informed of what

action, if any, will take place and the proposed timeline for any resolution.

The review of the minutes may also help identify agenda items for the upcoming focus groups.

17

http://www2.algonquincollege.com/directives/policy/advisory-committees/

http://www2.algonquincollege.com/directives/policy/aa03-program-council/

The Advisory Committee meeting minutes are not included in the final PQR report.

Further Study Pathways Review the laddering opportunities that exist for the program, to ensure that they are both current and have the correct scope. This includes both post-program opportunities (e.g., laddering into an Ontario College Credential) and any pre-College laddering opportunities, e.g., School to College to Work Initiative (SCWI) projects occurring in partnership with the program (i.e., program specific Specialist High Skills Majors (SHSM) or Dual Credit initiatives).

Review other established agreements for further study currently in place for the program, for example, articulation agreements, to ensure that curriculum changes made since the last issue have been reflected within the agreement.

The final PQR report must include a list and brief description of any such agreements that exist with other postsecondary institutions. If no such agreements currently exist and graduates from the program regularly seek further education, it may be prudent to recommend in the PQR final report that an articulation agreement be considered.

Information on degree pathways is found at: http://www.algonquincollege.com/prospective/programs_degrees.htm .

Reports from External Accreditation or Licensing Bodies If applicable to the program, information on the success or otherwise of graduates obtaining licenses or external accreditation should be collected and noted against quality criteria element #6.8. This information may also be summarized in the Executive Summary and Conclusions sections within the PQR final report.

Program Vocational Learning Outcomes Many College Certificates do not have Vocational Learning Outcomes (VLOs) established. When defined, current approved program Vocational Learning Outcomes (VLOs) are stored in GeneSIS, and are reviewed during Curriculum Mapping. When not yet defined, PQR is used as the starting point to establish the program VLOs. Input on the program VLOs should be requested at each focus group, and, where possible, draft VLOs captured. The plan to finalize the VLOs, and to review, approve and update curriculum based on the approved outcomes should be captured in the PQR implementation plan.

A listing of the program Vocational Learning Outcomes (draft or approved) is included in the final PQR report within Appendix A.

Program of Study The Program of Study is stored in GeneSIS, and is published on the Program Monograph. The Program of Study is reviewed during Curriculum Mapping.

When reviewing the Program of Study, the following needs to be considered:

18

http://www.algonquincollege.com/prospective/programs_degrees.htm

Are the courses sequenced properly? Are prerequisites and corequisites appropriate to ensure students have an adequate

foundation on which to build their learning without unnecessarily impeding progression in the program?

Are the courses offered at a frequency and in an order for students to complete the program within the published timelines? (This is especially relevant for programs offered through the Centre for Online and Continuing Education.)

The Program of Study forms part of the Program Monograph, which is included in the final PQR report within Appendix A.

Course Outlines and Course Section Information Course outlines and Course Section Information are reviewed by the course owners as part of annual curriculum review, with the output of this review utilized within PQR. Course outline review will be discussed further in the Curriculum Mapping section of this document. To note:

a course outline must be developed for each and every course offered in the program. all sections of the course outline must be completed. Policy AA26 – Course Outlines and Course Section Information provides additional

information regarding the expected content of course outlines and course section information.

Effective Fall 2008, course outlines and course section information for full-time Ontario College Credentials are maintained in COMMS. If a course offered through a College Certificate is also offered as part of an Ontario College Credential, it will be housed and maintained within COMMS.

19

http://www2.algonquincollege.com/directives/files/2011/08/AA-26.pdf

http://www.algonquincollege.com/ccol/

DATA REVIEWProgram specific data must be reviewed by the Academic Administrator, with a summary provided to the Team Leader, to gain an understanding of the program before focus group meetings are held. This aids with evaluation against the criteria and setting the agenda for focus group meetings.

Data for College Certificate programs are limited, although if the program is offered on a full-time basis, additional data may be available. Please consult with the PQAA regarding data availability. Up to five years of following data are provided, when available, for Program Quality Reviews for College Certificates:

Registration and Revenue DataThis includes, on a term by term basis, the following information for the courses in the program:

Revenue Students registered Early withdrawal Late withdrawal Pass Fail

These data should be reviewed in the context of specific constraints of the individual program, for example, the maximum registration allowed, or any physical (space) constraints. Items of concern may include

a course where large numbers of students withdraw term over term a course being repeatedly cancelled due to lack of registration few students completing the course compared to registrants a repeatedly high failure rate on a course.

Course Assessment Roll Up: This report is a roll up of all course section course assessments in a program on a term-by-term basis. (The roll up preserves the anonymity of the individual course section professors.) It provides student feedback at the program level and will identify program strengths and areas of concern.

The section “Evaluation Criteria Listing – with help text” within this document provides detailed guidance on which Course Assessment Roll Up questions relate to which quality criteria, and will not be repeated here. The questions referenced are from the Continuing Education survey form. The PQAA will advise if a different survey form was used for courses in the program under review.

20

CURRICULUM MAPPINGAnalyzing the mapping of a program’s curriculum to its Program Learning Outcomes is an essential part of PQR. Building on the course outline reviews conducted as part of annual curriculum review, it provides the opportunity to review cumulative effects of changes made to the program’s curriculum over the last five years, determine whether it meets the requirements of the Program Standards, and plan for updates to the program. The exercise of performing this analysis is called Curriculum Mapping.

Program Learning Outcomes At Algonquin College, all programs, including College Certificates, are required to have defined and approved Program Learning Outcomes. As many programs were developed and offered before this requirement was established, PQR is used to initiate development of the learning outcomes if they do not exist.

Learning outcomes represent culminating demonstrations of learning and achievement. They describe performances that demonstrate significant integrated learning by graduates of the program has been achieved and verified. Learning outcomes are interrelated and should not be viewed in isolation of one another.

Purposes of Curriculum Mapping A curriculum map can be used to

document curriculum and the inclusion of standards for accrediting/credentialing groups identify opportunities in the program for learners to demonstrate learning outcomes at

the required level serve as a planning tool to ensure that program learning outcomes are developed within

the program identify areas of redundancy where content is more than adequately covered identify paths that learners can follow to meet graduation requirements help faculty, learners and others “situate” courses and learning experiences within the

program curriculum provide an overview of the curriculum for the total program

Program Standards 11 For details on each credential, refer to the Credentials Framework in Appendix B of this guide.

Figure 2 illustrates how College Certificates fit within the Credentials Framework, and the interdependency between the Program Standards or Local Vocational Learning Outcomes and the entirety of the courses within that program.

11 As per the Credentials Framework, found in Appendix B, College Certificates do not have Program Standards. However, Vocational Learning Outcomes should be defined for each College Certificate.

21

Figure 2: Aligning Curriculum - A Roadmap

22

Performing Curriculum Mapping Course Outline ReviewEach year, as part of annual curriculum review, course owners are expected to formally review their course outlines and course section information and update courses as necessary to reflect changes made in the course content. Course outline updates may include changes to course learning requirements, learning activities, or assessment methodologies.

In the PQR year, each course teacher submits their course outline check-up worksheets or equivalent for each course taught to the Team Leader ahead of the faculty focus group meeting. Additionally, for each course section taught, the course outline review chart should be filled. The Team Leader uses this course specific information to formulate discussion points for the faculty focus group, and prepare a Course Outline Review summary sheet for the final PQR report.

Specifically, the course teacher and Team Leader need to assess each course against the five evaluation elements pertaining to course learning requirements, program outcomes, learning activities and resources, evaluation methods, and prior learning assessment and recognition (PLAR) to determine whether:

There is congruency between the course learning requirements and the program learning outcomes12 (#3.8).

There is a match between course learning requirements, course learning activities, and learning resources (#3.9).

Learning methods are published and are matched to the learning requirements (#4.3). Evaluation methods allow students to demonstrate the course learning requirements

(#4.7). PLAR opportunities exist and are based on course learning requirements (#1.4).

Key information is captured in each course outline, including whether each learning outcome is taught, assessed and if there is opportunity to demonstrate a culminating performance of the outcome is captured in the outline prior to approval.

Table 1 provides the guidelines used to categorize teach (T), assess (A) and culminating performance (CP) for each activity within the course outline.

For any curriculum gaps noted, an action plan is developed and included in the final recommendations.

Electronic versions of each course outline and course section information should be collected and submitted to the PQAA along with the PQR final report. Note that (1) if a course offered through a College Certificate is also offered as part of an Ontario College Credential, it will be housed and maintained within COMMS, and (2) the course outlines are not to be bound with the final report.

12 When Vocational (program) learning outcomes are not defined, state this in the ‘comments’ column, and mark the column NYA – Not Yet Applicable.

23

Reviewing the Learning Outcomes and Compiling Curriculum MapsIn order to conduct curriculum mapping for College Certificate programs, first the existence of defined Vocational Learning Outcomes must be confirmed.

If Vocational Learning Outcomes are not defined, a main activity for the faculty focus meeting and the External Stakeholders focus group is to draft the vocational learning outcomes for the program. The PQR recommendations and implementation plan will capture the go-forward plan regarding the finalizing and subsequent approval of these outcomes, and any additional curriculum modifications which need to be completed as a result. Note that course outline reviews, conducted as part of PQR, will greatly assist in the finalizing of the mapped curriculum.

When Program Learning Outcomes are defined, the Team Leader, with assistance from faculty members, uses the course outlines to produce curriculum maps by Level and by Program. These are produced ahead of the faculty focus group if possible.

Reviewing the Curriculum MapsAt the faculty focus group, the Team Leader facilitates the review of the Curriculum Maps (by Program and by Level) using the checklist Analysis and Review of Curriculum Maps for PQR . This checklist is included in the final PQR report.

In addition to analyzing the curriculum maps, the Program of Study is reviewed. Due consideration must be given to the sequencing of the courses – is it appropriate? – and to the prerequisites and corequisites – are they appropriate to ensure students have an adequate foundation on which to build their learning without unnecessarily impeding progression in the program?

Any actions arising are noted, and included in the recommendations in the final PQR report.

24

Teach Do you provide instruction/learning opportunities in this skill in your course?

Is this skill identified in one or more course learning requirements?

Do you devote a significant amount of time to facilitating student development of the skills and knowledge embedded in the outcome?

Assess Are there assignments, tests or projects which are designed to allow you to evaluate or assess student performance of this outcome or some of its elements?

Do you, in your evaluation of student performance, verify that this particular outcome (or a significant component of it) has been achieved?

Is this outcome reflected in your course outline in the course learning requirements and/or embedded knowledge and skills, and in the evaluation of the course outline?

Culminating Performance Culminating performances are tasks or activities designed to assess a learner’s ability to demonstrate one or more learning outcomes in their totality. While they do not necessarily occur at the end of a program of study, they do evaluate whether a learner is able to integrate and apply their learning to demonstrate the performance described in the learning outcome(s) at the exit level.

Is there an opportunity for you to evaluate the outcome in its totality?

Does the evaluation result in a final product or performance which allows you to determine whether the learner has integrated the knowledge and skills identified in the elements of performance?

Can you determine from this performance if the learner has demonstrated the outcome?

Program culminating performances should require learners to demonstrate learning similar to what would be expected of new graduates in as close to a “real world” context as possible.

If a culminating performance is assigned to a course, the professor teaching the course will be asked to evaluate whether or not the students successfully demonstrated the outcome(s) the culminating performance addresses

Table 1: Guidelines for Teach, Assess, Culminating Performance

25

EVALUATION AGAINST THE QUALITY CRITERIAThe criteria against which the program is evaluated are provided below. First, a listing of the evaluation criteria and its corresponding elements is provided. This is followed by a listing with help text to assist in the evaluation of each element of the quality criteria. Note that these elements are based on the PQAPA quality criteria, and at times there is necessary repetition.

Note: the PQAPA quality criteria were updated in Summer 2011, and the PQR quality criteria elements have been re-ordered and added to accordingly.

Evaluation Criteria Listing Criterion 1. Program learning outcomes are set, are consistent with the college mission and the program’s intended purpose, and are appropriate for the credential offered upon completion of the program.

1.1 Program learning outcomes are consistent with the College mission, appropriate to the level at which the qualification is offered, consistent with the requirements of the Credentials Framework, and appropriate to the occupational requirements of the program graduates.

1.2 Program learning outcomes are consistent with MTCU Provincial Program Standards where they exist.

1.3 Program learning outcomes are reflected in course outlines.1.4 Program learning outcomes are used in prior learning assessment and recognition

(PLAR).1.5 Changes to courses and program learning outcomes are introduced on a timely basis

and are designed to maintain the relevance of the program. 1.6 The capabilities of program graduates, including knowledge, understanding, skills, and

attitudes are consistent with the intended program learning outcomes.

Criterion 2. Admission, credit for prior learning, promotion, graduation, and other related academic policies support program development and student achievement of program learning outcomes.

2.1 The qualifications and prerequisites required of the applicant are published and are appropriate to allow the student to be successful without limiting access to the program.

2.2 Students have adequate information to allow them to make informed choices about: selecting the correct program to meet their career aspirations; the financial commitment needed; the workload commitment needed; and the study options available to them.

2.3 Students know how to get internal and external transfer of academic credits and recognition for prior learning.

2.4 Students know what is needed to ensure they will be able to demonstrate program outcomes and complete the program.

2.5 Students know how they will be evaluated.2.6 Students indicate the learning requirements are relevant and meaningful.2.7 Students indicate that assessment methods relate to the learning requirements.

26

Criterion 3. Programs conform to the Framework for Programs of Instruction and the Credentials Framework, are consistent with accepted college system nomenclature / program titling principles, and maintain relevance.

3.1 The duration and structure of the program are consistent with the program learning outcomes and the credential offered.

3.2 Appropriate credits are allocated for each component of the program, and transfer and laddering options are stated.

3.3 Prerequisites do not unnecessarily hinder progress in the program.3.4 The program title is consistent with college system nomenclature / titling principles. 3.5 The program has established articulation agreements.3.6 The program conforms to the College policy for the number of English courses.3.7 All curriculum documentation is up-to-date including course outlines and the program

monograph information.3.8 There is congruency between the course learning requirements and the program

learning outcomes.3.9 There is a match between course learning requirements, course learning activities, and

learning resources.3.10 Concepts of social, economic, and environmental sustainability are embedded in the

program curriculum.3.11 Off-campus experiential learning components are embedded in the program curriculum.3.12 Students have opportunities to develop the skills, knowledge, and attitudes necessary

to succeed in a global economy.

Criterion 4. Methods of program delivery and student evaluation are consistent with the program learning outcomes.

4.1 Program delivery, including that which takes place off-site, is consistent with the nature of the program, its learning outcomes, and the needs of the students.

4.2 There is a range of instruction methods consistent with a variety of learning styles and learner needs and abilities.

4.3 Learning methods are published and are matched to the learning outcomes.4.4 College designated targets regarding hybrid courses are met.4.5 Learners are provided the skills necessary to be successful with the learning strategies

selected.4.6 Evaluation criteria are published and students are aware of how and when they are

going to be evaluated.4.7 There is a match between course learning requirements and evaluation methods, i.e.,

evaluation methods allow students to demonstrate the course learning requirements.4.8 Evaluation methods are valid and reliable.4.9 Students indicate that feedback is timely and allows them to build on their learning.4.10 Students perceive evaluation to be fair.4.11 Resubmissions, supplementals, and appeals are published, appropriate, fair, valid, and

consistent.4.12 Student workload and assessment is balanced across the term at both the course and

program level.

27

4.13 There is a range of evaluation methods used consistent with a variety of learning styles.4.14 Learners can earn credit for up to 75% of the program hours using the PLAR process.4.15 Academic policies and practices that provide for the development and continuous

improvement of teaching and learning methods are valued, documented, and supported.

Criterion 5. Human, physical, financial, and support resources to support student achievement of program learning outcomes are available and accessible.

5.1 Program faculty members, as a whole have adequate academic preparation and workplace experience to deliver a quality program.

5.2 Program faculty members are formally evaluated every three years.5.3 Program faculty members engage in professional development activities that ensure

they are current in their field and developing teaching expertise.5.4 Program faculty members work within clear and well-structured instructional plans.5.5 Program faculty members participate in reflective practice.5.6 Students consider faculty to be available.5.7 Students consider faculty to be adequately prepared for class.5.8 Students consider faculty to promote a positive attitude to learning.5.9 All students are assigned an academic advisor.5.10 Academic Advisors contact their students early in the term with an invitation to meet

and to ensure that students know who their advisor is.5.11 Labs, clinical facilities and placement facilities are complementary to, and integrated

into, the program and allow the learner to demonstrate the program learning outcomes.5.12 Students indicate that there are adequate and accessible learning resource materials to

allow them to be successful, including: textbooks in the bookstore; online materials; print resources; equipment; and student support services.

5.13 The program is financially viable. 5.14 The demand for the program has been sustained for the last five years. 5.15 There is a future demand for graduates of this program.5.16 The learning environment is safe.5.17 The students are provided with the information they need to know to function safely in

both the College and workplace learning environments.

Criterion 6. Regular program quality assessment that involves faculty, students, industry representatives, and others as appropriate for the purpose of continual improvement is in place and happens.

6.1 Students indicate that they are satisfied with the program.6.2 Issues raised at Program Councils are addressed in a timely fashion and feedback is

provided to the council.6.3 Learners progress through the program, achieve program outcomes and graduate in a

timely fashion.6.4 Learners with a wide range of abilities demonstrate the expected learning outcomes.6.5 Items raised by the Program Advisory Committee are considered in a timely fashion

and feedback is provided to the committee.6.6 Graduates are satisfied with the overall program experience.

28

6.7 Graduates are obtaining employment in their fields. 6.8 Graduates are successful in obtaining external licenses or credentials where relevant.6.9 Employers are satisfied with graduate performance.

Responding to the Evaluation Criteria Each evaluation criterion is designed to examine a specific area or areas of quality in the program. In addition to responding Yes, No, or Not Applicable (NA) to each element of each quality criterion, the reason or reasons for the response must be provided. It is preferable to have each response supported by more than one piece of evidence, for example, a response to element 5.16 could be

5.16 The learning environment is safe (Yes)

Course assessment roll up data indicate that students feel safe (range is 84 to 91% of students over past 5 years).

This was confirmed by current students at the student focus group, although it was noted that in some instances level 01 students were reluctant to ask questions in large classroom environments.

Evaluation Criteria Listing – with help text The following text will help to evaluate the program against the quality criteria. In some situations there will be adequate survey data and additional information to state that an element is met without further inquiry. In other areas, there may be residual questions after reviewing the available documentation. These questions then need to be explored with any or all of, the Academic Administrator, the faculty, the students or external stakeholders.

Criterion 1. Program learning outcomes are set, are consistent with the college mission and the program’s intended purpose, and are appropriate for the credential offered upon completion of the program.


Algonquin College’s mission is‘Algonquin College will prepare students to achieve academic and career success’.

Program learning outcomes form the foundation of any program offered at Algonquin College. They provide the structure from which programs are developed and renewed, and as such, a review of the program’s learning outcomes is a fundamental component of PQR.

For College Certificates developed before 2003, program learning outcomes may not be defined for the program. If this is the case, then draft outcomes must be developed, and the plan to

29

review, approve and update curriculum based on the approved outcomes should be captured in the PQR implementation plan.

Where program learning outcomes are already defined, these are routinely reviewed by Faculty and with the Program Advisory Committee if established. During PQR these learning outcomes are examined to ensure that they remain valid and are at the level appropriate for the credential, and appropriate for the occupational requirements of program graduates (Curriculum Mapping and with the External Stakeholders). Any plans to revise the program learning outcomes or curriculum are captured as part of the PQR implementation plan.

The Credentials Framework is included in Appendix B of this guide for reference. Care should be taken to review all the expectations for the credential offered – duration, structure, General Education requirements etc. This information will be useful for responding to other quality criteria elements, for example #3.1, #3.2.

In a situation where curriculum does not meet the College or Ministry Standard in any area, a plan to ensure the curriculum will conform in the subsequent academic year must be clearly identified in the recommendations and implementation plan. Changes will be finalized during the annual curriculum review period.

1.2 Program learning outcomes are consistent with MTCU Provincial Program Standards where they exist.This element is not applicable for College Certificates.

1.3 Program learning outcomes are reflected in course outlines.This will be examined during Curriculum Mapping.

As per Policy AA26 – Course Outlines and Course Section Information the course outline must include the applicable program learning outcomes that are supported by the course. . It is an important part of PQR to review to ensure that these outcomes are easily identifiable and valid.

Again, for those programs without defined program learning outcomes, the plan to define and implement them should be captured within the PQR implementation plan.

1.4 Program learning outcomes are used in prior learning assessment and recognition (PLAR).As part of annual curriculum review, course outlines are individually reviewed using the course outline review worksheet or equivalent. PLAR options are recorded as part of each course outline review and the findings are documented on the Course Outline Review Summary sheet. NB: how a student can challenge the course using PLAR needs to be identified on each course outline.

30


As each course is reviewed, it needs to be determined how faculty can evaluate the outcomes for the purposes of PLAR. Usually a department has two weeks to set up an appropriate challenge. Therefore, it is prudent that the challenge be determined in advance. The two weeks lead time allows for the updating of the challenge.

1.5 Changes to courses and program learning outcomes are introduced on a timely basis and are designed to maintain the relevance of the program.

Against this element, describe how the department determines what updates are necessary and how they are incorporated into the curriculum. For example, state as a result of feedback from student program councils, agency staff, and advisory committee members, changes are made annually as part of the annual curriculum review process. Refer to relevant meeting minutes to indicate how this information was gathered and used.

1.6 The capabilities of program graduates, including knowledge, understanding, skills, and attitudes are consistent with the intended program learning outcomes.

Describe how the program provides experiential learning opportunities that allow the learners to perform, with support, as a practitioner in their field. Discuss planned learning activities that allow the student to perform as a practitioner.

Review the evaluation strategies for courses in the final level of the program. Are the students ready to function in the workplace at the entry level expected? This is an important question to pose at the external stakeholders focus group meeting.


2.1 The qualifications and prerequisites required of the applicant are published and are appropriate to allow the student to be successful without limiting access to the program.

To respond to this element, determine what is published and practiced for the program.

Prospective students in a College Certificate program apply on a course by course basis through the Centre for Continuing and Online Education. The program monograph indicates whether courses are to be taken in a specific order, and prerequisites and additional skills/knowledge required of the applicant are indicated for each course either in the monograph or on the CCOL webpage.

Students declare completion of the program once they have satisfactorily completed all courses in the program, using an application form available at http://www.algonquincollege.com/ccol/psCE/graduate.htm.

31

http://www.algonquincollege.com/ccol/psCE/graduate.htm

http://www.algonquincollege.com/ccol/index.htm

In order to assess this element, the Team Leader needs to review the current admission requirements with faculty, students, and the external stakeholder groups. Faculty teaching in the program will be able to provide valuable input about the degree of preparedness of the students entering each course. In addition, review the failure rates for courses in the program. If there are courses with consistently high failure rates, admission criteria may need to be reviewed or the course content may need to be adjusted.

Students in the student focus group will give their perspective about the admission criteria and how well they feel they were prepared to be successful in each course and the program overall.

Some programs offer remedial opportunities to assist students with specific deficiencies when the department does not want to inhibit access to the program, but recognizes that students need assistance to be successful. These opportunities should be highlighted in the report indicating the contribution they make to student success.


This information is published in the Program Narrative section of the Program Monograph. It is essential that the information given to applicants allows them to understand what skills they will learn, and how the program will prepare them for the workplace, as well as for which jobs in the workplace.

The items within this element should form key discussion points for the Student Focus Group. The program narrative can be reviewed with students. For example, ask Did they find the program description useful? How useful are the success factors and the employment information in helping the student with decisions regarding selecting the right program?

The members of the external stakeholders focus group will be able to provide feedback regarding the clarity of the Program Monograph.


Several College publications identify the process for students to follow to obtain credit for previous education and experience. A key item for discussion at student focus group meetings is whether students knew where to find this information and if they found this information useful. It is important to ask this question to students who did apply for credit to determine if they received the assistance they needed in a timely fashion, as students who did not need to apply for exemptions may not be knowledgeable on the process.

Course Assessment Roll Up report question #23 (post-2008) provides feedback on credit exemption.

32


Students need to know what the expectations and obligations are for them to advance in the program. This might include the necessity to submit health and immunization forms, proof of a driver’s licence, or a police records check. Identify any special requirements the program might have, ensuring that these are in line with College guidelines, and note how the students are informed.

The program may also have special progression requirements, for example, students need to obtain a higher grade than just a pass to proceed on to a course for which it is a prerequisite. Identify and review these unique situations to ensure they are legitimate requirements. Note this information, and how it is communicated to students.

2.5 Students know how they will be evaluated.

Students need to understand what to expect regarding evaluation methods, how to receive special accommodation, how to appeal a grade, and how progression is determined. Describe how the department informs students regarding assessment and progression. Examples of unique situations where students have received special accommodation can be briefly discussed. For example, cite information included in letters to students/applicants, within course descriptions, and course outlines. The course assessment roll up report question #4 (pre-2008 survey) and #19 (post-2008 survey) will indicate whether or not students feel the evaluation methods are clearly communicated.

2.6 Students indicate the learning requirements are relevant and meaningful.

Student feedback is available in the Course Assessment Roll Up Report with questions #1.6 and #5 (pre-2008 survey) and #18 and #21 (post-2008 survey). This may also be a question at the student focus group meeting.

2.7 Students indicate that assessment methods relate to the learning requirements.

Student feedback is available in the Course Assessment Roll Up Report with #2.1 (pre-2008 survey) and #13.1 (post-2008 survey). This may also be a question at the student focus group meeting.



The Credentials Framework document is included in Appendix B of this guide. Compare the program duration, structure and outcomes to the framework to ensure the program is in the appropriate credential level. If review shows that the certificate more closely resembles and

33

Ontario College Credential, including a Graduate Certificate, it may be appropriate to develop a proposal to alter the credential. The PQAA can provide guidance if this is the case.

The comparisons above will have been made during Curriculum Mapping.


It is important for faculty to ensure that courses are organized and sequenced to support student learning, are allocated appropriate hours of study, and are consistent with published requisites. (Note that this review is formally part of annual curriculum review and will be highlighted during Curriculum Mapping.)

A review of the courses which have a failure rate of more than 25% might help identify areas where students may need more foundational knowledge or where content needs to be reviewed.

Opportunities for graduates to move into the next credential either at the College or at university, opportunities for receiving external certifications etc, should be noted within the final PQR report. Also opportunities to specialize or select majors should be discussed if applicable.

3.3 Prerequisites do not unnecessarily hinder progress in the program.

If a College Certificate has prerequisites, this is indicated on the program monograph. For programs with prerequisites, it is important to review their necessity, how they are enforced, and what happens if the student does not have the prerequisite.

If there are situations where students, as a result of failing a prerequisite course, cannot continue in the program without waiting a whole year, it is valuable to review the prerequisite to ensure that the knowledge is indeed foundational and the student is not likely to be successful if allowed to progress without the prerequisite. When programs have identified strategies to help students stay in the program even though they may have to repeat a course, it should be noted here.

It may be that a prerequisite is actually required, but not present. A review of the courses with high failure rate may help to determine this.

3.4 The program title is consistent with college system nomenclature / titling principles.

College Certificates should be uniquely named in that they should not be identical to that of a program resulting in an Ontario College Credential either at Algonquin College or elsewhere in the College system. The title should clearly reflect the nature of the program’s vocation.

34

Note against this element if the title is shared with other credentials/institutions. Also note if an alternative name is attributed to the program, for example, for marketing purposes. If the name is shared, a recommendation may be to update the program name. The PQAA can advise in this circumstance.

3.5 The program has established agreements for further study, e.g., articulation agreements.

Review the laddering opportunities that exist for the program, to ensure that they are both current and have the correct scope. This includes both post-program opportunities (e.g., laddering into an Ontario College Credential) and any pre-College laddering opportunities, e.g., School to College to Work Initiative (SCWI) projects occurring in partnership with the program (i.e., program specific Specialist High Skills Majors (SHSM) or Dual Credit initiatives).Include a list and a brief description of laddering opportunities that exist.

Articulation agreements are not typically established for College Certificates, but any currently in place for the program should be reviewed to ensure that any major curriculum changes made since the last review have been reflected within the agreement. Include a list and brief description of any articulation agreements that exist with other postsecondary institutions in the PQR report. If no articulation agreements currently exist and graduates from the program regularly seek university education, the implementation plan may include a recommendation to consider establishing an articulation agreement.

Information on degree pathways is found at: http://www.algonquincollege.com/prospective/programs_degrees.htm.

3.6 The program conforms to the College policy for the number of English courses.

Although not formally applicable to College Certificates, employers do expect College graduates, regardless of the nature of the credential, to have a minimum proficiency in English. Describe here how English skills are assessed, if essential to your program.

3.7 All curriculum documentation is up-to-date including course outlines and the program monograph information.

As part of annual curriculum review, course outlines are individually reviewed using the course outline review worksheet or equivalent. This review identifies course outlines that need to be updated. Any work required to ensure course outlines are complete will be discussed with the Academic Administrator and captured in the Implementation Plan.

Program Monograph information is updated each year. If any areas have been designated as requiring modification as a result of this PQR, state what changes are being implemented. Otherwise, indicate the changes that are being made as a part of the annual update process.

35

http://www.algonquincollege.com/prospective/programs_degrees.htm

3.8 There is congruency between the course learning requirements and the program learning outcomes.

This will be shown during the analysis of the curriculum maps as per the Curriculum Mapping section of this manual. Faculty who teach in the program will be able to provide valuable insight into the sequencing of the courses and the culmination of the various course learning requirements to the program learning outcomes. Curriculum maps will demonstrate the distribution of the program outcomes across all levels and indicate if any areas need consideration, for example, where learning outcomes have more than adequate coverage the teaching may need to be balanced.

Questions #1.6 and #3 in the Course Assessment Roll Up report (pre-2008) and #12.7 (post-2008) may provide more student insight.

3.9 There is a match between course learning requirements, course learning activities, and learning resources.

Following directly from element #3.8, this is an important element for faculty to review as part of the Course Outline Review process. The match should be obvious. Additionally, there are several questions on the Course Assessment Roll Up Report that will provide student input: #2 (pre-2008), and #13 and #14 (post-2008).

3.10 Concepts of social, economic, and environmental sustainability are embedded in the program curriculum.

Describe how concepts of sustainability are, or are planned to be, introduced into the program curriculum. If embedding of sustainability into the curriculum is a new area for consideration, faculty need to discuss plans with the Academic Administrator. The resulting implementation plan may recommend discussions at advisory committee meetings (or equivalent) for the upcoming year to gauge required changes, with updates to the curriculum to be implemented in the following year.

As per the Algonquin Experience and the College Strategic Plan, in addition to embedding sustainability within the curriculum, faculty also need to identify how they will role-model sustainability concepts by embedding them into their day-to-day teaching. A three year faculty plan can be included in the PQR implementation plan.

3.11 Off-campus experiential learning components are embedded in the program curriculum.

As per the Algonquin Experience, programs should include an experiential component, characterized by practical, hands-on learning through co-operative education, clinical/field placement, work placement, field work and internship opportunities.

Describe how experiential learning is, or is planned to be, embedded into the program, and carry forward into the PQR implementation plan as appropriate.

36

http://www.algonquincollege.com/VPAcademic/TheAlgonquinExperience.pdf


3.12 Students have opportunities to develop the skills, knowledge, and attitudes necessary to succeed in a global economy.

Describe how concepts of cultural diversity, team work, flexibility, transfer of skills, interpersonal skills, problem-solving and decision-making ability, adaptability, creative thinking, self-motivation, the capacity for reflection, etc. are incorporated within the program curriculum13. Course outline review may highlight some cases where there are no specific learning outcomes or course learning requirements that directly reflect this learning. This should be noted against the element and the plans to update the specific course outlines should be included in the implementation plan.

Again, this element aligns with the expectations of the Algonquin Experience.


4.1 Program delivery, including that which takes place off-site, is consistent with the nature of the program, the learning outcomes, and the needs of the students.

This element should be discussed with students and faculty. With the broad mix of demographics in the classroom, curriculum design needs to be flexible. As well, students are not equally prepared to learn online, in groups, or using independent studies. It is important to note successes as well as challenges faced in the classroom.

If the program is delivered on more than one campus, discuss how regional differences are accommodated. (Aside: as much as possible, programs delivered at multiple campuses are reviewed in the same PQR cycle. Regardless, curriculum is mapped to the same outcomes.) Outline how communication between faculty of different campuses is facilitated.

If the program contains courses that are part of a full-time (day) credential, outline how communication between the programs’ faculty is facilitated to ensure that curriculum remains aligned.

If a program is delivered off-site, describe accommodations made to ensure there is consistency in the ability of the students to demonstrate the program outcomes. This could include specifications as outlined in the Memorandum of Understanding and the process for ensuring that off campus graduates truly meet the learning outcomes.


The Team Leader and faculty review the course outlines and the information in the course outline review form to evaluate this element for each course. Faculty need to discuss the current offering of learning activities, and determine whether the variety of activities is sufficient for the learners in each course.

13 Especially in Ontario College Credentials, through the embedding of Essential Employability Skills outcomes

37


State any anecdotal information about special accommodations that the program implements to aid student success. For example, discuss strategies used to engage learners, support special needs, or provide valuable learning opportunities.Question #14.1 in the Course Assessment Roll Up report (post-2008) may provide more student insight.

4.3 Learning methods are published and are matched to the learning outcomes.

The Course Assessment Roll Up report has student input for this element in questions #3 (pre-2008 survey) and #17 (post-2008 survey). The course outline review will also identify any areas where a match does not occur and changes are needed.

4.4 College designated targets regarding hybrid courses are met.

This element is Not Applicable for College Certificates as specific hybrid targets are not yet set for this credential.

However, any opportunities that exist for online learning in the program may be captured here. Where both in-class and online learning is available a comparison of the ratio taking and succeeding in each environment may assist in improving the quality in the program.

4.5 Learners are provided the skills necessary to be successful with the learning strategies selected.

As part of annual curriculum review activities, faculty review course learning activities and ensure that relevant skills are introduced to ensure students have the necessary skills to be successful.

Document how faculty provide orientation to online learning, or, if group projects are required, how orientation to group work is provided. Strategies used for any other skills needed by students to be successful learners may also be captured. If gaps are noted, captured the plan to address in the PQR Implementation Plan.

It is important to review the failure rate report for courses where special skills are required and identify whether students are successful.

4.6 Evaluation criteria are published and students are aware of how and when they are going to be evaluated.

The Course Assessment Roll Up report has student input in question # 4 (pre-2008 survey) and #19 (post-2008 survey). The course outline review process will also identify any gaps. Penalties for late assignments need to be reviewed to ensure that penalties are consistent across the program or that any differences in specific courses or course sections are justifiable, published, and implemented equitably. Briefly describe policies and how they impact student success.

38

4.7 There is a match between course learning requirements and evaluation methods, i.e., evaluation methods allow students to demonstrate the course learning requirements.

This match is assessed during course outline review. Student input for this element can be found in the Course Assessment Roll Up Report in question #1.5 and #2.1 (pre-2008 survey) and #12.5 and 13.1 (post-2008 survey).

It is important to include examples about the evaluation methods used and how they allow students to demonstrate the outcomes. Highlight special accommodations made or innovative methods used by the program faculty.

4.8 Evaluation methods are valid and reliable.

Describe processes that faculty use to ensure that evaluation methods are reliable and valid, for example, using rubrics, examining negatively discriminating questions on multiple choice examinations prior to submitting final grades, or the review of exam outcomes to determine questions that students answer poorly.

Where courses have a failure rate of 25% or higher, a plan is to be made to review the evaluation strategies.

4.9 Students indicate that feedback is timely and allows them to build on their learning.

The Course Assessment Roll Up data is limited but some insight is given in #2.1 (pre-2008 survey) and #13.1 (post-2008 survey), and can be asked at the student focus group. If the students indicate that feedback is not timely or helpful, note this in the PQR report being careful to maintain confidentiality. Recall that PQR is examining the program as a whole. Should matters arise that pertain to individuals, please work with the Academic Administrator/PQAA to ensure confidentiality is preserved.

4.10 Students perceive evaluation to be fair.

The Course Assessment Roll Up report offers student feedback in question #1.6 (pre-2008 survey) and #12.5 (post-2008 survey). This question may also be discussed at the student focus group meeting.

4.11 Resubmissions, supplementals, and appeals are published, appropriate, fair, valid, and consistent.

Describe the program policy and how it is published and implemented. In particular, department practices can be compared to Policy AA13 - Evaluation of Student Learning and Directive E29 - Program Progression and Graduation Requirements.

4.12 Student workload and assessment is balanced across the term at both the course and program level.

Discuss here the efforts faculty members make to ensure a balance of workload and assessment of all courses. For example, these can be mapped each term to ensure a balance. Student feedback can be found on the Course Assessment Roll Up Report for question #15 (post-2008 survey) and may be requested at the student focus group meeting.

39

http://www2.algonquincollege.com/directives/policy/program-progression-graduation-requirements/

http://www2.algonquincollege.com/directives/policy/aa13-evaluation-of-student-learning/

As most College Certificates are taken on a part-time basis, workload is usually balanced by the learner, and their choice of how many courses to take each term. Discussions may reveal an isolated course with a higher than expected workload. If this arises, then the course may be reviewed and possibly revised.

4.13 There is a range of evaluation methods used consistent with a variety of learning styles.

This element will need to be discussed with both faculty and students. The course outline review worksheets provide examples of the range of methods used. In the PQR report, highlight methods used that demonstrate meeting the needs of various learning styles, and highlight any innovative evaluation strategies used.

4.14 Learners can earn credit for up to 75% of the program hours using the PLAR process.

PLAR challenges are identified on each course outline. To determine whether the 75% threshold is being met, tally the number of hours in each course with documented PLAR challenges and compare this to the total number of program hours.

If the program has less than the required number of hours with PLAR challenges available, note this against the element, and include a recommendation to address the shortfall in the PQR implementation plan.

Note that PLAR challenges need to be established using the existing College and Ministry policies and fees structures. Assistance to define the challenges is available from the Centre for Organizational Learning. Recall that, on the course outline itself, it is only necessary to identify whether the course is challengeable, and to provide contact information. The actual challenge format should be identified and planned, although time will be needed and allowed to ensure the challenge tool is updated.

Aside: Once a challenge process is in place for a course, the PLAR flag needs to be set within GeneSIS for the course.

4.15 Academic policies and practices that provide for the development and continuous improvement of teaching and learning methods are valued, documented, and supported.

Describe how program faculty have experimented with new teaching methods and how they evaluate the effect on student success, and describe how the department supports teacher development. The department PD plan can be referenced here.

Identify any relevant Professional Development that faculty have attended and found useful. This can include how many faculty members have taken the TALL (Teaching Adult Lifelong Learners) program, and the part-time faculty orientation. This information will be found in the summary of faculty credentials and experience summarized by the Academic Administrator.

It is helpful to identify areas the faculty feel could be improved to help them develop their teaching skills, especially for Blackboard. If students identify online courses as challenging, a follow up action may be to assign PD for faculty to improve online course delivery.

40

http://www.algonquincollege.com/col/PTfacultyOrientation.html?sectionID=3

http://extraweb.algonquincollege.com/ce_programs/programOverview.aspx?id=6143X02PWO&

http://extraweb.algonquincollege.com/ce_programs/programOverview.aspx?id=6143X02PWO&

http://www.algonquincollege.com/col/

http://www.algonquincollege.com/col/

Should the faculty wish to test and measure a new learning strategy, opportunities may be available through Learning and Teaching Services, in particular, the Innovation Station. Plans can be captured in the PQR implementation plan.


5.1 Program faculty members, as a whole, have adequate academic preparation and workplace experience to deliver a quality program.

Information regarding the preparation and experience of the faculty team is gathered by the Academic Administrator, who submits a summary report to the Team Leader. Student perceptions of faculty knowledge and skill are available in the Course Assessment Roll Up Report questions #1.1-1.7 (pre-2008 survey) and #12.1-12.8 (post-2008) and may be gathered at the focus group meetings.

Describe the steps taken to ensure the quality of the faculty for programs delivered off site.

Any concerns that faculty have should be identified and considered in the recommendations. Similarly, celebrate the experience and expertise that is available to support student learning.

5.2 Program faculty members are formally evaluated every three years.

Data to support this element is within the summary report that the Academic Administrator will submit to the Team Leader. Summarize the data in the final report.

Describe how program faculty are evaluated both formally and informally, and how student feedback is provided to the faculty. The Academic Administrator can assist with this response.

5.3 Program faculty members engage in professional development activities that ensure they are current in their field and developing teaching expertise.


Feedback should also be solicited during the Faculty focus group meeting regarding the availability and usefulness of professional development.

5.4 Program faculty members work within clear and well-structured instructional plans.

All courses are expected to have documented course outlines and course section information within which faculty work. This will be verified during Curriculum Mapping. Should there be any issues at a program level, a plan to address should be made with the Academic Administrator, and captured within the PQR implementation plan.

5.5 Program faculty members participate in reflective practice.


41

http://lyceum.algonquincollege.com/innovationstation/

Feedback could also be solicited during the Faculty focus group meeting regarding the reflective practice methods used, successfully or otherwise, by faculty.

5.6 Students consider faculty to be available.

Some student feedback is available in the Course Assessment Roll Up #1.3-13.1 (pre-2008), #12.3 and #23.22 (post-2008). If this indicates that students do not consider faculty to be available, feedback should be requested from students during the Student Focus Group meeting. Additionally, the Academic Administrator can learn if there is a generic or individual issue as part of the course assessment process. It is certainly not intended that the Academic Administrator should release confidential information; however, when survey data reflects that student perception is that faculty are not available, assurance needs to be provided that any issues regarding faculty availability will be addressed in the Implementation Plan. Ask the PQAA for assistance with the writing of this recommendation if necessary.

5.7 Students consider faculty to be adequately prepared for class.

Some student feedback is available in the Course Assessment Roll Up #1.1-1.4 (pre-2008), #12.1-12.4, #14 and #24 (post-2008). It is necessary here, as above, that any concerns raised by students are discussed in a confidential manner and are addressed in the Implementation Plan accordingly. The PQAA may be of assistance when determining how to document areas which need to be addressed.

5.8 Students consider faculty to promote a positive attitude to learning.

A positive attitude to learning is considered to be an essential aspect of a quality learning environment. Please capture, if overall, the students consider program faculty to promote a positive learning environment. This question can be posed at the student focus group. Again, if there any concerns arise, these should be discussed in a confidential and respectful manner, and the PQAA is here to assist in documenting appropriately.

5.9 All students are assigned an academic advisor.

Describe how academic advisors are allocated for the program.

5.10 Academic Advisors contact their students early in the term with an invitation to meet and to ensure that students know who their advisor is.

Describe how students are informed about their academic advisor and in general, how the program implements Policy AA40 – Academic Advising. Although this question is not directly related to academic advising, if students are expressing dissatisfaction, a recommendation may need to be developed and implemented. This is a good topic for the Student Focus Group. Remember that not all departments use the same terminology, therefore a more generic question can be posed, like “Do you know where to get help if you have a problem with your program?”.

42


5.11 Labs, clinical facilities and placement facilities are complementary to, and integrated into, the program and allow the learner to demonstrate the learning outcomes.

Discuss how classrooms are used, types of labs available, and placement activities used to provide learning opportunities for students. Celebrate quality facilities and identify changing or future needs that may have been identified.The Course Assessment Roll Up Report provides student feedback in questions #13.9 (pre-2008) and #23.19 and #24.3 (post-2008 survey).

5.12 Students indicate that there are adequate and accessible learning resource materials to allow them to be successful, including: textbooks in the bookstore; online materials; print resources; equipment; and student support services.

The Course Assessment Roll Up Report provides student feedback in section #12 and #13 (pre-2008) and #23 (post-2008). Analyze components and trends.

5.13 The program is financially viable.

5.14 The demand for the program has been sustained for the last five years.

5.15 There is a future demand for graduates of this program.

Conclusions for these three elements can be drawn from the Registration and Revenues data. Registration and withdrawal statistics from the last 5 years will help identify any significant areas that may require follow up in the focus groups.

A review of the graduation statistics for the program will show how many students are completing the program and applying to graduate. If the graduation rate is low, but course registration is high, consider

are all successful students applying to graduate? does the program meet the needs of the students/employers? is the course mix correct for the current program?

The external stakeholders’ focus group, and the definition of the program level learning outcomes, will assist with responding to this element.

5.16 The learning environment is safe.

This question must be asked at the Student Focus Group meeting. If students identify areas where they feel unsafe, strategies to resolve issues should be discussed and feedback given. It is important that students feel safe asking questions, i.e., that the classroom environment is conducive to students being comfortable sharing their learning needs.

Student perception of the learning environment is found in the Course Assessment Roll Up report in question #43, 44, 45 (post-2008 survey).

43

5.17 The students are provided with the information they need to know to function safely in both the College and workplace learning environments.

Describe any special orientation processes provided for learners to help them function in the classroom, lab, or workplace environments. Any special precautions taken in lab settings, for example WHMIS protocols, and use of safety equipment can be discussed here.


6.1 Students indicate that they are satisfied with the program.

The Course Assessment Roll Up Report may be used to identify student satisfaction with program quality, especially question #5 (pre-2008 survey) and more directly with #20, 24 (post-2008). Any questions on this report which indicate student dissatisfaction may identify issues that need to be discussed at the student focus group meeting.

6.2 Issues raised at Program Councils are addressed in a timely fashion and feedback is provided to the Council.

Discuss Program Council (or equivalent) activities here, and how that process works in the program. A review of Program Council minutes can provide supporting data that indicates that issues from one term are followed up until resolved or identified as no longer needing action. The discussion of the feedback mechanism that is in place for the academic staff to communicate to students across the program can also be discussed.

6.3 Learners progress through the program, achieve program outcomes and graduate in a timely fashion.

When considering this element, examine retention, progression and graduation rate for the program.

Registration reports for the last five years help identify any terms in particular that appear to inhibit a student’s progress through the program.

Identification of courses which have high failure rates may need to be reviewed and strategies recommended that allow the students to be successful while maintaining the academic integrity of the program. Faculty and students can provide valuable feedback for this element.

Concerns here also include core courses that are only offered once a year. Consider whether there are strategies available to allow students to continue without having to lose a whole academic year if they fail the course. Additionally, consider the timing of when the courses are offered – does it match prospective student availability?

In cases where students are not successfully progressing through the program, endeavor to establish the cause – curriculum, financial, other inhibitor – in order that consideration may be made to provide additional support and/or adapt the curriculum.

44

6.4 Learners with a wide range of abilities demonstrate the expected learning outcomes.

While no statistics are available for the number of students with differing abilities and their success in the program, faculty and students may provide some anecdotal information relevant to this element. Highlight special accommodation that program staff members have made to assist students with special needs to meet the program outcomes and be successful in graduating from the program.

6.5 Items raised by the Program Advisory Committee are considered in a timely fashion and feedback is provided to the Committee

Although most College Certificate programs do not have an independent advisory committee, some do, or partner with similar/relevant full-time programs. Feedback from such meetings should be reviewed as part of PQR.

Include specifically how the relevance of the program to the field of practice it serves is assessed, and the program modified if required. A review of Program Advisory Committee minutes (or equivalent) can provide supporting data that indicates that program items are followed up until resolved or identified as no longer needing action.

6.6 Graduates are satisfied with the overall program experience.

No specific survey data is available for this element. Ensure that recent graduates are invited to the External Stakeholders focus group meeting and that their level of satisfaction with the program is captured.

6.7 Graduates are obtaining employment in their fields.

Recent graduates can give insight into this question. Determine if the College Certificate helped them gain, or advance in, current employment.

6.8 Graduates are successful in obtaining external licenses or credentials where relevant.

Include this information and its relevance as applicable. For example, if the students write a licencing examination to obtain provincial or national certification and the results of the success of your graduates on those examinations are available, include them here.

6.9 Employers14 are satisfied with graduate performance.

This question should be asked at the External Stakeholders focus group. Garner information about the strengths graduates possess, and any areas in which they may be better prepared for the workplace.

14 A listing of Employers to consult may be provided through the departmental office or the PQAA.

45

CONDUCTING FOCUS GROUPSThe memos and templates mentioned in this section are found in Appendix C.

Focus groups are held to gather feedback from the participants about their perception of the program’s effectiveness. Once the Team Leader and the Academic Administrator have reviewed the available survey data and program documentation, they determine which elements of the evaluation criteria require additional input from members of the focus groups and prepare the agendas for the specific focus group meetings.

Each program Team Leader plans the focus group meetings and prepares the agendas for their own program meeting, and, excepting the faculty focus group, facilitates the meetings for a Team Leader from another School. The report of the focus group meetings will be written by the facilitator, with an opportunity provided to the program Team Leader and Academic Administrator to review the report for accuracy before it is finalized.

Note: Facilitator partners need to work together to set meeting dates and room logistics for shared focus group meetings,

The following pages contain detailed directions to assist in planning and facilitating a focus group meeting and writing a report on the feedback obtained. It is understood that facilitators will dress and act professionally for these meetings. In the event of illness, the Academic Administrator acts as a facilitator, so please ensure that each Academic Administrator is informed of upcoming focus groups. Also, if a focus group is not feasible, an alternate means of collecting the information may be considered, for example, through an online survey. The survey would be composed by the program Team Leader, but conducted, collated and written into the report by the partner facilitator.

Memos from the Academic Administrator and Team Leader At the outset of the PQR, the Academic Administrator sends a notice to the faculty to introduce the Team Leader, and to ask each staff member to fill out a faculty credentials and experience sheet.

Once the Academic Administrator has sent the PQR introductory notice to the faculty, the Team Leader contacts faculty members to invite them to the faculty focus group meeting, and to prepare by filling out the program self audit form, and to bring the most recent review of each program course that they teach.

The Academic Administrator will also send out an invitation to the External Stakeholders (normally the advisory committee plus some recent graduates and relevant employers) to attend a focus group meeting for PQR. The invitation to the student focus group meeting can be sent by the Team Leader.

Faculty Education, Experience, and Professional Development SurveyAhead of the faculty focus group, each member of faculty fills and sends to the Academic Administrator a credentials and experience sheet. The Academic

46

Administrator then summarizes this and sends it to the Team Leader, who uses it to prepare for the upcoming focus groups, and to respond to the elements of the evaluation criteria.

Program Self-AuditThe program self-audit provides faculty teams an opportunity to review events, successes, challenges in the program within the context of the quality expectations of a College Certificate. Ahead of the faculty focus group, each member of faculty assesses the program against the quality criteria, and provides their assessment to the Team Leader to summarize, and use to prepare for the upcoming focus groups. The program self-audit summary is included in Appendix B of the final PQR report.

Course OutlinesEach course that contributes to the program needs to be self-assessed ahead of the faculty focus group meeting15. A tool for course writers (the “Course Outlines Check Up” tool, available in Appendix C) can be used to review each course, within which there is an option for peer review. The review of course outlines forms part of curriculum mapping. Course section information should also be self-assessed in order to respond to element #5.4 and possibly #5.5.

Setting the Focus Group Agendas An example agenda for each focus group is provided in Appendix C. Agendas are adjusted based on the findings from the document review and data analysis performed ahead of the focus group meetings. Discussion points are defined or expanded to address gaps or ambiguities. Where there is adequate evidence that an element of a criterion is met, it does not need to be brought forward to the focus group.

Tips for Planning and Facilitating Focus Groups Preparing and Planning for the Meeting

Objectives - identify the major objectives of the meeting Questions - develop lead questions

o Use five or six questions from the list of possible items for discussion, provided for each focus group

o Focus on areas that assist in projecting the future of the programo Identify areas of improvement within the program

Date and Time - set date and time (two hour meeting plus social/networking time) Location - reserve a conference room and order light refreshments Invitees - generate guest list and send invitations (four weeks prior to the meeting) Agenda - set the agenda based on the information gathered to that point (begin with an

simple topic) Materials - determine the meeting recorder and prepare/order necessary materials (flip

chart paper or laptop/projector, markers, etc.) Create name tent cards for participants

15 Course outline review happens annually as part of Annual Curriculum Review

47

Facilitating the Meeting Thank you – thank guests for their participation Introductions – establish positive rapport and comfortable environment by taking the time

for personal introductions Overview – review the meeting agenda and purpose and set the ground rules Questions – ask each question and record the responses. Listen attentively and request

clarification as necessary then reflect back to ensure the notation is accurate. Participation – encourage participation by all attendees Timing – be mindful of the meeting duration, allowing time for review and

recommendations. Closing – thanks guests once again for their valuable feedback and close the meeting SMILE – be positive and face the group as much as possible

Post Meeting Tasks Organize – gather notes, number pages, and comment on any important observations Report - write the report within two weeks Ensure that focus group participants are provided feedback on the meeting.

Faculty Focus Group The Team Leader will convene a meeting of as many faculty members who teach in the program as possible, including part-time faculty and professors of service courses. Part-time faculty members will be reimbursed for attending this meeting as per guidelines earlier in this document.

Conduct of the Faculty Focus Group sessions should be standardized as much as possible for all programs. The agenda for each session will closely resemble that in Appendix C.

Ahead of the faculty focus group meeting, all faculty must provide a completed Program Self-Audit, which the Team Leader will

have summarized in order to set discussion items for the agenda. course teachers are to provide a review of their course, using the course outline

summary sheet or equivalents

The main agenda items for the faculty focus group meeting are reviewing the mapping of Vocational Learning Outcomes to courses. (Where Vocational

Learning Outcomes are not defined, an initial draft should form part of the curriculum review exercise.)

review of Course Outlines against the 5 course specific elements of the quality criteria (as per section Curriculum Mapping)

discussing issues (arising from Program Self-Audit)

A second faculty meeting may be planned after all the focus groups have been completed, depending on the number of faculty on the team. At this time, the Team Leader and the Academic Administrator can share with faculty recommendations made at the external stakeholders focus group or the student focus group meeting. The input from this meeting,

48

should it be held, would be documented in a separate report and included in Appendix B, and conclusions and final recommendations formulated for the final report

Facilitating the focus groupThe Team Leader opens the meeting by welcoming everyone and providing an opportunity for introductions as necessary. After a brief overview of the Program Quality Review process, the Team Leader then reviews the Agenda, and explains the process and ground rules.

The ground rules for the discussion are:

Ask that only one person speak at a time. Members are asked to be as concise and focused as possible. All comments/perceptions should be freely stated and will be recorded by the Team

Leader/recorder All contributions are regarded as valid and no consensus is required with respect to the

perceptions and feedback stated and recorded. The Team Leader will attempt to achieve consensus only with respect to final

recommendations to be made to the program.

All elements of the discussion should be recorded during the meeting. The Team Leader may enlist the help of an external recorder, if one is available in the department. The meeting notes will serve as a permanent record of the discussion and will provide the basis for the focus group report.

Preparing the focus group reportThe Team Leader will prepare the report for the meeting which s/he has facilitated, using the notes from the meeting as the basis for the report. The Academic Administrator and the Team Leader must meet to review the report before it is finalized.

The faculty focus group report includes

the agenda, date, and time of the meeting the names of the faculty members who attended the meeting summary of the meeting – the highlights, key findings, and recommendations.

The final report does not contain the detailed discussion notes taken by the Team Leader/recorder.

External Stakeholders Focus Group Planning the External Stakeholders Focus Group The program Team Leader and the Academic Administrator meet to determine who to invite to the external stakeholders focus group. Participants include relevant Advisory Committee Members, additional employers, recent graduates and other guests as determined by the Academic Administrator and Team Leader. Faculty may be invited to the meeting. The Academic Administrator sends the invitation. A draft invitation is included in Appendix C.

49

The program Team Leader prepares the agenda, including discussion questions, for the external stakeholders focus group meeting, however Team Leaders facilitate the external focus group for a program other than their own. Conduct of the external stakeholders focus group meetings should be standardized as much as possible for all programs. The agenda for each meeting will closely resemble the sample included in Appendix C.

Note that the focus group leader will prepare the report for the meeting which s/he has facilitated, using the meeting notes as the basis for the report.

After the focus group meeting has been held, a letter is sent to each participant by the Academic Administrator to thank participants. This letter can include

a brief summary of the recommendations a note that these recommendations may be included in the final implementation plan how the department will inform the participants of the availability of the focus group

report.

Time Frame for the MeetingThe working part of the meeting should be about two hours in length. Refreshments should be available at the beginning of the meeting to minimize interruptions once the meeting commences.

Leadership of the MeetingThe Academic Administrator of the department states the purpose of the focus group meeting, introduces the focus group leader, and invites the participants to introduce themselves. The focus group facilitator then reviews the agenda, explains the PQR process and the ground rules for the focus group meeting.

Ground Rules for the Meeting Ask that only one person speak at a time. Participants are asked to be as concise and focused as possible. All comments/perceptions should be freely stated and are recorded by the facilitator /

recorder. All contributions are regarded as valid and no consensus is required with respect to the

perceptions and feedback stated and recorded. The facilitator will attempt to achieve consensus only with respect to final

recommendations to be made to the program. If consensus cannot be reached with respect to a recommendation, this should be noted.

The final decision rests with the Academic Administrator.

All elements of the discussion should be recorded during the meeting. The focus group facilitator may enlist the help of an external recorder, if one is available in the department. The meeting notes will serve as a permanent record of the discussion and will provide the basis for the focus group report. Note that the actual meeting notes are not included in the report.

50

Writing the external stakeholders focus group reportThe focus group leader will prepare the report for the meeting which s/he has facilitated, using the discussion notes from the meeting as the basis for the report. The final report will include

the agenda, date, and time of the meeting. the names and capacity (e.g., the company they represent) of the attendees at the

external stakeholders focus group a summary of the meeting - highlights, key findings, and recommendations.

The intent is to give the reader a clear understanding of the input and viewpoints of the attendees of the meeting, without the heavy overhead of verbatim minutes.

A draft of the external stakeholders focus group report should be verified by the program Team Leader and the Academic Administrator before it is finalized. A copy of the meeting notes should be available for consultation during this review.

The draft report is to be provided by the focus group facilitator to the program Team Leader within two weeks of the meeting and copied to the PQAA. If the program Team Leader wishes the final report earlier, a date can be negotiated.

Possible Items for DiscussionBelow are elements of the quality criteria that may be relevant for discussion at the external stakeholders focus group meeting. Some of these elements may not need to be addressed, as earlier review of the survey data and program documents may have concluded that no further information is required.

Criterion 1Program Learning Outcomes are appropriate for the program’s intended purpose, and the credential offered upon completion.

Does the group feel that the current (or draft) vocational learning outcomes are appropriate for the program and credential?

Are changes in the field anticipated such that the vocational learning outcomes should be changed in the next 5-10 years?

Criterion 2 Processes for granting of internal and external transfer of academic credits are clearly defined and available to students.

Does the group feel that current employees have opportunities to upgrade/receive credit for experience/have access to the courses needed to obtain a college credential?

Has anyone at the meeting used the advanced standing, external credit or Prior Learning Assessment and Recognition process?

Do people know where to look? Do they have suggestions about where they would expect to find it?

51

Students have adequate information to allow them to make informed choices about: selecting the correct program to meet their career aspirations; the financial commitment needed; the workload commitment needed; and the study options available to them.

Do employers have suggestions regarding the employment opportunities (or other program narrative information) as outlined in the program monograph?

Students know what is needed to ensure they will be able to demonstrate program outcomes and complete the program.

Do employers have any suggestions or changes to current job requirements like police records checks or special driver’s licences?

Criterion 3The duration and structure of the program are consistent with the program learning outcomes and the credential offered.

Does the group feel that the program curriculum needs to change in the next 5-10 years in anticipation of a change in the knowledge base needed to work in the field?

Criterion 5Labs, clinical facilities, and placement facilities are complementary to and integrated into the program to allow the learner to demonstrate the learning outcomes.

Does the group have any recommendations about the labs, equipment, and placement facilities?

The students are provided with the information they need to know, to function safely in both the College and workplace learning environments.

Is there any new information the College should know about, for example, WHMIS changes, new safety regulations?

Criterion 6Students are obtaining employment in their field.

What is the perspective of the group with respect to future employment opportunities for the next 5-10 years?

Graduates are successful in obtaining external licenses or credentials where relevant. Do the graduates understand the legal responsibilities of their job? Are they aware of

accreditations required?’

Employers are satisfied with graduate performance. Does the group feel that the graduates have adequate practical experience? Do the

graduates understand the reality of the workforce? Are they able to relate the skills they have learned to the workforce?

52

Student Focus Group Planning the Student Focus Group The program Team Leader invites students to participate in the focus group meeting. It is recommended that 20-30 students be invited, depending on the size of the program. There should be fairly equal representation from all years and all sections/groups. Students can be invited by telephone, email or letter, ensuring that they understand that it is expected that they respond to the invitation. This will allow time to find an alternate if the student cannot attend. Students can be selected randomly from a class list – e.g., every fourth or fifth student, depending on the size of the class. Faculty is not invited to the meeting. A sample invitation is included in Appendix C.

The program Team Leader prepares the agenda, including discussion questions, for the student focus group meeting, however Team Leaders facilitate the student focus group for a program other than their own. Conduct of the student focus group meetings should be standardized as much as possible for all programs. The agenda for each meeting will closely resemble the sample included in Appendix C.

Note that the focus group leader will prepare the report for the meeting which s/he has facilitated, using the meeting notes as the basis for the report. After the focus group meeting has been held, a letter is sent to each participant by the Academic Administrator to thank participants. This letter can include

a brief summary of the recommendations a note that these recommendations may be included in the final implementation plan how the department will inform the participant of the availability of the focus group report.

Time Frame for the MeetingThe working part of the meeting should be about two hours in length. Refreshments should be available at the beginning of the meeting to minimize interruptions once the meeting commences.

Leadership of the MeetingThe Academic Administrator of the department states the purpose of the focus group meeting, introduces the focus group leader, and invites the participants to introduce themselves. The Academic Administrator then leaves, allowing uninhibited discussion to take place. Thereafter, the focus group leader reviews the agenda, and explains the PQR process and the ground rules for the focus group meeting.

Ground Rules for the Meeting Ask that only one person speak at a time. Participants are asked to be as concise and focused as possible. All comments/perceptions should be freely stated and are recorded by the

facilitator/recorder. All contributions are regarded as valid and no consensus is required with respect to the

perceptions and feedback stated and recorded.

53

The focus group leader will attempt to achieve consensus only with respect to final recommendations to be made to the program.

This is an opportunity to discuss global issues regarding the program. Any students who have individual issues regarding faculty are invited to make an appointment to discuss them with the Academic Administrator.

All elements of the discussion should be recorded during the meeting. The focus group facilitator may enlist the help of an external recorder, if one is available in the department. The meeting notes will serve as a permanent record of the discussion and will provide the basis for the focus group report. Note that the actual meeting notes are not included in the report.

Writing the Student Focus Group ReportThe focus group leader will prepare the report for the meeting which s/he has facilitated, using the transcriptions of the discussion notes as the basis for the report. The final report will include

the agenda, date and time of the meeting. the number of students, and their current academic level (it is not necessary to list the

actual names or the participants, level is important as it provides context for the responses)16

a summary of the meeting - highlights, key findings, and recommendations.

The intent is to give the reader a clear understanding of the input and viewpoints of the attendees of the meeting, without the heavy overhead of verbatim minutes.

A draft of the student focus group report should be verified by the program Team Leader and the Academic Administrator before it is finalized. A copy of the meeting notes should be available for consultation during this review.

The draft report is to be provided by the focus group facilitator to the program Team Leader within two weeks of the meeting and copied to the PQAA. If the program Team Leader wishes the final report earlier, a date can be negotiated.

Possible Items for DiscussionBelow are elements of the quality criteria that may be relevant for discussion at the student focus group meeting. Some of these elements may not need to be addressed, as earlier review of the survey data and program documents may have indicated that no further information is required.

Criterion 1Program Learning Outcomes are appropriate for the program’s intended purpose, and the credential offered upon completion.

Did you/ do you know the expected outcomes for your program? Do you know where to locate them? If so, are they what you deem to be appropriate for the program?

Criterion 2

16 Note that the Team Leader will provide the Academic Administrator the list of participants, in order to send the thank you letter indicated earlier

54

The qualifications and prerequisites required of the applicant are published and are appropriate to allow the student to be successful without limiting access to the program.

Was the program description provided useful? Did you know what you needed to know to get into the program? Did you have enough of a Mathematics foundation to succeed in the program (if

applicable)? Did you know anyone who wanted to get into the same program as you but didn’t – and

why?

Students have adequate information to allow them to make informed choices about: selecting the correct program to meet their career aspirations; the financial commitment needed; the workload commitment needed; and the study options available to them.

Does the course content match what you expected to learn? Do you expect these studies will lead to the type of career you are seeking? Were you prepared for the financial and workload commitments needed?

Students know how to get internal and external transfer of academic credits and recognition for prior learning.

Did you know how to get any exemptions for courses taken at another postsecondary institution or here at Algonquin College in another program?

Were you eligible for exemptions or PLAR?

Students know what is needed to ensure they will be able to demonstrate program outcomes and complete the program.

Does your program have special requirements you must have before you can be able to demonstrate learning outcomes?

Was the information you needed readily available?

Students know how they will be evaluated. Is the published information adequately available and meaningful to ensure you know

how you will be evaluated?

Students indicate that assessment methods relate to the learning requirements. Are the assessment methods related to the learning requirements? How do you know?

Criterion 3Prerequisites do not unnecessarily hinder progress in the program.

Are there any courses that seem to obstruct student progress in the program? If so, do you have any suggestions about how this could be improved?

55

Courses are organized and sequenced to support student learning, are allocated appropriate hours of study, and are consistent with published prerequisites.

Does the sequencing of the courses make sense? Do you have the foundational knowledge needed as you progress in the program?

There is congruency between the course learning requirements and the program learning outcomes.

Do you think that each of the courses contribute to the overall learning outcomes?

There is a match between the course learning requirements, course learning activities and learning resources.

Do the learning activities and learning resources support your success in demonstrating the course learning requirements?

Curriculum design maximizes flexibility of student learning. Do you feel that you have adequate flexibility to accommodate your learning? Do you have any suggestions regarding how the program delivery could be more flexible

to support your learning? (What do the students consider to be needed for flexibility – courses available at different times to allow them flexible schedules; different learning offerings, e.g., online as well as in class; multiple resources available, a variety of learning activities; opportunities for PLAR challenges; laddering of curriculum?)

Criterion 4The program provides experiential learning opportunities that allow the learners to perform, with support, as a practitioner in their field.

Do you feel you are provided with adequate opportunities to perform as a practitioner in your field?

There is a range of instruction methods consistent with a variety of learning styles and learner needs and abilities.

Do the instruction methods support your learning style and your needs and abilities? Do you have any suggestions for improving this aspect of the program delivery?

Learning methods are published and are matched to the learning outcomes. Did you know what learning methods to expect and do you feel they match the learning

outcomes?

Learners are provided the skills necessary to be successful with the learning strategies selected.

Do you receive adequate preparation to be successful with the learning strategies being used, for example, online learning or group presentations?

56

Evaluation criteria are published and students are aware of how and when they are going to be evaluated.

(This element may be adequately answered in the student surveys.)

There is a match between course learning requirements and evaluation methods. Do you feel that the evaluation methods allow you to demonstrate the outcomes?

Students indicate that feedback is timely and allows them to build on their learning. (This element may be adequately answered in the student surveys.)

Students perceive evaluation to be fair. (This element may be adequately answered in the student surveys.)

Evaluation methods allow students to demonstrate the course learning outcomes. Do the ways the professors evaluate you allow you to demonstrate the course learning

requirements? Are there other ways that may better allow you to demonstrate your learning?

Student workload and assessment is balanced across the term at both the course and program level.

Do you find the workload to be balanced across the term for all your courses?

There is a range of evaluation methods used consistent with a variety of learning styles. Do the evaluation methods used match your learning style? Can you offer suggestions about other evaluation methods that would suit you better?

Criterion 5Students consider faculty to be available.


Students consider faculty to be adequately prepared. (This element may be adequately answered in the student surveys.)

Students consider faculty to promote a positive attitude to learning. Overall, has this been your experience with your program, and at the College?

Students are assigned an academic advisor. Do you know who your academic advisor is?

Academic Advisors contact their students early in the term with an invitation to meet and to ensure that students know who their advisor is.

Have you met with your academic advisor this term or connected in another way? Do you have any suggestions about how your academic advisor can better support your learning?

57

Labs, clinical facilities and placement facilities are complementary to and integrated into the program and allow the learner to demonstrate the learning outcomes.

Do the labs, clinical, and placement facilities allow you to demonstrate the learning outcomes?

Students indicate that there are adequate and accessible learning resource materials including: textbooks in the bookstore; online materials; print resources; equipment; and student support services, to allow them to be successful.


The learning environment is safe. Do you feel safe in the College? Is the classroom environment conducive to asking questions? Do you feel safe in all learning environments? Do you have adequate supervision when demonstrating new skills in a placement

environment? [If there is an area where students do not feel safe…] Do you have suggestions about

how this could be resolved?

The students are provided with the information they need to know how to function safely both in the College and workplace learning environments.

Do you feel that you are well orientated to practice lab and workplace learning environments?

Are you offered specialized safety training? Can you identify any situations where you felt you were not able to practice your skills in

a safe manner?

Criterion 6Students indicate that they are satisfied with the program.

What parts of the program provide you with the most satisfaction? What parts of the program provide the least satisfaction?

Issues raised at Program Councils are addressed in a timely fashion and feedback is provided to the council.

Are issues from Program Council (or equivalent) meetings addressed and progress communicated back to students? ( Aside: Note that if an issue cannot be addressed, this, with a reason, must be communicated back to the Program Council.)

Learners progress through the program, achieve program outcomes, and graduate in a timely fashion.

Do you feel that the learning outcomes are achievable in the designated time frame for the program?

Have you been successful in all your courses so far? If not, why not?

58

COMPLETING THE FINAL REPORT – AN OVERVIEWAs the Program Quality review is being conducted, the Team Leader should have

written the introduction section of the final report reviewed the available documentation and data responded iteratively to the elements of the evaluation criteria (i.e., after document and

data review and after each focus group meeting was held) completed curriculum mapping identified program Strengths, Weaknesses, Opportunities, and Threats started to identify recommendations.

To close the report, the Team Leader needs to complete responses to the elements of the evaluation criteria write the conclusions and prepare a draft of the recommendations and implementation

plan review the report with the Academic Administrator with the Academic Administrator, develop the Implementation Plan17

with the Academic Administrator, write the Executive Summary send the draft report to the PQAA for review make updates as required, and resubmit to the PQAA for printing and distribution.

The draft report should be submitted to the PQAA and the Program Academic Administrator as a .docx file.

All items, such as the Program Monograph and curriculum maps, should be included electronically into this final report. If administrative assistance is required in compiling the final report, please contact the PQAA.

On receipt of the draft report, the PQAA and the Program Academic Administrator will review and advise the Team Leader of any updates required. Once the report has been finalized, and formal review and sign-off procedure is followed, where the report is reviewed by the PQAA, the Academic Administrator and finally the Dean. Once approved by the Dean, the timing for the annual follow up report is provided. A formal sign-off sheet is used, and is provided in Appendix C.

In addition, unless the program curriculum is stored in COMMS, electronic versions of each course outline and course section information should be collected and submitted to the PQAA along with the PQR final report. The course outlines are not to be bound with the final report.

17 The Academic Administrator is responsible for actioning the recommendations

59

ANNUAL FOLLOW UPEach spring, the PQAA sends out a notice requesting follow up from each Academic Administrator regarding current PQR actions. Academic Administrators summarize the progress made, and update the program implementation plan, adjusting actions based on changes such as College expectations, program budget and availability of human resources.

In addition the Academic Administrator includes PQR recommendations in his/her annual

performance contract. the PQAA presents a summary report which includes the highlights and

recommendations for each program involved in PQR for that cycle. This report is submitted to the Vice President Academic.

The PQAA presents a report which summarizes the progress on open action items from all previous PQRs. This report is also submitted to the Vice President Academic.

60

GLOSSARY AND LIST OF ACRONYMS

CAAT Colleges of Applied Arts and TechnologyCOMMS Course Outline Mapping and Management SystemCCOL Centre for Continuing and Online LearningCOL Centre for Organizational LearningGeneSIS The College Student Information SystemKPI Key Performance IndicatorsMTCU Ministry of Training, Colleges and UniversityOCAS Ontario College Application ServiceOCQAS Ontario College Quality Assurance ServicePLAR Prior Learning Assessment and RecognitionPQAA Program Quality Assurance AdministratorPQR Program Quality ReviewTL Team Leader

61

APPENDIX A – THE FINAL REPORT

The template for the final report is found online at http://www.algonquincollege.com/acad_dev/CollegeApproved.htm.Help text for the main sections of the report – the executive summary, the introduction, the conclusions, and the implementation plan – are provided on the following pages of this Appendix.

62

http://www.algonquincollege.com/acad_dev/CollegeApproved.htm

The Executive Summary is a one-page, single-spaced review of the highlights of the Program Quality Review. The focus of this summary is to highlight the overall status of the program, its special strengths and achievements, and any challenges the program is facing and the strategies recommended to address these challenges. Four to five paragraphs can cover the information needed:

the purpose of the report, background - a brief history of the program and delivery information, key findings - strengths and highlights, challenges (SWOT analysis), a brief summary of the conclusions and recommendations.

Executive Summary

Example Executive Summary

(This is fictitious but uses some real situations from recent reports.)

This report is a summary of the findings of the Program Quality Review Process for the 2008-200 academic year for the Housemaking program offered at Algonquin College.

The Housemaking Program was first offered at Algonquin College in 1981 as a one-year certificate. In 1992, the program was converted to a two year diploma program to meet the demands for more highly skilled and knowledgeable workers, and in 1997, the delivery mode was compressed into a non-semester diploma program format. The program prepares graduates to enter the housebuilding industry in the Ottawa and surrounding area. The program is also offered at the Perth Campus.

There has been a consistently high demand for the program and its graduates except for a minor slump in the housing depression in the early 90s. There are a number of local builders who have supported the program over the last 30 years and our graduate employment rate is consistently above 90%. The KPI and Course Assessment Surveys note high satisfaction rates with respect to the dedication and expertise of the program faculty. This commitment from both full-time and part-time faculty members has ensured the program has maintained a high standard of delivery.

Two major concerns will be addressed in the implementation plan. The first is the need for the students to be ready to enter the job market in the spring rather than the fall. The faculty will review the delivery mode over the next year and determine ways to deliver the curriculum to meet the demands of the students to move quickly through the program yet be available at the start of the construction season. The other most significant concern is the expectation to lose the coordinator and the other most senior full-time professor in the next year as both of these faculty members are eligible to retire. Succession planning will be a high priority to ensure we have excellent teaching and curriculum development skills in the department along with our current practical expertise. A review of course outlines indicates that some outlines are not as complete as would be desired. A plan is in place to ensure this will be resolved in the May-June 2009 planning session.

This program has enjoyed a high demand, and high employment rates. Industry projections in the Ottawa area suggest that this demand will continue. Advice from our industry partners on an annual basis will assist us in keeping this program strong over the next few years.

63

The introduction ‘sets the tone’ for the entire PQR report. It introduces the program as it stands at the start of the PQR review period, and includes

the purpose of the program – why does it exist, who does it serve in terms of the student body and the future employers,

the programs’ evolution from its first inception to today, relevant external accreditation organizations linked to the program, if applicable, number of faculty, both full-time and part-time, the nature of the student body, special contributions that the program offers in the community, variations in delivery like co-op options, campus options, Winter intakes, multiple intakes

etc., a discussion of the use of classroom, labs, computer labs, placement opportunities, field

trips, etc.

If the program has previously gone through PQR, re-use material as appropriate within the introduction, and note the key recommendations and progress against them as applicable. If PQR actions remain open, they should be considered for current applicability, and carried forward as appropriate.

Introduction

This section does not include any information accumulated during the current PQR.

64

Findings

EVALUATION OF THE PROGRAM AGAINST THE CRITERIA

Program Quality Review Criteria

65

The findings section is where each of the elements of the evaluation criteria are responded to by Yes/No or occasionally N/A.

Each element of the criteria requires a justification, i.e. explain why the response is Yes, or No or N/A.

Previous PQR reports within the department can be used for reference when writing. Also, the PQAA can provide guidance on how to respond to each of the elements of the criteria.

To avoid repetition, the evaluation criteria and their elements are not listed here.

Discuss the review of the statistics and the identification of agenda items for the focus groups.

A brief paragraph can be dedicated to highlights of each of the three focus groups.

The last half or third of this section speaks to the evaluation of the various recommendations from each of the focus groups and the decisions taken for the formulation of the implementation plan. For example, students may suggest that the computer applications course is irrelevant and they do not like the hybrid courses. The external stakeholders provide feedback that the graduates are not computer literate. The final recommendation would probably indicate the need to make the computer applications course relevant to the industry by adding inventory control and invoicing software exercises and communication to the students that computer skills are desirable in the workplace. To address the students’ dislike of hybrid courses, a recommendation could identify the development of better orientation to Blackboard for both faculty and students. (Reminder: it is important to provide feedback to focus group participants regarding resolutions of issues raised at the focus group.)

Some recommendations may arise that will not be feasible to implement and therefore will not be addressed in the final recommendations. These need to be discussed here. For example, one group suggested that students have a university level course as a prerequisite for admission to the program, as the students need a strong science background. Given that this is not allowable by Ministry standards, and is not feasible, the recommendation might be modified to ensure that highest Grade 12 College course is listed as a required course, or if this is already the case, the recommendation may not warrant any action. If the students have the high school subject at the grade twelve level, there may need to be a college level course in level 01 to help the students with the material. This should be explained here, i.e., recommendations from the Focus Groups should not just disappear without explanation and feedback.

Some recommendations will be desirable to implement but may have budget constraints. These recommendations can be maintained but identified as having fiscal constraints. Sometimes recommendations may be carried forward until such time that the funds are available or it is determined that a less costly solution is needed and implemented.

Conclusions

This part of the report addresses the analysis of the data, documents, and input from the focus groups.

Note: Recommendations are identified under the relevant pillar of the College’s Mission and Strategic Plan.

66

Recommendations

Recommendations resulting from the PQR are listed, even if they are not able to be implemented, and the pillar(s) of the Strategic Plan under which they fall is noted.

The four pillars of the Strategic Plan are: Employee Development Service Excellence Academic Leadership Resource Management.

Examples of recommendations/actions which fall under these pillars are: Employee Development

o Professional development – teaching and learningo Professional development – adult learning principleso Professional development – integration of emerging technology

Service Excellenceo Integrating Student Services (e.g. Academic Advising)o Improving quality - program curriculum o Improving quality - program delivery

Academic Leadershipo Improve student retentiono Ensure the aspects of the Algonquin Experience are embedded in the program.o Expand Applied Research with the program curriculum

Resource Managemento Ensure equipment plan is sustainable

67

Implementation Plan

Actions arising from the stated recommendations are captured, and must be resourced and measureable. Any recommendations that are not able to be implemented at this time should be noted.

The PQR implementation plan is carried forward as a ‘living document’ to be used as part of the ongoing operations of the program, and is updated at least annually.

Recommendation 1: Strategic Plan: (one or more references here…)

Action 1.1Resource(s)Timing


Recommendation 2:Strategic Plan: (one or more references here…)





Not able to be actioned currently.



<…extend/adapt table as applicable…>

68

APPENDIX B – THE CREDENTIALS FRAMEWORKExtracted from the “Minister’s Binding Policy Directive “Framework for Programs of Instruction” found at: http: www.accc.ca/ftp/es-ce/MTCUCollege Framework .pdf

69

http://www.accc.ca/ftp/es-ce/MTCUCollegeFramework.pdf

APPENDIX C – TEMPLATESThe following templates and review aids are available to download from the PQR website at: http://www.algonquincollege.com/acad_dev/CollegeApproved.htm and are included in the following pages:

Introductory Memo from Academic Administrator Introductory Memo from Team Leader Faculty Education and Experience Form Faculty Education and Experience Summary Form Program Self-Audit Course Outline Check Up Review Tool Course Outline Review Chart Course Outline Review Summary Sheet Curriculum Mapping – Terms and Concepts Program Outcomes – Mapping by Level Program Outcomes – Mapping Summary Analysis of VLO maps (feedback form) Sample Faculty Focus Group Agenda Sample Invitation to External Stakeholders Sample External Stakeholders Focus Group Agenda Sample Student Focus Group Agenda Student Focus Group Purpose and Objectives Sample Thank You Letter to Focus Group participants PQR Approval (Submission Sign-off) form

The blank PQR final report template is also available to download at the above URL.

http://www.algonquincollege.com/acad_dev/CollegeApproved.htm

Introductory Memo from the Academic Administrator

MemoTo: Faculty members of <program title here>

From: <Academic Administrator’s name here>, Academic Administrator, <department name here>,

CC: <Team Leader’s name here>, Team Leader for 2011-2012 Program Quality Review

Date:Re: Program Quality Review

The <program title here> program is part of the College’s cyclical Program Quality Review (PQR) process for 2011-2012, and <Team Leader name here> has been assigned the task of reviewing the program against Ministry and College quality criteria.

Part of the process is to determine whether the program is appropriately staffed to ensure the delivery of a quality program. To that end, please complete the attached Faculty Education, Experience, and Professional Development Survey and submit it to me by <date>. I will summarize the information and submit cumulative totals for the final PQR report. If anyone has any concerns about how this information will be conveyed, please feel free to discuss them with me.

You will also be receiving documents from <Team Leader name here> to assist in reviewing the curriculum. Curriculum review is an important part of your work each and every year, and this is an opportunity for us to review the cumulative effects of changes made to the program’s curriculum over the last five years. In addition to assessing the level of compliance to College quality expectations, we need to ensure the congruency between the identification of course learning outcomes, learning activities, and assessment activities.

You will also receive a copy of the Program Self Audit document that lists the evaluation criteria used to assess the program. Please do consider the Program Self Audit document carefully, and identify any areas of concern that you might have. Please take the time to acknowledge the areas of quality in our program, so we can devise strategies to ensure that the quality is enhanced.

Thank you for your support for these PQR activities.

Attachment:

Faculty Education, Experience, and Professional Development Survey

78

Introductory Memo from the Team Leader

MemoTo: Faculty members of <program title here> program

From: <TL name here>, Team Leader for 2011-2012 Program Quality Review

CC: <Academic Administrator’s name here>, Academic Administrator, <department name>,

Date:Re: Faculty Focus Group

The <program title here> program is part of the College’s cyclical Program Quality Review (PQR) for 2011-2012. You are invited to a Faculty Meeting on <date here> in room <location here>. Please review and complete the following documents as outlined below. These documents will be discussed at the meeting and used by the Team Leader to compile summary reports.

Program Self-Audit: Please provide to the Team Leader one week ahead of Faculty Focus group meetingThis document includes Ministry and College criteria against which programs at the College are evaluated. Please read the document carefully and indicate your opinion regarding whether the program meets each of the elements of the criteria. Put a check mark in the appropriate column and include any comments you have. These may be discussed further at the meeting. Issues raised by the faculty members may be brought forward to the student and external stakeholder focus groups for further exploration if necessary.

Course outline review document documents: Please review your course(s) and bring reports to the Faculty Focus group meetingEach Faculty member is asked to complete the course outline review document for each course that they teach. This will help identify that the course outline meets Ministry criteria and also will provide valuable feedback that will be used the next time the course outline is updated.

If you require any assistance with these documents, please contact me at <email. phone, other contact details>.

Attachments:Curriculum Mapping Terms and ConceptsCourse Outline Check-up for Course WritersCourse Outline Review ChartProgram Self-Audit Template

79

Faculty Education, Experience, and Professional Development SurveyThis survey is to be submitted directly to the Academic Administrator of the program who will summarize the data for the final PQR report. This survey is considered confidential information once completed.

Name: Date:

Program:

Education

Response text here

Number of years relevant experience in the workplace

Response text here

Number of years teaching in this program

1. as a full-time professor Response text here

2. as a part-time professor Response text here

Date of hire: Response text here

Other relevant Credentials

Response text here

Other relevant Experience

Response text here

List any Professional Development activities attended in the last 5 years. Include formal courses, workshops, conferences, industry renewal opportunities, sabbatical leaves, etc.

Response text here

Describe any reflective practices used for Professional Development in the last 5 years. (For example, review of Course Assessments each term, peer-to-peer feedback, informal feedback from students, and the associated outcome(s)). Indicate if not regularly engaged in reflective practice.

Response text here

When was your last performance review? Response text here

Confidential once completed

80

Summary of Faculty Members’ Credentials and Experience

Program:

Date:

Total number of students in the programTotal number of faculty members teaching in the program:

Full-timePart-time

Education:Number of faculty members with PhDNumber of faculty members with EdDNumber of faculty members with Master’s DegreeNumber of faculty members with Bachelor’s DegreeNumber of faculty members with DiplomaNumber of faculty members with special credential(please list the relevant credentials and the number of Faculty with the credential)Number of Faculty with less than Diploma preparationTeaching Experience:Total number of years of full-time teaching experienceTotal number of years of part-time teaching experienceSpecial areas of strength of facultyAny areas of concern regarding faculty education and/or experienceWorkplace ExperienceNumber of years of relevant workplace experience combined.Professional Development:Number of Faculty members who have attended professional development within the last two years. (Please list some specifics, e.g. TALL program, part-time faculty orientation)Number of Faculty members participating in reflective practice. (please list examples of the reflective practices being used )Performance ReviewsNumber of faculty members with a written performance review done in the last year.Number of faculty members with written performance review done more than a year ago but within the last two years.Number of faculty members with written performance review done more that two years ago but within the last three years.Number of faculty members with no written performance reviews with the last three years.

81

Program Quality Review - Program Self-AuditEach faculty member completes this program self-audit form and submits it to the Team Leader who summarizes the input. The self-audit is intended as a quick reference check to capture faculty input regarding the program in order to guide discussions at the focus groups and to continue to evaluate against the quality criteria.

Criterion 1. Program learning outcomes are set, are consistent with the college mission and the program’s intended purpose, and are appropriate for the credential offered upon completion of the program.

Yes No Comments


1.2 Program learning outcomes are consistent with MTCU Provincial Program Standards where they exist.

N/A for College Certificates, but any relevant comments can be captured.

1.3 Program learning outcomes are reflected in course outlines.

1.4 Program learning outcomes are used in prior learning and assessment (PLAR).

1.5 Changes to courses and program learning outcomes are introduced on a timely basis and are designed to maintain the relevance of the program.

1.6 The capabilities of program graduates, including knowledge, understanding, skills, and attitudes are consistent with the intended program learning outcomes.


Yes No Comments

2.1 The qualifications and prerequisites required of the applicant are published and are appropriate to allow the student to be successful without limiting access to the

82

program.




2.5 Students know how they will be evaluated.

2.6 Students indicate the learning requirements are relevant and meaningful.

2.7 Students indicate that assessment methods relate to the learning requirements.


Yes No Comments



3.3 Prerequisites do not unnecessarily hinder progress in the program.

3.4 The program title is consistent with college system nomenclature / titling principles.

3.5 The program has established articulation agreements.

83

3.6 The program conforms to the College policy for the number of English courses.


3.7 All curriculum documentation is up-to-date including course outlines and the program monograph information.

3.8 There is congruency between the course learning requirements and the program learning outcomes.

3.9 There is a match between course learning requirements, course learning activities and learning resources.

3.10 Concepts of social, economic and environmental sustainability are embedded in the program curriculum.

3.11 Off-campus experiential learning components are embedded in the program curriculum.

3.12 Students have opportunities to develop the skills, knowledge and attitudes necessary to succeed in a global economy.


Yes No Comments

4.1 Program delivery, including that which takes place off-site, is consistent with the nature of the program, the learning outcomes, and the needs of the students.


4.3 Learning methods are published and are matched to the learning outcomes.

4.4 College designated targets regarding hybrid courses are met.


4.5 Learners are provided the skills necessary to be successful with the learning strategies selected.

4.6 Evaluation criteria are published and students are aware

84

of how and when they are going to be evaluated.

4.7 There is a match between course learning requirements and evaluation methods, i.e., evaluation methods allow students to demonstrate the course learning requirements.

4.8 Evaluation methods are valid and reliable.

4.9 Students indicate that feedback is timely and allows them to build on their learning.

4.10 Students perceive evaluation to be fair.

4.11 Resubmissions, supplementals, and appeals are published, appropriate, fair, valid, and consistent.

4.12 Student workload and assessment is balanced across the term at both the course and program level.

4.13 There is a range of evaluation methods used consistent with a variety of learning styles.

4.14 Learners can earn credit for up to 75% of the program hours using the PLAR process.

4.15 Academic policies and practices that provide for the development and continuous improvement of teaching and learning methods are valued, documented, and supported.


Yes No Comments

5.1 Program faculty members, as a whole have adequate academic preparation and workplace experience to deliver a quality program.

5.2 Program faculty members are evaluated every three years.

5.3 Program faculty members engage in professional development activities that ensure they are current in their field and developing teaching expertise.

5.4 Program faculty members work within clean and well-

85

structured instructional plans.

5.5 Program faculty members participate in reflective practice.

5.6 Students consider faculty to be available.

5.7 Students consider faculty to be adequately prepared for class.

5.8 Students consider faculty to promote a positive attitude to learning.

5.9 All students are assigned an academic advisor.

5.10 Academic Advisors contact their students early in the term with an invitation to meet and to ensure that students know who their advisor is.

5.11 Labs, clinical facilities and placement facilities are complementary to and integrated into the program and allow the learner to demonstrate the learning outcomes.

5.12 Students indicate that there are adequate and accessible learning resource materials including: textbooks in the bookstore; online materials; print resources; equipment and student support services, to allow them to be successful.

5.13 The program is financially viable.

5.14 The demand for the program has been sustained for the last five years.

5.15 There is a future demand for graduates of this program.

5.16 The learning environment is safe.

5.17 The students are provided with the information they need to know to function safely in both the College and workplace learning environments.

86


Yes No Comments

6.1 Students indicate that they are satisfied with the program.

6.2 Issues raised at Program Councils are addressed in a timely fashion and feedback is provided to the Council.

6.3 Learners progress through the program, achieve program outcomes and graduate in a timely fashion.

6.4 Learners with a wide range of abilities demonstrate the expected learning outcomes.

6.5 Items raised by the Program Advisory Committee are considered in a timely fashion and feedback is provided to the committee.

6.6 Graduates are satisfied with the overall program experience.

6.7 Graduates are obtaining employment in their fields.

6.8 Graduates are successful in obtaining external licenses or credentials where relevant.

6.9 Employers are satisfied with graduate performance.

87

Course Outline Check-Up Review Tool

88

Course Outline Review ChartThe following chart will help evaluate whether or not the curriculum meets the following quality criteria elements:

1. There is congruency between the course learning requirements and the program learning outcomes (#3.8).

2. There is a match between course learning requirements, course learning activities and learning resources (#3.9).

3. Learning methods are published and are matched to the learning requirements (#4.3).4. Evaluation methods allow students to demonstrate the course learning outcomes (#4.7).5. PLAR Challenges exist and are based on learning outcomes (#1.4).

Note: The course teacher should complete this sheet for each course section.DATE OF MAP:

Program Name: Academic Level:

Course name and code:

Course Learning Requirements

Learning Activities

Evaluation methods

Match with Criteria: Y/N

1. 1.2.3.4.5.

2. 1.2.3.4.5.

3. 1.2.3.4.5.

4. 1.2.3.4.5.

90

<extend table as required…>

91

Course Outline Review Summary Sheet

Program Name: Date:

1. There is congruency between the course learning requirements, and the program learning outcomes. #3.8

2. There is a match between course learning requirements, course learning activities and learning resources. #3.9

3. Learning methods are published and are matched to the learning requirements. #4.3

4. Evaluation methods allow students to demonstrate the course learning requirements. #4.7

5. PLAR opportunities exist and are based on course learning requirements. #1.7

Course Name and Number

Criteria (see Course Outline review chart)

Comments – particular strengths to be recognized or recommendations for

changes needing to be made1 2 3 4 5

<add rows as required…>

92

Program Outcomes – Mapping by Level

Program Name: Date:

Program Code:

Level 1Course Name

Course Number

Program Outcomes1 2 3 4 5 6 7 8

Level 2Course Name

Course Number


Level 3Course Name

Course Number


93

Program Outcomes – Mapping Summary

Program Name: Date:

Program Code:

Program Learning Outcome Level 1 Level 2 Level 3 Totals1. T

AC

2. TAC

3. TAC

4. TAC

5. TAC

6. TAC

7. TAC

8. TAC

9. TAC

94

Curriculum Mapping - Terms and Concepts

Teach Do you provide instruction/learning opportunities in this skill in your course?

Is this skill identified in one or more course learning requirements?

Do you devote a significant amount of time to facilitating student development of the skills and knowledge embedded in the outcome?

Assess Are there assignments, tests or projects which are designed to allow you to evaluate or assess student performance of this outcome or some of its elements?

Do you, in your evaluation of student performance, verify that this particular outcome (or a significant component of it) has been achieved?

Is this outcome reflected in your course outline in the course learning requirements and/or embedded knowledge and skills, and in the evaluation of the course outline?

Culminating Performance Culminating performances are tasks or activities designed to assess a learner’s ability to demonstrate one or more learning outcomes in their totality. While they do not necessarily occur at the end of a program of study, they do evaluate whether a learner is able to integrate and apply their learning to demonstrate the performance described in the learning outcome(s) at the exit level.

Is there an opportunity for you to evaluate the outcome in its totality?

Does the evaluation result in a final product or performance which allows you to determine whether the learner has integrated the knowledge and skills identified in the elements of performance?

Can you determine from this performance if the learner has demonstrated the outcome?

Program culminating performances should require learners to demonstrate learning similar to what would be expected of new graduates in as close to a “real world” context as possible.

If a culminating performance is assigned to a course, the professor teaching the course will be asked to evaluate whether or not the students successfully demonstrated the outcome(s) the culminating performance addresses

95

Analysis and Review of VLO Maps for PQR

96

Faculty Focus Group – Sample Agenda

SAMPLE AGENDAProgram Name:Date:Location:

Introductions Objectives of the session Process/ground rules/groups Tasks/activities

o Review of Course Outlines and Syllabus’o Review Mapping of Vocational Learning Outcomes / draft VLOso Identification of issues arising from Program Self-Audito Identification of agenda items for student and external stakeholder focus groupso Identification of recommendations for consideration

Anticipated outcomes

100

Sample Invitation to External StakeholdersAddress

Date

Dear <invitee name here>

As part of Quality Assurance at Algonquin College all programs participate in a cyclical Program Quality Review process. This year the <program title here> program is undergoing review. The College is seeking your participation in the External Stakeholder Focus Group session which will be held on <date, time, and location here>. Other participants will include members of the Program Advisory Committee, recent graduates, and employers.

During the review process the College solicits perceptions of the program held by various stakeholders who interact regularly with program faculty, students and graduates. A list of issues and topics which may be addressed is also included < or will be sent at a later date, adjust as applicable>. The Program Quality Review process seeks input from as broad a base as possible and from a wide range of perspectives held by various sectors linked to the program. The data collected will contribute valuable information and guidance to ensure that the program remains current and meets the changing needs of our students and the workplace.

I have enclosed a draft agenda for the focus group session. I anticipate that the focus group session will require two hours of your time, not including travel. <You may wish to include directions about parking.>

To ensure full participation for this special meeting, we ask that you respond to this invitation on or before <RSVP date here>, to allow us to invite an alternate in the event you are unable to attend.

The format for the Focus Group will be discussion-based and the session will be facilitated by one of the College’s Program Quality Review Team Leaders. Your perceptions are important to the program and we hope that you will be able to attend.

Thank you for your ongoing support of our program.

Sincerely,

Academic Administrator

101

External Stakeholder Focus Group


Introductions Objectives of the session Process/ground rules Program Overview (Academic Administrator)

o Events/Highlights/Issueso Environmental Influences

Discussion Itemso Review of Program Monographo Perceived strengths and weaknesses of the program (based on the responses to

quality criteria)o Present and anticipated trends in the vocation/workplaceo Review of vocational learning outcomeso Actions needed to address future trends.

Recommendations arising

102

Student Focus Group - Sample Agenda


Introductions Objectives of the session Process/ground rules Program Overview (Academic Administrator)

o Events/Highlights/Issueso Environmental Influences

Discussion Itemso Perceived strengths and weaknesses of the program (based on the responses to

quality criteria)o Present trends in the vocation/workplaceo Anticipated future trends in the vocation/workplaceo Actions needed to address future trends.

Recommendations arising

103

Student Focus Groups – Sample Purpose and Objectives

Purpose and Objectives

Purpose of the Student Focus Group

Focus group sessions have been convened to gather feedback from program clients. The sessions are an important part of the College’s Program Quality Review process.

Objective of the Focus Group Meeting

Focus group sessions are intended to

generate feedback from students about their experience with the College and their perception about the education they are receiving

identify factors students feel impact their learning environment

Format of the Meeting

An active process is used to generate as much feedback about a range of program related issues as possible within the time allotted. A full two hours is assigned to the meeting, which is facilitated by a program quality review Team Leader, or designate.

Anticipated Outcomes of the Session

The focus group is expected to provide

feedback about the program as it is currently delivered a record of the focus group members’ perceptions of the present status of the program,

and future trends that may affect the program recommendations for program updates/revision

104

Thank You Note to Focus Group Participants - SampleAddress

Date

Dear <participant name here>

Thank you for your recent participation in our focus group meeting, as part of the Program Quality review for the <program title here> program. Your contribution to our process for improving this program is greatly appreciated.

I have included a copy of the focus group report for your perusal. A copy of the final PQR report is anticipated to be available at the next advisory committee meeting in the fall.

Thank you for your ongoing support of the program.

Sincerely,

Academic Administrator

105

PQR Submission Sign Off Form

107

program quality assurance at algonquin college of applied ...€¦ · web viewprogram quality...

Documents