Program Assessment Process for Continuous Quality Improvement
(CQI)
Prepared by:Abul Fazal M. Arif
Member, ABET Steering CommitteeNovember 16, 2008.
ABET Steering Committee Program Assessment Process 2
Contents
ABET General Criteria, 2008-2009 Cycle Assessment Process – key components Tasks to be Done Closing the Loop Action Plan
ABET Steering Committee Program Assessment Process 3
ABET GENERAL CRITERIA
“Renumbered” Criteria:(2008-2009 Accreditation Cycle)
Criterion 1. Students Criterion 2. Program Educational ObjectivesCriterion 3. Program OutcomesCriterion 4. Continuous ImprovementCriterion 5. CurriculumCriterion 6. FacultyCriterion 7. FacilitiesCriterion 8. SupportCriterion 9. Program Criteria
ABET Steering Committee Program Assessment Process 4
Assessment Process for Quality Assurance
Criterion 1. Students Criterion 1. Students
Criterion 2. Program Educational ObjectivesCriterion 3. Program OutcomesCriterion 4. Continuous ImprovementCriterion 5. CurriculumCriterion 5. CurriculumCriterion 6. FacultyCriterion 6. FacultyCriterion 7. FacilitiesCriterion 7. FacilitiesCriterion 8. SupportCriterion 8. SupportCriterion 9. Program CriteriaCriterion 9. Program Criteria
Assessment is an ongoing process of setting high expectations for student learning, measuring progress toward established learning outcomes, and providing a basis for reflection, discussion and feedback to improve University academic programs. It is a systematic and cyclic process that makes expectations and standards explicit and public.
ABET Steering Committee Program Assessment Process 5
Key Components of Assessment Process
Performance Criteria
Assessment Methods Evaluation
Curriculum Mapping
Closing the Loop Action Taken / To be Taken
Program Learning Outcomes
Program Educational Objectives
ABET Steering Committee Program Assessment Process 6
Performance Criteria
Assessment Methods Evaluation
Curriculum Mapping
Closing the Loop Action Taken / To be Taken
Program Learning Outcomes
Program Educational Objectives
Educational objectives are statements that describe the expected accomplishments of graduates during the first few years after graduation—usually 3-5 years.
ABET Steering Committee Program Assessment Process 7
Performance Criteria
Assessment Methods Evaluation
Curriculum Mapping
Closing the Loop Action Taken / To be Taken
Program Learning Outcomes
Program Educational Objectives
Assessment Process for Quality Assurance
These are statements related to skills, knowledge and behaviors that describe what students are expected to know and/or be able to do by the time of graduation. If students have achieved these outcomes, it is anticipated that they will be able to achieve the educational objectives after graduation.
ABET Steering Committee Program Assessment Process 8
Assessment Process (cont.)
At the heart of assessment are the Program Objectives and Outcomes. Purpose: To assess the extent a program is
achieving its objectives and outcomes. To be conducted continuously at different stages
continuous improvement.
The assessment process itself should be evaluated and refined in light of emerging insights.
All accrediting bodies, including ABET, require evidence of continuous quality improvement of both, Process and Program.
ABET Steering Committee Program Assessment Process 9
Tasks to be done
Task 1: Review Program Educational Objectives (PEO)
Task 2: Review Program Learning OutcomesTask 3: Align Learning outcomes with PEOTask 4: Present to ConstituentsTask 5: Get Approval & PublishTask 6: Mapping Course Outcomes to Program
OutcomesTask 7: Develop Performance Indicators Task 8: Identify Assessment Tools/MethodsTask 9: Develop Assessment PlanTask 10: Data Collection and EvaluationTask 11: Closing the Loop
ABET Steering Committee Program Assessment Process 10
TASK 1: Review Program Educational Objectives (PEO)
Review process will be initiated by Department ABET Committee or PAC
Guidelines for review
1. PEO should be consistent with proper definition.
2. PEO should be aligned with mission of university/College/Department.
3. Numbers of PEO should be manageable.4. PEO should be measurable.
ABET Steering Committee Program Assessment Process 11
STEP 1: Revise Program Educational Objectives
Examples of PEO Prepare graduates who will be successful in
mechanical engineering-related careers and other diverse career paths
Prepare graduates to be leaders Prepare graduates to be good citizens of
the society Prepare graduates to pursue advanced
degree, if so desired Prepare graduates to be life long learners Prepare graduates to be entrepreneurs
ABET Steering Committee Program Assessment Process 12
TASK 1: Review Program Educational Objectives
Examples of PEO - Combined Prepare graduates who will have successful
careers, and become leaders, in industry and the public sector
Prepare graduates who will appropriately apply acquired knowledge, work well in team, effectively communicate ideas and technical information, and continue to gain knowledge
Prepare graduates who will continue professional development and will pursue continuing education opportunities relevant to their careers
Note: For more examples of PEO, refer to Aggarwal’s workshop notes.
ABET Steering Committee Program Assessment Process 13
TASK 2: Review Program Learning Outcomes
Task 2.1: Engineering programs must demonstrate
that their students attain the following (a) to (k)outcomes:
(a) an ability to apply knowledge of mathematics, science, and engineering
(b) an ability to design and conduct experiments, as well as to analyze and interpret data
(c) an ability to design a system, component, or process to meet desired needs within realistic constraints such as economic, environmental, social, political, ethical, health and safety, manufacturability, and sustainability
(d) an ability to function on multidisciplinary teams
ABET Steering Committee Program Assessment Process 14
Task 2: Review Program Learning Outcomes (cont.)
(e) an ability to identify, formulate, and solve engineering problems
(f) an understanding of professional and ethical responsibility
(g) an ability to communicate effectively(h) the broad education necessary to understand the
impact of engineering solutions in a global, economic, environmental, and societal context
(i) a recognition of the need for, and an ability to engage in life-long learning
(j) a knowledge of contemporary issues(k) an ability to use the techniques, skills, and modern
engineering tools necessary for engineering practice
ABET Steering Committee Program Assessment Process 15
Task 2: Review Program Learning Outcomes (cont.)
Task 2.2: Check if any additional outcomes that may be articulated by the program criteria.
Check Criterion 9 on Program Criteria from ABET 2008-2009, (For Mechanical Engineering)Mechanical Engineering program criteria include the requirement that the program must demonstrate that graduates have the ability to: apply principles of engineering, basic science, and mathematics (including multivariate calculus and differential equations) to model, analyze, design, and realize physical systems, components or processes; and work professionally in both thermal and mechanical systems areas.
ABET Steering Committee Program Assessment Process 16
TASK 3: Align Learning outcomes with Program Objectives. Program outcomes must foster attainment
of program educational objectives. Link outcomes with PEO All Learning Outcomes may or may not
link to all/any PEOProgram Learning Outcomes
PEO
(a) (b) (c) (d) (e) (f) (g) (h) (i) (j) (k)
1 X X X
2 X X X
3 X X X
4 X X X
5 X X X
ABET Steering Committee Program Assessment Process 17
TASK 4: Present to Constituents
Main Stakeholders/Constituents Faculty through Department Council Employers through Industrial Advisory
Committee Students through various Professional Groups
or Student Societies etc.Others: Administration Alumni Other Graduate Schools Recruiters Employers
Note: Mostly faculty define Program Learning Outcomes
ABET Steering Committee Program Assessment Process 18
TASK 5: Get Approval & Publish
Get Approval Final form of the Program Educational
Objectives and Learning Outcomes should be approved by the Department Council
Keep record in “ABET File”
Publish at: Department Website General Catalog Bulletin Boards etc
Keep minutes of meeting of constituent’s involvement and Council approval as evidence in record.
ABET Steering Committee Program Assessment Process 19
Performance Criteria
Assessment Methods Evaluation
Curriculum Mapping
Closing the Loop Action Taken / To be Taken
Program Learning Outcomes
Program Educational Objectives
Curriculum Mapping is the alignment of the program learning outcomes with the curriculum and, therefore, it needs development of performance indicators based on the curriculum.
ABET Steering Committee Program Assessment Process 20
Performance Criteria
Assessment Methods Evaluation
Curriculum Mapping
Closing the Loop Action Taken / To be Taken
Program Learning Outcomes
Program Educational Objectives
TASK 6: Mapping of Course Learning Outcomes to Program Learning Outcomes Include only Core (required) courses Select courses for Outcome Assessment
ABET Steering Committee Program Assessment Process 21
Mapping: Course Learning Outcomes and Program Outcomes
ABET Steering Committee Program Assessment Process 22
Mapping: Core Courses & Program Outcomes
ABET Steering Committee Program Assessment Process 23
Performance Criteria
Assessment Methods Evaluation
Curriculum Mapping
Closing the Loop Action Taken / To be Taken
Program Learning Outcomes
Program Educational Objectives
TASK 7: Develop Performance Indicators for each Program Outcome
ABET Steering Committee Program Assessment Process 24
Performance Criteria
Assessment Methods Evaluation
Curriculum Mapping
Closing the Loop Action Taken / To be Taken
Program Learning Outcomes
Program Educational Objectives
Performance Criteria (indicators, PIs) are defined as the specific, measurable statements identifying the specific knowledge, skills, attitudes and/or behavior students must demonstrate as indicators of achieving the outcomes.
ABET Steering Committee Program Assessment Process 25
Examples of Performance Indicators Outcome (a): Ability to apply knowledge of
mathematics………
1. Applies mathematics principles to obtain solutions.
2. Uses principles of sciences and engineering in solving engineering problems.
3. Combines scientific and engineering principles to formulate models of processes and systems.
4. Combines mathematics principles to formulate models of processes and systems.
ABET Steering Committee Program Assessment Process 26
Examples of Performance Indicators Outcome (a): Ability to apply knowledge of
mathematics………1. Applies mathematics principles to obtain Applies mathematics principles to obtain
solutions.solutions.2. Uses principles of sciences and engineering Uses principles of sciences and engineering
in solving engineering problems.in solving engineering problems.3. Combines scientific and engineering Combines scientific and engineering
principles to formulate models of processes principles to formulate models of processes and systems.and systems.
4. Combines mathematics principles to Combines mathematics principles to formulate models of processes and formulate models of processes and systems.systems.
Note: For more examples of PI, refer to Sub-Committee Report.
Generally, it is recommended to limit performance criteria to three-four for each outcome.
ABET Steering Committee Program Assessment Process 27
Performance Criteria
Assessment Methods Evaluation
Curriculum Mapping
Closing the Loop Action Taken / To be Taken
Program Learning Outcomes
Program Educational ObjectivesAssessment is one or more
processes that identify, collect, and prepare data to evaluate the achievement of program outcomes and program educational objectives.
ABET Steering Committee Program Assessment Process 28
TASK 8: Identify Assessment
Tools/Methods. Assessment tools are of two types:
Direct measures are based on facts (it also includes opinion based on facts).
Indirect measures are based on opinion only.
Multi-source or multi-method approach is generally recommended for data collection and assessment. Indirect measures are necessary but not sufficient. They
should be complemented with at least one direct measure.
Sometimes it may not be possible to use direct methods to measure some of the outcomes (for example team work). In those cases, we can rely on indirect methods.
ABET Steering Committee Program Assessment Process 29
Task 8: Identify Assessment Tools (cont.)
Examples of Direct measures
1. Student course portfolios and senior capstone course projects using overall score of learning outcome rubric
2. External review or performance appraisal through direct observation on internship
3. Locally developed exams.4. Oral exams 5. Standardized exams such as FE, GRE, GMAT6. Special student achievements (awards, student
publications and presentations at various meetings)
7. External examiners8. Graduate schools
ABET Steering Committee Program Assessment Process 30
Learning Outcome RubricPerformanc
e Criteria
Unsatisfactory1
Developing2
Satisfactory3
Exemplary4
Score
Contribute
Research & gather information.
Does not collect any
information that relates to the topic.
Collects very little
information – some
relates to the topic.
Collects some basic
information – most
relates to the topic
Collects a great deal of
information – all relates to the topic
Take Responsibility
Fulfill team role’s duties.
Does not perform any
duties of assigned
team role.
Perform very little duties.
Perform nearly all duties.
Perform all duties of assigned
team role.
Shear equally
Always relies on others to do
the work.
Rarely does the assigned
work – often needs
reminding.
Usually does the assigned work –
rarely needs reminding.
Always does the assigned work – without
having to be reminded.
Value Other’s Viewpoints
Listen to other
teammates.
Is always talking – never allows
anyone else to speak.
Usually doing most of the
talking – rarely allows
others to speak.
Listens, but sometimes
talk too much.
Listen and speaks a
fair amount
Average
Rubrics are developed from performance indicators and consist of sets of categories that define the process towards meeting the goal.
ABET Steering Committee Program Assessment Process 31
Example of Rubric on TeamworkPerformanc
e Criteria
Unsatisfactory1
Developing2
Satisfactory3
Exemplary4
Score
Contribute
Research & gather information.
Does not collect any
information that relates to the topic.
Collects very little
information – some
relates to the topic.
Collects some basic
information – most
relates to the topic
Collects a great deal of
information – all relates to the topic
Take Responsibility
Fulfill team role’s duties.
Does not perform any
duties of assigned
team role.
Perform very little duties.
Perform nearly all duties.
Perform all duties of assigned
team role.
Shear equally
Always relies on others to do
the work.
Rarely does the assigned
work – often needs
reminding.
Usually does the assigned work –
rarely needs reminding.
Always does the assigned work – without
having to be reminded.
Value Other’s Viewpoints
Listen to other
teammates.
Is always talking – never allows
anyone else to speak.
Usually doing most of the
talking – rarely allows
others to speak.
Listens, but sometimes
talk too much.
Listen and speaks a
fair amount
Average
ABET Steering Committee Program Assessment Process 32
Task 8: Identify Assessment Tools (cont.)
Examples of Indirect measures
Questionnaires and written surveys (Alumni and employer surveys)
Senior exit surveys and interviews Student course evaluations Grades ** Focus groups conducted by trained
moderator Students internship evaluations (if based on
opinion)
** Refer to Sub-Committee Report or Aggarwal Workshop Notes.
ABET Steering Committee Program Assessment Process 33
TASK 9: Development Assessment Plan
Selection of Courses Data Collection Cycle Identify Data To be Collected
ABET Steering Committee Program Assessment Process 34
Mapping: Core Courses & Program Outcomes
ABET Steering Committee Program Assessment Process 35
Selection of Courses for Outcome Assessment
ABET Steering Committee Program Assessment Process 36
Selected Courses for Outcome Assessment
ABET Steering Committee Program Assessment Process 37
Data Collection Cycle
Outcome
Year 1
Year 2 Year 3 Year 4 Year 5 Year 6
(a) X X
(b) X X
(c) X X
(d) X X
(e) X X
(f)
(g) X X
(h) X X
(i) X X
(j) X X
(k) X X
ABET Steering Committee Program Assessment Process 38
Data Collection Cycle
Outcome
Year 1
Year 2 Year 3 Year 4 Year 5 Year 6
(a) X X
(b) X X
(c) X X
(d) X X
(e) X X
(f)
(g) X X
(h) X X
(i) X X
(j) X X
(k) X X
Learning Outcomes are measured on a continuous basis (3 years cycle is acceptable). You do not have to measure every student for every outcome.
For PEOs, typically data is collected every three years from all constituents for every objective.
ABET Steering Committee Program Assessment Process 39
Data to be collected
For Program Learning Outcomes Students Course Evaluation using rubric Student Course Portfolios Grades Performance Appraisal Senior exit surveys and interviews Locally developed exams
For Program Educational Objectives Alumni Survey Employers Survey Industrial Advisory Board’s Feedback Placement Data
ABET Steering Committee Program Assessment Process 40
TASK 10: Collection of Data & Evaluation
ABET Steering Committee Program Assessment Process 41
Performance Criteria
Assessment Methods Evaluation
Curriculum Mapping
Closing the Loop Action Taken / To be Taken
Program Learning Outcomes
Program Educational Objectives
Evaluation is one or more processes for interpreting the data and evidence accumulated through assessment practices. Evaluation determines the extent to which program outcomes or program educational objectives are being achieved, and results in decisions and actions to improve the program.
ABET Steering Committee Program Assessment Process 42
Example of Evaluation
Evaluation using EXCEL file
ABET Steering Committee Program Assessment Process 43
TASK 11: Closing the LOOP
ABET Steering Committee Program Assessment Process 44
Performance Criteria
Assessment Methods Evaluation
Curriculum Mapping
Closing the Loop Action Taken / To be Taken
Program Learning Outcomes
Program Educational Objectives
The feedback process is critical to creating and maintaining a systematic quality assurance system. When successfully implemented, all elements of the quality assurance process interact with one another.
ABET Steering Committee Program Assessment Process 45
Performance Criteria
Assessment Methods Evaluation
Curriculum Mapping
Closing the Loop Action Taken / To be Taken
Program Learning Outcomes
Program Educational Objectives EXAMPLE of
Assessment Plan, Report & Rubrics
ABET Steering Committee Program Assessment Process 46
ABET Accreditation: key Dates
Task Responsibility Date
1. Complete the form for Programs Evaluation
Academic Departments 31-12-2008
2 Send Request for Programs Evaluation To ABET HQ
Dean Of Engineering Office
15-1-2009
3 Draft Self-Study Reports Ready
Academic Departments 1-4-2009
4 Forward Draft Self-Study Reports to Quality Assurance Committee & English Editing
Dean Of Engineering Office
9-4-2009
5 Receipt of Reviewed and Edited Reports
Quality Assurance Committee & ELC
9-5-2009
6 Final Self Study Report Ready
Academic Departments 30-5-2009
7 Preparation of the final Reports
Dean Of Engineering Office
15-6-2009
8 Dispatch The Final Reports to ABET HQ
Dean Of Engineering Office
20-6-2009
ABET Steering Committee Program Assessment Process 47
Tasks to be done
Task 1: Review Program Educational Objectives (PEO)
Task 2: Review Program Learning OutcomesTask 3: Align Learning outcomes with PEOTask 4: Present to ConstituentsTask 5: Get Approval & PublishTask 6: Mapping Course Outcomes to Program
OutcomesTask 7: Develop Performance Indicators Task 8: Identify Assessment Tools/MethodsTask 9: Develop Assessment PlanTask 10: Data Collection and EvaluationTask 11: Closing the Loop
ABET Steering Committee Program Assessment Process 48
Task 3: Draft Self-Study Reports ReadyBy: Academic DepartmentsSubmission date: 1-4-2009
Task 16-30 Nov
1-15 Dec
16-31 Dec
1-15 Jan
16-31 Jan
1-15 Feb
16-28 Feb
1-15 Mar
16-31 Mar
1
2
3
4
5
6
7
8
9
10
11
ABET Steering Committee Program Assessment Process 49
Sources of Information
http://www.abet.org/forms.shtml Aggarwal’s workshop # 1 lectures Aggarwal’s workshop # 2 lectures ABET Sub-Committee Report
ABET Steering Committee Program Assessment Process 50
THANK YOU