collection, maintenance and validation of effective evidence for program assessment
DESCRIPTION
Describes methods and tools for maintaining educational assessment information - primarily for program outcomes assessment. This is describing outcomes assessment processes and tools for an ABET-Accredited university undergraduate programs. Discusses direct and indirect evidence use, and how to use these to efficiently validate the program.TRANSCRIPT
ICEE 2006F. Mak & S. Frezza © 2006
Collection, Maintenance, and
Validation (CMV)
of a Set of
Effective Objective Evidence
Fong Mak, Ph.D., P.E.
Electrical and Computer Engineering
Steve Frezza, Ph.D., C.S.D.P.
Computer and Information Science
Gannon University
F. Mak & S. Frezza © 2006 ICEE 2006
Why CMV Objective Evidence ?
• Foundation:
Outcomes-Based Continuous Quality Improvement
• Objective Evidence …
– Shows we’ve engaged in…
– Reflects the quality of…
…the educational process
Documents…
Supports…
F. Mak & S. Frezza © 2006 ICEE 2006
Why Objective Evidence ?
• Process:
– Ensure the quality of the educational process
• Product:
– Support and ensure the quality of educational delivery
– Multiple constituents, measures
F. Mak & S. Frezza © 2006 ICEE 2006
What is Effective Evidence ?
Objective Evidence that
• Maximizes Information Content
– Process documents should provide decision-makers information to support their decisions
• Minimizes Item Count
– Collect as few items as needed
Golden Rule:
Documents Add Value
F. Mak & S. Frezza © 2006 ICEE 2006
Effective Objective Evidence
Does:
• Inform the process
• Support decisions
• Flow from the process
Key:
Process IntegrationTypical Educational Assessment Processes: dd
Untracked, Indirect, Direct
Does Not:
Just document it
Record decisions
Add steps
F. Mak & S. Frezza © 2006 ICEE 2006
DirectAssessments
Disconnected Assessment Process
Course
Syllabi
Course
Syllabi
Course
Syllabi
CourseImplementation
CorrectiveActions
ProcessObjectiveEvidence
Course
Portfolios
Course
Portfolios
CoursePortfolios
No feedbackMinimal documentationMinimal additional effort
Discontinuous improvement
F. Mak & S. Frezza © 2006 ICEE 2006
Indirect Assessment
• Respond to perceived effectiveness by:
– Students
– Faculty
– Others
F. Mak & S. Frezza © 2006 ICEE 2006
Indirect Assessment Process
Indirect Assessments
DirectAssessments
ProcessObjectiveEvidence
MeetingMinutes
ActionItem
Matrix
Course ExitSurvey Results
Senior ExitSurvey Results
EmployerSurvey Results
AlumniSurvey Results
Course
Syllabi
Course
Syllabi
Course
Syllabi
Course
Portfolios
Course
Portfolios
CoursePortfolios
CourseImplementation
FacultyReview
CorrectiveActions
Continuous feedbackStructured documentation
Some additional effortTracked improvement
Only indirect assessment
F. Mak & S. Frezza © 2006 ICEE 2006
Direct Assessment
• Direct measures of effectiveness:
– Student performance in meeting/exceeding
• Course objectives
• Program objectives (more difficult)
• Our Innovation:
Link grading & course outcomes
– Key Assignments: P/F to course
– Faculty-assigned, linked to course outcomes
F. Mak & S. Frezza © 2006 ICEE 2006
Minimized Direct AssessmentDirectAssessments
ProcessObjectiveEvidence
Course
Syllabi
Course
Syllabi
Course
Syllabi
Course
Portfolios
Course
Portfolios
CoursePortfolios
CourseImplementation
FacultyReview
CorrectiveActions
Course
Grade
Reports
Course
Grade
Reports
SummaryGrade
Reports
MeetingMinutes
ActionItem
Matrix
FCARsFCARs
FCARs
Continuous feedbackTracked improvement
Effort focused on
student performance of outcomes
F. Mak & S. Frezza © 2006 ICEE 2006
Complete Assessment Process
Indirect Assessments
DirectAssessments
ProcessObjectiveEvidence
MeetingMinutes
ActionItem
Matrix
Course ExitSurvey Results
Senior ExitSurvey Results
EmployerSurvey Results
AlumniSurvey Results
Course
Grade
Reports
Course
Grade
Reports
SummaryGrade
Reports
Course
Syllabi
Course
Syllabi
Course
Syllabi
Course
Portfolios
Course
Portfolios
CoursePortfolios
FCARsFCARs
FCARs
CourseImplementation
FacultyReview
CorrectiveActions
F. Mak & S. Frezza © 2006 ICEE 2006
Effective Collection
• Direct & indirect measures
– Improvements based on collected evidence
– Evidence, decisions and actions documented
– Continuous quality improvement tracked
Normal part of improvement process
F. Mak & S. Frezza © 2006 ICEE 2006
Effective Maintenance
• Automation
– Maximize measures collected/maintained online
• Survey creation, management: EvalTools
• Simple paper trails, used for review
– Standardized templates • simplify use and review
– E.g., Summary Grade Reports, FCARs, Meeting Minutes
F. Mak & S. Frezza © 2006 ICEE 2006
Effective Validation
• Review evidence as part of process
• Summarize decisions/planned actions
• Simple paper trails, used for validation
Analogy:
Bug TrackingDescribe (open) issuesAssign responsibility
Track changesTrack effectiveness (close issues)
F. Mak & S. Frezza © 2006 ICEE 2006
SummaryCMV Effective Objective Evidence
• Requires:
– Process design & commitment
• Benefits:
– Students
– Program Communicators
– Program Stakeholders (even faculty)
– Program Evaluators