program level assessment
DESCRIPTION
Program Level Assessment. Workshop for College of the Redwoods Fred Trapp August 18, 2008. Assessment of Program Effectiveness. What the program will do or achieve Curriculum Retention Graduation Placement Satisfaction (graduate and employer). Assessment of Student Learning Outcomes. - PowerPoint PPT PresentationTRANSCRIPT
Program Level Assessment
Workshop forCollege of the Redwoods
Fred TrappAugust 18, 2008
Assessment of Program Effectiveness• What the program will do or achieve
• Curriculum• Retention• Graduation• Placement • Satisfaction (graduate and employer)
Assessment of Student Learning Outcomes• What students will do or achieve• Direct methods of assessment
Direct Measures of Learning• Capstone experience• Standardized tests• Performance on national licensure certification or
professional exams • Locally developed tests • Essay questions blind scored by faculty• Juried review of projects• Externally reviewed exhibitions performances• Evaluation of internships based upon program
learning outcomes
Indirect Measures of Learning
• Alumni, employer, and student surveys (including satisfaction surveys)
• Exit interviews of graduates and focus groups graduate follow up studies
• Retention and transfer studies• Length of time to degree• Graduation and transfer rates• Job placement rates
Steps of Implementation• Step One: Developing Assessment
Plans• Step Two: Developing Implementation
Strategies• Step Three: Collecting Data, Analyzing
& Reflecting on its meaning• Step Four: Implementing Change as
the result of assessment
Assessment Language• Program Outcomes/ Topic Areas• Student Learning Outcomes• Means of Assessment/ Evidence• Criteria for Assessment• Standards of Success/ Quality of
Performance
Communication (program outcome topic)
SpeakRelate Listen ParticipateWrite
Program Outcome Topic & Student Learning Outcomes
Student learning outcomes
Means of Assessment/Evidence- Circumstances (learning events)
• Assignments (in class and out of class)• Feedback on practice• Self evaluation• Peer evaluation• Role Play• Pre Tests• Simulation
Goal
OutcomeOutcome OutcomeOutcomeOutcome
Object (deliverable or performance)
Learning events
Means of Assessment/Evidence- Deliverable or Performance
Communication
SpeakRelate Listen ParticipateWrite
Speech
Means of Assessment/Evidence- Deliverable or Performance
InterviewEssay
Communication
SpeakRelate Listen ParticipateWrite
Speech
Criteria for Assessment
Verbal deliveryNonverbal delivery
OrganizationContent
criteria
Communication
SpeakRelate Listen ParticipateWrite
Speech
Standards of Success/ Quality of Performance
Verbal deliveryNonverbal delivery
organizationcontent
Quality of Performance
Student Learning Outcomes
• Students should be able to<<action verb>> <<something>>
• Learner Centered• Specific• Action oriented• Cognitively appropriate for the
program level
KNOWLEDGECOMPREHENSION
APPLICATIONANALYSISSYNTHESISEVALUATION
CiteCountDefineDraw
IdentifyList
NamePointQuoteRead
ReciteRecordRepeatSelectState
TabulateTell
TraceUnderline
AssociateClassifyCompareComputeContrast
DifferentiateDiscuss
DistinguishEstimateExplainExpress
ExtrapolateInterpolate
LocatePredictReportRestateReview
TellTranslate
ApplyCalculateClassify
DemonstrateDetermineDramatize
EmployExamineIllustrateInterpretLocate
OperateOrder
PracticeReport
RestructureScheduleSketchSolve
TranslateUse
Write
AnalyzeAppraiseCalculate
CategorizeClassifyCompareDebate
DiagramDifferentiateDistinguish
ExamineExperiment
InspectInventoryQuestionSeparate
SummarizeTest
ArrangeAssemble
CollectComposeConstruct
CreateDesign
FormulateIntegrateManageOrganize
PlanPrepare
PrescribeProduceProposeSpecify
SynthesizeWrite
AppraiseAssessChoose
CompareCriticize
DetermineEstimateEvaluate
GradeJudge
MeasureRankRate
RecommendReviseScoreSelect
StandardizeTest
Validate
KNOWLEDGECOMPREHENSION
APPLICATIONANALYSISSYNTHESISEVALUATION
CiteCountDefineDraw
IdentifyList
NamePointQuoteRead
ReciteRecordRepeatSelectState
TabulateTell
TraceUnderline
AssociateClassifyCompareComputeContrast
DifferentiateDiscuss
DistinguishEstimateExplainExpress
ExtrapolateInterpolate
LocatePredictReportRestateReview
TellTranslate
ApplyCalculateClassify
DemonstrateDetermineDramatize
EmployExamineIllustrateInterpretLocate
OperateOrder
PracticeReport
RestructureScheduleSketchSolve
TranslateUse
Write
AnalyzeAppraiseCalculate
CategorizeClassifyCompareDebate
DiagramDifferentiateDistinguish
ExamineExperiment
InspectInventoryQuestionSeparate
SummarizeTest
ArrangeAssemble
CollectComposeConstruct
CreateDesign
FormulateIntegrateManageOrganize
PlanPrepare
PrescribeProduceProposeSpecify
SynthesizeWrite
AppraiseAssessChoose
CompareCriticize
DetermineEstimateEvaluate
GradeJudge
MeasureRankRate
RecommendReviseScoreSelect
StandardizeTest
Validate
Course outcomes
KNOWLEDGECOMPREHENSION
APPLICATIONANALYSISSYNTHESISEVALUATION
CiteCountDefineDraw
IdentifyList
NamePointQuoteRead
ReciteRecordRepeatSelectState
TabulateTell
TraceUnderline
AssociateClassifyCompareComputeContrast
DifferentiateDiscuss
DistinguishEstimateExplainExpress
ExtrapolateInterpolate
LocatePredictReportRestateReview
TellTranslate
ApplyCalculateClassify
DemonstrateDetermineDramatize
EmployExamineIllustrateInterpretLocate
OperateOrder
PracticeReport
RestructureScheduleSketchSolve
TranslateUse
Write
AnalyzeAppraiseCalculate
CategorizeClassifyCompareDebate
DiagramDifferentiateDistinguish
ExamineExperiment
InspectInventoryQuestionSeparate
SummarizeTest
ArrangeAssemble
CollectComposeConstruct
CreateDesign
FormulateIntegrateManageOrganize
PlanPrepare
PrescribeProduceProposeSpecify
SynthesizeWrite
AppraiseAssessChoose
CompareCriticize
DetermineEstimateEvaluate
GradeJudge
MeasureRankRate
RecommendReviseScoreSelect
StandardizeTest
Validate
Program outcomes
Possible Outcomes?
• Students will appreciate….• Students will be exposed to….• Students will demonstrate knowledge of
the history, theories and applications of…
• Students will attend….• Faculty will provide students with
opportunities to….
Possible Outcomes
• Students will appreciate…• Students will be exposed to….• Students will demonstrate knowledge of
the history, theories and applications of…• Students will attend….• Faculty will provide students with
opportunities to….
Writing Program Level Student Learning Outcomes• Student Learning Outcomes
• Students will recognize, analyze, and interpret human experience in terms of personal, intellectual, and social contexts.
• Students should be able to critically comprehend, interpret, and evaluate written, visual, and aural material.
Compound outcome statements will be problematic to assess.
Program Level Student Learning Outcomes• Students will be able to:
• Design a solution to a technical problem• Interpret performance data• Demonstrate the functions of key operating
systems• Articulate the ethical standards of the
profession• Produce diagnostic quality radiographic images• Evaluate the advantages of alterative solutions
Course LevelStudent Learning Outcomes• Students will be able to:
• Students will have a basic knowledge of human anatomy
• Students will be able to define 200 medical terms
• Students should know the history of the United States
• Students will be able to translate dollars into pounds and Euros
Course vs. Program Assessment• Single course• Course concepts
• Fewer confounds from outside the course detractors
• Faculty (indiv. & small groups)
• “Micrometer” precision• Less fear generated in faculty• Faculty professional
obligation, less overhead cost• No external certifications
• Multiple courses• Program performance
competencies (post grad)• More confounds from outside
the course detractors• Faculty (small & large groups)
• “Two sticks & a chain” precision
• Potential fear generated in faculty
• Faculty professional obligation, but some overhead cost
• External certifications possible
Identify Criteria for Assessing Program Level Outcomes• Must be shared across faculty• A way to further define the student
learning outcome• Provides a common language for
describing student learning• Must be outcome specific• Number of criteria will vary by outcome
Means of Assessment/Evidence- Deliverable or Performance + Criteria
Communication
SpeakRelate Listen ParticipateWrite
Salespresentation
deliverycontent
organizationOral
Report
Eulogy
volumepoise
conclusion
Speaking
Course Course Course Course Course
eye contactstyle
appearance
gesturesrate
evidence
sourcesexamples
organization
transitionsverbal variety
attention getter
The Reality of Assessing Program LevelStudent Learning Outcomes
volumepoise
conclusion
eye contactstyle
appearance
gesturesrate
evidence
sourcesexamples
organization
transitionsverbal variety
attention getter
Can our students deliver an effective public speech?
The Reality of Assessing Program LevelStudent Learning Outcomes
Program Learning Outcome Example #1
Gather factual information and apply it to a given problem in a manner that is relevant, clear, comprehensive, and conscious of possible bias in the information selected
BETTER: Students will be able to apply factual information to a problem
Criteria: RelevanceClarity
Comprehensiveness Aware of Bias
Program Learning Outcome Example #2
Imagine and seek out a variety of possible goals, assumptions, interpretations, or perspectives which can give alternative meanings or solutions to given situations or problems
BETTER: Students will be able to provide alternative solutions to situations or problems
Criteria: Variety of assumptions, perspectives, interpretationsAnalysis of comparative advantage
Program Learning Outcome Example #3
Formulate and test hypotheses by performing laboratory, simulation, or field experiments in at least two of the natural science disciplines (one of these experimental components should develop, in greater depth, students’ laboratory experience in the collection of data, its statistical and graphical analysis, and an appreciation of its sources of error and uncertainty)
BETTER: Students will be able to test hypotheses.Criteria:Data collectionStatistical AnalysisGraphical AnalysisIdentification of sources of error
Standards for Success/Quality of Performance
• Rubric• Scale or description for assessing each of
the criteria• Two to Five-point scales for each criteria• Anchored with descriptions and supported
by examples
• Portion of the Whole Group Which is Successful
ExceedsExpectations
MeetsExpectations
Does not meetExpectationsCriteria
Performance Anchors:
Descriptionsof
Performance
OUTCOME:
Standards for Success/Quality of Performance Rubric
YES!!!!AlmostNot YetComponents
Performance Anchors:OUTCOME:
Standards for Success/Quality of Performance Rubric
ExemplaryEssentialBasicComponents
Performance Anchors:
Standards for Success/Quality of Performance RubricOUTCOME:
Student LearningOutcome
3
2
1
4
Criteria Test Questions
1 2 3 4
5 6 7 8
10 11 12
1713 14 15 16
9
18
Student LearningOutcome
3
2
1
4
Criteria Test Questions
1 2 3 4
5 6 7 8
10 11 12
1713 14 15 16
9
18
78%
56%
82%
42%
%correct
Studentlearning outcomes Course
1
X
X
X
Course2
X
X
X
X
X
Course3
X
X
X
X
Course4
X
X
X
X
X
Course5
X
X
X
X
7
1
234
56
Identify Assessment Points in the Program
Studentlearning outcomes Course
1
X
X
X
Course2
X
X
X
X
X
Course3
X
X
X
X
Course4
X
X
X
X
X
Course5
X
X
X
X
7
1
234
56
Phase 1
Communicate EffectivelyDemonstrate Oral Communication Skills
VerbalDelivery
NonverbalDelivery
Organization
Does notmeet Meets Exceeds
20%
13%
24%
65%
57%
58%
15%
30%
18%
Program-wide Assessment Report
Is there a problem with performance in any of the criteria?