PROGRAM ASSESSMENT: DESIGNING AND DOCUMENTING LEARNING
Welcome!
March 29, 2010Forum Session 5-6 p.m.League for Innovation in the Community College ConferencePresenter: Robin Nickel, Ph.D.
AGENDA
Explore a process and tool for designing program assessment
Examine sample programs Identify criteria for program
assessment Connect program assessment
to performance-based curriculum
QUOTE OF THE DAY
“As long as you’re going to be thinking anyway, think big.
--Donald Trump quoted in Sky Magazine March 2010
Informed judgments about achievement of intended learning outcomes
Based on evidence
Assessments are valid, reliable, and fair
Assessment is built into the plan for teaching
Data-driven evaluation that is actively used for continual improvement of teaching and learning
Assessment of Learning
DRIVERS
Accreditation (NCA, AQIP, industry)
Increased accountability to improve teaching and learning
DRIVERS
Carl D. Perkins IV legislation and funding
Objectively measure student attainment of industry recognized skills upon graduation
ASSESSMENT IN COMMUNITY AND TECHNICAL COLLEGES
Focus on industry aligned skill setsStudents know they are acquiring
skills that have value and portability
Increases odds that students will be successful in their employment
Ensures that your college is educating students to meet the needs of employers
QUESTIONS ABOUT OUTCOMES
Who cares about outcomes?Which outcome is which?How are outcomes related?What is the desired outcome of
outcomes?What does this mean for students
and instructors?
QUESTIONS ABOUT OUTCOMES Who cares about outcomes? Which outcome is which? How are outcomes related?
What is the desired outcome of outcomes?
What does this mean for students and teachers
Stakeholders Types of outcomes Relationship of institutional,
program, and course level outcomes
Learning, assessment, documentation, etc.
Impact on course design, learning and assessment
“Assessment of student learning is a participatory, iterative process that:
Provides data/information you need on your students’ learning
Engages you and others in analyzing and using this data/information to confirm and improve teaching and learning
Produces evidence that students are learning the outcomes you intended
Guides you in making educational and institutional improvements
Evaluates whether changes made improve/impact student learning, and documents the learning and your efforts.”
NCA Higher Learning Commission
From “Student Learning, Assessment and Accreditation: Criteria and Contexts”, presented at Making a Difference in Student Learning: Assessment as a Core Strategy, a workshop from the Higher Learning Commission, July 26-28, 2006.
Assessment of Learning
HLC/AQIP asks 5 fundamental questions: How are your stated learning outcomes appropriate to
your mission, programs, students and degrees? How do you ensure shared responsibility for student
learning & assessment of student learning? What evidence do you have that students achieve your
stated learning outcomes? In what ways do you analyze and use evidence of student
learning? How do you evaluate and improve the effectiveness of
your efforts to assess and improve student learning?
From “Student Learning, Assessment and Accreditation: Criteria and Contexts”, presented at Making a Difference in Student Learning: Assessment as a Core Strategy, a workshop from the Higher Learning Commission, July 26-28, 2006.
Assessment of Learning
Curriculum and SACS
Core Requirement 2.5 Review of Programs Systematic Results in continuing improvement Demonstrates you accomplish mission
Curriculum and SACS
Core Requirements2.7.3 General Education component2.12 Quality Enhancement Plan relates to student learning
Curriculum and SACS
Comprehensive Standards 3.3.1 “institution identifies expected outcomes … assesses whether it achieves the outcomes; and provides evidence of improvement based on analysis of those results.”
3.4.1 “establish and evaluate program and learning outcomes
Curriculum and SACS
Comprehensive Standards 3.5.11 “general education core… provide evidence that graduates have attained these competencies”
Curriculum and SACS
Federal Requirements 4.2 “The institution maintains a curriculum that is directly related and appropriate to its purpose and goals….”
BENEFITS OF PROGRAM ASSESSMENT
• Feedback about skills needed on the job
• Added value for educationStudents
• Improvement of teaching and learning
• Supports overall missionColleges
• Assurance that your graduates have attained technical skills needed on the jobEmployers
Indirect Measures Data based on
inferences about why achievement is high or low: Enrollment Retention rates Course completion Student/graduate/
employer satisfaction surveys
Placement statistics
Direct Measures Data that provides
evidence that students have achieved the outcomes: Performance
assessments with rubrics Portfolios Artifacts Performances Outcome referenced
tests
Increased Accountability for Direct Measurement of Learning
Sound Assessment
Valid Outcomes based on standards
(industry) Measures intended outcomes Measures application and critical
thinking
Reliable Performance assessment based on
consistent rubrics, scoring guides, and rating scales
Fair Valid Reliable Learners informed of expectations
up front! Feedback to learners
Program Level or End of Course
ProgramQuality
Student Credentials
Course LevelFeedback to
studentImprovement of
Learning/Teaching
Summative/Formative Assessment
Continuous
Im
pro
vem
ent
PROGRAM OUTCOMES Measurable, observable, and field-specific
skills – major outcomesNumber 5-7 per program (guideline not
rule) Originate from:
Current DACUMsAccrediting AgenciesNational (or other) Skill StandardsAdvisory Committees
Threaded through courses Performance verified with summative
assessment of skill performance
RADIOGRAPHY PROGRAM OUTCOMES
A. Carry out the production and evaluation of radiographic images
B. Adhere to quality management processes in radiography
C. Apply computer skills in the radiographic clinical setting
D. Practice radiation safety principlesE. Provide quality patient careF. Model professional and ethical behavior
consistent with the A.R.R.T. Code of EthicsG. Apply critical thinking and problem solving
skills in the practice of diagnostic radiography
Balance of College Assessments and External Assessments
College A
ssessment
External A
ssessment
Indirect
2012-2013
New Accountability for Direct Measurement of Learning
INDIRECT ASSESSMENT STANDARDS
EXTERNAL ASSESSMENT STANDARDS
COLLEGE ASSESSMENT STANDARDS
is valid and reliableis approved by industry (advisory
committee or standards)meets quality guidelines for 3rd party
assessment if no rubric is used such as…
Program Assessment
2. EXTERNAL ASSESSMENT
Third party exam (i.e. NCLEX, Barb Cosmetology)
Required for job placement or minimal/no cost to student
Valid reliable Linked to technical skills required on the job
CONSIDERATIONS FOR SELECTING 3RD PARTY EXTERNAL ASSESSMENTS
Can we obtain the results/data? Are cost and administration
feasible? Does it add value or control entry to
the student/occupation? Is it valid, reliable, reputable,
recognized? Is it summative, cumulative or a
partial measurement of skills?
Concerns: Only measures part of the program outcomes-
“All Code-No Application”
Does not indicate job performance
Does it help employers?
Focused only on technical skills, what about academic skills and soft skills?
INDIRECT ASSESSMENT
GPA Course Completion Teacher developed exams that do not meet
the larger state or system standards
Proven processProgram and Course
Software ToolExpertise
What is WIDS?What is WIDS?Curriculum DesignCurriculum Design
For any discipline or delivery mode
Institutionally-defined
Instructor-defined
Exit Learning OutcomesProgram Outcomes Core Abilities Gen Ed Outcomes
WHAT CAN YOU CREATE WITH WIDS?
Program Documentation Outcome assessment charts Program profiles
Official Course Documentation Syllabi Assessments (rubrics and
checklists) Learning Plans/Teaching Plans Reporting Matrices
See Handout
Then… let’s see how they are generated!
See Handout
Then… let’s see how they are generated!