Strengthening Academic Assessment for Programs
A Systemic Approach
Timothy S. Brophy, Ph.D., Director of Institutional Assessment
University of Florida Office of the Provost
Today’s GoalsDescribe and explain assessment systems
Review UF’s program assessment components
Discuss program assessment systems and steps to strengthen them
Review examples
Assessment SystemsA system is a complex whole in which there are multiple parts or phases that can stand alone but are stronger when linked logically into a cohesive combinationThe system can use linear and nonlinear sequences of events or processes, or a combination of both
Assessment systems consist of curricular, educational, and procedural components
The UF Assessment System is one example
UF Assessment System
Establish Mission,
Goals, and Outcomes
Assessment Planning
Implement the Plan
and Gather Data
Interpret and
Evaluate the Data
Modify and Improve
May –
Assessm
ent Plans and
Effectiveness
Docum
entation R
eports subm
itted for the next AY
October - Assessment
Data, results, and use of results for
previous AY entered into Compliance
Assist!
Assessment and Institutional Effectiveness
Data Reporting
Elements of program assessment systems
Catalog• ALCs• Graduation requirements
Assessment Plans• SLOs• Measurement tools• Curriculum Maps and Assessment matrices
Compliance Assist!• Goals • Results• Use of results• Student work samples
The Academic Learning Compact
2012-13 UF Undergraduate Catalog
Academic Learning Compacts identify the skills students should acquire if they follow their major’s prescribed course of study. These skills, known collectively as Student Learning Outcomes, describe the core learning expectations that UF is required to assess for each baccalaureate degree program. Core learning expectations identify communication, critical thinking, and content knowledge skills, as well as additional learning outcomes specific to the major.
Student Learning Outcomes
What students are expected to learn by the completion of their baccalaureate degree
Individual Student AssessmentsThe different ways in which UF will measure whether students have successfully completed the learning outcomes for a major. These assessments can include a passing score on a particular test or final project, term paper, portfolio, etc.- 2012-13 UF Undergraduate Catalog
How are these components related?
Academic Learning
Compact
What we sometimes have
SLO
SLO
SLO
Assessm
en
t Assessment
Assessment
Course
Course
Course
What we should have in all programs
Academic Learning Compact
SLO
Assessment
Course
Assessment
Course Course
SLO
Assessment
Course
Assessment
Course
SLO
Assessment
Course
Assessment
Course
How do we get from
ALCSLO
Assessment
Course
Assessment
Course Course
SLO
Assessment
Course
Assessment
Course
SLO
Assessment
Course
Assessment
Course
SLO
Assessme
ntC
ou
rse
Course
Academic
Learning
Compact
SLO
SL
O
AssessmentCourse
THIS TO THIS?
And avoid this when dealing with assessment?
SLOs!!
Plans!!
!Rubr
ics!!!
Start with the ALC program description
Components of a strong ALC program description:
Establishes learning goals for the degree program – the “core learning expectations”
Addresses the relationship of the degree to the job or career
Characteristics of Program Learning Goals found in program descriptionsThese are broad statements that present multiple
measurable components so the statement itself may be “unmeasurable”
These often use verbs like “understand” and “learn” and “gain the ability to…” etc. in the program description
Why? These are the broad LEARNING GOALS of the program –
not the LEARNING OUTCOMESSLOs represent the measurable components of the SLOs
– they are “S.M.A.R.T.” SLOs are stated using active verbsHowever, the SLOs and goals share common language
and intent
This model allows you to develop assessments that measure the outcomes, and that then connect directly to the program goals
Specific Learning Outcome – the course-level SLO
General Learning Outcome – the program-level SLO – these describe what students will do to demonstrate the learning
goals
Learning Goal Level – programs establish learning goals for the degree – at UF these are described in the Academic
Learning Compact
Developing an Assessment System: A Three-level Model (Carriveau, 2010)
A sample description:Materials Science EngineeringThe major enables you to develop an understanding of materials systems and their role in engineering. Emphasis is placed on the ability to apply knowledge of mathematics, science and engineering principles to materials science and engineering; to design and conduct experiments, as well as to analyze and interpret data; and to design a system, component or process to meet desired needs within realistic constraints such as economic, environmental, social, political, ethical, health and safety, manufacturability and sustainability.
Learning Goals
Connection to the job or career
Students who complete the MSE degree will:
Understand materials systems and their role in engineering
Gain the ability to apply knowledge of mathematics, science and engineering principles to materials science and engineering
Level 1: Learning Goals based on the MSE ALC
Content Knowledge• Apply knowledge of mathematics, science and engineering
principles to materials science and engineering. • Design and conduct materials science and engineering
experiments and analyze and interpret the data.
Critical Thinking• Design a materials science and engineering system, component
or process to meet desired needs within realistic economic, environmental, social, political, ethical, health and safety, manufacturability and sustainability constraints.
Communication• Communicate technical data and design information effectively
in speech and in writing to other materials engineers.
Level 2 – Program Student Learning Outcomes for MSE
Learning Goals: Understand materials systems and their role in engineeringAbility to apply knowledge of mathematics, science and engineering principles to materials science and engineering.
Student Learning Outcomes: Design a materials science and engineering system, component or process to meet desired needs within realistic economic, environmental, social, political, ethical, health and safety, manufacturability and sustainability constraints.Communicate technical data and design information effectively in speech and in writing to other materials engineers
MSE: Connecting Goals to Outcomes
Goal
SLO
Learning Goals: Understand materials systems and their role in engineeringAbility to apply knowledge of mathematics, science and engineering principles to materials science and engineering.
Student Learning Outcomes: Apply knowledge of mathematics, science and engineering principles to materials science and engineering. Design and conduct materials science and engineering experiments and analyze and interpret the data
MSE: Connecting Goals to Outcomes
Level 1
Level 2
SLOs Additional Assess-ments
Content Knowledge
EMA3050 EMA3066 EMA4714 EMA3080
CEMA3513
C EMA4714
#1 I R A Senior exit survey
#2 I R A Senior exit survey
Critical Thinking
EMA3066 EMA4223 EMA4714
#3 I R A Senior exit survey
Communication
EMA3080C
EMA3013C
EMA3513C
#4 I R A Senior exit survey
Connecting Program SLOs to Courses MSE Curriculum Map
Assessments in the boxes marked A are conducted using specific homework, exam, or assignment questions aligned with that SLO.Source: 2011-12 MSE Academic Assessment Plan
Completing the System
Align course objectives with program SLOs
Develop assessments of the course objectives
Select course assessments as assessments of the program SLOs – vet these with the program faculty
Completing the System
Collect data
Analyze and interpret
ModifyReport to Compliance Assist!
A synopsis…
Program Description
Learning Goals
Program SLOs
Course objectives
and assessment
s
Data collection, analysis,
and interpretati
on
Annual Reporting
and Planning
All of this leads to
SUCCES
S!
KEEP UP THE GOOD WORK