What Works - Course and Program Assessment in STEM
PKAL Network – 7 October 2013Dr. Dianne Raubenheimer,
Director of Research, Planning and Assessment, Meredith College
By the end of this session, participants will be able to: Identify some purposes of assessmentExplain selected assessment terminologyArticulate ways to align outcomes,
teaching, learning and assessment Consider strategies for linking course and
program assessment
Ice Breaker: Write down some thoughts
A show of hands:Gathering information Measuring goals and outcomesUsing diverse measures and sourcesAnalyzing and interpreting informationReporting resultsEngaging stakeholdersMaking improvements
Assessment is the systematic collection, review, and use
of information about educational programs undertaken for the purpose
of improving student learning and development
(Palomba & Banta,1999).
Task: Individually - Write down different types of
assessment tasks you use in your courses. Use one small post-it for each type of assessment.
As a group - Sort the post-its into two columns – direct and indirect assessment methods Direct assessments provide for direct examination or
observation of student knowledge of skills against measureable learning outcomes.
Indirect assessments of student learning ascertain the perceived extent of value of learning experiences. They assess opinions or thoughts about student knowledge or skills.
Task: Looking at the different assessment
strategies. In your groups, discuss some thoughts about the purposes of assessment
You may be able to cluster some of the methods with particular purposes
Assessment of student learning (knowledge, skills and attitudes)
Improvement of student learning Improvement of teaching Program development, monitoring and
improvement SoTL, SoA, SoP
Accreditation Public accountability / Performance
scorecard
Internal / Intrinsic
External / Extrinsic
Outcomes
Reciprocal relationship
An example from NC State : Using an observation rubric, observed 15
instructors in different disciplines teaching All indicated they were using innovative
practices with technology Instructor noted the learning outcomes for the
lesson being taught Observer used the SOLO taxonomy to score the
SOLO level of instruction Observer used the SOLO taxonomy to score the
SOLO level at students were engaged in the class
Then gathered student work from the faculty that tested the outcome
Examined the assessment task and student work using the SOLO taxonomy (Raubenheimer, et al, 2007)
SOLO category Representation Type of outcome (Biggs, 2004)
Unanticipated Extension(Extended Abstract)
Create SynthesiseHypothesise ValidatePredict DebateTheorise
Logically Related (Relational)
Apply OutlineDistinguish AnalyseClassify ContrastSummarise Categorise
Multiple Points(Multistructural)
Explain DefineList SolveDescribe Interpret
Single Point (Unistructural)
State RecogniseRecall QuoteNote Name
Instructor SOLO
2
5
10
1
2
3
8
12
6
2
6
21
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Single point (e.g. tell,state, recall)
Multiple points (e.g.rework, clarify, explain)
Logically related (e.g.inquire, apply, distinguish)
Unanticipated extension(e.g. imagine, elaborate)
Always
Often
Sometimes
Rarely
Never
Student Solo
5
7
9
123
4
3
1
4
3
1
2
1
22
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Single point (e.g. tell, state, recall) Multiple points (e.g. clarify,
explain, define, describe)
Logically related (e.g. inquire,
apply, analyze)
Unanticipated extension (e.g.
imagine, elaborate, synthesize)
Always
Often
Sometimes
Rarely
Never
Scored Using SOLO Taxonomy
Level of outcome taught
Level of related student assessment task
Single 3 7
Multiple point 2 3
Logically related 8 2
Unanticipated extension
0 1
In your groups: Discuss ways in which you align teaching
and assessment of student learning outcomes?
Select 3 individuals to each act as chair, scribe and reporter for the group
http://www.arapahoe.edu/about-acc/presidents-office/assessment
On your own:Use the “Program Level-Assessment of Student
Learning: Self-assessment worksheet” to assess your program at your institution
In your groups: Discuss ways in which you align teaching
and assessment of student learning outcomes in your courses (columns 1-3)?
Select 3 individuals to each act as chair, scribe and reporter for the group
Ensure the program outcomes reflect what you want students to have learned/attained/ achieved by the time they graduateGenerating appropriate program outcomesDevelop program outcomes - course
assessment matricesSelecting appropriate assessment
tasks/methodsDevelop a system for keeping track of
program dataGather, review data and use data for ongoing,
systematic program review and improvement
Develop an implementation matrix associated with program outcomesH= high, M=medium; Low = L
Course1
Course2
Course3
Course4
Course5
Etc
Outcome 1
H M H
Outcome 2
M H M L
Outcome 3
H L M
Outcome 4
H L L H
Etc
Develop an implementation and assessment matrix associated with program outcomes I = implement; A = assessCourse
1Course2
Course3
Course4
Course5
Etc
Outcome 1
I & A I I & A
Outcome 2
I I & A I I
Outcome 3
I & A I I & A
Outcome 4
I & A I I I & A
Etc
Expected Outcomes
BIO183
BME 201
BME 203
BME 204
BME210
BME 252
BME 301
BME 302
Co-curricula/ other sources
Outcome (a) Graduating senior survey
Sub-Outcome a1 A A A
Sub-Outcome a2 A ASub-Outcome a3 A A ASub-Outcome a4 A A FE examOutcome (b) Graduating
senior surveySub-Outcome b1 x UG Research
Sub-Outcome b2 A A A UG Research
Sub-Outcome b3 A A
Sub-Outcome b4 A A
Outcome (e) Graduating senior survey
Sub-Outcome e1 A A A A Design day
Sub-Outcome e2 A A
Sub-Outcome e3 A
Sub-Outcome e4 ASub-Outcome e5 A
Articulate sub-outcomes on the matrix
GraduatingSenior Survey
FacultySurvey
WPS 332Test & homework
scores
WPS 360Math-
intensive test questions
WPS 475Math skills
test
Goal Actual Goal Actual Goal Actual Goal Actual Goal ActualE:
P: 30%+C:
NI:<10%
E: 16%P: 78%C: 7%NI: 0%
E: P: 30%+
C: NI:<10
%
E: 11%P: 67%C: 9%NI: 3%
Avg. scoresB or
higher
Avg. Score
B+
Avg. scoresB or
higher
Avg. Score
B+
Avg. scoresB or
higher
Avg. Score
B+
Determine assessment methods and expected performance criteria(level of performance )
Develop a system for recording and tracking results
Best practices indicate that assessment
is systematic and systemic. It occurs at multiple levels in an
institution, vertically and horizontally,
involving processes that are connected and integrated
Assessment: Arapahoe Community College http://www.arapahoe.edu/about-acc/presidents-office/assessment
Biggs, J. (2004). Teaching for quality learning at university. Berkshire, UK: Open University Press. Heer, R. (2009). A Model of Learning Objectives. Center for Excellence in Learning and Teaching,
Iowa State University. Maki, P.L. (2004). Assessing for learning: Building a sustainable commitment across the institution.
Sterling, VA: Stylus. Mertler, C.A. (2003). Classroom assessment: A practical guide for educators. Los Angeles: Pyrczak. Ozturk, H. & Raubenheimer, C.D. (2011). PAT: An online program A\assessment tool. Paper
presented at 2011 American Society for Engineering Education International Conference, Vancouver, Canada.
Palomba, C.A & Banta, T.W. (1999). Assessment essentials: Planning, implementing and improving assessment in higher education. San Francisco, CA: Jossey-Bass.
Raubenheimer, C. D., Spurlin, J., Martin, S., & Mehlenbacher, B. Faculty in Technology-Rich Contexts: Connecting Teaching, Learning, and Assessment in the Classroom. 2007 UNC TLT Conference Proceedings.
Rogers, G. (2004). Self-Assessment: Quality Assurance of Program Level-Assessment of Student Learning. Rose Hulman Institute of Technology.
Spurlin, J.E., Rajala, S.A., & Lavelle, J.P. (2008). Designing better engineering education through assessment: A practical resource for faculty and department chairs on using assessment and ABET criteria to improve student learning. Sterling, VA, Stylus Publishing.