assessment fyfa cmg
TRANSCRIPT
Assessment- Making Thinking Visible
Cormac McGrathDirector Unit for Medical Education (CLK, LIME)
05/01/23Centre for Medical Education, Karolinska Institutet
Agenda: Assessment and MCQs
Student learning Interpolated memory tests Formative and summative assessment Feedback
Outcome based education Constructive alignment
Criterion versus norm based assessment Sketching a way forward
Constructing MCQs Validating MCQs Think aloud protocols Digitalising exams
Centre for Medical Education, Karolinska Institutet
From idea to action: a series of four workshops MCQ construction
Building a database Testing your MCQs
Face validity Testing on students
Talk-aloud protocols Scaling up
digitalising the project
05/01/23Centre for Medical Education, Karolinska Institutet
Discussion
Positive experiences of assessing students’ work What concerns/ queries do you have about assessment?
Pair and Share2-3 min
05/01/23Centre for Medical Education, Karolinska Institutet
Do student know what they don’t know? Dunning-Kreuger effect Students may: fail to recognize their own
lack of skill fail to recognize genuine
skill in others fail to recognize the extent
of their inadequacy recognize and
acknowledge their own lack of skill, after they are exposed to training for that skill
05/01/23Centre for Medical Education, Karolinska Institutet
Student learning: What do we know?Backwash effect
May 1, 2023
6
ILOs Teaching methods Examination
Examination Study methods Student learning
What supports learning?
Interpolated memory testsstudents retained focus less likely to mind
wanderscored better on
cumulative tests Szpunar K,K, et al.,
2012 (Interpolated memory tests reduce mind wandering and improve learning of online learning)
05/01/23Centre for Medical Education, Karolinska Institutet
What supports learning?
Interconnectedness Meaningfulness
05/01/23Centre for Medical Education, Karolinska Institutet
Student learning: Feedback
Synthesis of 800 meta analysis, involving 80 million students Biggest effect on student learning: • Learners’ knowledge of goals/criteria• Results/feedback to the learner
• Feedback About the Task• Feedback About the Processing of the Task• Feedback About Self-Regulation
• The teacher's pedagogical skills• Classroom environment• Analyze teaching occasions together with colleagues
Hattie, Synligt lärande, rapport SKL , 2011
How do can we support student learning? Formative assessment
- Continuously runs during the teaching/ learning process - Diagnostic: gives feedback to students and teachers on
* strengths and weaknesses* difficulties* misunderstandings
- Gives an opportunity to modify/ improve
05/01/23Centre for Medical Education, Karolinska Institutet
Discussion
To which degree do to you provide opportunities for interpolated “memory” tests in your teaching today?
05/01/23Centre for Medical Education, Karolinska Institutet
Cormac McGrathCormac McGrath, UME 13
Outcome based education
Outcomes at different levels dictate student learning
Outcomes at programme level
Specific course goals
Teaching ocassion outcomes
Cormac McGrathCormac McGrath, UME 14
Course design and alingment
Learning outcomes
Assessment
Learning activities
Evaluation
Biggs, 2003
05/01/23Centre for Medical Education, Karolinska Institutet
Progression in the choice of verbsSOLO taxonomy
IdentifyMake simple procedures
NumberDescribeListCombine
CompareContrastExplain relationshipsAnalyseRelateApply
Misses the point
TheoriseGeneraliseMake a hypothesisReflect
(Biggs & Tang 2007)
Discussion
Review your intended learning outcomes using the SOLO taxonomy.
05/01/23Centre for Medical Education, Karolinska Institutet
Criterion V norm-referenced assessment Criterion-referenced
assessment: How well students have
learnt what we intended them to learn.
Criterion-referenced tests and assessments are designed to measure student performance against a fixed set of predetermined criteria or learning standards—i.e., concise, written descriptions of what students are expected to know and be able to do at a specific stage of their education.
Norm-referenced assessment: Comparing students
performances with each other, by ranking.
Compare to a cohort Relative to a group
05/01/23Centre for Medical Education, Karolinska Institutet
Discussion
Which resonates most strongly with you; norm referenced or criterion referenced assessment? 2-3 min
To which extent do the assessment criteria express distinct qualitative differences?
Can you give examples? How do you share this with the students? How well do you discuss examination and
assessment/assessment criteria?
05/01/23Centre for Medical Education, Karolinska Institutet
05/01/23Centre for Medical Education, Karolinska Institutet
Change of focus: Traditional MCQ assessment Emphasis on product and declarative knowledge Is fair (?) Is economical Encourages repetition of large amounts of literature, which can provide
opportunity for overview of the subject
Criticism: Encourages quick answers rather than slow investigation Correctness rather than risk taking Memory rather than conceptual development Reproduction rather than creative application Little space/time for feedback Difficult to measure deep understanding of concepts and context
(Lindström, 1997)
Powerful examinations?
The design is supposed to create the possibility to assess: specific cognitive competences such as problem-solving,
including formulating questions and critical thinking, information competences, such as searching for relevant
information, making informed judgements, efficient use of information, analysing data,
communication competences, such as presenting data communicatively, both oral and written;
meta-cognitive competences such as self-reflection and self-evaluation.
Bloom and SOLO
Taxonomy Level Representative Verbs SOLO
Evaluation•Critique•Summarize
S4
Synthesis•Organize•Design
S3
Analysis•Compare•Categorize
S2
Application•Organize•Solve
S2
Comprehension•Distinguish•Match
S1
Knowledge•Identify•Label
S1
05/01/23Centre for Medical Education, Karolinska Institutet
Can we achieve and assess higher order thinking with MCQs
05/01/23Centre for Medical Education, Karolinska Institutet
From idea to action: a series of four workshops MCQ construction
Building a database Testing your MCQs
Face validity Testing on students
Talk-aloud protocols Scaling up
digitalising the project
05/01/23Centre for Medical Education, Karolinska Institutet
Summary
Students’ learning is guided by examinations Backwash effect
Formative assessment facilitates learning Recurrent testing promotes learning
As course leader you can influence the design of your course and the ratio between teaching and examination.
05/01/23Centre for Medical Education, Karolinska Institutet
What do we want to assess? How do we use MCQs to achieve that? Different levels of learning/understanding
Identify (what is this muscle called) Describe/list (which muscle are involved in elbow flexion) Relate/apply (muscle x is injured how does it affect movement y) Theorise/generalise/Evolve
The different types of questions
Anatomy
”Problem solving”
Isolated structure
Single joint
Multiple Joints
1 2 3 4
Identify/list
Describe/apply
Describe/apply/evolve
Cormac McGrathPedagogik för universitetslärare 35
Progression of verbSOLO taxonomy
IdentifiesSimple procedures
NumbersDescribesListsCombines
ComparesContrastsExplains connectionsAnalysesApplies
Misses the point
Creates theoriesGeneralisesHypothesises Reflects
(Biggs 2003)
05/01/23Centre for Medical Education, Karolinska Institutet
References
Biggs, J. &C. Tang (2007). Teaching for Quality Learning at University. SRHE & Open University Press, Buckingham.
Carroll, J. &C.-M. Zetterling (2009). Guiding students away from plagiarism. KTH Learning Lab, Stockholm.
McConnell, DA., Steer, DN., & Owens, KD. (2003) Assessment and active learning strategies for introductory geology courses. Journal of Geoscience Education, v. 51 n:o 2, p. 205-216