![Page 1: Progress Testing – concept, history, and recent developments Arno Muijtjens Dept. of Educational Development & Research, Faculty of Health, Medicine, and](https://reader036.vdocument.in/reader036/viewer/2022062423/56649e7a5503460f94b7aa28/html5/thumbnails/1.jpg)
Progress Testing – concept, history, and recent
developments
Arno Muijtjens
Dept. of Educational Development & Research,Faculty of Health, Medicine, and Life Sciences,
University of Maastricht, The Netherlands
![Page 2: Progress Testing – concept, history, and recent developments Arno Muijtjens Dept. of Educational Development & Research, Faculty of Health, Medicine, and](https://reader036.vdocument.in/reader036/viewer/2022062423/56649e7a5503460f94b7aa28/html5/thumbnails/2.jpg)
Overview
Progress Testing (PT)
•Why PT?
•What is it?
•What can it do for you?
•What do you need for PT?
•Issues, How to improve PT
![Page 3: Progress Testing – concept, history, and recent developments Arno Muijtjens Dept. of Educational Development & Research, Faculty of Health, Medicine, and](https://reader036.vdocument.in/reader036/viewer/2022062423/56649e7a5503460f94b7aa28/html5/thumbnails/3.jpg)
Why Progress Testing
• 1974 Maastricht Medical school, Problem Based Learning (PBL)
• Educational aims Assessment
• PBL self-directed, discovery learningintegrated, not discipline oriented
• End-of-unit tests test-directed learningdiscouraged individual learning trajectories,stimulated rote memorizationinstead of insight and understanding
![Page 4: Progress Testing – concept, history, and recent developments Arno Muijtjens Dept. of Educational Development & Research, Faculty of Health, Medicine, and](https://reader036.vdocument.in/reader036/viewer/2022062423/56649e7a5503460f94b7aa28/html5/thumbnails/4.jpg)
• Solution (Wynand Wijnen):remove direct connection between Curriculum and Assessment
End Objectives
Curriculum:
Assessment:
CurriculumAssessment
End Objectives
![Page 5: Progress Testing – concept, history, and recent developments Arno Muijtjens Dept. of Educational Development & Research, Faculty of Health, Medicine, and](https://reader036.vdocument.in/reader036/viewer/2022062423/56649e7a5503460f94b7aa28/html5/thumbnails/5.jpg)
• Assess complete knowledge domain (end objectives)
• Like final exam, but taken several times a year by all students
• Assessing long term, functional knowledge
![Page 6: Progress Testing – concept, history, and recent developments Arno Muijtjens Dept. of Educational Development & Research, Faculty of Health, Medicine, and](https://reader036.vdocument.in/reader036/viewer/2022062423/56649e7a5503460f94b7aa28/html5/thumbnails/6.jpg)
What is Progress Testing
• Comprehensive written test of 200 MCQs (single best answer + don’t know option)
• Sample across all areas of undergraduate training program (organ systems, basic, behavioral, clinical disciplines)
• Blueprint representing end objectives of undergraduate medical training
• All students from all training levels (all classes) are submitted to the same test at the same time
• Four newly constructed tests per year• Students may keep their test booklet and all
questions
![Page 7: Progress Testing – concept, history, and recent developments Arno Muijtjens Dept. of Educational Development & Research, Faculty of Health, Medicine, and](https://reader036.vdocument.in/reader036/viewer/2022062423/56649e7a5503460f94b7aa28/html5/thumbnails/7.jpg)
Question format (clinical)You are a general practitioner. At the end of your morning surgery your assistant tells you that mrs. Rhines (24 years old) has come to your surgery. She is very worried about a loss of vision. On further history taking she tells you that it is not actually a loss of vision but a more blurred vision and flickering circles before the eyes. She has never had this before. In addition she has a headache above her eyes without nausea. The headache lasted for some hours, but is gone now. On examination you find a normal vision and a sharp papilla on fundoscopy on both sides.
The most probable diagnosis is:A: sinusitis.B: migraine.C: arteriitis temporalis.D: acuut glaucomaE: ablatio retinae (retinal detachment)
![Page 8: Progress Testing – concept, history, and recent developments Arno Muijtjens Dept. of Educational Development & Research, Faculty of Health, Medicine, and](https://reader036.vdocument.in/reader036/viewer/2022062423/56649e7a5503460f94b7aa28/html5/thumbnails/8.jpg)
Question format (Basic Science)
The diaphragm consists of a central tendon plate and a muscular part. Which of the structures below passes through the tendon plate?
A. Aorta.B. Oesophagus.C. Vagus nerve.D. Vena cava.
![Page 9: Progress Testing – concept, history, and recent developments Arno Muijtjens Dept. of Educational Development & Research, Faculty of Health, Medicine, and](https://reader036.vdocument.in/reader036/viewer/2022062423/56649e7a5503460f94b7aa28/html5/thumbnails/9.jpg)
Blueprint categories
• Respiratory system (16 questions)
• Hormones and Metabolism (10 questions)
• Preventive Healh Care (7 questions)
![Page 10: Progress Testing – concept, history, and recent developments Arno Muijtjens Dept. of Educational Development & Research, Faculty of Health, Medicine, and](https://reader036.vdocument.in/reader036/viewer/2022062423/56649e7a5503460f94b7aa28/html5/thumbnails/10.jpg)
• Institutions share same end objectives(not the same curriculum)cooperative progress test possible
Curriculum 1Assessment
Curriculum 2
End Objectives
![Page 11: Progress Testing – concept, history, and recent developments Arno Muijtjens Dept. of Educational Development & Research, Faculty of Health, Medicine, and](https://reader036.vdocument.in/reader036/viewer/2022062423/56649e7a5503460f94b7aa28/html5/thumbnails/11.jpg)
• 4x per year, 200 MCQ’s taken by students of 5 medical schools in The Netherlands
• Joint construction, administration at same time, same standards, same rules and regulations
• 6 year undergraduate curriculum,24 measurement moments
NetherlandsInteruniversity progress test in Medicine
![Page 12: Progress Testing – concept, history, and recent developments Arno Muijtjens Dept. of Educational Development & Research, Faculty of Health, Medicine, and](https://reader036.vdocument.in/reader036/viewer/2022062423/56649e7a5503460f94b7aa28/html5/thumbnails/12.jpg)
Measurement moment
4 8 12 16 20 24
Test score %
0
10
20
30
40
50
60
Scores of UM students (n=1600) in 7 academic years (2005-2011)
![Page 13: Progress Testing – concept, history, and recent developments Arno Muijtjens Dept. of Educational Development & Research, Faculty of Health, Medicine, and](https://reader036.vdocument.in/reader036/viewer/2022062423/56649e7a5503460f94b7aa28/html5/thumbnails/13.jpg)
What can Progress Testingdo for you
• Repeated measurement of all medical knowledge
• Monitor knowledge growthProgress to end objectives
• Total and for each subdomain
![Page 14: Progress Testing – concept, history, and recent developments Arno Muijtjens Dept. of Educational Development & Research, Faculty of Health, Medicine, and](https://reader036.vdocument.in/reader036/viewer/2022062423/56649e7a5503460f94b7aa28/html5/thumbnails/14.jpg)
Reference: distribution of scores in a peer group
70
50
95
155
percentiles:
student’s scoreHIGH
MID
LOW
![Page 15: Progress Testing – concept, history, and recent developments Arno Muijtjens Dept. of Educational Development & Research, Faculty of Health, Medicine, and](https://reader036.vdocument.in/reader036/viewer/2022062423/56649e7a5503460f94b7aa28/html5/thumbnails/15.jpg)
Longitudinal representation
![Page 16: Progress Testing – concept, history, and recent developments Arno Muijtjens Dept. of Educational Development & Research, Faculty of Health, Medicine, and](https://reader036.vdocument.in/reader036/viewer/2022062423/56649e7a5503460f94b7aa28/html5/thumbnails/16.jpg)
Cumulated score – noise reduction
![Page 17: Progress Testing – concept, history, and recent developments Arno Muijtjens Dept. of Educational Development & Research, Faculty of Health, Medicine, and](https://reader036.vdocument.in/reader036/viewer/2022062423/56649e7a5503460f94b7aa28/html5/thumbnails/17.jpg)
inspect results recent test
![Page 18: Progress Testing – concept, history, and recent developments Arno Muijtjens Dept. of Educational Development & Research, Faculty of Health, Medicine, and](https://reader036.vdocument.in/reader036/viewer/2022062423/56649e7a5503460f94b7aa28/html5/thumbnails/18.jpg)
Science and Methodology
![Page 19: Progress Testing – concept, history, and recent developments Arno Muijtjens Dept. of Educational Development & Research, Faculty of Health, Medicine, and](https://reader036.vdocument.in/reader036/viewer/2022062423/56649e7a5503460f94b7aa28/html5/thumbnails/19.jpg)
![Page 20: Progress Testing – concept, history, and recent developments Arno Muijtjens Dept. of Educational Development & Research, Faculty of Health, Medicine, and](https://reader036.vdocument.in/reader036/viewer/2022062423/56649e7a5503460f94b7aa28/html5/thumbnails/20.jpg)
Kidneys and Urinary tract
![Page 21: Progress Testing – concept, history, and recent developments Arno Muijtjens Dept. of Educational Development & Research, Faculty of Health, Medicine, and](https://reader036.vdocument.in/reader036/viewer/2022062423/56649e7a5503460f94b7aa28/html5/thumbnails/21.jpg)
Median score of 4 universities
Median score of 1 university
Compare groups to evaluate a curriculum
![Page 22: Progress Testing – concept, history, and recent developments Arno Muijtjens Dept. of Educational Development & Research, Faculty of Health, Medicine, and](https://reader036.vdocument.in/reader036/viewer/2022062423/56649e7a5503460f94b7aa28/html5/thumbnails/22.jpg)
What do you need for PT
• Homogeneous curriculum: no distinct (early) specializations
• Central co-ordination and organization
• Acceptence by students and staff
![Page 23: Progress Testing – concept, history, and recent developments Arno Muijtjens Dept. of Educational Development & Research, Faculty of Health, Medicine, and](https://reader036.vdocument.in/reader036/viewer/2022062423/56649e7a5503460f94b7aa28/html5/thumbnails/23.jpg)
A B C D E F G H Departments
Item pool
ReviewCommittee
Test administration
Item bank Test info(students, teachers)
- Statistical information- Student comments
Item review process
![Page 24: Progress Testing – concept, history, and recent developments Arno Muijtjens Dept. of Educational Development & Research, Faculty of Health, Medicine, and](https://reader036.vdocument.in/reader036/viewer/2022062423/56649e7a5503460f94b7aa28/html5/thumbnails/24.jpg)
Overall assessment programme
Unit
evaluation
Progress TestKnowledge
Clinical skills
Professional behaviour
Unit 1 Unit 2 Unit 3 Unit 4 Unit 5 Unit 6
Portfolio
Longitudinal
evaluation
![Page 25: Progress Testing – concept, history, and recent developments Arno Muijtjens Dept. of Educational Development & Research, Faculty of Health, Medicine, and](https://reader036.vdocument.in/reader036/viewer/2022062423/56649e7a5503460f94b7aa28/html5/thumbnails/25.jpg)
Advantages PT• prevents a test-driven learning approach• rewards each individual learning activity • measures functional knowledge• repeated measurement of the same
domain of knowledge• resits not needed• early detection of outliers• rich source of feedback (students and
staff)• rich research potentials
cross-sectional and longitudinal design• joint construction and administration
benchmarking
![Page 26: Progress Testing – concept, history, and recent developments Arno Muijtjens Dept. of Educational Development & Research, Faculty of Health, Medicine, and](https://reader036.vdocument.in/reader036/viewer/2022062423/56649e7a5503460f94b7aa28/html5/thumbnails/26.jpg)
Issues
• Formula scoring (don’t know option and penalty for wrong answer)Year 1: 85% don’t know
• Efficiency of the measurement• Item relevance • Standard setting• Paper and pencil test:
Logistics Lack of flexibility (time,place)No multimedia
![Page 27: Progress Testing – concept, history, and recent developments Arno Muijtjens Dept. of Educational Development & Research, Faculty of Health, Medicine, and](https://reader036.vdocument.in/reader036/viewer/2022062423/56649e7a5503460f94b7aa28/html5/thumbnails/27.jpg)
Adaptive progress test
Question 1
Question 3
Question 4
Question 6
Question 5
Question 7
Knowledge level/Difficulty Knowledge level Student
Student
Question 2
![Page 28: Progress Testing – concept, history, and recent developments Arno Muijtjens Dept. of Educational Development & Research, Faculty of Health, Medicine, and](https://reader036.vdocument.in/reader036/viewer/2022062423/56649e7a5503460f94b7aa28/html5/thumbnails/28.jpg)
Exploring adaptive progress test
• Not all PT items appropriate for adaptive testing23% “growth-items” (19% fit 2p IRT model)Masterphase (years 4-6)
• Growth-items cover blueprint categories• % growth-items 3.4 times higher with items of
high relevance• Prototype adaptive PT tested later this year• Exploration of item content characteristics in
relation to “relevance” and “growth”
![Page 29: Progress Testing – concept, history, and recent developments Arno Muijtjens Dept. of Educational Development & Research, Faculty of Health, Medicine, and](https://reader036.vdocument.in/reader036/viewer/2022062423/56649e7a5503460f94b7aa28/html5/thumbnails/29.jpg)
Applications of Progress Testing
• Other progress test networks– Germany– Sweden– UK (with NBME)
• Individual medical schools:– Gent, Berlin, Tampere-Finland, Manchester, Peninsula,
Liverpool, Pretoria, McMaster Hamilton, Canada• Psychology
– Maastricht, Rotterdam• Postgraduate training (Netherlands)
– General Practice – Pediatrics– OBGYN
![Page 30: Progress Testing – concept, history, and recent developments Arno Muijtjens Dept. of Educational Development & Research, Faculty of Health, Medicine, and](https://reader036.vdocument.in/reader036/viewer/2022062423/56649e7a5503460f94b7aa28/html5/thumbnails/30.jpg)
THANK YOU!
Additional information
and literature references:
www.ivtg.nl
(down left button “English”)
![Page 31: Progress Testing – concept, history, and recent developments Arno Muijtjens Dept. of Educational Development & Research, Faculty of Health, Medicine, and](https://reader036.vdocument.in/reader036/viewer/2022062423/56649e7a5503460f94b7aa28/html5/thumbnails/31.jpg)
![Page 32: Progress Testing – concept, history, and recent developments Arno Muijtjens Dept. of Educational Development & Research, Faculty of Health, Medicine, and](https://reader036.vdocument.in/reader036/viewer/2022062423/56649e7a5503460f94b7aa28/html5/thumbnails/32.jpg)
Item Relevance*
Criterion Very Relevant
A. Medical Knowledge requires a thorough study and understanding of the field medicine.
B. Ready Knowledge needed at the ready at any moment of the day.
C. Incidence in Practice needed in many medical practical situations
D. Prevalence or High-Risk essential to manage high prevalence or high risk medical situations.
E. Knowledge Foundations in the Medical Curriculum
forms the basis for one or more concepts and has to remain as explicit knowledge (eg, the Frank-Starling mechanism in the context of congestive heart failure).
*adapted from Schuwirth, L. Personal communication, July 2011.
![Page 33: Progress Testing – concept, history, and recent developments Arno Muijtjens Dept. of Educational Development & Research, Faculty of Health, Medicine, and](https://reader036.vdocument.in/reader036/viewer/2022062423/56649e7a5503460f94b7aa28/html5/thumbnails/33.jpg)
Cumulative score
10002000
30004000
Measurement moment1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Sco
re
-10
0
10
20
30
40
![Page 34: Progress Testing – concept, history, and recent developments Arno Muijtjens Dept. of Educational Development & Research, Faculty of Health, Medicine, and](https://reader036.vdocument.in/reader036/viewer/2022062423/56649e7a5503460f94b7aa28/html5/thumbnails/34.jpg)
![Page 35: Progress Testing – concept, history, and recent developments Arno Muijtjens Dept. of Educational Development & Research, Faculty of Health, Medicine, and](https://reader036.vdocument.in/reader036/viewer/2022062423/56649e7a5503460f94b7aa28/html5/thumbnails/35.jpg)