blueprint analysis and feedback based on the progress test lee coombes jeff stratford
TRANSCRIPT
Blueprint analysis and feedback based on the Progress Test
Lee CoombesJeff Stratford
Progress Test
• 125 MCQ items, 4 times a year• Don’t Know option (negative marking)• 1 formative test, 19 summative• All students take the same test• Set at the level of a newly qualified doctor
Progress Test
0.0
10.0
20.0
30.0
40.0
50.0
60.0
70.0
80.0PT
01PT
02PT
03PT
04PT
05PT
06PT
07PT
08PT
09PT
10PT
11PT
12PT
13PT
14PT
15PT
16PT
17PT
18PT
19PT
20PT
21PT
22PT
23PT
24PT
25PT
26PT
27PT
28PT
29PT
30PT
31PT
32PT
33PT
34PT
35PT
36PT
37PT
38PT
39PT
40PT
41PT
42PT
43PT
44PT
45
Perc
enta
ge
Student Progress02/03 Cohort03/04 Cohort04/05 Cohort05/06 Cohort06/07 Cohort07/08 Cohort08/09 Cohort09/10 Cohort10/11 Cohort11/12 Cohort12/13 Cohort
Student Feedback
Blueprinting
• Based on the GMC PLAB blueprint• 17 domains, 17 Skills• Classified by main question theme• Some areas have more questions than others• Aim has been to provide detailed feedback on
each area for students• Both single test and aggregated yearly average
Student Feedback
• ATLAS
Questions
• How can we use the information?– Provides valuable feedback
• Differences in gender/ethnic groups?– We’ve previously seen gender and ethnicity
differences in the data– Are these general, or specific to certain areas?
Methods
• Tests covering 7 year period – 2006/07 to 2012/13
• Examining the 20 tests students take• Different tests are grouped together• No adjustment for test difficulty• # items varies in each test/skill/domain
PCMD Progress Test
0
10
20
30
40
50
60
70
80
90
100
Test
1
Test
2
Test
3
Test
4
Test
5
Test
6
Test
7
Test
8
Test
9
Test
10
Test
11
Test
12
Test
13
Test
14
Test
15
Test
16
Test
17
Test
18
Test
19
Test
20
Applied Medical Knowledge ScoresScoreCorrectIncorrectDon't Know
Gender
0
10
20
30
40
50
60
70
80
90
100
Test
1
Test
2
Test
3
Test
4
Test
5
Test
6
Test
7
Test
8
Test
9
Test
10
Test
11
Test
12
Test
13
Test
14
Test
15
Test
16
Test
17
Test
18
Test
19
Test
20
Applied Medical Knowledge ScoresFemale ScoreFemale CorrectFemale IncorrectFemale Don't KnowMale ScoreMale CorrectMale IncorrectMale Don't Know
Gender
Domain: Neurological Skill: Pathology
0
10
20
30
40
50
60
70
80
90
100
Test
1
Test
2
Test
3
Test
4
Test
5
Test
6
Test
7
Test
8
Test
9
Test
10
Test
11
Test
12
Test
13
Test
14
Test
15
Test
16
Test
17
Test
18
Test
19
Test
20
NeurologicalScore (F)Correct (F)Incorrect (F)Don't Know (F)Score (M)Correct (M)Incorrect (M)Don't Know (M)
0
10
20
30
40
50
60
70
80
90
100
Test
1
Test
2
Test
3
Test
4
Test
5
Test
6
Test
7
Test
8
Test
9
Test
10
Test
11
Test
12
Test
13
Test
14
Test
15
Test
16
Test
17
Test
18
Test
19
Test
20
SBM PathologyScore (F)Correct (F)Incorrect (F)Don't Know (F)Score (M)Correct (M)Incorrect (M)Don't Know (M)
Gender
Domain: Women's Health Skill: Diagnosis
0
10
20
30
40
50
60
70
80
90
100
Test
1
Test
2
Test
3
Test
4
Test
5
Test
6
Test
7
Test
8
Test
9
Test
10
Test
11
Test
12
Test
13
Test
14
Test
15
Test
16
Test
17
Test
18
Test
19
Test
20
Womens HealthScore (F)Correct (F)Incorrect (F)Don't Know (F)Score (M)Correct (M)Incorrect (M)Don't Know (M)
0
10
20
30
40
50
60
70
80
90
100
Test
1
Test
2
Test
3
Test
4
Test
5
Test
6
Test
7
Test
8
Test
9
Test
10
Test
11
Test
12
Test
13
Test
14
Test
15
Test
16
Test
17
Test
18
Test
19
Test
20
DiagnosisScore (F)Correct (F)Incorrect (F)Don't Know (F)Score (M)Correct (M)Incorrect (M)Don't Know (M)
Ethnicity
0
10
20
30
40
50
60
70
80
90
100
Test
1
Test
2
Test
3
Test
4
Test
5
Test
6
Test
7
Test
8
Test
9
Test
10
Test
11
Test
12
Test
13
Test
14
Test
15
Test
16
Test
17
Test
18
Test
19
Test
20
Applied Medical Knowledge ScoresBME ScoreBME CorrectBME IncorrectBME Don't KnowWhite ScoreWhite CorrectWhite IncorrectWhite Don't Know
Ethnicity
Domain: Neurological Domain: Digestive
0
10
20
30
40
50
60
70
80
90
100
Test
1
Test
2
Test
3
Test
4
Test
5
Test
6
Test
7
Test
8
Test
9
Test
10
Test
11
Test
12
Test
13
Test
14
Test
15
Test
16
Test
17
Test
18
Test
19
Test
20
NeurologicalScore (BME)Correct (BME)Incorrect (BME)Don't Know (BME)Score (W)Correct (W)Incorrect (W)Don't Know (W)
0
10
20
30
40
50
60
70
80
90
100
Test
1
Test
2
Test
3
Test
4
Test
5
Test
6
Test
7
Test
8
Test
9
Test
10
Test
11
Test
12
Test
13
Test
14
Test
15
Test
16
Test
17
Test
18
Test
19
Test
20
DigestiveScore (BME)Correct (BME)Incorrect (BME)Don't Know (BME)Score (W)Correct (W)Incorrect (W)Don't Know (W)
Ethnicity
Skill: Management Plan Skill: Diagnosis
0
10
20
30
40
50
60
70
80
90
100
Test
1
Test
2
Test
3
Test
4
Test
5
Test
6
Test
7
Test
8
Test
9
Test
10
Test
11
Test
12
Test
13
Test
14
Test
15
Test
16
Test
17
Test
18
Test
19
Test
20
Management PlanScore (BME)Correct (BME)Incorrect (BME)Don't Know (BME)Score (W)Correct (W)Incorrect (W)Don't Know (W)
0
10
20
30
40
50
60
70
80
90
100
Test
1
Test
2
Test
3
Test
4
Test
5
Test
6
Test
7
Test
8
Test
9
Test
10
Test
11
Test
12
Test
13
Test
14
Test
15
Test
16
Test
17
Test
18
Test
19
Test
20
DiagnosisScore (BME)Correct (BME)Incorrect (BME)Don't Know (BME)Score (W)Correct (W)Incorrect (W)Don't Know (W)
Conclusions
• Gender differences are skill/domain specific• Ethnic difference appears more general• Caused by the test, or is the test accurately
measuring pre-existing differences?– Evidence of similar differences elsewhere
• What role do other demographics play?– Home\EU\Other– Entry Type (Graduate\Adult)
Thank You
Lee CoombesJeff Stratford