implementing a new multilingual selection model across 27 european countries gilles guillard, markus...
TRANSCRIPT
![Page 1: Implementing a New Multilingual Selection Model Across 27 European Countries Gilles Guillard, Markus Nussbaum](https://reader038.vdocument.in/reader038/viewer/2022110322/56649d145503460f949e93e9/html5/thumbnails/1.jpg)
Implementing a New Multilingual Selection Model Across 27 European Countries Gilles Guillard, Markus NussbaumGilles Guillard, Markus Nussbaum
![Page 2: Implementing a New Multilingual Selection Model Across 27 European Countries Gilles Guillard, Markus Nussbaum](https://reader038.vdocument.in/reader038/viewer/2022110322/56649d145503460f949e93e9/html5/thumbnails/2.jpg)
The European Union
Economic &Social Committee
Committee of the Regions
Ombudsman
Data ProtectionSupervisor
Council
Parliament
Court of Justice
Court of Auditors
Commission
![Page 3: Implementing a New Multilingual Selection Model Across 27 European Countries Gilles Guillard, Markus Nussbaum](https://reader038.vdocument.in/reader038/viewer/2022110322/56649d145503460f949e93e9/html5/thumbnails/3.jpg)
The EU Institutions
Posts in 2012
European Parliament 6655
Council of the European Union 3153
European Commission 25478
Court of Justice 1952
European Court of Auditors 887
European Economic and Social Committee 724
Committee of the Regions 531
European Ombudsman 66
European Data Protection Supervisor 43
European External Action Service 1670
Total 41159
European External
Action Service4% Others
3%
European Parliament
16%
Council of the
European Union
8%
European Commission
62%
European Court
of Auditors2%
Court of Justice
5%
![Page 4: Implementing a New Multilingual Selection Model Across 27 European Countries Gilles Guillard, Markus Nussbaum](https://reader038.vdocument.in/reader038/viewer/2022110322/56649d145503460f949e93e9/html5/thumbnails/4.jpg)
Administrators (AD)e.g. policy officers, lawyers, auditors, scientific officers, translators, interpreters, communication & press officers …
Qualification: University degree (bachelor)
Assistants (AST)e.g. secretaries, HR assistants, financial assistants, conference technicians …
Qualification: Secondary education (at least) and relevant professional experience
Administrators (AD)e.g. policy officers, lawyers, auditors, scientific officers, translators, interpreters, communication & press officers …
Qualification: University degree (bachelor)
Assistants (AST)e.g. secretaries, HR assistants, financial assistants, conference technicians …
Qualification: Secondary education (at least) and relevant professional experience
Entry Streams
![Page 5: Implementing a New Multilingual Selection Model Across 27 European Countries Gilles Guillard, Markus Nussbaum](https://reader038.vdocument.in/reader038/viewer/2022110322/56649d145503460f949e93e9/html5/thumbnails/5.jpg)
Selection in Cycles
![Page 6: Implementing a New Multilingual Selection Model Across 27 European Countries Gilles Guillard, Markus Nussbaum](https://reader038.vdocument.in/reader038/viewer/2022110322/56649d145503460f949e93e9/html5/thumbnails/6.jpg)
Competition launched in March 2012
45000+ candidates for 296 posts
5 fields: Audit, EU Public Administration, Communication, Law, External Relations
Competition launched in March 2012
45000+ candidates for 296 posts
5 fields: Audit, EU Public Administration, Communication, Law, External Relations
The AD Cycle 2012
![Page 7: Implementing a New Multilingual Selection Model Across 27 European Countries Gilles Guillard, Markus Nussbaum](https://reader038.vdocument.in/reader038/viewer/2022110322/56649d145503460f949e93e9/html5/thumbnails/7.jpg)
Verbal reasoning20 items from a database, 35 min, 23 languages
Numerical reasoning10 Items from a database, 20 min, 23 languages
Abstract reasoning10 Items in fixed test forms, 10 min
Situational Judgement20 Items in fixed test forms, 30 min, EN/DE/FR
Verbal reasoning20 items from a database, 35 min, 23 languages
Numerical reasoning10 Items from a database, 20 min, 23 languages
Abstract reasoning10 Items in fixed test forms, 10 min
Situational Judgement20 Items in fixed test forms, 30 min, EN/DE/FR
Computer Based Tests
![Page 8: Implementing a New Multilingual Selection Model Across 27 European Countries Gilles Guillard, Markus Nussbaum](https://reader038.vdocument.in/reader038/viewer/2022110322/56649d145503460f949e93e9/html5/thumbnails/8.jpg)
Verbal Reasoning
![Page 9: Implementing a New Multilingual Selection Model Across 27 European Countries Gilles Guillard, Markus Nussbaum](https://reader038.vdocument.in/reader038/viewer/2022110322/56649d145503460f949e93e9/html5/thumbnails/9.jpg)
Numerical Reasoning
![Page 10: Implementing a New Multilingual Selection Model Across 27 European Countries Gilles Guillard, Markus Nussbaum](https://reader038.vdocument.in/reader038/viewer/2022110322/56649d145503460f949e93e9/html5/thumbnails/10.jpg)
Abstract Reasoning
![Page 11: Implementing a New Multilingual Selection Model Across 27 European Countries Gilles Guillard, Markus Nussbaum](https://reader038.vdocument.in/reader038/viewer/2022110322/56649d145503460f949e93e9/html5/thumbnails/11.jpg)
X
X
Situational Judgement
![Page 12: Implementing a New Multilingual Selection Model Across 27 European Countries Gilles Guillard, Markus Nussbaum](https://reader038.vdocument.in/reader038/viewer/2022110322/56649d145503460f949e93e9/html5/thumbnails/12.jpg)
Results
EU Public Admin Audit Communication Law External Rel
Sum
AD5 AD7 AD5 AD7 AD5 AD7 AD5 AD7 AD5 AD7
validated application 12287 7892 2675 1389 3467 2023 4636 2130 6519 2338 45356
booked CBT 9557 6076 2182 1005 2525 1430 3615 1592 5016 1706 34704
participated CBT 8323 5336 1942 893 2132 1263 3113 1330 4412 1470 30214
reached CBT pass-marks 5502 3767 1427 603 1253 723 2076 830 2935 948 20064
succeeded CBT 190 102 108 34 77 51 144 43 64 21 834
![Page 13: Implementing a New Multilingual Selection Model Across 27 European Countries Gilles Guillard, Markus Nussbaum](https://reader038.vdocument.in/reader038/viewer/2022110322/56649d145503460f949e93e9/html5/thumbnails/13.jpg)
Score Distribution - VR
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
0250500750
10001250150017502000225025002750300032503500
3750
AD177: VR
participated
admitted
reserve list
Test score
Co
un
t
![Page 14: Implementing a New Multilingual Selection Model Across 27 European Countries Gilles Guillard, Markus Nussbaum](https://reader038.vdocument.in/reader038/viewer/2022110322/56649d145503460f949e93e9/html5/thumbnails/14.jpg)
0 1 2 3 4 5 6 7 8 9 10
0
500
1000
1500
2000
2500
3000
3500
4000
4500
5000
5500
6000
AD177: NR
participated
admitted
reserve list
Test score
Co
un
t
Score Distribution - NR
![Page 15: Implementing a New Multilingual Selection Model Across 27 European Countries Gilles Guillard, Markus Nussbaum](https://reader038.vdocument.in/reader038/viewer/2022110322/56649d145503460f949e93e9/html5/thumbnails/15.jpg)
0 1 2 3 4 5 6 7 8 9 10
0500
1000150020002500300035004000450050005500600065007000
7500
AD177: AR
participated
admitted
reserve list
Test score
Co
un
t
Score Distribution - AR
![Page 16: Implementing a New Multilingual Selection Model Across 27 European Countries Gilles Guillard, Markus Nussbaum](https://reader038.vdocument.in/reader038/viewer/2022110322/56649d145503460f949e93e9/html5/thumbnails/16.jpg)
Situational Judgement
AD-2011: SJ-AP
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Score
Co
un
t
AD-2011: SJ-RE
0
1000
2000
3000
4000
5000
6000
7000
8000
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Score
Co
un
t
AD-2011: SJ-QR
0
1000
2000
3000
4000
5000
6000
7000
8000
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Score
Co
un
t
AD-2011: SJ-PO
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Score
Co
un
t
AD-2011: SJ-WO
0
1000
2000
3000
4000
5000
6000
7000
8000
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Score
Co
un
t
![Page 17: Implementing a New Multilingual Selection Model Across 27 European Countries Gilles Guillard, Markus Nussbaum](https://reader038.vdocument.in/reader038/viewer/2022110322/56649d145503460f949e93e9/html5/thumbnails/17.jpg)
Intercorrelations
Verbal R
easoning
Num
erical Reasoning
Abstract R
easoning
SJT
Analysis &
Problem
S
olving
SJT
Planning &
Organising
SJT
Quality &
Results
SJT
Resilience
SJT
Working w
ith others
Verbal Reasoning - .41 .29 .16 .23 .15 .17 .18
Numerical Reasoning .41 - .32 .15 .20 .14 .15 .15
Abstract Reasoning .29 .32 - .14 .17 .12 .14 .14
SJT Analysis & Problem Solving .16 .15 .14 - .40 .33 .31 .35
SJT Planning & Organising .23 .20 .17 .40 - .32 .35 .35
SJT Quality & Results .15 .14 .12 .33 .32 - .35 .33
SJT Resilience .17 .15 .14 .31 .35 .35 - .32
SJT Working with others .18 .15 .14 .35 .35 .33 .32 -
![Page 18: Implementing a New Multilingual Selection Model Across 27 European Countries Gilles Guillard, Markus Nussbaum](https://reader038.vdocument.in/reader038/viewer/2022110322/56649d145503460f949e93e9/html5/thumbnails/18.jpg)
How EPSO monitors the behaviour of items
How EPSO monitors the behaviour of items
Item Analysis
![Page 19: Implementing a New Multilingual Selection Model Across 27 European Countries Gilles Guillard, Markus Nussbaum](https://reader038.vdocument.in/reader038/viewer/2022110322/56649d145503460f949e93e9/html5/thumbnails/19.jpg)
Based on the dicotomous / polytomous Rasch model
Conducted after every majorcompetition
Calibrate items and monitor their behaviour
Making psychometrics as vivid as possible
Based on the dicotomous / polytomous Rasch model
Conducted after every majorcompetition
Calibrate items and monitor their behaviour
Making psychometrics as vivid as possible
Item Analysis
Georg Rasch (1901-1980)
![Page 20: Implementing a New Multilingual Selection Model Across 27 European Countries Gilles Guillard, Markus Nussbaum](https://reader038.vdocument.in/reader038/viewer/2022110322/56649d145503460f949e93e9/html5/thumbnails/20.jpg)
![Page 21: Implementing a New Multilingual Selection Model Across 27 European Countries Gilles Guillard, Markus Nussbaum](https://reader038.vdocument.in/reader038/viewer/2022110322/56649d145503460f949e93e9/html5/thumbnails/21.jpg)
![Page 22: Implementing a New Multilingual Selection Model Across 27 European Countries Gilles Guillard, Markus Nussbaum](https://reader038.vdocument.in/reader038/viewer/2022110322/56649d145503460f949e93e9/html5/thumbnails/22.jpg)
![Page 23: Implementing a New Multilingual Selection Model Across 27 European Countries Gilles Guillard, Markus Nussbaum](https://reader038.vdocument.in/reader038/viewer/2022110322/56649d145503460f949e93e9/html5/thumbnails/23.jpg)
![Page 24: Implementing a New Multilingual Selection Model Across 27 European Countries Gilles Guillard, Markus Nussbaum](https://reader038.vdocument.in/reader038/viewer/2022110322/56649d145503460f949e93e9/html5/thumbnails/24.jpg)
![Page 25: Implementing a New Multilingual Selection Model Across 27 European Countries Gilles Guillard, Markus Nussbaum](https://reader038.vdocument.in/reader038/viewer/2022110322/56649d145503460f949e93e9/html5/thumbnails/25.jpg)
![Page 26: Implementing a New Multilingual Selection Model Across 27 European Countries Gilles Guillard, Markus Nussbaum](https://reader038.vdocument.in/reader038/viewer/2022110322/56649d145503460f949e93e9/html5/thumbnails/26.jpg)
![Page 27: Implementing a New Multilingual Selection Model Across 27 European Countries Gilles Guillard, Markus Nussbaum](https://reader038.vdocument.in/reader038/viewer/2022110322/56649d145503460f949e93e9/html5/thumbnails/27.jpg)
![Page 28: Implementing a New Multilingual Selection Model Across 27 European Countries Gilles Guillard, Markus Nussbaum](https://reader038.vdocument.in/reader038/viewer/2022110322/56649d145503460f949e93e9/html5/thumbnails/28.jpg)
Test Responses Candidates Items
VR 2 295 360 114 768 5 071
NR 1 147 680 114 768 3 490
Total 3 443 040 8 561
Example:Cumulative Analysis VR+NR
Oct 2012
![Page 29: Implementing a New Multilingual Selection Model Across 27 European Countries Gilles Guillard, Markus Nussbaum](https://reader038.vdocument.in/reader038/viewer/2022110322/56649d145503460f949e93e9/html5/thumbnails/29.jpg)
Item Difficulty Adjustment Across Languages (IDAAL)
How EPSO ensures fair opportunities for all candidates although testing in 23 different languages
How EPSO ensures fair opportunities for all candidates although testing in 23 different languages
![Page 30: Implementing a New Multilingual Selection Model Across 27 European Countries Gilles Guillard, Markus Nussbaum](https://reader038.vdocument.in/reader038/viewer/2022110322/56649d145503460f949e93e9/html5/thumbnails/30.jpg)
Aligned Averages Algorithm (AAA)
A
0
1
2
3
4
-4
-3
-2
-1
5
EN
0
1
2
3
4
-4
-3
-2
-1
5
EN
A
-4-3
-2-1
0+
1+
2+
3+
4
![Page 31: Implementing a New Multilingual Selection Model Across 27 European Countries Gilles Guillard, Markus Nussbaum](https://reader038.vdocument.in/reader038/viewer/2022110322/56649d145503460f949e93e9/html5/thumbnails/31.jpg)
Aligned Averages Algorithm (AAA)
citem = common item in en and lang
lang
m
encitemencitem
n
langcitemlangcitem
mn
1)(
)(1)(
)(
![Page 32: Implementing a New Multilingual Selection Model Across 27 European Countries Gilles Guillard, Markus Nussbaum](https://reader038.vdocument.in/reader038/viewer/2022110322/56649d145503460f949e93e9/html5/thumbnails/32.jpg)
Minimum Meansquare Method (MMM)
0
1
2
3
4
-4
-3
-2
-1
5
EN A B C
A
B
C
0
1
2
3
4
-4
-3
-2
-1
5
EN
![Page 33: Implementing a New Multilingual Selection Model Across 27 European Countries Gilles Guillard, Markus Nussbaum](https://reader038.vdocument.in/reader038/viewer/2022110322/56649d145503460f949e93e9/html5/thumbnails/33.jpg)
Minimum Meansquare Method (MMM)
min!
2
1 1
1,
,
n
item
m
lang
m
langlanglangitem
langlangitem m
![Page 34: Implementing a New Multilingual Selection Model Across 27 European Countries Gilles Guillard, Markus Nussbaum](https://reader038.vdocument.in/reader038/viewer/2022110322/56649d145503460f949e93e9/html5/thumbnails/34.jpg)
Person Parameter Contrast Measure (PPCM)
0
1
2
3
4
-4
-3
-2
-1
5
Cdts EN
Cdts A
![Page 35: Implementing a New Multilingual Selection Model Across 27 European Countries Gilles Guillard, Markus Nussbaum](https://reader038.vdocument.in/reader038/viewer/2022110322/56649d145503460f949e93e9/html5/thumbnails/35.jpg)
Person Parameter Contrast Measure (PPCM)
lang
m
encdtencdt
n
langcdtlangcdt
mn
1)(
)(1)(
)(
![Page 36: Implementing a New Multilingual Selection Model Across 27 European Countries Gilles Guillard, Markus Nussbaum](https://reader038.vdocument.in/reader038/viewer/2022110322/56649d145503460f949e93e9/html5/thumbnails/36.jpg)
... is a combination of these three approaches:
Aligned Averages Algorithm (AAA)
Minimum Meansquare Method (MMM)
Person Parameter Contrast Measure (PPCM)
... is a combination of these three approaches:
Aligned Averages Algorithm (AAA)
Minimum Meansquare Method (MMM)
Person Parameter Contrast Measure (PPCM)
Item Difficulty Adjustment Across Languages (IDAAL)
![Page 37: Implementing a New Multilingual Selection Model Across 27 European Countries Gilles Guillard, Markus Nussbaum](https://reader038.vdocument.in/reader038/viewer/2022110322/56649d145503460f949e93e9/html5/thumbnails/37.jpg)
EPSO Rasch Item Combination Algorithm (ERICA)
EPSO Rasch Item Combination Algorithm (ERICA)
How to assemble test forms
![Page 38: Implementing a New Multilingual Selection Model Across 27 European Countries Gilles Guillard, Markus Nussbaum](https://reader038.vdocument.in/reader038/viewer/2022110322/56649d145503460f949e93e9/html5/thumbnails/38.jpg)
Candidates have to be treated equally: Same number of items Same difficulty of test form
➾ Adaptive testing is not possible
EPSO has developed an algorithm involving two steps:
1. Set up a "master test form"
2. Create test forms corresponding to master test form
Candidates have to be treated equally: Same number of items Same difficulty of test form
➾ Adaptive testing is not possible
EPSO has developed an algorithm involving two steps:
1. Set up a "master test form"
2. Create test forms corresponding to master test form
Framework
![Page 39: Implementing a New Multilingual Selection Model Across 27 European Countries Gilles Guillard, Markus Nussbaum](https://reader038.vdocument.in/reader038/viewer/2022110322/56649d145503460f949e93e9/html5/thumbnails/39.jpg)
Actual test forms correspond to master test form
Limited overlap between test forms
No over-exposed items
Further improving gender balance
Actual test forms correspond to master test form
Limited overlap between test forms
No over-exposed items
Further improving gender balance
Target (quality criteria)
![Page 40: Implementing a New Multilingual Selection Model Across 27 European Countries Gilles Guillard, Markus Nussbaum](https://reader038.vdocument.in/reader038/viewer/2022110322/56649d145503460f949e93e9/html5/thumbnails/40.jpg)
Defining the master test form
easy difficult
Master test form difficulty: 8
Master test form difficulty: 2
Master test form difficulty: 5
![Page 41: Implementing a New Multilingual Selection Model Across 27 European Countries Gilles Guillard, Markus Nussbaum](https://reader038.vdocument.in/reader038/viewer/2022110322/56649d145503460f949e93e9/html5/thumbnails/41.jpg)
Items are approximately standard normal distributed
Most items are around the average (0) available
Weighting was implemented to better take account of item availabilities
Items are approximately standard normal distributed
Most items are around the average (0) available
Weighting was implemented to better take account of item availabilities
Weighting
![Page 42: Implementing a New Multilingual Selection Model Across 27 European Countries Gilles Guillard, Markus Nussbaum](https://reader038.vdocument.in/reader038/viewer/2022110322/56649d145503460f949e93e9/html5/thumbnails/42.jpg)
Sample Test FormTarget Item Actual
-1.00 EN1626VEN -1.001
-0.56 EN1555V -0.550
-0.22 EN1170V -0.227
0.06 EN1063V 0.062
0.30 EN2361V 0.295
0.51 EN1003VEN 0.517
0.71 EN1686V 0.703
0.89 EN2089V 0.891
1.06 EN2041VEN 1.064
1.22 EN2133V 1.211
1.38 EN2191V 1.394
1.54 EN2261V 1.532
1.69 EN2260VEN 1.684
1.84 EN2180VEN 1.811
2.00 EN2196VEN 2.032
2.16 EN2264V 2.198
2.33 EN2142V 2.333
2.51 EN1578VEN 2.451
2.73 EN1292VEN 2.809
3.00 EN2371V 2.941
0.9Sum SqDev:
![Page 43: Implementing a New Multilingual Selection Model Across 27 European Countries Gilles Guillard, Markus Nussbaum](https://reader038.vdocument.in/reader038/viewer/2022110322/56649d145503460f949e93e9/html5/thumbnails/43.jpg)
Thank you very much for your attention!
Any questions? Any questions?