![Page 1: The training needs of different stakeholders in language testing and assessment Angela Hasselgreen, Ari Huhta, Jayanti Banerjee](https://reader031.vdocument.in/reader031/viewer/2022032605/56649e755503460f94b76e69/html5/thumbnails/1.jpg)
The training needsof different stakeholders
in language testingand assessment
Angela Hasselgreen, Ari Huhta, Jayanti Banerjee
![Page 2: The training needs of different stakeholders in language testing and assessment Angela Hasselgreen, Ari Huhta, Jayanti Banerjee](https://reader031.vdocument.in/reader031/viewer/2022032605/56649e755503460f94b76e69/html5/thumbnails/2.jpg)
Outline
• The design and implementation of the questionnaire
• Some overall themes• Comparing results by regions• Implications
![Page 3: The training needs of different stakeholders in language testing and assessment Angela Hasselgreen, Ari Huhta, Jayanti Banerjee](https://reader031.vdocument.in/reader031/viewer/2022032605/56649e755503460f94b76e69/html5/thumbnails/3.jpg)
The timetable
• First meeting:• Preliminary design:• Web-based
questionnaire (roll-out date):
• Data collection:• Data preparation (for
analysis):• Analysis + preparation
for conference:
24 – 25 January 2004
1 February 2004
1 March 2004
1 March – 15 April 2004
18 – 21 April 2004
23 April - 12 May 2004
![Page 4: The training needs of different stakeholders in language testing and assessment Angela Hasselgreen, Ari Huhta, Jayanti Banerjee](https://reader031.vdocument.in/reader031/viewer/2022032605/56649e755503460f94b76e69/html5/thumbnails/4.jpg)
![Page 5: The training needs of different stakeholders in language testing and assessment Angela Hasselgreen, Ari Huhta, Jayanti Banerjee](https://reader031.vdocument.in/reader031/viewer/2022032605/56649e755503460f94b76e69/html5/thumbnails/5.jpg)
![Page 6: The training needs of different stakeholders in language testing and assessment Angela Hasselgreen, Ari Huhta, Jayanti Banerjee](https://reader031.vdocument.in/reader031/viewer/2022032605/56649e755503460f94b76e69/html5/thumbnails/6.jpg)
Part 2: Questions for teacher trainers/educatorsPart 3: Questions for people who work, full time or part time, for language testing/examination boards or organisations that design school-external, regional or nationwide tests/examinations
![Page 7: The training needs of different stakeholders in language testing and assessment Angela Hasselgreen, Ari Huhta, Jayanti Banerjee](https://reader031.vdocument.in/reader031/viewer/2022032605/56649e755503460f94b76e69/html5/thumbnails/7.jpg)
Summary of responses
• 914 respondents to at least one part of the questionnaire
– Part 1: 741
– Part 2: 709
– Part 3: 689
• 501 respondents have provided contact details
![Page 8: The training needs of different stakeholders in language testing and assessment Angela Hasselgreen, Ari Huhta, Jayanti Banerjee](https://reader031.vdocument.in/reader031/viewer/2022032605/56649e755503460f94b76e69/html5/thumbnails/8.jpg)
Summary of responses
• Point to Note– Respondents did not necessarily answer every
item in each section for which they submitted data.
• Responses for parts 2 & 3 are particularly low– Part 1: 522 – 614 responses per item– Part 2: 181 – 212 responses per item– Part 3: 198 – 228 responses per item
![Page 9: The training needs of different stakeholders in language testing and assessment Angela Hasselgreen, Ari Huhta, Jayanti Banerjee](https://reader031.vdocument.in/reader031/viewer/2022032605/56649e755503460f94b76e69/html5/thumbnails/9.jpg)
Where do the respondents work?
Non-European countri
Other European count
United Kingdom
Turkey
Sw eden
Spain
Slovenia
Romania
Poland
Norw ay
Netherlands
Ireland
Hungary
Greece
Finland
Estonia
Bulgaria
Belgium
Missing
![Page 10: The training needs of different stakeholders in language testing and assessment Angela Hasselgreen, Ari Huhta, Jayanti Banerjee](https://reader031.vdocument.in/reader031/viewer/2022032605/56649e755503460f94b76e69/html5/thumbnails/10.jpg)
’Teachers only’: degree of formal training in assessment
Formal education on assessment
4321
Formal education on assessment
Fre
qu
en
cy
160
140
120
100
80
60
40
20
0
Std. Dev = 1,09
Mean = 3
N = 360,00
1= none2= short3= medium4= long
![Page 11: The training needs of different stakeholders in language testing and assessment Angela Hasselgreen, Ari Huhta, Jayanti Banerjee](https://reader031.vdocument.in/reader031/viewer/2022032605/56649e755503460f94b76e69/html5/thumbnails/11.jpg)
Correlation between assessment-related activities teachers are engaged in
and absence of professional training
1 Classroom-focussed assessment
05
1015202530354045
Prepa
ring
your
ow
...
Using
read
y m
ade.
..
Inte
rpre
ting
test
re...
Givi
ng fe
edba
ck
Using
self/
pee
r a...
Using
info
rmal
, co.
..
Using
portf
olio
![Page 12: The training needs of different stakeholders in language testing and assessment Angela Hasselgreen, Ari Huhta, Jayanti Banerjee](https://reader031.vdocument.in/reader031/viewer/2022032605/56649e755503460f94b76e69/html5/thumbnails/12.jpg)
Teachers’ perceived needs: do you need training in this area?
1 Classroom-focussed assessment
0102030405060
prep
arin
g yo
ur o
wn...
usin
g re
ady-
mad
e ...
inte
rpre
ting
test
re...
givi
ng fe
edba
ck
usin
g se
lf/ p
eer a
...
usin
g in
form
al, c
on...
usin
g po
rtfol
io
no
yes
![Page 13: The training needs of different stakeholders in language testing and assessment Angela Hasselgreen, Ari Huhta, Jayanti Banerjee](https://reader031.vdocument.in/reader031/viewer/2022032605/56649e755503460f94b76e69/html5/thumbnails/13.jpg)
Correlation between assessment-related activities teachers are engaged in
and absence of professional training
2 Purposes of assessment
0
5
10
15
20
25
30
35
To givegrades
To find outwhat needsto be taught
To placestudents
To awardfinal
certificates
![Page 14: The training needs of different stakeholders in language testing and assessment Angela Hasselgreen, Ari Huhta, Jayanti Banerjee](https://reader031.vdocument.in/reader031/viewer/2022032605/56649e755503460f94b76e69/html5/thumbnails/14.jpg)
Teachers’ perceived needs: do you need training in this area?
2 Purposes of assessment
05
1015202530354045
to givegrades
to find outwhat needs
to betaught
to placestudents
to awardfinal
certificates
no
yes
![Page 15: The training needs of different stakeholders in language testing and assessment Angela Hasselgreen, Ari Huhta, Jayanti Banerjee](https://reader031.vdocument.in/reader031/viewer/2022032605/56649e755503460f94b76e69/html5/thumbnails/15.jpg)
Correlation between assessment-related activities teachers are engaged in
and absence of professional training
3 Content and concepts
05
1015202530
T. receptive skills
T. productive skills
T. microlinguistic a...
T. integrated skills
T. culture
Est. reliability
Est. validity
Using statistics
![Page 16: The training needs of different stakeholders in language testing and assessment Angela Hasselgreen, Ari Huhta, Jayanti Banerjee](https://reader031.vdocument.in/reader031/viewer/2022032605/56649e755503460f94b76e69/html5/thumbnails/16.jpg)
Teachers’ perceived needs: do you need training in this area?
3 Content and concepts
0 10 20 30 40 50 60
testing receptive skills
testing productive skills
testing grammar/ vocabulary
testing integrated language skills
testing aspects of culture
establishing reliability
establishing validity
using statistics
yes
no
![Page 17: The training needs of different stakeholders in language testing and assessment Angela Hasselgreen, Ari Huhta, Jayanti Banerjee](https://reader031.vdocument.in/reader031/viewer/2022032605/56649e755503460f94b76e69/html5/thumbnails/17.jpg)
Correlation between assessment-related activities teachers are engaged in
and absence of professional training
4 External tests and exams
05
101520253035404550
Rating oral/ written perf.
Using statistics
Writing item
s
Reviewing items
Acting as an interviewer
Defining assessment ...
![Page 18: The training needs of different stakeholders in language testing and assessment Angela Hasselgreen, Ari Huhta, Jayanti Banerjee](https://reader031.vdocument.in/reader031/viewer/2022032605/56649e755503460f94b76e69/html5/thumbnails/18.jpg)
Teachers’ perceived needs : do you need training in this area?
4 External tests and exams
0
10
20
30
40
50
60
External:taking part in
rating
usingstatistics
w riting items review ingitems
acting as aninterview er
definingassessment
criteria
no
yes
![Page 19: The training needs of different stakeholders in language testing and assessment Angela Hasselgreen, Ari Huhta, Jayanti Banerjee](https://reader031.vdocument.in/reader031/viewer/2022032605/56649e755503460f94b76e69/html5/thumbnails/19.jpg)
Tentative conclusions on teachers’ needs
• Teachers generally perceive a need for training across the board
• Areas of particular need:– ELP/other portfolios, preparing own tests, peer/self
assessment, interpreting results and giving feedback– To award grades, to find out what needs to be taught,
placing students, awarding certificates– Aspects of culture, integrated skills, establishing reliability
and validity, statistics– Reviewing and writing items, statistics, defining criteria,
rating, interviewing
![Page 20: The training needs of different stakeholders in language testing and assessment Angela Hasselgreen, Ari Huhta, Jayanti Banerjee](https://reader031.vdocument.in/reader031/viewer/2022032605/56649e755503460f94b76e69/html5/thumbnails/20.jpg)
Teacher trainers: what they train
1 Classroom-based
0
20
40
60
80
100
120
Preparing yourown classroom
tests
Using ready-made tests
Givingfeedback
Using self /peer
assessment
Using informal,continuous,
non-test
Using portfolio
Not at all
Both
In-service
Pre-service
![Page 21: The training needs of different stakeholders in language testing and assessment Angela Hasselgreen, Ari Huhta, Jayanti Banerjee](https://reader031.vdocument.in/reader031/viewer/2022032605/56649e755503460f94b76e69/html5/thumbnails/21.jpg)
Correlation between assessment-related activities teacher trainers give training in, and absence of their own professional training
1 Classroom-based
0
10
20
30
40
50
60
Preparing own classroom te
sts
using ready m
ade tests
giving feedback
using self/peer a
ssessment
using inform
al, contin
uous tests
using portfolio
![Page 22: The training needs of different stakeholders in language testing and assessment Angela Hasselgreen, Ari Huhta, Jayanti Banerjee](https://reader031.vdocument.in/reader031/viewer/2022032605/56649e755503460f94b76e69/html5/thumbnails/22.jpg)
Teacher trainers: what they train
2 Purposes
0
20
40
60
80
100
120
To give grades To find out whatneeds to be
taught
To placestudents
To award finalcertificates
Not at all
Both
In-service
Pre-service
![Page 23: The training needs of different stakeholders in language testing and assessment Angela Hasselgreen, Ari Huhta, Jayanti Banerjee](https://reader031.vdocument.in/reader031/viewer/2022032605/56649e755503460f94b76e69/html5/thumbnails/23.jpg)
Correlation between assessment-related activities teacher trainers give training in
and absence of their own professional training
2 Purposes
0
10
20
30
40
50
To give gradesTo find out whatneeds to be
taught
To placestudents
To award finalcertificates
![Page 24: The training needs of different stakeholders in language testing and assessment Angela Hasselgreen, Ari Huhta, Jayanti Banerjee](https://reader031.vdocument.in/reader031/viewer/2022032605/56649e755503460f94b76e69/html5/thumbnails/24.jpg)
Teacher trainers: what they train
3 Contents and concepts
0
20
40
60
80
100
120
Test
ing re
cept
ive s
kills
Test
ing p
rodu
ctive
skil
ls
Test
ing g
ram
mar
/voc
abula
ry
Test
ing in
tegr
ated
lang
uage
skil
ls
Test
ing a
spec
ts o
f cult
ure
Estab
lishin
g re
liabil
ity
Estab
lishin
g va
lidity
Using
stat
istics
Not at all
Both
In-service
Pre-service
![Page 25: The training needs of different stakeholders in language testing and assessment Angela Hasselgreen, Ari Huhta, Jayanti Banerjee](https://reader031.vdocument.in/reader031/viewer/2022032605/56649e755503460f94b76e69/html5/thumbnails/25.jpg)
Correlation between assessment-related activities teacher trainers give training in
and absence of their own professional training
3 Contents and concepts
05
1015202530354045
Receptive skills
Productive skills
Microlinguistic aspects
Integrated language skills
Aspects of culture
Establishing reliability
Establishing validity
Using statistics
![Page 26: The training needs of different stakeholders in language testing and assessment Angela Hasselgreen, Ari Huhta, Jayanti Banerjee](https://reader031.vdocument.in/reader031/viewer/2022032605/56649e755503460f94b76e69/html5/thumbnails/26.jpg)
Tentative conclusions from teacher trainer data
• Teacher trainers perceive their own needs similarly to teachers• Much of what they train in is in areas in which they have no
professional training themselves (generally 30%-50%)• Areas of greatest need, where teacher training is not given, or
given by people with little professional training, largely coincide with the areas of greatest need in teachers:– ELP/other portfolios, preparing own tests, peer/self assessment,
interpreting results and giving feedback + informal continuous assessment
– To award grades, to find out what needs to be taught, placement, awarding certificates
– Aspects of culture, integrated skills, establishing reliability and validity, statistics + productive skills
![Page 27: The training needs of different stakeholders in language testing and assessment Angela Hasselgreen, Ari Huhta, Jayanti Banerjee](https://reader031.vdocument.in/reader031/viewer/2022032605/56649e755503460f94b76e69/html5/thumbnails/27.jpg)
Correlation between activities experts are engaged in and absence of professional training
1 Item writing and rating
0
5
10
15
20
25
ratin
g sp
eakin
g or
wri.
..
mak
ing o
pen-
ende
d r..
.
writing
item
s/ta
sks
revie
wing it
ems/
test
s
actin
g as
inte
rview
er
mak
ing d
ecisi
ons
abo.
.
![Page 28: The training needs of different stakeholders in language testing and assessment Angela Hasselgreen, Ari Huhta, Jayanti Banerjee](https://reader031.vdocument.in/reader031/viewer/2022032605/56649e755503460f94b76e69/html5/thumbnails/28.jpg)
Experts’ perceived needs : do you need training in this area?
1 Item writing and rating
0
5
10
15
20
25R
atin
g sp
eaki
ngor
writ
ing
Mar
king
ope
n-en
ded
resp
onse
s
Writ
ing
item
s/de
sign
ing
test
task
s
Rev
iew
ing
item
s or
test
s
Act
ing
as a
nin
terv
iew
er
Mak
ing
deci
sion
s ab
out
com
posi
tion
ofte
sts
No
Yes
![Page 29: The training needs of different stakeholders in language testing and assessment Angela Hasselgreen, Ari Huhta, Jayanti Banerjee](https://reader031.vdocument.in/reader031/viewer/2022032605/56649e755503460f94b76e69/html5/thumbnails/29.jpg)
Correlation between which activities experts are engaged in and absence of professional training.
2 Developing tests and assessment systems
05
1015202530
planning new tests
designing te
st sp
eci...
defining ass
essmen...
pilotin
g items a
nd tests
training ra
ters etc.
statis
tical a
nalysis
o...
settin
g pass m
arks...
item banks
valid
ation re
search
![Page 30: The training needs of different stakeholders in language testing and assessment Angela Hasselgreen, Ari Huhta, Jayanti Banerjee](https://reader031.vdocument.in/reader031/viewer/2022032605/56649e755503460f94b76e69/html5/thumbnails/30.jpg)
Experts’ perceived needs : do you need training in this area?
2 Developing tests and assessment systems
0
5
10
15
20
25
Plannin
g ne
w test
s, s
yste
ms,
fram
ewor
ks
To fin
d ou
t wha
t nee
ds to
be
taug
ht
Design
ing te
st s
pecif
icatio
ns
Design
ing a
sses
smen
t crit
eria
Pilotin
g ite
ms
and
test
s
Traini
ng ra
ters
, mar
kers
, inte
rview
ers
Statis
tical
analy
sis o
f tes
ts a
nd ite
ms
Settin
g pa
ss m
arks
/cut
-off
scor
es
Creat
ing a
nd m
ainta
ining
item
ban
ks
Doing
valid
ation
rese
arch
No
Yes
![Page 31: The training needs of different stakeholders in language testing and assessment Angela Hasselgreen, Ari Huhta, Jayanti Banerjee](https://reader031.vdocument.in/reader031/viewer/2022032605/56649e755503460f94b76e69/html5/thumbnails/31.jpg)
Correlation between which activities experts are engaged in and absence of professional training
3 CEF related
23
24
25
26
27
28
29
30
using CEF as the basis fornew tests
relating existing tests tothe CEF
![Page 32: The training needs of different stakeholders in language testing and assessment Angela Hasselgreen, Ari Huhta, Jayanti Banerjee](https://reader031.vdocument.in/reader031/viewer/2022032605/56649e755503460f94b76e69/html5/thumbnails/32.jpg)
Experts’ perceived needs : do you need training in this area?
3 CEF related
0
5
10
15
20
25
30
Using CEF as basis for new testsystems
Relating existing tests to CEF
No
Yes
![Page 33: The training needs of different stakeholders in language testing and assessment Angela Hasselgreen, Ari Huhta, Jayanti Banerjee](https://reader031.vdocument.in/reader031/viewer/2022032605/56649e755503460f94b76e69/html5/thumbnails/33.jpg)
Tentative conclusions on experts’ needs
• Experts generally perceive need for training across the board
• Areas of particular need:– Item writing, reviewing items or tests, making decisions
about composition of tests– Statistical analysis, setting pass marks, creating/maintaining
item banks, doing validation research– Relating existing tests to the CEF, using CEF as basis for
new test systems
![Page 34: The training needs of different stakeholders in language testing and assessment Angela Hasselgreen, Ari Huhta, Jayanti Banerjee](https://reader031.vdocument.in/reader031/viewer/2022032605/56649e755503460f94b76e69/html5/thumbnails/34.jpg)
Regions compared
1. Southern Europe– Portugal, Spain, Andorra, Italy, Malta, Greece, Turkey2. Western Europe– Ireland, UK, France, Belgium, Netherlands, Germany,
Austria, Switzerland3. Northern Europe– Iceland, Norway, Sweden, Denmark, Finland4. Eastern Europe– Estonia, Latvia, Lithuania, Poland, Russia, Czech Rep.,
Slovakia, Hungary, Bulgaria, Romania, Ukraine, Slovenia, Croatia, Serbia and Montenegro, Macedonia, Azerbaijan
5. non-European countries
• Questionnaire 1 / Teachers
![Page 35: The training needs of different stakeholders in language testing and assessment Angela Hasselgreen, Ari Huhta, Jayanti Banerjee](https://reader031.vdocument.in/reader031/viewer/2022032605/56649e755503460f94b76e69/html5/thumbnails/35.jpg)
Teachers’ previous training
2 = A little (1-2 days)
1,8
1,9
2
2,1
2,2
South West North East non-Eur.
![Page 36: The training needs of different stakeholders in language testing and assessment Angela Hasselgreen, Ari Huhta, Jayanti Banerjee](https://reader031.vdocument.in/reader031/viewer/2022032605/56649e755503460f94b76e69/html5/thumbnails/36.jpg)
Teachers’ need for training- average number of activities, etc where need
for assessment training was reported (out of max 25)
02468101214161820
South West North East non-Eur.
Advanced training
![Page 37: The training needs of different stakeholders in language testing and assessment Angela Hasselgreen, Ari Huhta, Jayanti Banerjee](https://reader031.vdocument.in/reader031/viewer/2022032605/56649e755503460f94b76e69/html5/thumbnails/37.jpg)
Teachers’ reported need for advanced training 1
(activities out of max 25)
0 5 10 15 20
Belgium
Bulgaria
Estonia
Finland
Greece
Hungary
Ireland
Netherlands
Norway
![Page 38: The training needs of different stakeholders in language testing and assessment Angela Hasselgreen, Ari Huhta, Jayanti Banerjee](https://reader031.vdocument.in/reader031/viewer/2022032605/56649e755503460f94b76e69/html5/thumbnails/38.jpg)
Teachers’ reported need for advanced training 2
(activities out of max 25)
0 5 10 15 20
Norway
Poland
Romania
Slovenia
Spain
Sweden
Turkey
UK
Other Eur.
![Page 39: The training needs of different stakeholders in language testing and assessment Angela Hasselgreen, Ari Huhta, Jayanti Banerjee](https://reader031.vdocument.in/reader031/viewer/2022032605/56649e755503460f94b76e69/html5/thumbnails/39.jpg)
Statistically significant differences
Teachers’ previous training• no differences between regions
Teachers’ need for advanced training:• Western Europe < Eastern Europe• Western Europe < Southern EuropeTeachers’ need for basic training:• no statistically significant differences
• small n size made it impossible to calculate differences between individual countries
![Page 40: The training needs of different stakeholders in language testing and assessment Angela Hasselgreen, Ari Huhta, Jayanti Banerjee](https://reader031.vdocument.in/reader031/viewer/2022032605/56649e755503460f94b76e69/html5/thumbnails/40.jpg)
Top 5 needs (teachers / regions)
1st 2nd 3rd 4th 5th
South Portfolio
73%
Interpretation 65%
Statistics ext. 64%
Continuous 62%
Criteria
61%
West Reliability
55%
Validity
54%
Statistics
50%
Portfolio
45%
Self / peer
45%
North Portfolio
70%
Self / peer
63%
Reliability
62%
Validity
62%
Statistics
61%
East Portfolio
72%
Reliability
72%
Statistics
71%
Validity
71%
Interpretation 65%
![Page 41: The training needs of different stakeholders in language testing and assessment Angela Hasselgreen, Ari Huhta, Jayanti Banerjee](https://reader031.vdocument.in/reader031/viewer/2022032605/56649e755503460f94b76e69/html5/thumbnails/41.jpg)
Implications for training event(s)
• Common training events could be designed for both teachers and teacher trainers
• These should aim to cover the needs revealed here as pressing; however, it is likely that more ’core’ needs may exist for grass roots teachers (further investigation needed)
• A separate, specialist, training event would be necessary to cater for experts’ needs
• Regional events may be adapted to reflect different priorities
![Page 42: The training needs of different stakeholders in language testing and assessment Angela Hasselgreen, Ari Huhta, Jayanti Banerjee](https://reader031.vdocument.in/reader031/viewer/2022032605/56649e755503460f94b76e69/html5/thumbnails/42.jpg)
And finally..
Beneficial to belong to EALTA?
0
10
20
30
40
50
60
Teachers Teacher trainers Experts
Serie1
Serie2