dubeau testing research bucharest

7/25/2019 Dubeau Testing Research Bucharest

1/37

STANAG 6001-OPI Testing

Julie J. Dubeau

Bucharest BILC 2008


2/37

Julie J. Dubeau

Bill Who???


3/37

Julie J. Dubeau


4/37

Julie J. Dubeau

Are We All On the Same

Page?

An Exploratory Study of OPI RatingsAcross NATO CountriesUsing the NATO STANAG !!" Scale#

*This research was completed in 2006 as part of a M.A. Thesis inApplied Linguistics


5/37

Julie J. Dubeau

Pesentation Outline

Context

Research Questions

Literature Review

Methodolo!

Results

" Ratins

" Raters" #cale

Conclusion


6/37

Julie J. Dubeau

NATO !anguage Testing "onte#t

#tandardi$ed Lanuae %ro&ile '#L%(based on the )*+, #+*)D*RDI-*+I,)

*R//M/)+ ')*+, #+*)*( 001Lanuae %ro&icienc! Levels '/d 1 /d 2(

" 2 )*+, countries3 20 %artnershi4 &or %eace

'%&%( countries 5 others6


7/37

Julie J. Dubeau

InteroperabilityProblem?

Lanuae trainin is central within ar7ed &orces dueto the increasin nu7ber o& 4eacesu44ort

o4erations3 and is considered as havin an

i74ortant role in achievin interoperability

a7on the various 4la!ers.

9+he sinle 7ost i74ortant 4roble7 identi&iedb! al7ost all 4artners as an i74edi7ent to

develo4in intero4erabilit! with the *lliancehas been shortco7ins inco77unications: '/*%C '%*R%( D3 1;;


8/37

Julie J. Dubeau

O$ea%hing &esea%h 'uestion

#ince no =nown stud! had investiated interrater

reliabilit! in this context3 the 7ain research >uestion

was?

How comparable or consistent are ratings across

AT! raters and countries"


9/37

Julie J. Dubeau

&esea%h 'uestions

Research questions pertaining to the ratings

RQ1

Research questions pertaining raters training

and bacground RQ!

Research questions pertaining to the rating

process and to the scale RQ"


10/37

Julie J. Dubeau

&esea%h 'uestions

RQ1#Ratings$ (o) *o atings o+ the sa,e oal o+i%ien%

inte$ie)s /OPIs %o,ae +o, ate to ate?

Woul* the use o+ lus le$els in%ease ateagee,ent?

(o) *o the atings o+ the OPIs %o,ae +o,

%ount to %ount?

Ae thee *i++een%es in s%oes )ithin the sa,e

%ount?


11/37

Julie J. Dubeau

&esea%h 'uestions

RQ!#Raters training and bacground$ Ae thee *i++een%es in atings bet)een ates )ho

ha$e e%ei$e* $aing *egees o+ testeatetaining an* STANAG taining?

Di* $e e#eien%e* ates s%oe ,oe eliablthan lesse e#eien%e* ones? Ae e#eien%e*ates s%oing as eliabl as taine* ates?

Ae thee *i++een%es in atings bet)eenati%iants )ho test at-ti,e $esus +ull-ti,e2ae nati$e o non-nati$e sea3es o+ 4nglish2 an*ae +o, 5Ol*e an* 5Ne)e NATO %ounties?


12/37

Julie J. Dubeau

&esea%h 'uestions

RQ"#Rating process and scale use$

Do *i++eing ating a%ti%es a++e%t atings? Do ates aea to use the s%ale in si,ila

)as?

What ae the ates %o,,ents ega*ing theuse an* ali%ation o+ the s%ale?


13/37

Julie J. Dubeau

!iteatue &e$ie)

+estin Constructs

" @hat are we testin

A eneral 4ro&icienc! 5 @h!

A Ratin scales

Rater ariance

" ow do raters var!

A Raterscale interactionA Rater trainin 5 bac=round


14/37

Julie J. Dubeau

7etho*olog

#esign of study$/x4lorator! surve!" 2 ,ral %ro&icienc! Interviews ',%Is * 5 B(

" Rater data >uestionnaire

" Questionnaire acco74an!in each sa74le ,%I

%artici4ants ? Countries recruited at BILC #e7inarin #o&ia 200E

10F raters &ro7 18 countries and 2 )*+, units


15/37

Julie J. Dubeau

Analsis8" Ratin co74arisons

" ,riinal ratins" G%lusH ratins

" Rater co74arisons

" +rainin

" Bac=round

" Countr! to countr! co74arisons

A @ithin countr! dis4ersion

" Ratin 4rocess

A Ratin &actors

" Raterscale interaction

A #cale user&riendliness


16/37

Julie J. Dubeau

&esults &'1- Su,,a

Ratins ? +o co74are ,%I ratins and to

ex4lore the e&&icac! o& G4lus ratinsH.

" #o7e ratertorater di&&erences

" G%lusH levels brouht ratins closer to the 7ean

" #o7e countr!tocountr! di&&erences" reater Gwithincountr!H dis4ersion in so7e

countries


17/37

Julie J. Dubeau

withinlevel Fwithinlevel 2withinlevel 1%tac&ed'iewofA

I0F020100

View of OPI ratings sample A


18/37

Julie J. Dubeau

&esultsSample A %&1'All Ratings %(ith )'

Levels )u7bers

@ithin Level 1 rane


19/37

Julie J. Dubeau

All *ountries +eans ,or Sample A

2.02.202.001.801.01.01.201.00

!'erall (ountry Mean

(ountrynu

mbers 20

))+

),)6

)-)

)/)2

)))0

+

,

6-

/

2)


20/37

Julie J. Dubeau

All Ratings ,or Sample -%le.el !'

Levels )u7bers

1 2 1.;

1K 1 1.0

2 < E.

2K 8


21/37

Julie J. Dubeau

/ie( o, OPI ratings sample -

withinlevel Iwithinlevel Fwithinlevel 2withinlevel 1%tac&ed'iewof0I0F020100


22/37

Julie J. Dubeau

All *ountries +eans ,or Sample -

.F.002.


23/37

Julie J. Dubeau

&esults &'9- Su,,a Raters? +o investiate rater trainin and scale trainin

and see how 'or i&( the! i74acted the ratins3 and to

ex4lore how various bac=round characteristics

i74acted the ratins

" +rained raters scored within the 7ean3 es4eciall!&or sa74le B

" /x4erienced raters did not do as well as scale

trained raters

" ullti7e raters scored closer to 7ean" G)ewH )*+, raters scored slihtl! closer to 7ean

" ))# raters scored slihtl! closer to 7ean


24/37

Julie J. Dubeau

substantial to lotsnone to little


25/37

Julie J. Dubeau

:eas o+ 4#eien%e

E !ears K to E !ears2 to F !ears0 to 1 !ear

E0

0

F0

20

10

0

2re3uency

.-4

)-.+4

).+4

).+-4


26/37

Julie J. Dubeau

substantial to lotsnone to little

0

E0

0

F0

20

10

0

5ercent

0.04

60.04

%TAA %cale Training


27/37

Julie J. Dubeau

5Ol* $s. 5Ne) NATO "ounties

Summary of Tester Trg

Little Lots

Total

14

20

236

Newer NATO

member? Yes

No

Total

6

23

29

30

28

58

36

51

87


28/37

Julie J. Dubeau

5Ol* $s. 5Ne) NATO "ounties

Rating OPI B Corret?

Yes No

Total

14

20

236

Newer NATO

member? Yes

No

Total

27

27

54

6

!6

32

37

55

92

Other!issi"g

4

2

6


29/37

Julie J. Dubeau

&esults &ates Ba%3goun*

Conducts +estin ullti7eA Nes F 'FF.0 (

A )o < 'E.0 (A ullti7e testers 7ore reliable 'accurate(

" ))# '0( raters better trained

" G)ewH raters better trained


30/37

Julie J. Dubeau

&esults &';- Su,,a

#cale? +o ex4lore the wa!s in which raters used the

various #+*)* state7ents and ratin &actors to

arrive at their ratins.

" Ratin 4rocess did not a&&ect ratins sini&icantl!

" F 7ain Gt!4esH o& raters e7ered?

A /videncebased

A Intuitive

A /xtracontextual


31/37

Julie J. Dubeau

&esults

*n GevidencedbasedH ratin &or #a74le B 'level 2(?

I co74ared the candidateHs 4er&or7ance with

the #+*)* criteria 'levels 2 and F( and

decided that he did not 7eet the

re>uire7ents &or level F with reard to

&lexibilit! and the use o& structural devices.

/rrors were &re>uent not onl! in low&re>uenc! structures3 but in so7e hih

&re>uenc! areas as well. $Rater %! & rated '(


32/37

Julie J. Dubeau

&esults

*n GintuitiveH ratin &or #a74le * 'level 1(?

I )oul* sa that


33/37

Julie J. Dubeau

&esults

*n GextracontextualH ratin &or #a74le * 'level 1(?

Level 3 is the basic level needed for officers in

(my country). I think the candidate could performthe tasks required of him. He could easily be

bulldozed by native speakers in a meeting but

!ould hold his o!n !ith non"native speakers. Hemakes mistakes that very rarely distort meaning

and are rarely disturbing.$Rater %) & rated '(


34/37

Julie J. Dubeau

I,li%ations

+rainin not e>ual in all countries

#cale inter4retation

%lus levels use&ul

Di&&erent rids3 s4ea=in tests

Institutional 4ers4ectives


35/37

Julie J. Dubeau

!i,itations = >utue &esea%h

%artici4ants 7a! not have rated thiswa! in their own countries

,%Is new to so7e 4artici4ants

uture research could

" et 4artici4ants to test" Investiate ratin rids

" Loo= at other s=ills


36/37

Julie J. Dubeau

"on%lusion o+ &esea%h

#o3 are we all on the sa7e 4ae

N/#O *UT+

%lus levels were instru7ental in bridina4

+rainin was &ound to be =e! to reliabilit!

More incountr! trainin should be the &irst

ste4 toward international bench7ar=in.


37/37

Than3 :ouAre 7e All !n the %ame 5age"

An Exploratory Study of OPI Ratings

Across NATO Countries

Using the NATO STANAG !!" Scale

#ubeau.889forces.gc.ca

The full thesis is a'ailable on the (#A website

http$::www.cda;acd.forces.gc.ca!r google #ubeau thesis

dubeau testing research bucharest

Documents