the meaning and benefits of standardistion meaning

ICAO LPR Examiner Training/LTAS/SMC/2011‐11 1

1ICAO Language Proficiency Requirements Implementation Workshop

Vienna, Austria 21-23 February 2012

The meaning and benefits of standardistion

MeaningStandardisation: the process of developing and implementing technical standards.

To standardise: to make all the things of a particular type have the same features or level of quality.

Benefits• A means of facilitating processes and tasks;• A means by which, through set guidelines, different organisations can

more easily communicate in order to maintain focus;• A solution for a harmonisation and coordination problem

2

• A solution for a harmonisation and coordination problem.

• A situation in which all parties can realise mutual gains……but only by making mutually consistent decisions.

ICAO Language Proficiency Requirements Implementation Workshop Vienna, Austria 21-23 February 2012


The role of ICAOThe Organisation serves as the forum for cooperation in all fields of civil aviation among its diverse Member States.

MissionMissionTo foster a global civil aviation system that consistently and uniformly operates at peak efficiency and provides optimum safety, security and sustainability.

Strategic Objectives• Safety• Security • Environmental protection

3

Environmental protection• Sustainable development of air transport

Method• Setting standards and regulations (SARPs)

http://www.icao.int/


Forms of standards & recommended practices• Standards and Recommended Practices – SARPS • Procedures for Air Navigation Services – PANS;• Regional supplementary procedures – SUPPs;• Guidance material in several formats • Guidance material in several formats.

Forms of standards and recommended practices in relation to the ICAO LPRs (inter alia)

• ICAO Holistic Descriptors of operational language proficiency (2003)• ICAO Language Proficiency Rating Scale (2003)• Amendments to Annexes 1, 6 10 & 11 • ICAO Doc 4444 PANS ATM• ICAO Doc. 9835 AN/453 (1st ed. 2004)

4

ICAO Doc. 9835 AN/453 (1 ed. 2004)• ICAO AUD 001 (V1.1 2006 & V1.2 2007)• ICAO EAN PG 48 Report (2006)• ICAO Circular 318-AN/180• ICAO Doc. 9835 AN/453 (2nd ed. 2010)• ICAO Test Endorsement Procedure (2011)



A question of compliance

ICAO Holistic Descriptors of operational language proficiency ICAO Language Proficiency Rating Scale (2003)

• Appendix to Annex 1 – Personnel Licensing• Appendix to Annex 1 Personnel Licensing

to convey a notion of a standard;a guide to good judgement;a frame of reference for teachers and assessors to be able to

make consistent judgements about pilot and controller language proficiency. ICAO Doc. 9835 Ch.2

5

Standardisation of interpretation and application is required.


A question of adherence

ICAO Doc 9835 Manual on the Implementation of ICAO Language Proficiency Requirements (First ed. Sept 2004)

Ch 4 Table 4-1 Note 4Ch 4 Table 4 1 Note 4Training in test familiarity and rater calibration is necessary for all interlocutors and raters.

6.5.4 …essential elements of the test development process include… …establishing a rater training process and a quality control process…

6.7.1 …consider the role of good, experienced, wise and guided judgement in language testing.

Appendix D Standards for language training and testing

6

Those doing the grading should be trained for the task and both inter and intra-rater reliability should be calculated and published.



A question of adherence

ICAO Doc 9835 Manual on the Implementation of ICAO Language Proficiency Requirements (First ed. Sept 2004)

6 7 4 The use of a rating scale requires that6.7.4 The use of a rating scale requires that…• a body of well-informed and experienced raters be formed;• the raters inspire trust and can gain the confidence of the community;• the raters reflect the values of the community;• the raters understand the (assessment)criteria and the context in which

the criteria occur;• the raters agree upon standardized procedures for the

implementation of the criteria; and,• trained raters commit to best practice.

7

trained raters commit to best practice.

After these conditions are met, then the rating scale is used to make informed judgements about candidates.


A question of adherenceICAO Language Proficiency Requirements Rated Speech

SamplesForwardForwardA training aid

• to assist and support personnel involved in implementing the ICAO LPRs consistently and reliably worldwide.

• to assist in training language &operational raters (inter alia)• to provide support in calibrating the scoring of raters

Purpose• to raise awareness of the criteria by which proficiency is evaluated;

8

to raise awareness of the criteria by which proficiency is evaluated;• to provide specific examples of speech with justifications• to establish benchmarked samples of authentic speech at ICAO

Levels 3, 4 and 5;



A question of adherenceICAO Language Proficiency Requirements Rated Speech

SamplesForwardForward

The training aid of is of use to personnel undertaking a range of functions related to the implementation of the ICAO LPRs.For each of these functions, minimum qualifications in terms of education, training and experience should be met. Depending on the function they carry out, personnel will need to demonstrate a set of competencies.

E O f th t i li i ti t ld h t

9

E.g. One of the competencies a linguistic rater would have to demonstrate is the ability to characterize the performance of the test taker for each language proficiency skill at one of the six ICAO Levels.


A question of adherenceReport of the 48th Meeting of the European Air Navigation Planning Group (November 2006)Test Team Recommended QualificationsTest Team Recommended Qualifications

• Members of the test team should meet a set of minimum qualifications as well as comply with rigorous competency requirements. (Appendix I)

• Raters should meet further specified requirements necessary for efficient evaluation of language proficiency. (Appendix J)

EANPG Conclusion 48/31 – Recommended qualifications for ratersThat States be invited to use the recommended qualification and competency requirements in order to ensure that personnel responsible

10

co pe e cy equ e e s o de o e su e a pe so e espo s b e for rating comply with the minimum professional qualifications necessary for the accurate assessment of the language proficiency of air traffic controllers and pilots.



A question of adherenceReport of the 48th Meeting of the European Air Navigation Planning Group (November 2006)Appendix I ICAO Policy on Language TestingAppendix I ICAO Policy on Language Testing3.4.4 Consistency among rating teams should also be carefully monitored in order to avoid unacceptable levels of rating variance. The rater training of personnel involved in the rating process is the key to achieving consistency.



A question of adherenceReport of the 48th Meeting of the European Air Navigation Planning Group (November 2006)

A di L R d d L P fi i i l t ti Appendix L Recommended Language Proficiency implementation internal audit protocol

12

Has my State formulated, developed and approved qualification and competencyrequirements for raters?

Yes

No

Documented evidence

Satisfactory

Not satisfactory

Not applicable

12

15Do raters pass an annual refresher course to maintain standards in rating and assessment techniques?

Yes

No

Documented evidence

Satisfactory

Not satisfactory

Not applicable



A question of adherenceICAO Circular 318-AN/180 / ICAO Doc. 9835 AN/453 (2010)

Ch.2/3.3 Initial and recurrent rater training should be documented; the rater training records should be gmaintained, and audits of raters should be conducted and documented periodically. (6.3.4.3)

Ch.2/6.4.3 Raters should have successfully completed initial rater training. (6.3.8.12)

Ch.2/6.4.4 Raters should successfully complete recurrent rater training at least once each year. (6.3.8.13)

Initial and recurrent training aiming to standardize rater behaviour is vital to

13

g gobjectivity. As a language testing standard, raters should undergo approximately 40 hours of initial rater training and 24 to 40 hours of recurrent training per year. (6.3.8.13)


A question of adherenceICAO Circular 318-AN/180 / ICAO Doc. 9835 AN/453 (2010)

Checklist for Aviation Language Testing

Is initial and recurrent rater training documented? Are rater training records 3.3 Is initial and recurrent rater training documented? Are rater training records maintained? Are raters audited periodically and reports documented?

6.1.1 Are all testing team members familiar with the (listed) ICAO documents and publications?

6.4.1Do raters demonstrate language proficiency of at least ICAO Extended Level 5 in the language to be tested (and Expert Level 6 if the test is designed to assess ICAO Level 6 proficiency)?A t f ili ith i ti E li h d ith b l d

14

6.4.2 Are raters familiar with aviation English and with any vocabulary and structures that will likely be elicited by the test prompts and interactions?

6.4.3 Have raters successfully completed initial rater training?

6.4.4 Have raters successfully completed recurrent rater training at least once each year?



A question of adherenceICAO Test Endorsement Process (Oct 2011)

8. Professional qualificationsDescribe the professional qualifications you require of your test Describe the professional qualifications you require of your test administrators, raters, interlocutors and other members of your testing team.

Include information on initial and recurrent training, and any other activity completed regularly or as needed with regards to professional qualifications.

Who provides accreditation for your testing personnel?



A question of good practice in language assessmentMaximising test usefulness – the VRIP Principles

Validity • Construct• Content• Criterion-referenced

ICAO Holistic DescriptorsICAO Rating Scale

• Criterion-relatedConcurrentPredictive

Reliability • Accuracy of measure• Consistency• Stability

I t Positive examination ‘wash back’

16

Impact • Positive examination ‘wash-back’ Language training good practiceOperational good practice

Practicality • Efficient• Sustainable• Cost-effective



Aspects of test reliability

The extent to which test results are: • Stable• Consistent• Free from error of measurement

Methods of reporting reliability• Standard errors of measurement• Confidence intervals• Dependability indices

Classic reliability coefficients:

17

• Alternate form coefficients• Test/retest coefficients• Internal consistency coefficients


Aspects of test reliability – Tests of Speaking

When the scoring of a test involves judgement by examiners or raters, it is important to consider reliability in terms of the accuracy and consistency of the ratings which are made. o e a gs c a e ade.

The reliability of subjective assessments (using examiners) is typically estimated using correlations, e.g. intra- and inter-rater correlations.

Intra rater reliability: consistency of judgement of a single rater of the same or similar performances over time

Inter-rater reliability: consistency of judgement of a group of raters of the same performance

18

raters of the same performance



Achieving intra-rater and inter-rater reliabilityStandardisation

• Standardisation of selection criteriaICAO recommended MPRs

• Standardisation of training programmeDuration, content and refresher training

• Standardisation of interpretation of HDs and LPRS descriptorsCorrespondence of rater performance to benchmarked performances

• Standardisation of monitoring proceduresMeasurement, analysis and reporting of rater performances and

19

y p g premedial activities as required


Purpose of rater training programmes• to familiarise participants with the principles and procedures of

assessing speaking performances; • to introduce the idea and practice of using analytic rating scales for

i l fassessing oral performances;• to provide participants with sufficient information concerning the

assessment tool to be used (format, task types etc);• to familiarise participants with standard interlocutor and candidate

behaviour in the assessment tool to be used;• to enable participants to develop the required skills in assessment for

ICAO compliance;• to ensure valid and reliable assessment of live performances through

20

to ensure valid and reliable assessment of live performances through standardisation.



Typical content of rater training programmesStage 1: Pre-course distance learningIndividual study of an Introductory Training Pack containing:

• Introduction to the principles and practice of language assessment for Introduction to the principles and practice of language assessment for ICAO LPR compliance

• Overview of the speaking examination to be used• Guidelines for interlocutor behaviour• Guidelines for assessor behaviour• Self-assessment questions• Audio/audio visual sample candidate performances• Pre-course assessment tasks

21

Pre course assessment tasks• Appendices

e.g. benchmarks and justifications for the sample performance assessments; samples of candidate language; glossary of selected language assessment terms, etc.


Outline of rater training programme modelStage 2: The Training Seminar

Stage 2A: The theory• Discussion of experiences of the distance learning stage • Discussion of experiences of the distance learning stage • Further guidance in the assessment of oral performances: models and

techniques of assessment• Further familiarisation with and discussion of the Holistic Descriptors• Further familiarisation with and discussion of the Rating Scale

Stage 2A: The practice• Standardisation of the assessment procedure

22

• Blind rating of benchmarked candidate performances at all levels

Stage 2C: Mock ‘live’ testing• Practical application of the acquired skills in practice speaking tests



Potential issues in rater behaviour• avoidance of bias (cultural/linguistic/national/notional)• awareness of accommodation / adaptation of language• preconceived notions of what is ‘proficient’p eco ce ed o o s o a s p o c e• central tendency• generalising tendency• ‘masking’ effect• ‘horns’ / ‘halo’ effect• contrast effect• drift• pendulum effect

i fit / i fit

23

• infit / misfit

The solution: • continuous monitoring approach• remedial activities• regular, recurrent refresher training


A question of good practice in language assessmentMaximising test usefulness – the VRIP Principles

Validity • Construct• Content• Criterion-referenced

ICAO Holistic DescriptorsICAO Rating Scale

• Criterion-relatedConcurrentPredictive

Reliability • Accuracy of measure• Consistency• Stability

I t Positive examination ‘wash back’

24

Impact • Positive examination ‘wash-back’ Language training good practiceOperational good practice

Practicality • Efficient• Sustainable• Cost-effective



Conclusion

A question of compliance with the ICAO standard and adherence A question of compliance with the ICAO standard and adherence to its guidance material and a question of testing best practice.

25

Questions?

[email protected]


the meaning and benefits of standardistion meaning

Documents