maintaining quality and effectiveness in a testing system ... icao aviation langu… · relta and...
TRANSCRIPT
![Page 1: Maintaining Quality and Effectiveness in a Testing System ... ICAO Aviation Langu… · RELTA and the LPRs The test addresses Annex 1, 6 and 10 and the Appendix by assessing: •](https://reader036.vdocument.in/reader036/viewer/2022071401/60eb425f8b60f63b8b191bd9/html5/thumbnails/1.jpg)
Maintaining Quality and Effectiveness in a Testing
System – The RELTA
![Page 2: Maintaining Quality and Effectiveness in a Testing System ... ICAO Aviation Langu… · RELTA and the LPRs The test addresses Annex 1, 6 and 10 and the Appendix by assessing: •](https://reader036.vdocument.in/reader036/viewer/2022071401/60eb425f8b60f63b8b191bd9/html5/thumbnails/2.jpg)
• Aviation English training and testing materials - 15 years• Release of ICAO language proficiency requirements –
development of new testing system: • RMIT English Language Test for Aviation (RELTA)
RMITUniversityMelbourne, Australia
![Page 3: Maintaining Quality and Effectiveness in a Testing System ... ICAO Aviation Langu… · RELTA and the LPRs The test addresses Annex 1, 6 and 10 and the Appendix by assessing: •](https://reader036.vdocument.in/reader036/viewer/2022071401/60eb425f8b60f63b8b191bd9/html5/thumbnails/3.jpg)
ICAO LPRs and TestingAnnex 1
• … pilots and air traffic controllers …. shall demonstrate the ability to speak and understand the language used for radiotelephony communications.
Appendix• LPRs applicable to both phraseology and plain English.
Holistic Descriptors • communicate in radiotelephone and face-to-face situations;
• communicate on common, concrete and work-related topics;
• use communicative strategies to exchange messages and resolve misunderstandings in general or work-related contexts;
• manage linguistic challenges presented by an unexpected turn of events that occur within the context of a routine work situation or communicative task which is unfamiliar.
![Page 4: Maintaining Quality and Effectiveness in a Testing System ... ICAO Aviation Langu… · RELTA and the LPRs The test addresses Annex 1, 6 and 10 and the Appendix by assessing: •](https://reader036.vdocument.in/reader036/viewer/2022071401/60eb425f8b60f63b8b191bd9/html5/thumbnails/4.jpg)
ICAO-compliant tests
Quality in proficiency tests• Validity• Reliability• Authenticity• Interactivity• Fairness• Practicality
+
• Validity• Reliability• Authenticity• Interactivity• Fairness• Practicality
• ICAO LPRs:• Radiotelephony• Phraseology• Plain English• Routine non-routine• Voice-only• Face-to-face• 6 ICAO criteria• 6 bands of ICAO scale
![Page 5: Maintaining Quality and Effectiveness in a Testing System ... ICAO Aviation Langu… · RELTA and the LPRs The test addresses Annex 1, 6 and 10 and the Appendix by assessing: •](https://reader036.vdocument.in/reader036/viewer/2022071401/60eb425f8b60f63b8b191bd9/html5/thumbnails/5.jpg)
RELTA and the LPRsThe test addresses Annex 1, 6 and 10 and the Appendix by assessing:
• understanding and production of phraseology for communication in routine voice-only radiotelephone situations (Annex 1, Annex 6, Annex 10 and Holistic descriptors);
• understanding and production of plain English in contexts where phraseology does not suffice in non-routine radiotelephonesituations (Annex 1, Annex 6, Holistic descriptors and Appendix to Annex 1)
• understanding and production of plain English in face-to-face situations (Appendix to Annex 1)
![Page 6: Maintaining Quality and Effectiveness in a Testing System ... ICAO Aviation Langu… · RELTA and the LPRs The test addresses Annex 1, 6 and 10 and the Appendix by assessing: •](https://reader036.vdocument.in/reader036/viewer/2022071401/60eb425f8b60f63b8b191bd9/html5/thumbnails/6.jpg)
RELTA Delivery
![Page 7: Maintaining Quality and Effectiveness in a Testing System ... ICAO Aviation Langu… · RELTA and the LPRs The test addresses Annex 1, 6 and 10 and the Appendix by assessing: •](https://reader036.vdocument.in/reader036/viewer/2022071401/60eb425f8b60f63b8b191bd9/html5/thumbnails/7.jpg)
Air Traffic Controllers Pilots
ATC Listening
(30 minutes)
Pilot Listening
(30 minutes)
Pilot Speaking
(25 minutes)
Test Design
- TowerSpeaking - Enroute
- Approach
(25 minutes)
![Page 8: Maintaining Quality and Effectiveness in a Testing System ... ICAO Aviation Langu… · RELTA and the LPRs The test addresses Annex 1, 6 and 10 and the Appendix by assessing: •](https://reader036.vdocument.in/reader036/viewer/2022071401/60eb425f8b60f63b8b191bd9/html5/thumbnails/8.jpg)
Listening Speaking
Radiotelephony: Phraseology
(Routine)
Radiotelephony: Plain English
(Non-routine)
Conversation: Plain English
(Complex language)
Radiotelephony: Phraseology
(Routine)
Radiotelephony: Plain English
(Non-routine)
Conversation: Plain English
(Complex language)
Section 1(Voice-only)
Test Design
Section 2(Voice-only)
Section 3(Face-to-face)
![Page 9: Maintaining Quality and Effectiveness in a Testing System ... ICAO Aviation Langu… · RELTA and the LPRs The test addresses Annex 1, 6 and 10 and the Appendix by assessing: •](https://reader036.vdocument.in/reader036/viewer/2022071401/60eb425f8b60f63b8b191bd9/html5/thumbnails/9.jpg)
Delivery: Speaking
Section 2: Assesses PLAIN ENGLISH in radiotelephony
![Page 10: Maintaining Quality and Effectiveness in a Testing System ... ICAO Aviation Langu… · RELTA and the LPRs The test addresses Annex 1, 6 and 10 and the Appendix by assessing: •](https://reader036.vdocument.in/reader036/viewer/2022071401/60eb425f8b60f63b8b191bd9/html5/thumbnails/10.jpg)
Tower
Tower stream
![Page 11: Maintaining Quality and Effectiveness in a Testing System ... ICAO Aviation Langu… · RELTA and the LPRs The test addresses Annex 1, 6 and 10 and the Appendix by assessing: •](https://reader036.vdocument.in/reader036/viewer/2022071401/60eb425f8b60f63b8b191bd9/html5/thumbnails/11.jpg)
Enroute
Tower stream
![Page 12: Maintaining Quality and Effectiveness in a Testing System ... ICAO Aviation Langu… · RELTA and the LPRs The test addresses Annex 1, 6 and 10 and the Appendix by assessing: •](https://reader036.vdocument.in/reader036/viewer/2022071401/60eb425f8b60f63b8b191bd9/html5/thumbnails/12.jpg)
Approach
Tower stream
![Page 13: Maintaining Quality and Effectiveness in a Testing System ... ICAO Aviation Langu… · RELTA and the LPRs The test addresses Annex 1, 6 and 10 and the Appendix by assessing: •](https://reader036.vdocument.in/reader036/viewer/2022071401/60eb425f8b60f63b8b191bd9/html5/thumbnails/13.jpg)
DevelopmentRELTA for PilotsRELTA for Air Traffic ControllersTrialling and evaluation
![Page 14: Maintaining Quality and Effectiveness in a Testing System ... ICAO Aviation Langu… · RELTA and the LPRs The test addresses Annex 1, 6 and 10 and the Appendix by assessing: •](https://reader036.vdocument.in/reader036/viewer/2022071401/60eb425f8b60f63b8b191bd9/html5/thumbnails/14.jpg)
Trialling partners
![Page 15: Maintaining Quality and Effectiveness in a Testing System ... ICAO Aviation Langu… · RELTA and the LPRs The test addresses Annex 1, 6 and 10 and the Appendix by assessing: •](https://reader036.vdocument.in/reader036/viewer/2022071401/60eb425f8b60f63b8b191bd9/html5/thumbnails/15.jpg)
Evaluation and Maintenance
0.00 100.00 200.00 300.00 400.00 500.00
Externally Validated Test
2.00
3.00
4.00
5.00O
VER
ALL
ICA
O S
core
R Sq Linear = 0.643
![Page 16: Maintaining Quality and Effectiveness in a Testing System ... ICAO Aviation Langu… · RELTA and the LPRs The test addresses Annex 1, 6 and 10 and the Appendix by assessing: •](https://reader036.vdocument.in/reader036/viewer/2022071401/60eb425f8b60f63b8b191bd9/html5/thumbnails/16.jpg)
Administration: Models
RELTA Examining
RELTA Rating
Issuing of results
Partnerroles
RMITroles
Local delivery Local examining
RELTA Examining
RELTA Rating
Issuing of results
RMIT delivery
RELTA Examining
RELTA Rating
Issuing of results
![Page 17: Maintaining Quality and Effectiveness in a Testing System ... ICAO Aviation Langu… · RELTA and the LPRs The test addresses Annex 1, 6 and 10 and the Appendix by assessing: •](https://reader036.vdocument.in/reader036/viewer/2022071401/60eb425f8b60f63b8b191bd9/html5/thumbnails/17.jpg)
Test versions dispatched
Speech file dispatched for independent rating
Speech files rated and checked
Results issued
Administration: EfficienciesInternet-based test data management (SRMS)
![Page 18: Maintaining Quality and Effectiveness in a Testing System ... ICAO Aviation Langu… · RELTA and the LPRs The test addresses Annex 1, 6 and 10 and the Appendix by assessing: •](https://reader036.vdocument.in/reader036/viewer/2022071401/60eb425f8b60f63b8b191bd9/html5/thumbnails/18.jpg)
Administration: SRMS
• Test managed centrally• Candidate registration • Test dispatch for local delivery• Distributes candidate speaking files for remote rating• Computes ICAO scores• Identifies rater discrepancies• Communication portal • Stores candidate details centrally
![Page 19: Maintaining Quality and Effectiveness in a Testing System ... ICAO Aviation Langu… · RELTA and the LPRs The test addresses Annex 1, 6 and 10 and the Appendix by assessing: •](https://reader036.vdocument.in/reader036/viewer/2022071401/60eb425f8b60f63b8b191bd9/html5/thumbnails/19.jpg)
Listen to candidate sound file
Enter scores for Sections
Download rating scripts
SRMS InterfaceRater 1 (Guangzhou)
![Page 20: Maintaining Quality and Effectiveness in a Testing System ... ICAO Aviation Langu… · RELTA and the LPRs The test addresses Annex 1, 6 and 10 and the Appendix by assessing: •](https://reader036.vdocument.in/reader036/viewer/2022071401/60eb425f8b60f63b8b191bd9/html5/thumbnails/20.jpg)
SRMS InterfaceRater 1 (Muscat)
Enter scores for Sections
Download rating scripts
Listen to candidate sound file
![Page 21: Maintaining Quality and Effectiveness in a Testing System ... ICAO Aviation Langu… · RELTA and the LPRs The test addresses Annex 1, 6 and 10 and the Appendix by assessing: •](https://reader036.vdocument.in/reader036/viewer/2022071401/60eb425f8b60f63b8b191bd9/html5/thumbnails/21.jpg)
SRMS Interface(Melbourne)
Calculates overall scores
Determines ICAO Level
Imports scores from R1 and R2
Imports Listening score
![Page 22: Maintaining Quality and Effectiveness in a Testing System ... ICAO Aviation Langu… · RELTA and the LPRs The test addresses Annex 1, 6 and 10 and the Appendix by assessing: •](https://reader036.vdocument.in/reader036/viewer/2022071401/60eb425f8b60f63b8b191bd9/html5/thumbnails/22.jpg)
Results(Melbourne)
![Page 23: Maintaining Quality and Effectiveness in a Testing System ... ICAO Aviation Langu… · RELTA and the LPRs The test addresses Annex 1, 6 and 10 and the Appendix by assessing: •](https://reader036.vdocument.in/reader036/viewer/2022071401/60eb425f8b60f63b8b191bd9/html5/thumbnails/23.jpg)
RELTA Sites
![Page 24: Maintaining Quality and Effectiveness in a Testing System ... ICAO Aviation Langu… · RELTA and the LPRs The test addresses Annex 1, 6 and 10 and the Appendix by assessing: •](https://reader036.vdocument.in/reader036/viewer/2022071401/60eb425f8b60f63b8b191bd9/html5/thumbnails/24.jpg)
Conclusion
• Washback on training• Need development and maintenance
resources• Need administration resources
capability• Test developers and users have
obligations: − apply ICAO standard − safety