Rater effects and foci in the oral assessment process of English test for aviation personnel

Abstract

Miscommunication in global aeronautical transmissions may lead to severe incidents and accidents in the aviation sector. In order to maintain flawless communication, International Civil Aviation Organization (ICAO) introduced Language Proficiency Requirements (LPR) and a Proficiency Rating Scale to aviation English test holders in 2003 (Alderson, 2009). However, several aviation English tests fail to provide detailed information about their reliability and validity (Alderson, 2011). Thus, this study intends to present evidence for rater effects and the areas they focus in the oral assessment part of English Test for Aviation Personnel (ETAP). Based on a theoretical model of a language for specific purposes (LSP) system by O’Sullivan and Weir (2011), the accuracy of raters’ decisions in ETAP was investigated. Four ETAP raters participated in the study and evaluated eight pilots’ oral performances while they were taking an eight-week-long online rater training. The assigned scores by ETAP raters were analyzed via Multi-Faceted Rasch Model (MFRM) (Linacre, 1989) by concentrating on rater severity/leniency, consistency, bias and central tendency effect. Through qualitative data including the raters’ verbal reports, the study also aimed to explore ETAP raters’ foci when evaluating oral performances. The results demonstrated raters’ rating consistency with rare rater effects and high rater reliability. Besides, six common foci were drawn from the study that can be utilized in future rater training sessions in order to improve ETAP, its testing process and ETAP rating scale.