Validating multiple-text reading tasks in foreign language proficiency tests through verbal protocols and eye tracking

Abstract

The purpose of this study is to investigate how multiple texts reading skill in tasks in existing English proficiency tests are operationalized, whether the subskills and strategies specified in these exams match the theoretical explanations, and whether the actual use of skills and strategies reflects a sufficient and accurate coverage of theoretically designated multiple texts reading skill. ISE II, MET, and ECCE have been found to aim at assessing multiple-text reading comprehension. The tasks purportedly measuring multiple-text reading comprehension in these proficiency exams were administered to 10 participants of varying nationalities. Data were collected through eye tracking and retrospective think aloud method. The results revealed that ISE II does not attempt to operationalize multiple texts reading skill representatively, while MET specifications are not specific enough on multiple-text reading skill. ECCE specifications show that this task attempts to operationalize these skills representatively. When it comes to the operationalization of the specified multiple texts reading skill in each task, ISE II and MET do not sufficiently operationalize multiple-text reading skill, while ECCE is found to operationalize these skills to some extent. The findings also have implications for the design of multiple texts reading comprehension test tasks.