The effect of item format on the choice of reading and test-taking strategies

Abstract

Investigating the processes that test takers undergo when they answer reading comprehension questions is of utmost importance to make claims about cognitive validity of a test. As such, this study was motivated by the need to explore reading processes, reading and test-taking strategies that test takers utilize when they take a test in different formats, namely multiple-choice and open-ended formats. In order to find out whether there were any differences in terms of reading processes and strategy use in these question formats, the data were collected through a triangulation of eye tracking technology, retrospective verbal reports and short semi-structured interviews. The results showed that the scores of open-ended questions were higher than their multiple-choice equivalents. Eye-tracking data showed that the percentage of careful reading was higher in open-ended test format and a further analysis on eye movement data showed that test takers spent a longer time in the interest areas while answering open-ended questions. The results of verbal report data displayed that test takers used test-taking strategies more in multiple-choice items while they use reading strategies more in open-ended items. Lastly, interview results showed that the majority of test takers reported to have comprehended the text in open-ended format more. This study is important in terms of proving that test format has an effect on the reading processes of test takers and multiple-choice format reduces the amount of careful reading and alters normal reading processes. The study also shows the promising and valuable contribution of eye-tracking technology in investigating cognitive validity.