Morphological annotation of a corpus with a collaborative multiplayer game

Abstract

In most of the natural language processing tasks, state of the art systems usually rely on machine learning methods for building their mathematical models. Given that the majority of these systems employ supervised learning strategies, a corpus that is annotated for the problem area is essential. The current method for annotating a corpus is to hire several experts and make them annotate the corpus manually or -in its best practice- by using a helper software. However, this method is costly and timeconsuming if not error free. We propose a method that aims to solve these problems at once. By employing a multiplayer collaborative game that is playable by ordinary people on the Internet, it seems possible to direct the covert labour force so that people can contribute by just playing a fun game. Through a game site which incorporates some functionality inherited from social networking sites, people are motivated to contribute to the annotation process by answering questions about the underlying morphological features of a target word. The results reported in the thesis are compiled from the first eleven days of the experiment which is planned to continue until an indeterminate date. It is reported that the 63.5 per cent of the actual question types are successful based on two phases. The current 74 question types cover 58.3 per cent of the corpus completely while increasing this number to only 100 types increases the coverage rate to 70.7 per cent. Due to the time constraints and the relatively low traffic to the site, we were not able to annotate the corpus completely, but we can nevertheless estimate a hypothetical rate of successful morphological disambiguation as 51.4 per cent of the whole corpus which is calculated to be completed in two and a half months if the game were to be hosted on a major web site. This is indeed a relatively short duration for a bootstrapping of this size when compared with the current methods.