At DALI, we achieve accurate and quality data annotation with crowdsourcing via Games With a Purpose. We intend to lead our players into contributing to science in a fun and intuitive way!
Phrase Detectives is an online crowdsourcing game with a validation stage, primarily designed to collect data about English (and subsequently Italian) anaphoric co-reference. Anaphoric coreference is a type of linguistic reference where one expression depends on another referential element. An example would be the relation between the entity 'Jon' and the pronoun 'his' in the text 'Jon rode his bike to school.'
The game uses two styles of text annotation for players to complete a linguistic task. Initially text is presented in Annotation Mode (called Name the Culprit in the game). This is a traditional annotation method in which the player makes an interpretation (annotation decision) about a highlighted markable (section of text). If different players enter different interpretations for a markable then each interpretation is presented to more players in Validation Mode (called Detectives Conference in the game). The players in Validation Mode have to agree or disagree with the interpretation. Players may also make comments about the task and/or skip the task if they do not want to provide an interpretation.
Players could label markables as DN (discourse-new, where the markable refers to a newly introduced entity), DO (discourse-old, where the markable refers to an already mentioned entity in the text, NR (non-referring, where the markable does not refer to anything or PR (where the markable represents a property of a previously mentioned entity).
The first publicly available dataset (Phrase Detectives Corpus 1.0) was used to determine what the collective quality of the players were, as well as the quality of individual decisions. Full details of the game and corpora, including processing pipeline, descriptive statistics and gold standard creation, have been published (see publications section).
Tile Attack is a two player blind game in which players are awarded points based on player agreement on tokens that they identify as being nounphrases.
The design of the game is inspired by scrabble with a tile like visualisation shown in figure 1. The game includes a point system and leaderboard that is shown to the player between rounds.
Wormingo is our newest game. Currently in beta version, the game attempts to gather quality data by employing a technique called 'motivation'.
The gameplay switches between motivation and annotation modes. Motivation mode removes some tokens from the text and asks these tokens to the players through different puzzles. The objective of the motivation mode is to entertain the player for better engagement, meanwhile solving the puzzles reinforces text comprehension.
After motivation mode, the game switches to annotation mode and asks the player to complete an annotation task. This mode resembles Phrase Detectives, in that the player is presented with the text and their task in hand.
One of the gamification techniques Wormingo employs is the Categories setting. This setting allows the player to choose which categories of text should appear more often, letting them customise their gaming experience to their taste of reading.