Games

At DALI, we achieve accurate and quality data annotation with crowdsourcing via Games With a Purpose. We intend to lead our players into contributing to science in a fun and intuitive way!

Phrase Detectives

Phrase Detectives is an online crowdsourcing game with a validation stage, primarily designed to collect data about English (and subsequently Italian) anaphoric co-reference. Anaphoric coreference is a type of linguistic reference where one expression depends on another referential element. An example would be the relation between the entity 'Jon' and the pronoun 'his' in the text 'Jon rode his bike to school.'

The game uses two styles of text annotation for players to complete a linguistic task. Initially text is presented in Annotation Mode (called Name the Culprit in the game). This is a traditional annotation method in which the player makes an interpretation (annotation decision) about a highlighted markable (section of text). If different players enter different interpretations for a markable then each interpretation is presented to more players in Validation Mode (called Detectives Conference in the game). The players in Validation Mode have to agree or disagree with the interpretation. Players may also make comments about the task and/or skip the task if they do not want to provide an interpretation.

Players could label markables as DN (discourse-new, where the markable refers to a newly introduced entity), DO (discourse-old, where the markable refers to an already mentioned entity in the text, NR (non-referring, where the markable does not refer to anything or PR (where the markable represents a property of a previously mentioned entity).

The first publicly available dataset (Phrase Detectives Corpus 1.0) was used to determine what the collective quality of the players were, as well as the quality of individual decisions. Full details of the game and corpora, including processing pipeline, descriptive statistics and gold standard creation, have been published (see publications section).

Tile Attack

Tile Attack is a two player blind game in which players are awarded points based on player agreement on tokens that they identify as being nounphrases.

The design of the game is inspired by scrabble with a tile like visualisation shown in figure 1. The game includes a point system and leaderboard that is shown to the player between rounds.

Quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum.

Quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum.