Seminars
Aggregating and Learning from Multiple Annotators
by
Silviu Paun
2021-04-19
aggregation
The success of NLP research is founded on high-quality annotated datasets, which are usually obtained from multiple expert annotators or crowd workers. The standard practice to training machine learning models is to first adjudicate the disagreements and then perform the training. To this end, there has been a lot of work on aggregating annotations, particularly for classification tasks. However, many other tasks, particularly in NLP, have unique characteristics not considered by standard models of annotation, e.g., label interdependencies in sequence labelling tasks, unrestricted labels for anaphoric annotation, or preference labels for ranking texts. In recent years, researchers have picked up on this and are covering the gap. A first objective of this tutorial is to connect NLP researchers with state-of-the-art aggregation models for a diverse set of canonical language annotation tasks. There is also a growing body of recent work arguing that following the convention and training with adjudicated labels ignores any uncertainty the labellers had in their classifications, which results in models with poorer generalisation capabilities. Therefore, a second objective of this tutorial is to teach NLP workers how they can augment their (deep) neural models to learn from data with multiple interpretations.
Link:
https://sites.google.com/view/alma-tutorial
The Design and Development of Games with a Purpose for AI Systems
by
Doruk Kıcıkoğlu
2020-12-15
Games with a purpose
Collecting training data is often time-consuming, expensive and imposes a bottleneck on many machine learning tasks. Much of training data used to train ML systems is a result of the work of crowdworkers who are paid to do routinizable mundane tasks. Games with a purpose leverage game mechanics to use the perceptive capacities of users to collect data in a way that is more enjoyable to crowdworkers. Different machine learning tasks require different types of training data. In this paper, we discuss the design and development of building two games with a purpose: Guess the Word and Fool the AI, designed to collect data from both crowdworkers and domain experts for two very different machine learning problems. To make these games enjoyable and interactive, a team of engineers, research scientists and designers create new games with a purpose around various machine learning tasks. In this paper, we describe the design of these games, how we incorporate game mechanics within these games to make the collection of annotation tasks more efficient but also enjoyable
Link:
https://openreview.net/pdf?id=SHlZmg_ks7j
Computer Games in Education
by
Doruk Kıcıkoğlu
2020-05-26
Computer games, Video games, Educational games, Serious games, Computer-based learning, Instructional design, Cognitive skill training
Visionaries offer strong claims for the educational benefits of computer games, but there is a need to test those claims with rigorous scientific research and ground them in evidence-based theories of how people learn. Three genres of game research are (a) value-added research, which compares the learning outcomes of groups that learn academic material from playing a base version of a game to the outcomes of those playing the same game with one feature added; (b) cognitive consequences research, which compares improvements in cognitive skills of groups that play an off-the-shelf game to the skill improvements of those who engage in a control activity; and (c) media comparison research, which compares the learning outcomes of groups that learn academic material in a game to the outcomes of those who learn with conventional media. Value-added research suggests five promising features to include in educational computer games: modality, personalization, pretraining, coaching, and self-explanation. Cognitive consequences research suggests two promising approaches to cognitive training with computer games: using first-person shooter games to train perceptual attention skills and using spatial puzzle games to train two-dimensional mental rotation skills. Media comparison research suggests three promising areas where games may be more effective than conventional media: science, mathematics, and second-language learning. Future research is needed to pinpoint the cognitive, motivational, affective, and social processes that underlie learning with educational computer games.
Link:
https://www.annualreviews.org/doi/abs/10.1146/annurev-psych-010418-102744
Coordinating Advanced Crowd Work: Extending Citizen Science
by
Doruk Kıcıkoğlu
2020-03-03
Citizen science, Coordination theory, Dependencies, Writing, Advanced work
Crowdsourcing work with high levels of coupling between tasks poses challenges for coordination. This paper presents a study of an online citizen science project that involved volunteers in such tasks: not just analyzing bulk data but also interpreting data and writing a paper for publication. However, extending the reach of citizen science adds tasks with more dependencies, which calls for more elaborate coordination mechanisms but the relationship between the project and volunteers limits how work can be coordinated. Contrariwise, a mismatch between dependencies and available coordination mechanisms can be expected to lead to performance problems. The results of the study offer recommendations for design of crowdsourcing of more complex tasks.
Authors:
Crowston, K., Mitchell, E., & Østerlund, C. (2019)
Link:
https://theoryandpractice.citizenscienceassociation.org/articles/10.5334/cstp.166/
Development of a ‘Game with a Purpose’ for Acquisition of Brain-Computer Interface Data
by
Doruk Kıcıkoğlu
2020-03-03
Brain-computer interfaces, EEG, Games with a Purpose
Brain-computer interfaces (BCIs) have the potential to significantly change the ways in which humans interact with technology, the environment, and even each other. Unfortunately, BCI technologies are seldom robust enough for use in real-world applications, in part due to the large amount of data that must be collected, processed, and classified in order to develop models of task-related neural activity that account for two of the most important and least-understood drivers of BCI illiteracy: individual differences in neural signals and intra-individual differences across interdependent, time-varying neural states. This paper describes the feasibility of using a game with a purpose (GWAP) as a viable instrument for collecting data from BCI-relevant research tasks. By leveraging game-related reward processes to maintain participant interest and engagement, this approach will enable large amounts of BCI data to be acquired, both across many individuals and longitudinally from specific individuals as neural states vary naturally over time. Pilot and technical testing results are presented here to demonstrate that the BCI-relevant tasks embedded within the research game elicit neural signals similar to those that would be expected from more traditional BCI tasks. These preliminary data provide support and validation of the use of GWAPs as promising tools to enable long-term collection of BCI-relevant data in an engaging environment.
Authors:
Rexwinkle, J. T., Lieberman, G., Jaswa, M., & Lance, B. J. (2019)
Link:
https://arxiv.org/pdf/1910.00106.pdf
BERT for Coreference Resolution: Baselines and Analysis
SpanBERT: Improving Pre-training by Representing and Predicting Spans
by
Juntao Yu
2020-01-21
BERT - Coreference
The first paper BERT for coreference resolution demonstrates how original BERT models can be used for coreference resolution task. The proposed system is an extension of the previous SOTA Lee et al system that replaces the LSTMs with BERT and fine-tune the BERT model for the coreference task. The paper introduces the first BERT fine-tune approach for coreference resolution.
The second paper introduces the SpanBERT, which transforms the original BERT to more robust for the span prediction tasks. They replace the random subword masking scheme used in the original BERT model with a random span masking scheme to make the BERT better for the span prediction tasks. They also replace the next sentence prediction (NSP) with their span boundary objective (SBO) to encourage the system to encode information of the spans in their boundary tokens. The SBO has been found especially helpful for coreference task that directly uses the boundary token to represent the spans. Overall they show that the resulted model (SpanBERT) works better than the original BERT on various tasks evaluated.
Papers:
Joshi, Mandar, et al. "BERT for Coreference Resolution: Baselines and Analysis." Proceedings of EMNLP-IJCNLP. 2019.
Joshi, Mandar, et al. "Spanbert: Improving pre-training by representing and predicting spans." arXiv preprint arXiv:1907.10529 (2019).
Link:
https://www.aclweb.org/anthology/D19-1588/
https://arxiv.org/abs/1907.10529
Quizz: Targeted Crowdsourcing with a Billion (Potential) Users (2014)
by
Doruk Kıcıkoğlu
2019-05-21
Gamification - Crowdsourcing - Expert Assessment - Ad Targeting
Quizz is described as a gamified crowdsourcing system that simultaneously assesses the knowledge of users and acquires new knowledge from them. Quizz operates by asking users to complete short quizzes on specific topics; as a user answers the quiz questions, Quizz estimates the user's competence. To acquire new knowledge, Quizz also incorporates questions for which we do not have a known answer; the answers given by competent users provide useful signals for selecting the correct answers for these questions. Quizz actively tries to identify knowledgeable users on the Internet by running advertising campaigns, effectively leveraging the targeting capabilities of existing, publicly available, ad placement services. Quizz quantifies the contributions of the users using information theory and sends feedback to the advertisingsystem about each user. The feedback allows the ad targeting mechanism to further optimize ad placement.
Paper:
Ipeirotis, P., & Gabrilovich, E. (2014). Quizz. Proceedings Of The 23Rd International Conference On World Wide Web - WWW '14. doi: 10.1145/2566486.2567988
Link:
https://dl.acm.org/citation.cfm?doid=2566486.2567988
Jerry R. Hobbs: Granularity (1985)
by
Janosch Haber
2019-05-07
Granularity - Abstraction - Conceptualisation
In 1985 Jerry Hobbs introduced the granularity framework as an attempt to formalise the abstraction of complex global theories into simplified and computationally tractable local theories, something that for example often is seen during planning. In the article discussed during this session, Hobbs presents four central concepts to his granularity framework; abstraction, simplification, idealization and articulation, and deals with the notion of indistinguishability under relevant predicates, fuzzy indistinguishability boundaries, the importance of an explicit representation of granularity levels and the link between seamless granularity shifting and human intelligence.
Paper:
Jerry R. Hobbs. 1985. Granularity. In Proceedings of the 9th international joint conference on Artificial intelligence - Volume 1 (IJCAI'85), Aravind Joshi (Ed.), Vol. 1. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 432-435.
Link:
https://www.isi.edu/~hobbs/granularity-web.pdf
Boaz Keysar et al.: Taking Perspective in Conversation: The Role of Mutual Knowledge in Comprehension (2000)
by
Janosch Haber
2019-02-26
Visually-grounded Deixis - Collaborative Dialogue - Egocentric Perspective
Many dialogue theories assume a collaborative interaction between participants, often expecting speakers to choose expressions that maximise the probability of being understood correctly under the current conversation model. In the presented paper, Keysar and colleagues investigate the notion of shared perspective in dialogue and ask whether listeners interpret expressions based on a model containing only shared information or whether they rather act based on a personal (egocentric) model. In their experiments, participants move items following the instructions of a confederate instructor. Some of the items are clearly hidden from the instructor’s view and thus only exist in the participants personal model. Under the shared perspective hypothesis, this setup should not lead the participant to consider these hidden items as potential referents as they know that they cannot be the intended referents of the instructor’s expressions. Keysar and colleagues however observe that participants often still look at occluded items and start reaching for them in more than 20% of the trials.
Paper:
Keysar, B., Barr, D. J., Balin, J. A., & Brauner, J. S. (2000). Taking Perspective in Conversation: The Role of Mutual Knowledge in Comprehension. Psychological Science, 11(1), 32–38.
Link:
https://journals.sagepub.com/doi/pdf/10.1111/1467-9280.00211
Barbara Hall Partee: Opacity, Coreference and Pronouns (1972)
by
Janosch Haber
2019-02-05
Pronoun-Antecedent Relations - Coreference - Pronouns of Laziness
In her seminal 1972 paper, Barbara Hall Partee discusses different approaches of explaining and formalising ambiguous pronoun-antecedent relations, considering first a number of previously proposed attempts such as stipulating antecedent presupposition, quantifier scope modification or explicit specificity marking, finding all of them to be insufficient for explaining the full set of observed phenomena. She then moves to elaborate on pronouns of laziness, definite noun phrases and coreference, and indefinite noun phrases and actualisation to point at a range of future research avenues.
Paper:
Partee B.H. (1972) Opacity, Coreference, and Pronouns. In: Davidson D., Harman G. (eds) Semantics of Natural Language. Synthese Library (Monographs on Epistemology, Logic, Methodology, Philosophy of Science, Sociology of Science and of Knowledge, and on the Mathematical Methods of Social and Behavioral Sciences), vol 40. Springer, Dordrecht