Coordinating Advanced Crowd Work: Extending Citizen Science
Citizen science, Coordination theory, Dependencies, Writing, Advanced work
Crowdsourcing work with high levels of coupling between tasks poses challenges for coordination. This paper presents a study of an online citizen science project that involved volunteers in such tasks: not just analyzing bulk data but also interpreting data and writing a paper for publication. However, extending the reach of citizen science adds tasks with more dependencies, which calls for more elaborate coordination mechanisms but the relationship between the project and volunteers limits how work can be coordinated. Contrariwise, a mismatch between dependencies and available coordination mechanisms can be expected to lead to performance problems. The results of the study offer recommendations for design of crowdsourcing of more complex tasks.
Crowston, K., Mitchell, E., & Østerlund, C. (2019)
Development of a ‘Game with a Purpose’ for Acquisition of Brain-Computer Interface Data
Brain-computer interfaces, EEG, Games with a Purpose
Brain-computer interfaces (BCIs) have the potential to significantly change the ways in which humans interact with technology, the environment, and even each other. Unfortunately, BCI technologies are seldom robust enough for use in real-world applications, in part due to the large amount of data that must be collected, processed, and classified in order to develop models of task-related neural activity that account for two of the most important and least-understood drivers of BCI illiteracy: individual differences in neural signals and intra-individual differences across interdependent, time-varying neural states. This paper describes the feasibility of using a game with a purpose (GWAP) as a viable instrument for collecting data from BCI-relevant research tasks. By leveraging game-related reward processes to maintain participant interest and engagement, this approach will enable large amounts of BCI data to be acquired, both across many individuals and longitudinally from specific individuals as neural states vary naturally over time. Pilot and technical testing results are presented here to demonstrate that the BCI-relevant tasks embedded within the research game elicit neural signals similar to those that would be expected from more traditional BCI tasks. These preliminary data provide support and validation of the use of GWAPs as promising tools to enable long-term collection of BCI-relevant data in an engaging environment.
Rexwinkle, J. T., Lieberman, G., Jaswa, M., & Lance, B. J. (2019)
BERT for Coreference Resolution: Baselines and Analysis
SpanBERT: Improving Pre-training by Representing and Predicting Spans
BERT - Coreference
The first paper BERT for coreference resolution demonstrates how original BERT models can be used for coreference resolution task. The proposed system is an extension of the previous SOTA Lee et al system that replaces the LSTMs with BERT and fine-tune the BERT model for the coreference task. The paper introduces the first BERT fine-tune approach for coreference resolution.
The second paper introduces the SpanBERT, which transforms the original BERT to more robust for the span prediction tasks. They replace the random subword masking scheme used in the original BERT model with a random span masking scheme to make the BERT better for the span prediction tasks. They also replace the next sentence prediction (NSP) with their span boundary objective (SBO) to encourage the system to encode information of the spans in their boundary tokens. The SBO has been found especially helpful for coreference task that directly uses the boundary token to represent the spans. Overall they show that the resulted model (SpanBERT) works better than the original BERT on various tasks evaluated.
Joshi, Mandar, et al. "BERT for Coreference Resolution: Baselines and Analysis." Proceedings of EMNLP-IJCNLP. 2019.
Joshi, Mandar, et al. "Spanbert: Improving pre-training by representing and predicting spans." arXiv preprint arXiv:1907.10529 (2019).
Quizz: Targeted Crowdsourcing with a Billion (Potential) Users (2014)
Gamification - Crowdsourcing - Expert Assessment - Ad Targeting
Quizz is described as a gamified crowdsourcing system that simultaneously assesses the knowledge of users and acquires new knowledge from them. Quizz operates by asking users to complete short quizzes on specific topics; as a user answers the quiz questions, Quizz estimates the user's competence. To acquire new knowledge, Quizz also incorporates questions for which we do not have a known answer; the answers given by competent users provide useful signals for selecting the correct answers for these questions. Quizz actively tries to identify knowledgeable users on the Internet by running advertising campaigns, effectively leveraging the targeting capabilities of existing, publicly available, ad placement services. Quizz quantifies the contributions of the users using information theory and sends feedback to the advertisingsystem about each user. The feedback allows the ad targeting mechanism to further optimize ad placement.
Ipeirotis, P., & Gabrilovich, E. (2014). Quizz. Proceedings Of The 23Rd International Conference On World Wide Web - WWW '14. doi: 10.1145/2566486.2567988
Jerry R. Hobbs: Granularity (1985)
Granularity - Abstraction - Conceptualisation
In 1985 Jerry Hobbs introduced the granularity framework as an attempt to formalise the abstraction of complex global theories into simplified and computationally tractable local theories, something that for example often is seen during planning. In the article discussed during this session, Hobbs presents four central concepts to his granularity framework; abstraction, simplification, idealization and articulation, and deals with the notion of indistinguishability under relevant predicates, fuzzy indistinguishability boundaries, the importance of an explicit representation of granularity levels and the link between seamless granularity shifting and human intelligence.
Jerry R. Hobbs. 1985. Granularity. In Proceedings of the 9th international joint conference on Artificial intelligence - Volume 1 (IJCAI'85), Aravind Joshi (Ed.), Vol. 1. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 432-435.
Boaz Keysar et al.: Taking Perspective in Conversation: The Role of Mutual Knowledge in Comprehension (2000)
Visually-grounded Deixis - Collaborative Dialogue - Egocentric Perspective
Many dialogue theories assume a collaborative interaction between participants, often expecting speakers to choose expressions that maximise the probability of being understood correctly under the current conversation model. In the presented paper, Keysar and colleagues investigate the notion of shared perspective in dialogue and ask whether listeners interpret expressions based on a model containing only shared information or whether they rather act based on a personal (egocentric) model. In their experiments, participants move items following the instructions of a confederate instructor. Some of the items are clearly hidden from the instructorâ€™s view and thus only exist in the participants personal model. Under the shared perspective hypothesis, this setup should not lead the participant to consider these hidden items as potential referents as they know that they cannot be the intended referents of the instructorâ€™s expressions. Keysar and colleagues however observe that participants often still look at occluded items and start reaching for them in more than 20% of the trials.
Keysar, B., Barr, D. J., Balin, J. A., & Brauner, J. S. (2000). Taking Perspective in Conversation: The Role of Mutual Knowledge in Comprehension. Psychological Science, 11(1), 32â€“38.
Barbara Hall Partee: Opacity, Coreference and Pronouns (1972)
Pronoun-Antecedent Relations - Coreference - Pronouns of Laziness
In her seminal 1972 paper, Barbara Hall Partee discusses different approaches of explaining and formalising ambiguous pronoun-antecedent relations, considering first a number of previously proposed attempts such as stipulating antecedent presupposition, quantifier scope modification or explicit specificity marking, finding all of them to be insufficient for explaining the full set of observed phenomena. She then moves to elaborate on pronouns of laziness, definite noun phrases and coreference, and indefinite noun phrases and actualisation to point at a range of future research avenues.
Partee B.H. (1972) Opacity, Coreference, and Pronouns. In: Davidson D., Harman G. (eds) Semantics of Natural Language. Synthese Library (Monographs on Epistemology, Logic, Methodology, Philosophy of Science, Sociology of Science and of Knowledge, and on the Mathematical Methods of Social and Behavioral Sciences), vol 40. Springer, Dordrecht