What Is Unclear?

Computational Assessment of Task Clarity in Crowdsourcing

authored by: Zahra Nouri, Ujwal Gadiraju, Gregor Engels, Henning Wachsmuth
Abstract: Designing tasks clearly to facilitate accurate task completion is a challenging endeavor for requesters on crowdsourcing platforms. Prior research shows that inexperienced requesters fail to write clear and complete task descriptions which directly leads to low quality submissions from workers. By complementing existing works that have aimed to address this challenge, in this paper we study whether clarity flaws in task descriptions can be identified automatically using natural language processing methods. We identify and synthesize seven clarity flaws in task descriptions that are grounded in relevant literature. We build both BERT-based and feature-based binary classifiers, in order to study the extent to which clarity flaws in task descriptions can be computationally assessed, and understand textual properties of descriptions that affect task clarity. Through a crowdsourced study, we collect annotations of clarity flaws in 1332 real task descriptions. Using this dataset, we evaluate several configurations of the classifiers. Our results indicate that nearly all the clarity flaws in task descriptions can be assessed reasonably by the classifiers. We found that the content, style, and readability of tasks descriptions are particularly important in shaping their clarity. This work has important implications on the design of tools to help requesters in improving task clarity on crowdsourcing platforms. Flaw-specific properties can provide for valuable guidance in improving task descriptions.
External Organisation(s): Paderborn University
Delft University of Technology
Type: Conference contribution
Pages: 165-175
No. of pages: 11
Publication date: 29.08.2021
Publication status: Published
Peer reviewed: Yes
ASJC Scopus subject areas: Artificial Intelligence, Computer Graphics and Computer-Aided Design, Human-Computer Interaction, Software
Electronic version(s): https://doi.org/10.1145/3465336.3475109 (Access: Closed)

BibTeX

@inproceedings{5dd61c646d5d407eb1559fc682975afe,
title = "What Is Unclear?: Computational Assessment of Task Clarity in Crowdsourcing",
abstract = "Designing tasks clearly to facilitate accurate task completion is a challenging endeavor for requesters on crowdsourcing platforms. Prior research shows that inexperienced requesters fail to write clear and complete task descriptions which directly leads to low quality submissions from workers. By complementing existing works that have aimed to address this challenge, in this paper we study whether clarity flaws in task descriptions can be identified automatically using natural language processing methods. We identify and synthesize seven clarity flaws in task descriptions that are grounded in relevant literature. We build both BERT-based and feature-based binary classifiers, in order to study the extent to which clarity flaws in task descriptions can be computationally assessed, and understand textual properties of descriptions that affect task clarity. Through a crowdsourced study, we collect annotations of clarity flaws in 1332 real task descriptions. Using this dataset, we evaluate several configurations of the classifiers. Our results indicate that nearly all the clarity flaws in task descriptions can be assessed reasonably by the classifiers. We found that the content, style, and readability of tasks descriptions are particularly important in shaping their clarity. This work has important implications on the design of tools to help requesters in improving task clarity on crowdsourcing platforms. Flaw-specific properties can provide for valuable guidance in improving task descriptions.",
keywords = "BERT-based binary classification, crowdsourcing, feature-based binary classification, task clarity assessment, task design, unclear task descriptions",
author = "Zahra Nouri and Ujwal Gadiraju and Gregor Engels and Henning Wachsmuth",
year = "2021",
month = aug,
day = "29",
doi = "10.1145/3465336.3475109",
language = "English",
isbn = "9781450385510",
pages = "165--175",
booktitle = "HT 2021 - Proceedings of the 32nd ACM Conference on Hypertext and Social Media",
publisher = "Association for Computing Machinery, Inc",
note = "32nd ACM Conference on Hypertext and Social Media, HT 2021 ; Conference date: 30-08-2021 Through 02-09-2021",
}

Publication Details

What Is Unclear?

Computational Assessment of Task Clarity in Crowdsourcing

Funded by