Abstract
Collecting high-quality annotations for Natural Language Processing (NLP) tasks poses challenges. Gamified annotation systems, like Games-with-a-Purpose (GWAP), have become popular tools for data annotation. For GWAPs to be effective, they must be user-friendly and produce high-quality annotations to ensure the collected data’s usefulness. This paper investigates the effectiveness of a gamified approach through two specific studies on an existing GWAP designed for collecting NLP coreference judgments. The first study involved preliminary usability testing using the concurrent think-aloud method to gather open-ended feedback. This feedback was crucial in pinpointing design issues. Following this, we conducted semi-structured interviews with our participants, and the insights collected from these interviews were instrumental in crafting player personas, which informed design improvements aimed at enhancing user experience. The outcomes of our research have been generalized to benefit other GWAP implementations. The second study evaluated the linguistic acceptability and reliability of the data collected through our GWAP. Our findings indicate that our GWAP produced reliable corpora with 91.49% accuracy and 0.787 Cohen’s kappa.- Anthology ID:
- 2024.games-1.10
- Volume:
- Proceedings of the 10th Workshop on Games and Natural Language Processing @ LREC-COLING 2024
- Month:
- May
- Year:
- 2024
- Address:
- Torino, Italia
- Editors:
- Chris Madge, Jon Chamberlain, Karen Fort, Udo Kruschwitz, Stephanie Lukin
- Venues:
- games | WS
- SIG:
- Publisher:
- ELRA and ICCL
- Note:
- Pages:
- 85–96
- Language:
- URL:
- https://aclanthology.org/2024.games-1.10
- DOI:
- Cite (ACL):
- Wateen Abdullah Aliady and Massimo Poesio. 2024. Linguistic Acceptability and Usability Enhancement: A Case Study of GWAP Evaluation and Redesign. In Proceedings of the 10th Workshop on Games and Natural Language Processing @ LREC-COLING 2024, pages 85–96, Torino, Italia. ELRA and ICCL.
- Cite (Informal):
- Linguistic Acceptability and Usability Enhancement: A Case Study of GWAP Evaluation and Redesign (Aliady & Poesio, games-WS 2024)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/2024.games-1.10.pdf