Abstract
This study investigates learning with disagreement in NLP tasks and evaluates its performance on four datasets. The results suggest that the model performs best on the experimental dataset and faces challenges in minority languages. Furthermore, the analysis indicates that annotator demographics play a significant role in the interpretation of such tasks. This study suggests the need for greater consideration of demographic differences in annotators and more comprehensive evaluation metrics for NLP models.- Anthology ID:
- 2023.semeval-1.272
- Volume:
- Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023)
- Month:
- July
- Year:
- 2023
- Address:
- Toronto, Canada
- Editors:
- Atul Kr. Ojha, A. Seza Doğruöz, Giovanni Da San Martino, Harish Tayyar Madabushi, Ritesh Kumar, Elisa Sartori
- Venue:
- SemEval
- SIG:
- SIGLEX
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 1978–1982
- Language:
- URL:
- https://aclanthology.org/2023.semeval-1.272
- DOI:
- 10.18653/v1/2023.semeval-1.272
- Cite (ACL):
- Ruyuan Wan and Karla Badillo-Urquiola. 2023. Dragonfly_captain at SemEval-2023 Task 11: Unpacking Disagreement with Investigation of Annotator Demographics and Task Difficulty. In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 1978–1982, Toronto, Canada. Association for Computational Linguistics.
- Cite (Informal):
- Dragonfly_captain at SemEval-2023 Task 11: Unpacking Disagreement with Investigation of Annotator Demographics and Task Difficulty (Wan & Badillo-Urquiola, SemEval 2023)
- PDF:
- https://preview.aclanthology.org/ingest-acl-2023-videos/2023.semeval-1.272.pdf