Improving Label Quality by Jointly Modeling Items and Annotators
Tharindu Cyril Weerasooriya, Alexander Ororbia, Christopher Homan
Abstract
We propose a fully Bayesian framework for learning ground truth labels from noisy annotators. Our framework ensures scalability by factoring a generative, Bayesian soft clustering model over label distributions into the classic David and Skene joint annotator-data model. Earlier research along these lines has neither fully incorporated label distributions nor explored clustering by annotators only or data only. Our framework incorporates all of these properties within a graphical model designed to provide better ground truth estimates of annotator responses as input to any black box supervised learning algorithm. We conduct supervised learning experiments with variations of our models and compare them to the performance of several baseline models.- Anthology ID:
- 2022.nlperspectives-1.12
- Volume:
- Proceedings of the 1st Workshop on Perspectivist Approaches to NLP @LREC2022
- Month:
- June
- Year:
- 2022
- Address:
- Marseille, France
- Editors:
- Gavin Abercrombie, Valerio Basile, Sara Tonelli, Verena Rieser, Alexandra Uma
- Venue:
- NLPerspectives
- SIG:
- Publisher:
- European Language Resources Association
- Note:
- Pages:
- 95–99
- Language:
- URL:
- https://aclanthology.org/2022.nlperspectives-1.12
- DOI:
- Cite (ACL):
- Tharindu Cyril Weerasooriya, Alexander Ororbia, and Christopher Homan. 2022. Improving Label Quality by Jointly Modeling Items and Annotators. In Proceedings of the 1st Workshop on Perspectivist Approaches to NLP @LREC2022, pages 95–99, Marseille, France. European Language Resources Association.
- Cite (Informal):
- Improving Label Quality by Jointly Modeling Items and Annotators (Weerasooriya et al., NLPerspectives 2022)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/2022.nlperspectives-1.12.pdf
- Data
- GoEmotions, LabelMe, SBIC