Inconsistencies in Crowdsourced Slot-Filling Annotations: A Typology and Identification Methods
Stefan Larson, Adrian Cheung, Anish Mahendran, Kevin Leach, Jonathan K. Kummerfeld
Abstract
Slot-filling models in task-driven dialog systems rely on carefully annotated training data. However, annotations by crowd workers are often inconsistent or contain errors. Simple solutions like manually checking annotations or having multiple workers label each sample are expensive and waste effort on samples that are correct. If we can identify inconsistencies, we can focus effort where it is needed. Toward this end, we define six inconsistency types in slot-filling annotations. Using three new noisy crowd-annotated datasets, we show that a wide range of inconsistencies occur and can impact system performance if not addressed. We then introduce automatic methods of identifying inconsistencies. Experiments on our new datasets show that these methods effectively reveal inconsistencies in data, though there is further scope for improvement.- Anthology ID:
- 2020.coling-main.442
- Volume:
- Proceedings of the 28th International Conference on Computational Linguistics
- Month:
- December
- Year:
- 2020
- Address:
- Barcelona, Spain (Online)
- Editors:
- Donia Scott, Nuria Bel, Chengqing Zong
- Venue:
- COLING
- SIG:
- Publisher:
- International Committee on Computational Linguistics
- Note:
- Pages:
- 5035–5046
- Language:
- URL:
- https://aclanthology.org/2020.coling-main.442
- DOI:
- 10.18653/v1/2020.coling-main.442
- Cite (ACL):
- Stefan Larson, Adrian Cheung, Anish Mahendran, Kevin Leach, and Jonathan K. Kummerfeld. 2020. Inconsistencies in Crowdsourced Slot-Filling Annotations: A Typology and Identification Methods. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5035–5046, Barcelona, Spain (Online). International Committee on Computational Linguistics.
- Cite (Informal):
- Inconsistencies in Crowdsourced Slot-Filling Annotations: A Typology and Identification Methods (Larson et al., COLING 2020)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-5/2020.coling-main.442.pdf