Abstract
Standard agreement measures for interannotator reliability are neither necessary nor sufficient to ensure a high quality corpus. In a case study of word sense annotation, conventional methods for evaluating labels from trained annotators are contrasted with a probabilistic annotation model applied to crowdsourced data. The annotation model provides far more information, including a certainty measure for each gold standard label; the crowdsourced data was collected at less than half the cost of the conventional approach.- Anthology ID:
- Q14-1025
- Erratum e1:
- Q14-1025e1
- Volume:
- Transactions of the Association for Computational Linguistics, Volume 2
- Month:
- Year:
- 2014
- Address:
- Cambridge, MA
- Editors:
- Dekang Lin, Michael Collins, Lillian Lee
- Venue:
- TACL
- SIG:
- Publisher:
- MIT Press
- Note:
- Pages:
- 311–326
- Language:
- URL:
- https://preview.aclanthology.org/add_missing_videos/Q14-1025/
- DOI:
- 10.1162/tacl_a_00185
- Cite (ACL):
- Rebecca J. Passonneau and Bob Carpenter. 2014. The Benefits of a Model of Annotation. Transactions of the Association for Computational Linguistics, 2:311–326.
- Cite (Informal):
- The Benefits of a Model of Annotation (Passonneau & Carpenter, TACL 2014)
- PDF:
- https://preview.aclanthology.org/add_missing_videos/Q14-1025.pdf