Abstract
Estimating label proportions in a target corpus is a type of measurement that is useful for answering certain types of social-scientific questions. While past work has described a number of relevant approaches, nearly all are based on an assumption which we argue is invalid for many problems, particularly when dealing with human annotations. In this paper, we identify and differentiate between two relevant data generating scenarios (intrinsic vs. extrinsic labels), introduce a simple but novel method which emphasizes the importance of calibration, and then analyze and experimentally validate the appropriateness of various methods for each of the two scenarios.- Anthology ID:
- N18-1148
- Volume:
- Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)
- Month:
- June
- Year:
- 2018
- Address:
- New Orleans, Louisiana
- Editors:
- Marilyn Walker, Heng Ji, Amanda Stent
- Venue:
- NAACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 1636–1646
- Language:
- URL:
- https://aclanthology.org/N18-1148
- DOI:
- 10.18653/v1/N18-1148
- Cite (ACL):
- Dallas Card and Noah A. Smith. 2018. The Importance of Calibration for Estimating Proportions from Annotations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1636–1646, New Orleans, Louisiana. Association for Computational Linguistics.
- Cite (Informal):
- The Importance of Calibration for Estimating Proportions from Annotations (Card & Smith, NAACL 2018)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/N18-1148.pdf