Abstract
A basic step in any annotation effort is the measurement of the Inter Annotator Agreement (IAA). An important factor that can affect the IAA is the presence of annotator bias. In this paper we introduce a new interpretation and application of the Item Response Theory (IRT) to detect annotators’ bias. Our interpretation of IRT offers an original bias identification method that can be used to compare annotators’ bias and characterise annotation disagreement. Our method can be used to spot outlier annotators, improve annotation guidelines and provide a better picture of the annotation reliability. Additionally, because scales for IAA interpretation are not generally agreed upon, our bias identification method is valuable as a complement to the IAA value which can help with understanding the annotation disagreement.- Anthology ID:
- 2020.coling-main.421
- Volume:
- Proceedings of the 28th International Conference on Computational Linguistics
- Month:
- December
- Year:
- 2020
- Address:
- Barcelona, Spain (Online)
- Venue:
- COLING
- SIG:
- Publisher:
- International Committee on Computational Linguistics
- Note:
- Pages:
- 4787–4797
- Language:
- URL:
- https://aclanthology.org/2020.coling-main.421
- DOI:
- 10.18653/v1/2020.coling-main.421
- Cite (ACL):
- Jacopo Amidei, Paul Piwek, and Alistair Willis. 2020. Identifying Annotator Bias: A new IRT-based method for bias identification. In Proceedings of the 28th International Conference on Computational Linguistics, pages 4787–4797, Barcelona, Spain (Online). International Committee on Computational Linguistics.
- Cite (Informal):
- Identifying Annotator Bias: A new IRT-based method for bias identification (Amidei et al., COLING 2020)
- PDF:
- https://preview.aclanthology.org/auto-file-uploads/2020.coling-main.421.pdf