Validating Label Consistency in NER Data Annotation

Qingkai Zeng, Mengxia Yu, Wenhao Yu, Tianwen Jiang, Meng Jiang


Abstract
Data annotation plays a crucial role in ensuring your named entity recognition (NER) projects are trained with the right information to learn from. Producing the most accurate labels is a challenge due to the complexity involved with annotation. Label inconsistency between multiple subsets of data annotation (e.g., training set and test set, or multiple training subsets) is an indicator of label mistakes. In this work, we present an empirical method to explore the relationship between label (in-)consistency and NER model performance. It can be used to validate the label consistency (or catches the inconsistency) in multiple sets of NER data annotation. In experiments, our method identified the label inconsistency of test data in SCIERC and CoNLL03 datasets (with 26.7% and 5.4% label mistakes). It validated the consistency in the corrected version of both datasets.
Anthology ID:
2021.eval4nlp-1.2
Volume:
Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems
Month:
November
Year:
2021
Address:
Punta Cana, Dominican Republic
Editors:
Yang Gao, Steffen Eger, Wei Zhao, Piyawat Lertvittayakumjorn, Marina Fomicheva
Venue:
Eval4NLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
11–15
Language:
URL:
https://aclanthology.org/2021.eval4nlp-1.2
DOI:
10.18653/v1/2021.eval4nlp-1.2
Bibkey:
Cite (ACL):
Qingkai Zeng, Mengxia Yu, Wenhao Yu, Tianwen Jiang, and Meng Jiang. 2021. Validating Label Consistency in NER Data Annotation. In Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems, pages 11–15, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Cite (Informal):
Validating Label Consistency in NER Data Annotation (Zeng et al., Eval4NLP 2021)
Copy Citation:
PDF:
https://preview.aclanthology.org/emnlp22-frontmatter/2021.eval4nlp-1.2.pdf
Data
CoNLL++SciERC