Turning silver into gold: error-focused corpus reannotation with active learning

Pierre André Ménard, Antoine Mougeot


Abstract
While high quality gold standard annotated corpora are crucial for most tasks in natural language processing, many annotated corpora published in recent years, created by annotators or tools, contains noisy annotations. These corpora can be viewed as more silver than gold standards, even if they are used in evaluation campaigns or to compare systems’ performances. As upgrading a silver corpus to gold level is still a challenge, we explore the application of active learning techniques to detect errors using four datasets designed for document classification and part-of-speech tagging. Our results show that the proposed method for the seeding step improves the chance of finding incorrect annotations by a factor of 2.73 when compared to random selection, a 14.71% increase from the baseline methods. Our query method provides an increase in the error detection precision on average by a factor of 1.78 against random selection, an increase of 61.82% compared to other query approaches.
Anthology ID:
R19-1088
Volume:
Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019)
Month:
September
Year:
2019
Address:
Varna, Bulgaria
Venue:
RANLP
SIG:
Publisher:
INCOMA Ltd.
Note:
Pages:
758–767
Language:
URL:
https://aclanthology.org/R19-1088
DOI:
10.26615/978-954-452-056-4_088
Bibkey:
Cite (ACL):
Pierre André Ménard and Antoine Mougeot. 2019. Turning silver into gold: error-focused corpus reannotation with active learning. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), pages 758–767, Varna, Bulgaria. INCOMA Ltd..
Cite (Informal):
Turning silver into gold: error-focused corpus reannotation with active learning (Ménard & Mougeot, RANLP 2019)
Copy Citation:
PDF:
https://preview.aclanthology.org/update-css-js/R19-1088.pdf