Quality and Efficiency of Manual Annotation: Pre-annotation Bias

Marie Mikulová, Milan Straka, Jan Štěpánek, Barbora Štěpánková, Jan Hajic

Abstract
This paper presents an analysis of annotation using an automatic pre-annotation for a mid-level annotation complexity task - dependency syntax annotation. It compares the annotation efforts made by annotators using a pre-annotated version (with a high-accuracy parser) and those made by fully manual annotation. The aim of the experiment is to judge the final annotation quality when pre-annotation is used. In addition, it evaluates the effect of automatic linguistically-based (rule-formulated) checks and another annotation on the same data available to the annotators, and their influence on annotation quality and efficiency. The experiment confirmed that the pre-annotation is an efficient tool for faster manual syntactic annotation which increases the consistency of the resulting annotation without reducing its quality.
Anthology ID:
2022.lrec-1.312
Volume:
Proceedings of the Thirteenth Language Resources and Evaluation Conference
Month:
June
Year:
2022
Address:
Marseille, France
Editors:
Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Jan Odijk, Stelios Piperidis
Venue:
LREC
SIG:
Publisher:
European Language Resources Association
Note:
Pages:
2909–2918
Language:
URL:
https://aclanthology.org/2022.lrec-1.312
DOI:
Bibkey:
Cite (ACL):
Marie Mikulová, Milan Straka, Jan Štěpánek, Barbora Štěpánková, and Jan Hajic. 2022. Quality and Efficiency of Manual Annotation: Pre-annotation Bias. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2909–2918, Marseille, France. European Language Resources Association.
Cite (Informal):
Quality and Efficiency of Manual Annotation: Pre-annotation Bias (Mikulová et al., LREC 2022)
Copy Citation:
PDF:
https://preview.aclanthology.org/naacl24-info/2022.lrec-1.312.pdf
PDF Search