DELA Corpus - A Document-Level Corpus Annotated with Context-Related Issues

Sheila Castilho, João Lucas Cavalheiro Camargo, Miguel Menezes, Andy Way

Abstract
Recently, the Machine Translation (MT) community has become more interested in document-level evaluation especially in light of reactions to claims of “human parity”, since examining the quality at the level of the document rather than at the sentence level allows for the assessment of suprasentential context, providing a more reliable evaluation. This paper presents a document-level corpus annotated in English with context-aware issues that arise when translating from English into Brazilian Portuguese, namely ellipsis, gender, lexical ambiguity, number, reference, and terminology, with six different domains. The corpus can be used as a challenge test set for evaluation and as a training/testing corpus for MT as well as for deep linguistic analysis of context issues. To the best of our knowledge, this is the first corpus of its kind.
Anthology ID:
2021.wmt-1.63
Volume:
Proceedings of the Sixth Conference on Machine Translation
Month:
November
Year:
2021
Address:
Online
Editors:
Loic Barrault, Ondrej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jussa, Christian Federmann, Mark Fishel, Alexander Fraser, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Tom Kocmi, Andre Martins, Makoto Morishita, Christof Monz
Venue:
WMT
SIG:
SIGMT
Publisher:
Association for Computational Linguistics
Note:
Pages:
566–577
Language:
URL:
https://aclanthology.org/2021.wmt-1.63
DOI:
Bibkey:
Cite (ACL):
Sheila Castilho, João Lucas Cavalheiro Camargo, Miguel Menezes, and Andy Way. 2021. DELA Corpus - A Document-Level Corpus Annotated with Context-Related Issues. In Proceedings of the Sixth Conference on Machine Translation, pages 566–577, Online. Association for Computational Linguistics.
Cite (Informal):
DELA Corpus - A Document-Level Corpus Annotated with Context-Related Issues (Castilho et al., WMT 2021)
Copy Citation:
PDF:
https://preview.aclanthology.org/emnlp-22-attachments/2021.wmt-1.63.pdf
Video:
 https://preview.aclanthology.org/emnlp-22-attachments/2021.wmt-1.63.mp4
Code
 sheilacastilho/dela-project
PDF Search Code Video