Abstract
Providing quality scores along with Machine Translation (MT) output, so-called reference-free Quality Estimation (QE), is crucial to inform users about the reliability of the translation. We propose a model-specific, unsupervised QE approach, termed kNN-QE, that extracts information from the MT model’s training data using k-nearest neighbors. Measuring the performance of model-specific QE is not straightforward, since they provide quality scores on their own MT output, thus cannot be evaluated using benchmark QE test sets containing human quality scores on premade MT output. Therefore, we propose an automatic evaluation method that uses quality scores from reference-based metrics as gold standard instead of human-generated ones. We are the first to conduct detailed analyses and conclude that this automatic method is sufficient, and the reference-based MetricX-23 is best for the task.- Anthology ID:
- 2024.eamt-1.14
- Volume:
- Proceedings of the 25th Annual Conference of the European Association for Machine Translation (Volume 1)
- Month:
- June
- Year:
- 2024
- Address:
- Sheffield, UK
- Editors:
- Carolina Scarton, Charlotte Prescott, Chris Bayliss, Chris Oakley, Joanna Wright, Stuart Wrigley, Xingyi Song, Edward Gow-Smith, Rachel Bawden, Víctor M Sánchez-Cartagena, Patrick Cadwell, Ekaterina Lapshinova-Koltunski, Vera Cabarrão, Konstantinos Chatzitheodorou, Mary Nurminen, Diptesh Kanojia, Helena Moniz
- Venue:
- EAMT
- SIG:
- Publisher:
- European Association for Machine Translation (EAMT)
- Note:
- Pages:
- 133–146
- Language:
- URL:
- https://preview.aclanthology.org/add_missing_videos/2024.eamt-1.14/
- DOI:
- Cite (ACL):
- Tu Anh Dinh, Tobias Palzer, and Jan Niehues. 2024. Quality Estimation with k-nearest Neighbors and Automatic Evaluation for Model-specific Quality Estimation. In Proceedings of the 25th Annual Conference of the European Association for Machine Translation (Volume 1), pages 133–146, Sheffield, UK. European Association for Machine Translation (EAMT).
- Cite (Informal):
- Quality Estimation with k-nearest Neighbors and Automatic Evaluation for Model-specific Quality Estimation (Anh Dinh et al., EAMT 2024)
- PDF:
- https://preview.aclanthology.org/add_missing_videos/2024.eamt-1.14.pdf