Simpson’s Paradox and the Accuracy-Fluency Tradeoff in Translation

Zheng Wei Lim, Ekaterina Vylomova, Trevor Cohn, Charles Kemp

Abstract
A good translation should be faithful to the source and should respect the norms of the target language. We address a theoretical puzzle about the relationship between these objectives. On one hand, intuition and some prior work suggest that accuracy and fluency should trade off against each other, and that capturing every detail of the source can only be achieved at the cost of fluency. On the other hand, quality assessment researchers often suggest that accuracy and fluency are highly correlated and difficult for human raters to distinguish (Callison-Burch et al., 2007). We show that the tension between these views is an instance of Simpson’s paradox, and that accuracy and fluency are positively correlated at the level of the corpus but trade off at the level of individual source segments. We further suggest that the relationship between accuracy and fluency is best evaluated at the segment (or sentence) level, and that the trade off between these dimensions has implications both for assessing translation quality and developing improved MT systems.
Anthology ID:
2024.acl-short.9
Volume:
Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
Month:
August
Year:
2024
Address:
Bangkok, Thailand
Editors:
Lun-Wei Ku, Andre Martins, Vivek Srikumar
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
92–103
Language:
URL:
https://aclanthology.org/2024.acl-short.9
DOI:
10.18653/v1/2024.acl-short.9
Bibkey:
Cite (ACL):
Zheng Wei Lim, Ekaterina Vylomova, Trevor Cohn, and Charles Kemp. 2024. Simpson’s Paradox and the Accuracy-Fluency Tradeoff in Translation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 92–103, Bangkok, Thailand. Association for Computational Linguistics.
Cite (Informal):
Simpson’s Paradox and the Accuracy-Fluency Tradeoff in Translation (Lim et al., ACL 2024)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-5/2024.acl-short.9.pdf
PDF Search