@inproceedings{saba-skiena-2025-evaluating,
    title = "Evaluating Language Translation Models by Playing Telephone",
    author = "Saba, Syeda Jannatus  and
      Skiena, Steven",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-main.524/",
    pages = "10332--10347",
    ISBN = "979-8-89176-332-6",
    abstract = "Our ability to efficiently and accurately evaluate the quality of machine translation systems has been outrun by the effectiveness of current language models{---}which limits the potential for further improving these models on more challenging tasks like long-form and literary translation. We propose an unsupervised method to generate training data for translation evaluation over different document lengths and application domains by repeated rounds of translation between source and target languages. We evaluate evaluation systems trained on texts mechanically generated using both model rotation and language translation approaches, demonstrating improved performance over a popular translation evaluation system (xCOMET) on two different tasks: (i) scoring the quality of a given translation against a human reference and (ii) selecting which of two translations is generationally closer to an original source document."
}