@inproceedings{chen-etal-2025-benchmarking-llms,
    title = "Benchmarking {LLM}s for Translating Classical {C}hinese Poetry: Evaluating Adequacy, Fluency, and Elegance",
    author = "Chen, Andong  and
      Lou, Lianzhang  and
      Chen, Kehai  and
      Bai, Xuefeng  and
      Xiang, Yang  and
      Yang, Muyun  and
      Zhao, Tiejun  and
      Zhang, Min",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-main.1678/",
    pages = "33007--33024",
    ISBN = "979-8-89176-332-6",
    abstract = "Large language models (LLMs) have shown remarkable performance in general translation tasks. However, the increasing demand for high-quality translations that are not only adequate but also fluent and elegant. To assess the extent to which current LLMs can meet these demands, we introduce a suitable benchmark (PoetMT) for translating classical Chinese poetry into English. This task requires not only adequacy in translating culturally and historically significant content but also a strict adherence to linguistic fluency and poetic elegance. Our study reveals that existing LLMs fall short of this task. To address these issues, we propose RAT, a Retrieval-Augmented machine Translation method that enhances the translation process by incorporating knowledge related to classical poetry. Additionally, we propose an automatic evaluation metric based on GPT-4, which better assesses translation quality in terms of adequacy, fluency, and elegance, overcoming the limitations of traditional metrics."
}