@inproceedings{manakhimova-etal-2024-investigating,
    title = "Investigating the Linguistic Performance of Large Language Models in Machine Translation",
    author = {Manakhimova, Shushen  and
      Macketanz, Vivien  and
      Avramidis, Eleftherios  and
      Lapshinova-Koltunski, Ekaterina  and
      Bagdasarov, Sergei  and
      M{\"o}ller, Sebastian},
    editor = "Haddow, Barry  and
      Kocmi, Tom  and
      Koehn, Philipp  and
      Monz, Christof",
    booktitle = "Proceedings of the Ninth Conference on Machine Translation",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/jlcl-multiple-ingestion/2024.wmt-1.28/",
    doi = "10.18653/v1/2024.wmt-1.28",
    pages = "355--371",
    abstract = "This paper summarizes the results of our test suite evaluation on 39 machine translation systems submitted at the Shared Task of the Ninth Conference of Machine Translation (WMT24). It offers a fine-grained linguistic evaluation of machine translation outputs for English{--}German and English{--}Russian, resulting from significant manual linguistic effort. Based on our results, LLMs are inferior to NMT in English{--}German, both in overall scores and when translating specific linguistic phenomena, such as punctuation, complex future verb tenses, and stripping. LLMs show quite a competitive performance in English-Russian, although top-performing systems might struggle with some cases of named entities and terminology, function words, mediopassive voice, and semantic roles. Additionally, some LLMs generate very verbose or empty outputs, posing challenges to the evaluation process."
}