@inproceedings{armannsson-etal-2024-killing,
    title = "Killing Two Flies with One Stone: An Attempt to Break {LLM}s Using {E}nglish-{I}celandic Idioms and Proper Names",
    author = "{\'A}rmannsson, Bjarki  and
      Hafsteinsson, Hinrik  and
      Jasonarson, Atli  and
      Steingr{\'i}msson, Stein{\th}{\'o}r",
    editor = "Haddow, Barry  and
      Kocmi, Tom  and
      Koehn, Philipp  and
      Monz, Christof",
    booktitle = "Proceedings of the Ninth Conference on Machine Translation",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/jlcl-multiple-ingestion/2024.wmt-1.31/",
    doi = "10.18653/v1/2024.wmt-1.31",
    pages = "451--458",
    abstract = "The submission of the {\'A}rni Magn{\'u}sson Institute`s team to the WMT24 test suite subtask focuses on idiomatic expressions and proper names for the English{\textrightarrow}Icelandic translation direction. Intuitively and empirically, idioms and proper names are known to be a significant challenge for neural translation models. We create two different test suites. The first evaluates the competency of MT systems in translating common English idiomatic expressions, as well as testing whether systems can distinguish between those expressions and the same phrases when used in a literal context. The second test suite consists of place names that should be translated into their Icelandic exonyms (and correctly inflected) and pairs of Icelandic names that share a surface form between the male and female variants, so that incorrect translations impact meaning as well as readibility. The scores reported are relatively low, especially for idiomatic expressions and place names, and indicate considerable room for improvement."
}