@inproceedings{niklaus-etal-2025-swiltra,
    title = "{S}wi{LT}ra-Bench: The {S}wiss Legal Translation Benchmark",
    author = {Niklaus, Joel  and
      Merane, Jakob  and
      Nenadic, Luka  and
      Ahmadi, Sina  and
      Gao, Yingqiang  and
      Chevalley, Cyrill A. H.  and
      Humbel, Claude  and
      G{\"o}sken, Christophe  and
      Tanzi, Lorenzo  and
      L{\"u}thi, Thomas  and
      Palombo, Stefan  and
      Poff, Spencer  and
      Yang, Boling  and
      Wu, Nan  and
      Guillod, Matthew  and
      Mami{\'e}, Robin  and
      Brunner, Daniel  and
      Pereyra, Julio  and
      Grupen, Niko},
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.725/",
    pages = "14894--14916",
    ISBN = "979-8-89176-251-0",
    abstract = "In Switzerland legal translation is uniquely important due to the country{'}s four official languages and requirements for multilingual legal documentation. However, this process traditionally relies on professionals who must be both legal experts and skilled translators{---}creating bottlenecks and impacting effective access to justice. To address this challenge, we introduce SwiLTra-Bench, a comprehensive multilingual benchmark of over 180K aligned Swiss legal translation pairs comprising laws, headnotes, and press releases across all Swiss languages along with English, designed to evaluate LLM-based translation systems. Our systematic evaluation reveals that frontier models achieve superior translation performance across all document types, while specialized translation systems excel specifically in laws but under-perform in headnotes. Through rigorous testing and human expert validation, we demonstrate that while fine-tuning open SLMs significantly improves their translation quality, they still lag behind the best zero-shot prompted frontier models such as Claude-3.5-Sonnet. Additionally, we present SwiLTra-Judge, a specialized LLM evaluation system that aligns best with human expert assessments."
}