@inproceedings{fluckiger-etal-2025-comparison,
    title = "A comparison of translation performance between {D}eep{L} and Supertext",
    author = {Fl{\"u}ckiger, Alex  and
      Amrhein, Chantal  and
      Graf, Tim  and
      Odermatt, Fr{\'e}d{\'e}ric  and
      P{\"o}msl, Martin  and
      Schl{\"a}pfer, Philippe  and
      Schottmann, Florian  and
      L{\"a}ubli, Samuel},
    editor = {Bouillon, Pierrette  and
      Gerlach, Johanna  and
      Girletti, Sabrina  and
      Volkart, Lise  and
      Rubino, Raphael  and
      Sennrich, Rico  and
      L{\"a}ubli, Samuel  and
      Volk, Martin  and
      Espl{\`a}-Gomis, Miquel  and
      Vandeghinste, Vincent  and
      Moniz, Helena  and
      Szoc, Sara},
    booktitle = "Proceedings of Machine Translation Summit XX: Volume 2",
    month = jun,
    year = "2025",
    address = "Geneva, Switzerland",
    publisher = "European Association for Machine Translation",
    url = "https://preview.aclanthology.org/mtsummit-25-ingestion/2025.mtsummit-2.6/",
    pages = "52--57",
    ISBN = "978-2-9701897-1-8",
    abstract = "As strong machine translation (MT) systems are increasingly based on large language models (LLMs), reliable quality benchmarking requires methods that capture their ability to leverage extended context. This study compares two commercial MT systems {--} DeepL and Supertext {--} by assessing their performance on unsegmented texts. We evaluate translation quality across four language directions with professional translators assessing segments with full document-level context. While segment-level assessments indicate no strong preference between the systems in most cases, document-level analysis reveals a preference for Supertext in three out of four language directions, suggesting superior consistency across longer texts. We advocate for more context-sensitive evaluation methodologies to ensure that MT quality assessments reflect real-world usability. We release all evaluation data and scripts for further analysis and reproduction at https://github.com/supertext/evaluation{\_}deepl{\_}supertext."
}