@inproceedings{lavie-etal-2025-findings,
    title = "Findings of the {WMT}25 Shared Task on Automated Translation Evaluation Systems: Linguistic Diversity is Challenging and References Still Help",
    author = "Lavie, Alon  and
      Hanneman, Greg  and
      Agrawal, Sweta  and
      Kanojia, Diptesh  and
      Lo, Chi-Kiu  and
      Zouhar, Vil{\'e}m  and
      Blain, Frederic  and
      Zerva, Chrysoula  and
      Avramidis, Eleftherios  and
      Deoghare, Sourabh  and
      Sindhujan, Archchana  and
      Wang, Jiayi  and
      Adelani, David Ifeoluwa  and
      Thompson, Brian  and
      Kocmi, Tom  and
      Freitag, Markus  and
      Deutsch, Daniel",
    editor = "Haddow, Barry  and
      Kocmi, Tom  and
      Koehn, Philipp  and
      Monz, Christof",
    booktitle = "Proceedings of the Tenth Conference on Machine Translation",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.wmt-1.24/",
    pages = "436--483",
    ISBN = "979-8-89176-341-8",
    abstract = "The WMT25 Shared Task on Automated Translation Evaluation Systems evaluates metrics and quality estimation systems that assess the quality of language translation systems. This task unifies and consolidates the separate WMT shared tasks on Machine Translation Evaluation Metrics and Quality Estimation from previous years. Our primary goal is to encourage the development and assessment of new state-of-the-art translation quality evaluation systems. The shared task this year consisted of three subtasks: (1) segment-level quality score prediction, (2) span-level translation error annotation, and (3) quality-informed segment-level error correction. The evaluation data for the shared task were provided by the General MT shared task and were complemented by ``challenge sets'' from both the organizers and participants. Task 1 results indicate the strong performance of large LLMs at the system level, whilereference-based baseline metrics outperform LLMs at the segment level. Task 2 results indicate that accurate error detection and balancing precision and recall are persistent challenges. Task 3 results show that minimal editing is challenging even when informed by quality indicators. Robustness across the broad diversity of languages remains a major challenge across all three subtasks."
}