@inproceedings{li-etal-2025-evaluating-wmt,
    title = "Evaluating {WMT} 2025 Metrics Shared Task Submissions on the {SSA}-{MTE} {A}frican Challenge Set",
    author = "Li, Senyu  and
      Ali, Felermino Dario Mario  and
      Wang, Jiayi  and
      Sousa-Silva, Rui  and
      Lopes Cardoso, Henrique  and
      Stenetorp, Pontus  and
      Cherry, Colin  and
      Adelani, David Ifeoluwa",
    editor = "Haddow, Barry  and
      Kocmi, Tom  and
      Koehn, Philipp  and
      Monz, Christof",
    booktitle = "Proceedings of the Tenth Conference on Machine Translation",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/author-page-yu-wang-polytechnic/2025.wmt-1.65/",
    doi = "10.18653/v1/2025.wmt-1.65",
    pages = "913--919",
    ISBN = "979-8-89176-341-8",
    abstract = "This paper presents the evaluation of submissions to the WMT 2025 Metrics Shared Task on the SSA-MTE challenge set, a large-scale benchmark for machine translation evaluation (MTE) in Sub-Saharan African languages. The SSA-MTE test sets contains over 12,768 human-annotated adequacy scores across 11 language pairs sourced from English, French, and Portuguese, spanning 6 commercial and open-source MT systems. Results show that correlations with human judgments remain generally low, with most systems falling below the 0.4 Spearman threshold for medium-level agreement. Performance varies widely across language pairs, with most correlations under 0.4; in some extremely low-resource cases, such as Portuguese{--}Emakhuwa, correlations drop to around 0.1, underscoring the difficulty of evaluating MT for very low-resource African languages. These findings highlight the urgent need for more research on robust, generalizable MT evaluation methods tailored for African languages."
}