@inproceedings{li-etal-2025-ssa,
    title = "{SSA}-{COMET}: Do {LLM}s Outperform Learned Metrics in Evaluating {MT} for Under-Resourced {A}frican Languages?",
    author = "Li, Senyu  and
      Wang, Jiayi  and
      Ali, Felermino D. M. A.  and
      Cherry, Colin  and
      Deutsch, Daniel  and
      Briakou, Eleftheria  and
      Sousa-Silva, Rui  and
      Lopes Cardoso, Henrique  and
      Stenetorp, Pontus  and
      Adelani, David Ifeoluwa",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-main.656/",
    pages = "12990--13009",
    ISBN = "979-8-89176-332-6",
    abstract = "Evaluating machine translation (MT) quality for under-resourced African languages remains a significant challenge, as existing metrics often suffer from limited language coverage and poor performance in low-resource settings. While recent efforts, such as AfriCOMET, have addressed some of the issues, they are still constrained by small evaluation sets, a lack of publicly available training data tailored to African languages, and inconsistent performance in extremely low-resource scenarios. In this work, we introduce SSA-MTE, a large-scale human-annotated MT evaluation (MTE) dataset covering 13 African language pairs from the News domain, with over 63,000 sentence-level annotations from a diverse set of MT systems. Based on this data, we develop SSA-COMET and SSA-COMET-QE, improved reference-based and reference-free evaluation metrics. We also benchmark prompting-based approaches using state-of-the-art LLMs like GPT-4o and Claude. Our experimental results show that SSA-COMET models significantly outperform AfriCOMET and are competitive with the strongest LLM (Gemini 2.5 Pro) evaluated in our study, particularly on low-resource languages such as Twi, Luo, and Yoruba. All resources are released under open licenses to support future research."
}