@inproceedings{piergentili-etal-2025-llm,
    title = "An {LLM}-as-a-judge Approach for Scalable Gender-Neutral Translation Evaluation",
    author = "Piergentili, Andrea  and
      Savoldi, Beatrice  and
      Negri, Matteo  and
      Bentivogli, Luisa",
    editor = "Hackenbuchner, Jani{\c{c}}a  and
      Bentivogli, Luisa  and
      Daems, Joke  and
      Manna, Chiara  and
      Savoldi, Beatrice  and
      Vanmassenhove, Eva",
    booktitle = "Proceedings of the 3rd Workshop on Gender-Inclusive Translation Technologies (GITT 2025)",
    month = jun,
    year = "2025",
    address = "Geneva, Switzerland",
    publisher = "European Association for Machine Translation",
    url = "https://preview.aclanthology.org/mtsummit-25-ingestion/2025.gitt-1.3/",
    pages = "46--63",
    ISBN = "978-2-9701897-4-9",
    abstract = "Gender-neutral translation (GNT) aims to avoid expressing the gender of human referents when the source text lacks explicit cues about the gender of those referents. Evaluating GNT automatically is particularly challenging, with current solutions being limited to monolingual classifiers. Such solutions are not ideal because they do not factor in the source sentence and require dedicated data and fine-tuning to scale to new languages. In this work, we address such limitations by investigating the use of large language models (LLMs) as evaluators of GNT. Specifically, we explore two prompting approaches: one in which LLMs generate sentence-level assessments only, and another{---}akin to a chain-of-thought approach{---}where they first produce detailed phrase-level annotations before a sentence-level judgment. Through extensive experiments on multiple languages with five models, both open and proprietary, we show that LLMs can serve as evaluators of GNT. Moreover, we find that prompting for phrase-level annotations before sentence-level assessments consistently improves the accuracy of all models, providing a better and more scalable alternative to current solutions."
}