@inproceedings{clifford-etal-2025-gaeilge,
    title = "Gaeilge Bhriste {\'o} Shamhlacha Cliste: How Clever Are {LLM}s When Translating {I}rish Text?",
    author = "Clifford, Teresa  and
      Walsh, Abigail  and
      Davis, Brian  and
      {\'O} Meachair, M{\'i}che{\'a}l J.",
    editor = "Davis, Brian  and
      Fransen, Theodorus  and
      Dhonnchadha, Elaine U{\'i}  and
      Walsh, Abigail",
    booktitle = "Proceedings of the 5th Celtic Language Technology Workshop",
    month = jan,
    year = "2025",
    address = "Abu Dhabi [Virtual Workshop]",
    publisher = "International Committee on Computational Linguistics",
    url = "https://preview.aclanthology.org/jlcl-multiple-ingestion/2025.cltw-1.5/",
    pages = "46--51",
    abstract = "Large Language Models have been widely adopted in NLP tasks and applications, how- ever, their ability to accurately process Irish and other minority languages has not been fully explored. In this paper we describe prelim- inary experiments examining the capacity of publicly-available machine translation engines (Google Translate, Microsoft Bing, and eTrans- lation) and prompt-based AI systems systems (ChatGPT 3.5, Llama 2) for translating and handling challenging language features of Irish. A hand-crafted selection of challenging Irish language features were incorporated into trans- lation prompts, and the output from each model was examined by a human evaluator. The re- sults of these experiments indicate that these LLM-based models still struggle with translat- ing rare linguistic phenomena and ambiguous constructions. This preliminary analysis helps to inform further research in this field, pro- viding a simple ranking of publicly-available models, and indicating which language features require particular attention when evaluating model capacity."
}