@inproceedings{devine-etal-2026-gaeleval,
    title = "{G}ael{E}val: Benchmarking {LLM} Performance for {S}cottish {G}aelic",
    author = "Devine, Peter  and
      Lamb, William  and
      Alex, Beatrice  and
      Ezeani, Ignatius  and
      Knight, Dawn  and
      {\'O} Meachair, M{\'i}che{\'a}l J.  and
      Rayson, Paul  and
      Wynne, Martin",
    editor = "Montejo-Raez, Arturo  and
      Grisot, Cristina  and
      Blochowiak, Joanna  and
      Ljube{\v{s}}i{\'c}, Nikola  and
      Battaner, Elena  and
      Rigau, German",
    booktitle = "Proceedings of Shaping Multilingual, Multimodal {AI} for the Social Sciences and Humanities ({LLM}s4{SSH}) @ {LREC} 2026",
    month = may,
    year = "2026",
    address = "Palma de Mallorca (Spain)",
    publisher = "ELRA Language Resources Association (ELRA)",
    url = "https://preview.aclanthology.org/ingest-lrec/2026.llms4ssh-1.8/",
    doi = "10.63317/2yqmx3es6ad5",
    pages = "73--85",
    abstract = "Multilingual large language models (LLMs) often exhibit emergent `shadow' capabilities in languages without official support, yet their performance on these languages remains uneven and under-measured. This is particularly acute for morphosyntactically rich minority languages such as Scottish Gaelic, where translation benchmarks fail to capture structural competence. We introduce GaelEval, the first multi-dimensional benchmark for Gaelic, comprising: (i) an expert-authored morphosyntactic MCQA task; (ii) a culturally-grounded translation benchmark and (iii) a large-scale cultural knowledge Q{\&}A task. Evaluating 19 LLMs against a fluent-speaker human baseline (n = 30), we find that Gemini 3 Pro Preview achieves 83.3{\%} accuracy on the linguistic task, surpassing the human baseline (78.1{\%}). Proprietary models consistently outperform open-weight systems, and in-language (Gaelic) prompting yields a small but stable advantage (+2.4pp). On the cultural task, leading models exceed 90{\%} accuracy, though most systems perform worse under Gaelic prompting and absolute scores are inflated relative to the manual benchmark. Overall, GaelEval reveals that frontier models achieve above-human performance on several dimensions of Gaelic grammar, demonstrates the effect of Gaelic prompting and shows a consistent performance gap favouring proprietary over open-weight models."
}