@inproceedings{galeshchuk-etal-2026-toward,
    title = "Toward a Gold-Standard Benchmark for Evaluating {U}krainian Language Proficiency in {LLM}s",
    author = "Galeshchuk, Svitlana  and
      Maksymiuk, Yuliia  and
      Chernobrov, Yuliia  and
      Stankevych, Nina  and
      Antoniv, Oleksandra  and
      Faryna, Nataliia  and
      Popkova, Oksana",
    editor = "Romanyshyn, Mariana",
    booktitle = "Proceedings of the Fifth {U}krainian Natural Language Processing Conference ({UNLP} 2026)",
    month = may,
    year = "2026",
    address = "Lviv, Ukraine",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/corrections-2026-06/2026.unlp-1.12/",
    pages = "121--135",
    ISBN = "979-8-89176-359-3",
    abstract = "The paper presents an expert-curated benchmark for assessing Ukrainian proficiency in LLMs, focusing on grammar and orthography as core components of language competence. Prepared by professional linguists, the proposed gold-standard dataset is designed to test normative Ukrainian usage.The benchmark is further used to evaluate a range of LLMs, including Ukrainian-focused, multilingual, and large-scale models, under zero-shot and few-shot prompting in Ukrainian and English. Across these settings, smaller models achieve no more than 42.1{\%} accuracy, while large-scale LLMs reach up to 59.6{\%}. These results show that standard Ukrainian remains challenging for current LLMs and highlight the need for stronger language-specific evaluation and adaptation."
}