@inproceedings{aparovich-etal-2025-belarusianglue,
    title = "{B}elarusian{GLUE}: Towards a Natural Language Understanding Benchmark for {B}elarusian",
    author = "Aparovich, Maksim  and
      Harytskaya, Volha  and
      Poritski, Vladislav  and
      Volchek, Oksana  and
      Smrz, Pavel",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.25/",
    pages = "511--527",
    ISBN = "979-8-89176-251-0",
    abstract = "In the epoch of multilingual large language models (LLMs), it is still challenging to evaluate the models' understanding of lower-resourced languages, which motivates further development of expert-crafted natural language understanding benchmarks. We introduce BelarusianGLUE {---} a natural language understanding benchmark for Belarusian, an East Slavic language, with {\ensuremath{\approx}}15K instances in five tasks: sentiment analysis, linguistic acceptability, word in context, Winograd schema challenge, textual entailment. A systematic evaluation of BERT models and LLMs against this novel benchmark reveals that both types of models approach human-level performance on easier tasks, such as sentiment analysis, but there is a significant gap in performance between machine and human on a harder task {---} Winograd schema challenge. We find the optimal choice of model type to be task-specific: e.g. BERT models underperform on textual entailment task but are competitive for linguistic acceptability. We release the datasets (https://hf.co/datasets/maaxap/BelarusianGLUE) and evaluation code (https://github.com/maaxap/BelarusianGLUE)."
}