@article{koeva-etal-2026-bulgarian,
    title = "{B}ulgarian Massive Multitask Language Understanding Benchmark",
    author = "Koeva, Svetla Peneva  and
      Stoyanova, Ivelina  and
      Georgiev, Dimiter  and
      Leseva, Svetlozara  and
      Stefanova, Valentina  and
      Todorova, Maria  and
      Dimitrova, Tsvetana Ivanova  and
      Kukova, Hristina  and
      Moskova, Mihaela  and
      Tinchev, Tinko",
    editor = "Piperidis, Stelios  and
      Bel, N{\'u}ria  and
      van den Heuvel, Henk  and
      Ide, Nancy  and
      Krek, Simon  and
      Toral, Antonio",
    journal = "International Conference on Language Resources and Evaluation",
    volume = "main",
    month = may,
    year = "2026",
    address = "Palma de Mallorca, Spain",
    publisher = "ELRA Language Resource Association",
    url = "https://preview.aclanthology.org/ingest-lrec/2026.lrec-main.366/",
    pages = "4658--4672",
    abstract = "Assessing the broad general knowledge of Large Language Models (LLMs) across multiple domains in Bulgarian remains challenging due to the limited availability of Bulgarian evaluation benchmarks. To address this gap, we introduce the Bulgarian Massive Multitask Language Understanding benchmark (MMLU-BG), designed to evaluate whether LLMs possess generalised knowledge capabilities beyond simple text prediction in Bulgarian. This paper presents the structure, the development protocol, and the size of the MMLU-BG benchmark. It is tested in comparison with the original MMLU for English across seven LLMs selected according to specific criteria. The experiments demonstrate that the MMLU-BG benchmark assesses multi-domain versatility and highlights the models' strengths and weaknesses across different subject areas."
}