@inproceedings{paniv-etal-2025-benchmarking,
    title = "Benchmarking Multimodal Models for {U}krainian Language Understanding Across Academic and Cultural Domains",
    author = "Paniv, Yurii  and
      Kiulian, Artur  and
      Chaplynskyi, Dmytro  and
      Khandoga, Mykola  and
      Polishko, Anton  and
      Bas, Tetiana  and
      Gabrielli, Guillermo",
    editor = "Romanyshyn, Mariana",
    booktitle = "Proceedings of the Fourth Ukrainian Natural Language Processing Workshop (UNLP 2025)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria (online)",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/acl25-workshop-ingestion/2025.unlp-1.2/",
    pages = "14--26",
    ISBN = "979-8-89176-269-5",
    abstract = "While the evaluation of multimodal English-centric models is an active area of research with numerous benchmarks, there is a profound lack of benchmarks or evaluation suites for low- and mid-resource languages. We introduce ZNO-Vision, a comprehensive multimodal Ukrainian-centric benchmark derived from the standardized university entrance examination (ZNO). The benchmark consists of over 4300 expert-crafted questions spanning 12 academic disciplines, including mathematics, physics, chemistry, and humanities. We evaluated the performance of both open-source models and API providers, finding that only a handful of models performed above baseline. Alongside the new benchmark, we performed the first evaluation study of multimodal text generation for the Ukrainian language: we measured caption generation quality on the Multi30K-UK dataset. Lastly, we tested a few models from a cultural perspective on knowledge of national cuisine. We believe our work will advance multimodal generation capabilities for the Ukrainian language and our approach could be useful for other low-resource languages."
}