Ukrainian Visual Word Sense Disambiguation Benchmark
Yurii Laba, Yaryna Mohytych, Ivanna Rohulia, Halyna Kyryleyza, Hanna Dydyk-Meush, Oles Dobosevych, Rostyslav Hryniv
Abstract
This study presents a benchmark for evaluating the Visual Word Sense Disambiguation (Visual-WSD) task in Ukrainian. The main goal of the Visual-WSD task is to identify, with minimal contextual information, the most appropriate representation of a given ambiguous word from a set of ten images. To construct this benchmark, we followed a methodology similar to that proposed by (CITATION), who previously introduced benchmarks for the Visual-WSD task in English, Italian, and Farsi. This approach allows us to incorporate the Ukrainian benchmark into a broader framework for cross-language model performance comparisons. We collected the benchmark data semi-automatically and refined it with input from domain experts. We then assessed eight multilingual and multimodal large language models using this benchmark. All tested models performed worse than the zero-shot CLIP-based baseline model (CITATION) used by (CITATION) for the English Visual-WSD task. Our analysis revealed a significant performance gap in the Visual-WSD task between Ukrainian and English.- Anthology ID:
- 2024.unlp-1.8
- Volume:
- Proceedings of the Third Ukrainian Natural Language Processing Workshop (UNLP) @ LREC-COLING 2024
- Month:
- May
- Year:
- 2024
- Address:
- Torino, Italia
- Editors:
- Mariana Romanyshyn, Nataliia Romanyshyn, Andrii Hlybovets, Oleksii Ignatenko
- Venue:
- UNLP
- SIG:
- Publisher:
- ELRA and ICCL
- Note:
- Pages:
- 61–66
- Language:
- URL:
- https://aclanthology.org/2024.unlp-1.8
- DOI:
- Cite (ACL):
- Yurii Laba, Yaryna Mohytych, Ivanna Rohulia, Halyna Kyryleyza, Hanna Dydyk-Meush, Oles Dobosevych, and Rostyslav Hryniv. 2024. Ukrainian Visual Word Sense Disambiguation Benchmark. In Proceedings of the Third Ukrainian Natural Language Processing Workshop (UNLP) @ LREC-COLING 2024, pages 61–66, Torino, Italia. ELRA and ICCL.
- Cite (Informal):
- Ukrainian Visual Word Sense Disambiguation Benchmark (Laba et al., UNLP 2024)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/2024.unlp-1.8.pdf