Riki Shimizu
2025
XCOMPS: A Multilingual Benchmark of Conceptual Minimal Pairs
Linyang He
|
Ercong Nie
|
Sukru Samet Dindar
|
Arsalan Firoozi
|
Van Nguyen
|
Corentin Puffay
|
Riki Shimizu
|
Haotian Ye
|
Jonathan Brennan
|
Helmut Schmid
|
Hinrich Schuetze
|
Nima Mesgarani
Proceedings of the 7th Workshop on Research in Computational Linguistic Typology and Multilingual NLP
In this work, we introduce XCOMPS, a multilingual conceptual minimal pair dataset that covers 17 languages.Using this dataset, we evaluate LLMs’ multilingual conceptual understanding through metalinguistic prompting, direct probability measurement, and neurolinguistic probing. We find that: 1) LLMs exhibit weaker conceptual understanding for low-resource languages, and accuracy varies across languages despite being tested on the same concept sets. 2) LLMs excel at distinguishing concept-property pairs that are visibly different but exhibit a marked performance drop when negative pairs share subtle semantic similarities. 3) More morphologically complex languages yield lower concept understanding scores and require deeper layers for conceptual reasoning.
Search
Fix author
Co-authors
- Jonathan Brennan 1
- Sukru Samet Dindar 1
- Arsalan Firoozi 1
- Linyang He 1
- Nima Mesgarani 1
- show all...