Riki Shimizu


2025

pdf bib
XCOMPS: A Multilingual Benchmark of Conceptual Minimal Pairs
Linyang He | Ercong Nie | Sukru Samet Dindar | Arsalan Firoozi | Van Nguyen | Corentin Puffay | Riki Shimizu | Haotian Ye | Jonathan Brennan | Helmut Schmid | Hinrich Schuetze | Nima Mesgarani
Proceedings of the 7th Workshop on Research in Computational Linguistic Typology and Multilingual NLP

In this work, we introduce XCOMPS, a multilingual conceptual minimal pair dataset that covers 17 languages.Using this dataset, we evaluate LLMs’ multilingual conceptual understanding through metalinguistic prompting, direct probability measurement, and neurolinguistic probing. We find that: 1) LLMs exhibit weaker conceptual understanding for low-resource languages, and accuracy varies across languages despite being tested on the same concept sets. 2) LLMs excel at distinguishing concept-property pairs that are visibly different but exhibit a marked performance drop when negative pairs share subtle semantic similarities. 3) More morphologically complex languages yield lower concept understanding scores and require deeper layers for conceptual reasoning.