@inproceedings{he-etal-2025-xcomps,
    title = "{XCOMPS}: A Multilingual Benchmark of Conceptual Minimal Pairs",
    author = "He, Linyang  and
      Nie, Ercong  and
      Dindar, Sukru Samet  and
      Firoozi, Arsalan  and
      Nguyen, Van  and
      Puffay, Corentin  and
      Shimizu, Riki  and
      Ye, Haotian  and
      Brennan, Jonathan  and
      Schmid, Helmut  and
      Schuetze, Hinrich  and
      Mesgarani, Nima",
    editor = "Hahn, Michael  and
      Rani, Priya  and
      Kumar, Ritesh  and
      Shcherbakov, Andreas  and
      Sorokin, Alexey  and
      Serikov, Oleg  and
      Cotterell, Ryan  and
      Vylomova, Ekaterina",
    booktitle = "Proceedings of the 7th Workshop on Research in Computational Linguistic Typology and Multilingual NLP",
    month = aug,
    year = "2025",
    address = "Vinenna. Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/landing_page/2025.sigtyp-1.9/",
    pages = "75--81",
    ISBN = "979-8-89176-281-7",
    abstract = "In this work, we introduce XCOMPS, a multilingual conceptual minimal pair dataset that covers 17 languages.Using this dataset, we evaluate LLMs' multilingual conceptual understanding through metalinguistic prompting, direct probability measurement, and neurolinguistic probing. We find that: 1) LLMs exhibit weaker conceptual understanding for low-resource languages, and accuracy varies across languages despite being tested on the same concept sets. 2) LLMs excel at distinguishing concept-property pairs that are visibly different but exhibit a marked performance drop when negative pairs share subtle semantic similarities. 3) More morphologically complex languages yield lower concept understanding scores and require deeper layers for conceptual reasoning."
}