Riki Shimizu


Fixing paper assignments

  1. Please select all papers that belong to the same person.
  2. Indicate below which author they should be assigned to.
Provide a valid ORCID iD here. This will be used to match future papers to this author.
Provide the name of the school or the university where the author has received or will receive their highest degree (e.g., Ph.D. institution for researchers, or current affiliation for students). This will be used to form the new author page ID, if needed.

TODO: "submit" and "cancel" buttons here


2025

pdf bib
XCOMPS: A Multilingual Benchmark of Conceptual Minimal Pairs
Linyang He | Ercong Nie | Sukru Samet Dindar | Arsalan Firoozi | Adrian Florea | Van Nguyen | Corentin Puffay | Riki Shimizu | Haotian Ye | Jonathan Brennan | Helmut Schmid | Hinrich Schütze | Nima Mesgarani
Proceedings of the 7th Workshop on Research in Computational Linguistic Typology and Multilingual NLP

In this work, we introduce XCOMPS, a multilingual conceptual minimal pair dataset that covers 17 languages.Using this dataset, we evaluate LLMs’ multilingual conceptual understanding through metalinguistic prompting, direct probability measurement, and neurolinguistic probing. We find that: 1) LLMs exhibit weaker conceptual understanding for low-resource languages, and accuracy varies across languages despite being tested on the same concept sets. 2) LLMs excel at distinguishing concept-property pairs that are visibly different but exhibit a marked performance drop when negative pairs share subtle semantic similarities. 3) More morphologically complex languages yield lower concept understanding scores and require deeper layers for conceptual reasoning.