@article{liu-etal-2026-systematic,
    title = "A Systematic Assessment of Language Models with Linguistic Minimal Pairs in {C}hinese",
    author = "Liu, Yikang  and
      Shen, Yeting  and
      Zhu, Hongao  and
      Xu, Lilong  and
      Qian, Zhiheng  and
      Song, Siyuan  and
      Zhang, Kejia  and
      Tang, Jialong  and
      Zhang, Pei  and
      Yang, Baosong  and
      Wang, Rui  and
      Hu, Hai",
    journal = "Transactions of the Association for Computational Linguistics",
    volume = "14",
    year = "2026",
    address = "Cambridge, MA",
    publisher = "MIT Press",
    url = "https://preview.aclanthology.org/ingest-latest-mitpress-cl-tacl/2026.tacl-1.34/",
    doi = "10.1162/tacl.a.648",
    pages = "755--771",
    abstract = "We present ZhoBLiMP, the largest linguistic minimal pair benchmark for Chinese, with over 100 paradigms, ranging from topicalization to the Ba construction. We then train from scratch a suite of Chinese language models (LMs) with different tokenizers, parameter sizes, and token volumes, to study the learning curves of LMs on Chinese. To mitigate the biases introduced by unequal lengths of the sentences in a minimal pair, we propose a new metric named sub-linear length normalized log-probabilities (SLLN-LP). Using SLLN-LP as the metric, our results show that Anaphor, Quantifiers, and Ellipsis in Chinese are difficult for LMs even up to 32B parameters, and that SLLN-LP successfully mitigates biases in ZhoBLiMP, JBLiMP and BLiMP. We conclude that future evaluations should be more carefully designed to consider the intricate relations between linking functions, LMs, and targeted minimal pairs."
}