@inproceedings{zhao-etal-2026-cnsl,
    title = "{CNSL}-bench: Benchmarking the Sign Language Understanding Capabilities of {MLLM}s on {C}hinese National Sign Language",
    author = "Zhao, Rui  and
      Zhong, Xuewen  and
      Zheng, Xiaoyun  and
      Su, Jinsong  and
      Chen, Yidong",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.acl-long.1896/",
    pages = "40869--40890",
    ISBN = "979-8-89176-390-6",
    abstract = "Sign language research has achieved significant progress due to the advances in large language models (LLMs). However, the intrinsic ability of LLMs to understand sign language, especially in multimodal contexts, remains underexplored. To address this limitation, we introduce $\textbf{CNSL-bench}$, the first comprehensive $\textbf{C}$hinese $\textbf{N}$ational $\textbf{S}$ign $\textbf{L}$anguage $\textbf{bench}$mark designed for evaluating multimodal large language models (MLLMs) in sign language understanding. The proposed CNSL-bench is characterized by: 1) Authoritative grounding, as it is anchored to the officially standardized $\textit{National Common Sign Language Dictionary}$, mitigating ambiguity from regional or non-canonical variants and ensuring consistent semantic definitions; 2) Multimodal coverage, providing aligned textual descriptions, illustrative images, and sign language videos; and 3) Articulatory diversity, supporting fine-grained analysis across key manual articulatory forms, including air-writing, finger-spelling, and the Chinese manual-alphabet. Using CNSL-bench, we extensively evaluate 21 open-source and proprietary up-to-date MLLMs. Our results reveal that, despite recent advances in multimodal modeling, current MLLMs remain substantially inferior to human performance, exhibiting systematic disparities across input modalities and manual articulatory forms. Additional diagnostic analyses suggest that several performance limitations persist beyond improvements in reasoning and that instruction-following robustness varies substantially across models."
}