@inproceedings{zhang-etal-2025-mllms,
    title = "Can {MLLM}s Understand the Deep Implication Behind {C}hinese Images?",
    author = "Zhang, Chenhao  and
      Feng, Xi  and
      Bai, Yuelin  and
      Du, Xeron  and
      Hou, Jinchang  and
      Deng, Kaixin  and
      Han, Guangzeng  and
      Li, Qinrui  and
      Wang, Bingli  and
      Liu, Jiaheng  and
      Qu, Xingwei  and
      Zhang, Yifei  and
      Zhao, Qixuan  and
      Liang, Yiming  and
      Liu, Ziqiang  and
      Fang, Feiteng  and
      Yang, Min  and
      Huang, Wenhao  and
      Lin, Chenghua  and
      Zhang, Ge  and
      Ni, Shiwen",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.700/",
    pages = "14369--14402",
    ISBN = "979-8-89176-251-0",
    abstract = "As the capabilities of Multimodal Large Language Models (MLLMs) improve, the need for higher-order evaluation of them is increasing. However, there is a lack of work evaluating MLLM for higher-order perception and understanding of Chinese visual content. To address this, we introduce the CII-Bench, which aims to assess MLLMs' such capabilities for Chinese images. To ensure the authenticity of the Chinese context, images in CII-Bench are sourced from the Chinese Internet and manually reviewed, with corresponding answers also manually crafted. Additionally, CII-Bench incorporates images that represent Chinese traditional culture, such as famous Chinese traditional paintings, which can deeply reflect the model{'}s understanding of Chinese traditional culture. Through experiments on multiple MLLMs using CII-Bench, significant findings emerged. There is a large gap between MLLMs and humans in performance. The highest MLLM accuracy is 64.4{\%}, while the human average is 78.2{\%} and the peak is 81.0{\%}. MLLMs perform poorly on traditional culture images, indicating limitations in understanding high-level semantics and lacking a deep knowledge base of Chinese traditional culture. Moreover, most models have higher accuracy when image emotion hints are added to the prompts. We believe CII-Bench will help MLLMs better understand Chinese semantics and specific images, and move forward the development of expert artificial general intelligence (AGI). Our project is publicly available at https://cii-bench.github.io."
}