@inproceedings{zhao-etal-2025-realbench,
    title = "{R}eal{B}ench: A {C}hinese Multi-image Understanding Benchmark Close to Real-world Scenarios",
    author = "Zhao, Fei  and
      Lu, Chengqiang  and
      Shen, Yufan  and
      Wang, Qimeng  and
      Qian, Yicheng  and
      Zhang, Haoxin  and
      Gao, Yan  and
      Yiwu  and
      Hu, Yao  and
      Wu, Zhen  and
      Xing, Shangyu  and
      Dai, Xinyu",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2025",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/author-page-yu-wang-polytechnic/2025.findings-emnlp.1039/",
    doi = "10.18653/v1/2025.findings-emnlp.1039",
    pages = "19097--19115",
    ISBN = "979-8-89176-335-7",
    abstract = "While various multimodal multi-image evaluation datasets have been emerged, but these datasets are primarily based on English, and there has yet to be a Chinese multi-image dataset. To fill this gap, we introduce RealBench, the first Chinese multimodal multi-image dataset, which contains 9393 samples and 69910 images. RealBench distinguishes itself by incorporating real user-generated content, ensuring high relevance to real-world applications. Additionally, the dataset covers a wide variety of scenes, image resolutions, and image structures, further increasing the difficulty of multi-image understanding. Ultimately, we conduct a comprehensive evaluation of RealBench using 21 multimodal LLMs of different sizes, including closed-source models that support multi-image inputs as well as open-source visual and video models. The experimental results indicate that even the most powerful closed-source models still face challenges when handling multi-image Chinese scenarios. Moreover, there remains a noticeable performance gap of around 71.8{\%} on average between open-source visual/video models and closed-source models. These results show that RealBench provides an important research foundation for further exploring multi-image understanding capabilities in the Chinese context. Our datasets will be publicly available."
}