@inproceedings{xue-etal-2025-mmrc,
    title = "{MMRC}: A Large-Scale Benchmark for Understanding Multimodal Large Language Model in Real-World Conversation",
    author = "Xue, Haochen  and
      Tang, Feilong  and
      Hu, Ming  and
      Liu, Yexin  and
      Huang, Qidong  and
      Li, Yulong  and
      Liu, Chengzhi  and
      Xu, Zhongxing  and
      Zhang, Chong  and
      Feng, Chun-Mei  and
      Xie, Yutong  and
      Razzak, Imran  and
      Ge, Zongyuan  and
      Su, Jionglong  and
      He, Junjun  and
      Qiao, Yu",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.1096/",
    pages = "22477--22503",
    ISBN = "979-8-89176-251-0",
    abstract = "Recent multimodal large language models (MLLMs) have demonstrated significant potential in open-ended conversation, generating more accurate and personalized responses. However, their abilities to memorize, recall, and reason in sustained interactions within real-world scenarios remain underexplored. This paper introduces MMRC, a Multi-Modal Real-world Conversation benchmark for evaluating six core open-ended abilities of MLLMs: information extraction, multi-turn reasoning, information update, image management, memory recall, and answer refusal. With data collected from real-world scenarios, MMRC comprises 5,120 conversations and 28,720 corresponding manually labeled questions, posing a significant challenge to existing MLLMs. Evaluations on 20 MLLMs in MMRC indicate an accuracy drop during open-ended interactions. We identify four common failure patterns: long-term memory degradation, inadequacies in updating factual knowledge, accumulated assumption of error propagation, and reluctance to ``say no.'' To mitigate these issues, we propose a simple yet effective NOTE-TAKING strategy, which can record key information from the conversation and remind the model during its responses, enhancing conversational capabilities. Experiments across six MLLMs demonstrate significant performance improvements."
}