@inproceedings{liu-etal-2025-mdseval,
    title = "{MDSE}val: A Meta-Evaluation Benchmark for Multimodal Dialogue Summarization",
    author = "Liu, Yinhong  and
      He, Jianfeng  and
      Su, Hang  and
      Lian, Ruixue  and
      Nian, Yi  and
      Vincent, Jake W.  and
      Vishnubhotla, Srikanth  and
      Piramuthu, Robinson  and
      Mansour, Saab",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2025",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/author-page-yu-wang-polytechnic/2025.findings-emnlp.794/",
    doi = "10.18653/v1/2025.findings-emnlp.794",
    pages = "14707--14727",
    ISBN = "979-8-89176-335-7",
    abstract = "Multimodal Dialogue Summarization (MDS) is a critical task with wide-ranging applications. To support the development of effective MDS models, robust automatic evaluation methods are essential for reducing both cost and human effort. However, such methods require a strong meta-evaluation benchmark grounded in human annotations. In this work, we introduce MDSEval, the first meta-evaluation benchmark for MDS, consisting image-sharing dialogues, corresponding summaries, and human judgments across eight well-defined quality aspects. To ensure data quality and richfulness, we propose a novel filtering framework leveraging Mutually Exclusive Key Information (MEKI) across modalities. Our work is the first to identify and formalize key evaluation dimensions specific to MDS. Finally, we benchmark state-of-the-art modal evaluation methods, revealing their limitations in distinguishing summaries from advanced MLLMs and their susceptibility to various bias."
}