@inproceedings{min-etal-2025-towards,
    title = "Towards Multi-dimensional Evaluation of {LLM} Summarization across Domains and Languages",
    author = "Min, Hyangsuk  and
      Lee, Yuho  and
      Ban, Minjeong  and
      Deng, Jiaqi  and
      Kim, Nicole Hee-Yeon  and
      Yun, Taewon  and
      Su, Hang  and
      Cai, Jason  and
      Song, Hwanjun",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.702/",
    pages = "14417--14450",
    ISBN = "979-8-89176-251-0",
    abstract = "Evaluation frameworks for text summarization have evolved in terms of both domain coverage and metrics. However, existing benchmarks still lack domain-specific assessment criteria, remain predominantly English-centric, and face challenges with human annotation due to the complexity of reasoning. To address these, we introduce MSumBench, which provides a multi-dimensional, multi-domain evaluation of summarization in English and Chinese. It also incorporates specialized assessment criteria for each domain and leverages a multi-agent debate system to enhance annotation quality. By evaluating eight modern summarization models, we discover distinct performance patterns across domains and languages. We further examine large language models as summary evaluators, analyzing the correlation between their evaluation and summarization capabilities, and uncovering systematic bias in their assessment of self-generated summaries. Our benchmark dataset is publicly available at https://github.com/DISL-Lab/MSumBench."
}