@inproceedings{ridenour-etal-2022-assessing,
    title = "Assessing Inter-metric Correlation for Multi-document Summarization Evaluation",
    author = "Ridenour, Michael  and
      Agrawal, Ameeta  and
      Olabisi, Olubusayo",
    editor = "Bosselut, Antoine  and
      Chandu, Khyathi  and
      Dhole, Kaustubh  and
      Gangal, Varun  and
      Gehrmann, Sebastian  and
      Jernite, Yacine  and
      Novikova, Jekaterina  and
      Perez-Beltrachini, Laura",
    booktitle = "Proceedings of the 2nd Workshop on Natural Language Generation, Evaluation, and Metrics (GEM)",
    month = dec,
    year = "2022",
    address = "Abu Dhabi, United Arab Emirates (Hybrid)",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/add-emnlp-2024-awards/2022.gem-1.40/",
    doi = "10.18653/v1/2022.gem-1.40",
    pages = "428--438",
    abstract = "Recent advances in automatic text summarization have contemporaneously been accompanied by a great deal of new metrics of automatic evaluation. This in turn has inspired recent research to re-assess these evaluation metrics to see how well they correlate with each other as well as with human evaluation, mostly focusing on single-document summarization (SDS) tasks. Although many of these metrics are typically also used for evaluating multi-document summarization (MDS) tasks, so far, little attention has been paid to studying them under such a distinct scenario. To address this gap, we present a systematic analysis of the inter-metric correlations for MDS tasks, while comparing and contrasting the results with SDS models. Using datasets from a wide range of domains (news, peer reviews, tweets, dialogues), we thus study a unified set of metrics under both the task setups. Our empirical analysis suggests that while most reference-based metrics show fairly similar trends across both multi- and single-document summarization, there is a notable lack of correlation between reference-free metrics in multi-document summarization tasks."
}