A Data Set for the Analysis of Text Quality Dimensions in Summarization Evaluation

Margot Mieskes, Eneldo Loza Mencía, Tim Kronsbein

Abstract
Automatic evaluation of summarization focuses on developing a metric to represent the quality of the resulting text. However, text qualityis represented in a variety of dimensions ranging from grammaticality to readability and coherence. In our work, we analyze the depen-dencies between a variety of quality dimensions on automatically created multi-document summaries and which dimensions automaticevaluation metrics such as ROUGE, PEAK or JSD are able to capture. Our results indicate that variants of ROUGE are correlated tovarious quality dimensions and that some automatic summarization methods achieve higher quality summaries than others with respectto individual summary quality dimensions. Our results also indicate that differentiating between quality dimensions facilitates inspectionand fine-grained comparison of summarization methods and its characteristics. We make the data from our two summarization qualityevaluation experiments publicly available in order to facilitate the future development of specialized automatic evaluation methods.
Anthology ID:
2020.lrec-1.826
Volume:
Proceedings of the Twelfth Language Resources and Evaluation Conference
Month:
May
Year:
2020
Address:
Marseille, France
Editors:
Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, Stelios Piperidis
Venue:
LREC
SIG:
Publisher:
European Language Resources Association
Note:
Pages:
6690–6699
Language:
English
URL:
https://aclanthology.org/2020.lrec-1.826
DOI:
Bibkey:
Cite (ACL):
Margot Mieskes, Eneldo Loza Mencía, and Tim Kronsbein. 2020. A Data Set for the Analysis of Text Quality Dimensions in Summarization Evaluation. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 6690–6699, Marseille, France. European Language Resources Association.
Cite (Informal):
A Data Set for the Analysis of Text Quality Dimensions in Summarization Evaluation (Mieskes et al., LREC 2020)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-2/2020.lrec-1.826.pdf
PDF Search