Earlier Isn’t Always Better: Sub-aspect Analysis on Corpus and System Biases in Summarization

Taehee Jung, Dongyeop Kang, Lucas Mentch, Eduard Hovy

Abstract
Despite the recent developments on neural summarization systems, the underlying logic behind the improvements from the systems and its corpus-dependency remains largely unexplored. Position of sentences in the original text, for example, is a well known bias for news summarization. Following in the spirit of the claim that summarization is a combination of sub-functions, we define three sub-aspects of summarization: position, importance, and diversity and conduct an extensive analysis of the biases of each sub-aspect with respect to the domain of nine different summarization corpora (e.g., news, academic papers, meeting minutes, movie script, books, posts). We find that while position exhibits substantial bias in news articles, this is not the case, for example, with academic papers and meeting minutes. Furthermore, our empirical study shows that different types of summarization systems (e.g., neural-based) are composed of different degrees of the sub-aspects. Our study provides useful lessons regarding consideration of underlying sub-aspects when collecting a new summarization dataset or developing a new system.
Anthology ID:
D19-1327
Volume:
Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
Month:
November
Year:
2019
Address:
Hong Kong, China
Editors:
Kentaro Inui, Jing Jiang, Vincent Ng, Xiaojun Wan
Venues:
EMNLP | IJCNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
3324–3335
Language:
URL:
https://aclanthology.org/D19-1327
DOI:
10.18653/v1/D19-1327
Bibkey:
Cite (ACL):
Taehee Jung, Dongyeop Kang, Lucas Mentch, and Eduard Hovy. 2019. Earlier Isn’t Always Better: Sub-aspect Analysis on Corpus and System Biases in Summarization. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3324–3335, Hong Kong, China. Association for Computational Linguistics.
Cite (Informal):
Earlier Isn’t Always Better: Sub-aspect Analysis on Corpus and System Biases in Summarization (Jung et al., EMNLP-IJCNLP 2019)
Copy Citation:
PDF:
https://preview.aclanthology.org/dois-2013-emnlp/D19-1327.pdf
Attachment:
 D19-1327.Attachment.pdf
Data
NEWSROOM, PeerRead
PDF Search Attachment