Abstract
In this paper, we introduce an important yet relatively unexplored NLP task called Semantic Overlap Summarization (SOS), which entails generating a single summary from multiple alternative narratives which can convey the common information provided by those narratives. As no benchmark dataset is readily available for this task, we created one by collecting 2,925 alternative narrative pairs from the web and then, went through the tedious process of manually creating 411 different reference summaries by engaging human annotators. As a way to evaluate this novel task, we first conducted a systematic study by borrowing the popular ROUGE metric from text-summarization literature and discovered that ROUGE is not suitable for our task. Subsequently, we conducted further human annotations to create 200 document-level and 1,518 sentence-level ground-truth overlap labels. Our experiments show that the sentence-wise annotation technique with three overlap labels, i.e., Absent (A), Partially-Present (PP), and Present (P), yields a higher correlation with human judgment and higher inter-rater agreement compared to the ROUGE metric.- Anthology ID:
- 2022.coling-1.541
- Volume:
- Proceedings of the 29th International Conference on Computational Linguistics
- Month:
- October
- Year:
- 2022
- Address:
- Gyeongju, Republic of Korea
- Editors:
- Nicoletta Calzolari, Chu-Ren Huang, Hansaem Kim, James Pustejovsky, Leo Wanner, Key-Sun Choi, Pum-Mo Ryu, Hsin-Hsi Chen, Lucia Donatelli, Heng Ji, Sadao Kurohashi, Patrizia Paggio, Nianwen Xue, Seokhwan Kim, Younggyun Hahm, Zhong He, Tony Kyungil Lee, Enrico Santus, Francis Bond, Seung-Hoon Na
- Venue:
- COLING
- SIG:
- Publisher:
- International Committee on Computational Linguistics
- Note:
- Pages:
- 6195–6207
- Language:
- URL:
- https://aclanthology.org/2022.coling-1.541
- DOI:
- Cite (ACL):
- Naman Bansal, Mousumi Akter, and Shubhra Kanti Karmaker Santu. 2022. Semantic Overlap Summarization among Multiple Alternative Narratives: An Exploratory Study. In Proceedings of the 29th International Conference on Computational Linguistics, pages 6195–6207, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
- Cite (Informal):
- Semantic Overlap Summarization among Multiple Alternative Narratives: An Exploratory Study (Bansal et al., COLING 2022)
- PDF:
- https://preview.aclanthology.org/corrections-2024-05/2022.coling-1.541.pdf