Attention Optimization for Abstractive Document Summarization

Min Gui, Junfeng Tian, Rui Wang, Zhenglu Yang


Abstract
Attention plays a key role in the improvement of sequence-to-sequence-based document summarization models. To obtain a powerful attention helping with reproducing the most salient information and avoiding repetitions, we augment the vanilla attention model from both local and global aspects. We propose attention refinement unit paired with local variance loss to impose supervision on the attention model at each decoding step, and we also propose a global variance loss to optimize the attention distributions of all decoding steps from the global perspective. The performances on CNN/Daily Mail dataset verify the effectiveness of our methods.
Anthology ID:
D19-1117
Volume:
Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
Month:
November
Year:
2019
Address:
Hong Kong, China
Editors:
Kentaro Inui, Jing Jiang, Vincent Ng, Xiaojun Wan
Venues:
EMNLP | IJCNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
1222–1228
Language:
URL:
https://aclanthology.org/D19-1117
DOI:
10.18653/v1/D19-1117
Bibkey:
Cite (ACL):
Min Gui, Junfeng Tian, Rui Wang, and Zhenglu Yang. 2019. Attention Optimization for Abstractive Document Summarization. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1222–1228, Hong Kong, China. Association for Computational Linguistics.
Cite (Informal):
Attention Optimization for Abstractive Document Summarization (Gui et al., EMNLP-IJCNLP 2019)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingest-bitext-workshop/D19-1117.pdf