DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization
Yu Li, Baolin Peng, Pengcheng He, Michel Galley, Zhou Yu, Jianfeng Gao
Abstract
Dialogue summarization has recently garnered significant attention due to its wide range of applications. However, existing methods for summarizing dialogues have limitations because they do not take into account the inherent structure of dialogue and rely heavily on labeled data, which can lead to poor performance in new domains. In this work, we propose DIONYSUS (dynamic input optimization in pre-training for dialogue summarization), a pre-trained encoder-decoder model for summarizing dialogues in any new domain. To pre-train DIONYSUS, we create two pseudo summaries for each dialogue example: one from a fine-tuned summarization model and the other from important dialogue turns. We then choose one of these pseudo summaries based on information distribution differences in different types of dialogues. This selected pseudo summary serves as the objective for pre-training DIONYSUS using a self-supervised approach on a large dialogue corpus. Our experiments show that DIONYSUS outperforms existing methods on six datasets, as demonstrated by its ROUGE scores in zero-shot and few-shot settings- Anthology ID:
- 2023.acl-long.76
- Volume:
- Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Month:
- July
- Year:
- 2023
- Address:
- Toronto, Canada
- Venue:
- ACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 1368–1386
- Language:
- URL:
- https://aclanthology.org/2023.acl-long.76
- DOI:
- 10.18653/v1/2023.acl-long.76
- Cite (ACL):
- Yu Li, Baolin Peng, Pengcheng He, Michel Galley, Zhou Yu, and Jianfeng Gao. 2023. DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1368–1386, Toronto, Canada. Association for Computational Linguistics.
- Cite (Informal):
- DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization (Li et al., ACL 2023)
- PDF:
- https://preview.aclanthology.org/remove-xml-comments/2023.acl-long.76.pdf