Abstract
Neural dialogue models, despite their successes, still suffer from lack of relevance, diversity, and in many cases coherence in their generated responses. On the other hand, transformer-based models such as GPT-2 have demonstrated an excellent ability to capture long-range structures in language modeling tasks. In this paper, we present DLGNet, a transformer-based model for dialogue modeling. We specifically examine the use of DLGNet for multi-turn dialogue response generation. In our experiments, we evaluate DLGNet on the open-domain Movie Triples dataset and the closed-domain Ubuntu Dialogue dataset. DLGNet models, although trained with only the maximum likelihood objective, achieve significant improvements over state-of-the-art multi-turn dialogue models. They also produce best performance to date on the two datasets based on several metrics, including BLEU, ROUGE, and distinct n-gram. Our analysis shows that the performance improvement is mostly due to the combination of (1) the long-range transformer architecture with (2) the injection of random informative paddings. Other contributing factors include the joint modeling of dialogue context and response, and the 100% tokenization coverage from the byte pair encoding (BPE).- Anthology ID:
- 2020.nlp4convai-1.7
- Volume:
- Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI
- Month:
- July
- Year:
- 2020
- Address:
- Online
- Editors:
- Tsung-Hsien Wen, Asli Celikyilmaz, Zhou Yu, Alexandros Papangelis, Mihail Eric, Anuj Kumar, Iñigo Casanueva, Rushin Shah
- Venue:
- NLP4ConvAI
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 54–62
- Language:
- URL:
- https://aclanthology.org/2020.nlp4convai-1.7
- DOI:
- 10.18653/v1/2020.nlp4convai-1.7
- Cite (ACL):
- Olabiyi Oluwatobi and Erik Mueller. 2020. DLGNet: A Transformer-based Model for Dialogue Response Generation. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pages 54–62, Online. Association for Computational Linguistics.
- Cite (Informal):
- DLGNet: A Transformer-based Model for Dialogue Response Generation (Oluwatobi & Mueller, NLP4ConvAI 2020)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/2020.nlp4convai-1.7.pdf