Abstract
Recent advances in NLP have been attributed to the emergence of large-scale pre-trained language models. GPT-2, in particular, is suited for generation tasks given its left-to-right language modeling objective, yet the linguistic quality of its generated text has largely remain unexplored. Our work takes a step in understanding GPT-2’s outputs in terms of discourse coherence. We perform a comprehensive study on the validity of explicit discourse relations in GPT-2’s outputs under both organic generation and fine-tuned scenarios. Results show GPT-2 does not always generate text containing valid discourse relations; nevertheless, its text is more aligned with human expectation in the fine-tuned scenario. We propose a decoupled strategy to mitigate these problems and highlight the importance of explicitly modeling discourse information.- Anthology ID:
- 2020.inlg-1.8
- Volume:
- Proceedings of the 13th International Conference on Natural Language Generation
- Month:
- December
- Year:
- 2020
- Address:
- Dublin, Ireland
- Editors:
- Brian Davis, Yvette Graham, John Kelleher, Yaji Sripada
- Venue:
- INLG
- SIG:
- SIGGEN
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 52–59
- Language:
- URL:
- https://aclanthology.org/2020.inlg-1.8
- DOI:
- 10.18653/v1/2020.inlg-1.8
- Cite (ACL):
- Wei-Jen Ko and Junyi Jessy Li. 2020. Assessing Discourse Relations in Language Generation from GPT-2. In Proceedings of the 13th International Conference on Natural Language Generation, pages 52–59, Dublin, Ireland. Association for Computational Linguistics.
- Cite (Informal):
- Assessing Discourse Relations in Language Generation from GPT-2 (Ko & Li, INLG 2020)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/2020.inlg-1.8.pdf