A Stable Variational Autoencoder for Text Modelling
Ruizhe Li, Xiao Li, Chenghua Lin, Matthew Collinson, Rui Mao
Abstract
Variational Autoencoder (VAE) is a powerful method for learning representations of high-dimensional data. However, VAEs can suffer from an issue known as latent variable collapse (or KL term vanishing), where the posterior collapses to the prior and the model will ignore the latent codes in generative tasks. Such an issue is particularly prevalent when employing VAE-RNN architectures for text modelling (Bowman et al., 2016; Yang et al., 2017). In this paper, we present a new architecture called Full-Sampling-VAE-RNN, which can effectively avoid latent variable collapse. Compared to the general VAE-RNN architectures, we show that our model can achieve much more stable training process and can generate text with significantly better quality.- Anthology ID:
- W19-8673
- Volume:
- Proceedings of the 12th International Conference on Natural Language Generation
- Month:
- October–November
- Year:
- 2019
- Address:
- Tokyo, Japan
- Editors:
- Kees van Deemter, Chenghua Lin, Hiroya Takamura
- Venue:
- INLG
- SIG:
- SIGGEN
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 594–599
- Language:
- URL:
- https://aclanthology.org/W19-8673
- DOI:
- 10.18653/v1/W19-8673
- Cite (ACL):
- Ruizhe Li, Xiao Li, Chenghua Lin, Matthew Collinson, and Rui Mao. 2019. A Stable Variational Autoencoder for Text Modelling. In Proceedings of the 12th International Conference on Natural Language Generation, pages 594–599, Tokyo, Japan. Association for Computational Linguistics.
- Cite (Informal):
- A Stable Variational Autoencoder for Text Modelling (Li et al., INLG 2019)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-3/W19-8673.pdf
- Code
- ruizheliUOA/HR-VAE
- Data
- Penn Treebank