Abstract
Injecting desired geometric properties into text representations has attracted a lot of attention. A property that has been argued for, due to its better utilisation of representation space, is isotropy. In parallel, VAEs have been successful in areas of NLP, but are known for their sub-optimal utilisation of the representation space. To address an aspect of this, we investigate the impact of injecting isotropy during training of VAEs. We achieve this by using an isotropic Gaussian posterior (IGP) instead of the ellipsoidal Gaussian posterior. We illustrate that IGP effectively encourages isotropy in the representations, inducing a more discriminative latent space. Compared to vanilla VAE, this translates into a much better classification performance, robustness to input perturbation, and generative behavior. Additionally, we offer insights about the representational properties encouraged by IGP.- Anthology ID:
- 2022.acl-short.78
- Volume:
- Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
- Month:
- May
- Year:
- 2022
- Address:
- Dublin, Ireland
- Editors:
- Smaranda Muresan, Preslav Nakov, Aline Villavicencio
- Venue:
- ACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 694–701
- Language:
- URL:
- https://aclanthology.org/2022.acl-short.78
- DOI:
- 10.18653/v1/2022.acl-short.78
- Cite (ACL):
- Lan Zhang, Wray Buntine, and Ehsan Shareghi. 2022. On the Effect of Isotropy on VAE Representations of Text. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 694–701, Dublin, Ireland. Association for Computational Linguistics.
- Cite (Informal):
- On the Effect of Isotropy on VAE Representations of Text (Zhang et al., ACL 2022)
- PDF:
- https://preview.aclanthology.org/ingest-acl-2023-videos/2022.acl-short.78.pdf
- Code
- lanzhang128/IGPVAE