On the Effect of Isotropy on VAE Representations of Text

Lan Zhang, Wray Buntine, Ehsan Shareghi


Abstract
Injecting desired geometric properties into text representations has attracted a lot of attention. A property that has been argued for, due to its better utilisation of representation space, is isotropy. In parallel, VAEs have been successful in areas of NLP, but are known for their sub-optimal utilisation of the representation space. To address an aspect of this, we investigate the impact of injecting isotropy during training of VAEs. We achieve this by using an isotropic Gaussian posterior (IGP) instead of the ellipsoidal Gaussian posterior. We illustrate that IGP effectively encourages isotropy in the representations, inducing a more discriminative latent space. Compared to vanilla VAE, this translates into a much better classification performance, robustness to input perturbation, and generative behavior. Additionally, we offer insights about the representational properties encouraged by IGP.
Anthology ID:
2022.acl-short.78
Volume:
Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
Month:
May
Year:
2022
Address:
Dublin, Ireland
Editors:
Smaranda Muresan, Preslav Nakov, Aline Villavicencio
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
694–701
Language:
URL:
https://aclanthology.org/2022.acl-short.78
DOI:
10.18653/v1/2022.acl-short.78
Bibkey:
Cite (ACL):
Lan Zhang, Wray Buntine, and Ehsan Shareghi. 2022. On the Effect of Isotropy on VAE Representations of Text. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 694–701, Dublin, Ireland. Association for Computational Linguistics.
Cite (Informal):
On the Effect of Isotropy on VAE Representations of Text (Zhang et al., ACL 2022)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingest-acl-2023-videos/2022.acl-short.78.pdf
Software:
 2022.acl-short.78.software.zip
Video:
 https://preview.aclanthology.org/ingest-acl-2023-videos/2022.acl-short.78.mp4
Code
 lanzhang128/IGPVAE