Abstract
Recent studies on neural networks with pre-trained weights (i.e., BERT) have mainly focused on a low-dimensional subspace, where the embedding vectors computed from input words (or their contexts) are located. In this work, we propose a new approach, called OoMMix, to finding and regularizing the remainder of the space, referred to as out-of-manifold, which cannot be accessed through the words. Specifically, we synthesize the out-of-manifold embeddings based on two embeddings obtained from actually-observed words, to utilize them for fine-tuning the network. A discriminator is trained to detect whether an input embedding is located inside the manifold or not, and simultaneously, a generator is optimized to produce new embeddings that can be easily identified as out-of-manifold by the discriminator. These two modules successfully collaborate in a unified and end-to-end manner for regularizing the out-of-manifold. Our extensive evaluation on various text classification benchmarks demonstrates the effectiveness of our approach, as well as its good compatibility with existing data augmentation techniques which aim to enhance the manifold.- Anthology ID:
- 2021.acl-long.49
- Volume:
- Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)
- Month:
- August
- Year:
- 2021
- Address:
- Online
- Editors:
- Chengqing Zong, Fei Xia, Wenjie Li, Roberto Navigli
- Venues:
- ACL | IJCNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 590–599
- Language:
- URL:
- https://aclanthology.org/2021.acl-long.49
- DOI:
- 10.18653/v1/2021.acl-long.49
- Cite (ACL):
- Seonghyeon Lee, Dongha Lee, and Hwanjo Yu. 2021. OoMMix: Out-of-manifold Regularization in Contextual Embedding Space for Text Classification. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 590–599, Online. Association for Computational Linguistics.
- Cite (Informal):
- OoMMix: Out-of-manifold Regularization in Contextual Embedding Space for Text Classification (Lee et al., ACL-IJCNLP 2021)
- PDF:
- https://preview.aclanthology.org/naacl24-info/2021.acl-long.49.pdf
- Data
- AG News