Contrastive Learning with Prompt-derived Virtual Semantic Prototypes for Unsupervised Sentence Embedding
Jiali Zeng, Yongjing Yin, Yufan Jiang, Shuangzhi Wu, Yunbo Cao
Abstract
Contrastive learning has become a new paradigm for unsupervised sentence embeddings.Previous studies focus on instance-wise contrastive learning, attempting to construct positive pairs with textual data augmentation. In this paper, we propose a novel Contrastive learning method with Prompt-derived Virtual semantic Prototypes (ConPVP). Specifically, with the help of prompts, we construct virtual semantic prototypes to each instance, and derive negative prototypes by using the negative form of the prompts.Using a prototypical contrastive loss, we enforce the anchor sentence embedding to be close to its corresponding semantic prototypes, and far apart from the negative prototypes as well as the prototypes of other sentences.Extensive experimental results on semantic textual similarity, transfer, and clustering tasks demonstrate the effectiveness of our proposed model compared to strong baselines.Code is available at https://github.com/lemon0830/promptCSE.- Anthology ID:
- 2022.findings-emnlp.522
- Volume:
- Findings of the Association for Computational Linguistics: EMNLP 2022
- Month:
- December
- Year:
- 2022
- Address:
- Abu Dhabi, United Arab Emirates
- Editors:
- Yoav Goldberg, Zornitsa Kozareva, Yue Zhang
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 7042–7053
- Language:
- URL:
- https://aclanthology.org/2022.findings-emnlp.522
- DOI:
- 10.18653/v1/2022.findings-emnlp.522
- Cite (ACL):
- Jiali Zeng, Yongjing Yin, Yufan Jiang, Shuangzhi Wu, and Yunbo Cao. 2022. Contrastive Learning with Prompt-derived Virtual Semantic Prototypes for Unsupervised Sentence Embedding. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 7042–7053, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
- Cite (Informal):
- Contrastive Learning with Prompt-derived Virtual Semantic Prototypes for Unsupervised Sentence Embedding (Zeng et al., Findings 2022)
- PDF:
- https://preview.aclanthology.org/naacl24-info/2022.findings-emnlp.522.pdf