Abstract
The recent success of SimCSE has greatly advanced state-of-the-art sentence representations. However, the original formulation of SimCSE does not fully exploit the potential of hard negative samples in contrastive learning. This study introduces an unsupervised contrastive learning framework that combines SimCSE with hard negative mining, aiming to enhance the quality of sentence embeddings. The proposed focal-InfoNCE function introduces self-paced modulation terms in the contrastive objective, downweighting the loss associated with easy negatives and encouraging the model focusing on hard negatives. Experimentation on various STS benchmarks shows that our method improves sentence embeddings in terms of Spearman’s correlation and representation alignment and uniformity.- Anthology ID:
- 2023.findings-emnlp.315
- Volume:
- Findings of the Association for Computational Linguistics: EMNLP 2023
- Month:
- December
- Year:
- 2023
- Address:
- Singapore
- Editors:
- Houda Bouamor, Juan Pino, Kalika Bali
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 4757–4762
- Language:
- URL:
- https://aclanthology.org/2023.findings-emnlp.315
- DOI:
- 10.18653/v1/2023.findings-emnlp.315
- Cite (ACL):
- Pengyue Hou and Xingyu Li. 2023. Improving Contrastive Learning of Sentence Embeddings with Focal InfoNCE. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 4757–4762, Singapore. Association for Computational Linguistics.
- Cite (Informal):
- Improving Contrastive Learning of Sentence Embeddings with Focal InfoNCE (Hou & Li, Findings 2023)
- PDF:
- https://preview.aclanthology.org/emnlp-22-attachments/2023.findings-emnlp.315.pdf