Abstract
Fine-tuning a pre-trained language model via the contrastive learning framework with a large amount of unlabeled sentences or labeled sentence pairs is a common way to obtain high-quality sentence representations. Although the contrastive learning framework has shown its superiority on sentence representation learning over previous methods, the potential of such a framework is under-explored so far due to the simple method it used to construct positive pairs. Motivated by this, we propose a method that makes hard positives from the original training examples. A pivotal ingredient of our approach is the use of prefix that attached to a pre-trained language model, which allows for differentiable data augmentation during contrastive learning. Our method can be summarized in two steps: supervised prefix-tuning followed by joint contrastive fine-tuning with unlabeled or labeled examples. Our experiments confirm the effectiveness of our data augmentation approach. The proposed method yields significant improvements over existing methods under both semi-supervised and supervised settings. Our experiments under a low labeled data setting also show that our method is more label-efficient than the state-of-the-art contrastive learning methods.- Anthology ID:
- 2022.emnlp-main.520
- Volume:
- Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
- Month:
- December
- Year:
- 2022
- Address:
- Abu Dhabi, United Arab Emirates
- Editors:
- Yoav Goldberg, Zornitsa Kozareva, Yue Zhang
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 7640–7653
- Language:
- URL:
- https://aclanthology.org/2022.emnlp-main.520
- DOI:
- 10.18653/v1/2022.emnlp-main.520
- Cite (ACL):
- Tianduo Wang and Wei Lu. 2022. Differentiable Data Augmentation for Contrastive Sentence Representation Learning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7640–7653, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
- Cite (Informal):
- Differentiable Data Augmentation for Contrastive Sentence Representation Learning (Wang & Lu, EMNLP 2022)
- PDF:
- https://preview.aclanthology.org/ingest-acl-2023-videos/2022.emnlp-main.520.pdf