Abstract
This paper aims to investigate the possibility of exploiting original semantic features of PLMs (pre-trained language models) during contrastive learning in the context of SRL (sentence representation learning). In the context of feature modification, we identified a method called IFM (implicit feature modification), which reduces the tendency of contrastive models for VRL (visual representation learning) to rely on feature-suppressing shortcut solutions. We observed that IFM did not work well for SRL, which may be due to differences between the nature of VRL and SRL. We propose BYOP, which boosts well-represented features, taking the opposite idea of IFM, under the assumption that SimCSE’s dropout-noise-based augmentation may be too simple to modify high-level semantic features, and that the features learned by PLMs are semantically meaningful and should be boosted, rather than removed. Extensive experiments lend credence to the logic of BYOP, which considers the nature of SRL.- Anthology ID:
- 2024.findings-eacl.38
- Volume:
- Findings of the Association for Computational Linguistics: EACL 2024
- Month:
- March
- Year:
- 2024
- Address:
- St. Julian’s, Malta
- Editors:
- Yvette Graham, Matthew Purver
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 560–569
- Language:
- URL:
- https://aclanthology.org/2024.findings-eacl.38
- DOI:
- Cite (ACL):
- Yoo Hyun Jeong, Myeong Soo Han, and Dong-Kyu Chae. 2024. Bootstrap Your Own PLM: Boosting Semantic Features of PLMs for Unsuperivsed Contrastive Learning. In Findings of the Association for Computational Linguistics: EACL 2024, pages 560–569, St. Julian’s, Malta. Association for Computational Linguistics.
- Cite (Informal):
- Bootstrap Your Own PLM: Boosting Semantic Features of PLMs for Unsuperivsed Contrastive Learning (Jeong et al., Findings 2024)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-1/2024.findings-eacl.38.pdf