@inproceedings{jeong-etal-2024-simple,
    title = "Simple Temperature Cool-down in Contrastive Framework for Unsupervised Sentence Representation Learning",
    author = "Jeong, Yoo Hyun  and
      Han, Myeong Soo  and
      Chae, Dong-Kyu",
    editor = "Graham, Yvette  and
      Purver, Matthew",
    booktitle = "Findings of the Association for Computational Linguistics: EACL 2024",
    month = mar,
    year = "2024",
    address = "St. Julian{'}s, Malta",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2024.findings-eacl.37/",
    pages = "550--559",
    abstract = "In this paper, we proposes a simple, tricky method to improve sentence representation of unsupervised contrastive learning. Even though contrastive learning has achieved great performances in both visual representation learning (VRL) and sentence representation learning (SRL) fields, we focus on the fact that there is a gap between characteristics and training dynamics of VRL and SRL. We first examine the role of temperature to bridge the gap between VRL and SRL, and find some temperature-dependent elements in SRL; \textit{i.e.}, a higher temperature causes overfitting of the uniformity while improving the alignment in earlier phase of training. Then, we design a \textit{temperature cool-down} technique based on this observation, which helps PLMs to be more suitable for contrastive learning via preparation of uniform representation space. Our experimental results on widely-utilized benchmarks demonstrate the effectiveness and extensiblity of our method."
}