Abstract
Sentence embedding is a cornerstone in NLP. Whitening has been claimed to be an effective method to improve embeddings obtained from Large Language Models (LLMs) for sentence embedding. However, we find that the effectiveness of whitening is model-dependent and task-dependent. In particular, whitening degenerates embeddings for classification tasks. The conclusion is supported by extensive experiments. A by-product of our research is embedding evaluation platform for LLMs called SentEval+- Anthology ID:
- 2024.repl4nlp-1.21
- Volume:
- Proceedings of the 9th Workshop on Representation Learning for NLP (RepL4NLP-2024)
- Month:
- August
- Year:
- 2024
- Address:
- Bangkok, Thailand
- Editors:
- Chen Zhao, Marius Mosbach, Pepa Atanasova, Seraphina Goldfarb-Tarrent, Peter Hase, Arian Hosseini, Maha Elbayad, Sandro Pezzelle, Maximilian Mozes
- Venues:
- RepL4NLP | WS
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 285–289
- Language:
- URL:
- https://aclanthology.org/2024.repl4nlp-1.21
- DOI:
- Cite (ACL):
- Ali Forooghi, Shaghayegh Sadeghi, and Jianguo Lu. 2024. Whitening Not Recommended for Classification Tasks in LLMs. In Proceedings of the 9th Workshop on Representation Learning for NLP (RepL4NLP-2024), pages 285–289, Bangkok, Thailand. Association for Computational Linguistics.
- Cite (Informal):
- Whitening Not Recommended for Classification Tasks in LLMs (Forooghi et al., RepL4NLP-WS 2024)
- PDF:
- https://preview.aclanthology.org/dois-2013-emnlp/2024.repl4nlp-1.21.pdf