Abstract
Detecting out-of-distribution (OOD) instances is significant for the safe deployment of NLP models. Among recent textual OOD detection works based on pretrained language models (PLMs), distance-based methods have shown superior performance. However, they estimate sample distance scores in the last-layer CLS embedding space and thus do not make full use of linguistic information underlying in PLMs. To address the issue, we propose to boost OOD detection by deriving more holistic sentence embeddings. On the basis of the observations that token averaging and layer combination contribute to improving OOD detection, we propose a simple embedding approach named Avg-Avg, which averages all token representations from each intermediate layer as the sentence embedding and significantly surpasses the state-of-the-art on a comprehensive suite of benchmarks by a 9.33% FAR95 margin. Furthermore, our analysis demonstrates that it indeed helps preserve general linguistic knowledge in fine-tuned PLMs and substantially benefits detecting background shifts. The simple yet effective embedding method can be applied to fine-tuned PLMs with negligible extra costs, providing a free gain in OOD detection. Our code is available at https://github.com/lancopku/Avg-Avg.- Anthology ID:
- 2022.findings-emnlp.497
- Volume:
- Findings of the Association for Computational Linguistics: EMNLP 2022
- Month:
- December
- Year:
- 2022
- Address:
- Abu Dhabi, United Arab Emirates
- Editors:
- Yoav Goldberg, Zornitsa Kozareva, Yue Zhang
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 6676–6686
- Language:
- URL:
- https://aclanthology.org/2022.findings-emnlp.497
- DOI:
- 10.18653/v1/2022.findings-emnlp.497
- Cite (ACL):
- Sishuo Chen, Xiaohan Bi, Rundong Gao, and Xu Sun. 2022. Holistic Sentence Embeddings for Better Out-of-Distribution Detection. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6676–6686, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
- Cite (Informal):
- Holistic Sentence Embeddings for Better Out-of-Distribution Detection (Chen et al., Findings 2022)
- PDF:
- https://preview.aclanthology.org/ingest-acl-2023-videos/2022.findings-emnlp.497.pdf