Abstract
“Learning sentence representation is a fundamental task in natural language processing and has been studied extensively. Recently, many works have obtained high-quality sentence representation based on contrastive learning from pre-trained models. However, these works suffer the inconsistency of input forms between the pre-training and fine-tuning stages. Also, they typically encode a sentence independently and lack feature interaction between sentences. To conquer these issues, we propose a novel Contrastive framework with Inter-sentence Interaction (ConIsI), which introduces a sentence-level objective to improve sentence representation based on contrastive learning by fine-grained interaction between sentences. The sentence-level objective guides the model to focus on fine-grained semantic information by feature interaction between sentences, and we design three different sentence construction strategies to explore its effect. We conduct experiments on seven Semantic Textual Similarity (STS) tasks. The experimental results show that our ConIsI models based on BERTbase and RoBERTabase achieve state-ofthe-art performance, substantially outperforming previous best models SimCSE-BERTbase and SimCSE-RoBERTabase by 2.05% and 0.77% respectively.”- Anthology ID:
- 2022.ccl-1.67
- Volume:
- Proceedings of the 21st Chinese National Conference on Computational Linguistics
- Month:
- October
- Year:
- 2022
- Address:
- Nanchang, China
- Editors:
- Maosong Sun (孙茂松), Yang Liu (刘洋), Wanxiang Che (车万翔), Yang Feng (冯洋), Xipeng Qiu (邱锡鹏), Gaoqi Rao (饶高琦), Yubo Chen (陈玉博)
- Venue:
- CCL
- SIG:
- Publisher:
- Chinese Information Processing Society of China
- Note:
- Pages:
- 748–760
- Language:
- English
- URL:
- https://aclanthology.org/2022.ccl-1.67
- DOI:
- Cite (ACL):
- Sun Meng and Huang Degen. 2022. ConIsI: A Contrastive Framework with Inter-sentence Interaction for Self-supervised Sentence Representation. In Proceedings of the 21st Chinese National Conference on Computational Linguistics, pages 748–760, Nanchang, China. Chinese Information Processing Society of China.
- Cite (Informal):
- ConIsI: A Contrastive Framework with Inter-sentence Interaction for Self-supervised Sentence Representation (Meng & Degen, CCL 2022)
- PDF:
- https://preview.aclanthology.org/ml4al-ingestion/2022.ccl-1.67.pdf
- Data
- SICK