Mitigating Shortcuts in Language Models with Soft Label Encoding

Zirui He, Huiqi Deng, Haiyan Zhao, Ninghao Liu, Mengnan Du

Abstract
Recent research has shown that large language models rely on spurious correlations in the data for natural language understanding (NLU) tasks. In this work, we aim to answer the following research question: Can we reduce spurious correlations by modifying the ground truth labels of the training data? Specifically, we propose a simple yet effective debiasing framework, named Soft Label Encoding (SoftLE). First, we train a teacher model to quantify each sample’s degree of relying on shortcuts. Then, we encode this shortcut degree into a dummy class and use it to smooth the original ground truth labels, generating soft labels. These soft labels are used to train a more robust student model that reduces spurious correlations between shortcut features and certain classes. Extensive experiments on two NLU benchmark tasks via two language models demonstrate that SoftLE significantly improves out-of-distribution generalization while maintaining satisfactory in-distribution accuracy. Our code is available at https://github.com/ZiruiHE99/sle
Anthology ID:
2024.lrec-main.991
Volume:
Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)
Month:
May
Year:
2024
Address:
Torino, Italia
Editors:
Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, Nianwen Xue
Venues:
LREC | COLING
SIG:
Publisher:
ELRA and ICCL
Note:
Pages:
11341–11348
Language:
URL:
https://aclanthology.org/2024.lrec-main.991
DOI:
Bibkey:
Cite (ACL):
Zirui He, Huiqi Deng, Haiyan Zhao, Ninghao Liu, and Mengnan Du. 2024. Mitigating Shortcuts in Language Models with Soft Label Encoding. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 11341–11348, Torino, Italia. ELRA and ICCL.
Cite (Informal):
Mitigating Shortcuts in Language Models with Soft Label Encoding (He et al., LREC-COLING 2024)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-4/2024.lrec-main.991.pdf
PDF Search