Abstract
The biases present in training datasets have been shown to affect models for sentence pair classification tasks such as natural language inference (NLI) and fact verification. While fine-tuning models on additional data has been used to mitigate them, a common issue is that of catastrophic forgetting of the original training dataset. In this paper, we show that elastic weight consolidation (EWC) allows fine-tuning of models to mitigate biases while being less susceptible to catastrophic forgetting. In our evaluation on fact verification and NLI stress tests, we show that fine-tuning with EWC dominates standard fine-tuning, yielding models with lower levels of forgetting on the original (biased) dataset for equivalent gains in accuracy on the fine-tuning (unbiased) dataset.- Anthology ID:
- 2021.eacl-main.82
- Volume:
- Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume
- Month:
- April
- Year:
- 2021
- Address:
- Online
- Venue:
- EACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 957–964
- Language:
- URL:
- https://aclanthology.org/2021.eacl-main.82
- DOI:
- 10.18653/v1/2021.eacl-main.82
- Cite (ACL):
- James Thorne and Andreas Vlachos. 2021. Elastic weight consolidation for better bias inoculation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 957–964, Online. Association for Computational Linguistics.
- Cite (Informal):
- Elastic weight consolidation for better bias inoculation (Thorne & Vlachos, EACL 2021)
- PDF:
- https://preview.aclanthology.org/paclic-22-ingestion/2021.eacl-main.82.pdf
- Code
- j6mes/eacl2021-debias-finetuning
- Data
- FEVER, MultiFC, MultiNLI