Abstract
The increasing use of foundation models highlights the urgent need to address and eliminate implicit biases present in them that arise during pretraining. In this paper, we introduce PEFTDebias, a novel approach that employs parameter-efficient fine-tuning (PEFT) to mitigate the biases within foundation models. PEFTDebias consists of two main phases: an upstream phase for acquiring debiasing parameters along a specific bias axis, and a downstream phase where these parameters are incorporated into the model and frozen during the fine-tuning process. By evaluating on four datasets across two bias axes namely gender and race, we find that downstream biases can be effectively reduced with PEFTs. In addition, we show that these parameters possess axis-specific debiasing characteristics, enabling their effective transferability in mitigating biases in various downstream tasks.- Anthology ID:
- 2023.emnlp-main.122
- Volume:
- Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
- Month:
- December
- Year:
- 2023
- Address:
- Singapore
- Editors:
- Houda Bouamor, Juan Pino, Kalika Bali
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 1992–2000
- Language:
- URL:
- https://aclanthology.org/2023.emnlp-main.122
- DOI:
- 10.18653/v1/2023.emnlp-main.122
- Cite (ACL):
- Sumit Agarwal, Aditya Veerubhotla, and Srijan Bansal. 2023. PEFTDebias : Capturing debiasing information using PEFTs. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1992–2000, Singapore. Association for Computational Linguistics.
- Cite (Informal):
- PEFTDebias : Capturing debiasing information using PEFTs (Agarwal et al., EMNLP 2023)
- PDF:
- https://preview.aclanthology.org/emnlp-22-attachments/2023.emnlp-main.122.pdf