Abstract
Due to biases inherently present in data for pre-training, current pre-trained Large Language Models (LLMs) also ubiquitously manifest the same phenomena. Since the bias influences the output from the LLMs across various tasks, the widespread deployment of the LLMs is hampered. We propose a simple method that utilizes structured knowledge to alleviate this issue, aiming to reduce the bias embedded within the LLMs and ensuring they have an encompassing perspective when used in applications. Experimental results indicated that our method has good debiasing ability when applied to existing both autoregressive and masked language models. Additionally, it could ensure that the performances of LLMs on downstream tasks remain uncompromised.Our method outperforms state-of-the-art (SOTA) baselines in the debiasing ability. Importantly, our method obviates the need for training from scratch, thus offering enhanced scalability and cost-effectiveness.- Anthology ID:
- 2024.findings-acl.612
- Volume:
- Findings of the Association for Computational Linguistics ACL 2024
- Month:
- August
- Year:
- 2024
- Address:
- Bangkok, Thailand and virtual meeting
- Editors:
- Lun-Wei Ku, Andre Martins, Vivek Srikumar
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 10274–10287
- Language:
- URL:
- https://aclanthology.org/2024.findings-acl.612
- DOI:
- Cite (ACL):
- Congda Ma, Tianyu Zhao, and Manabu Okumura. 2024. Debiasing Large Language Models with Structured Knowledge. In Findings of the Association for Computational Linguistics ACL 2024, pages 10274–10287, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.
- Cite (Informal):
- Debiasing Large Language Models with Structured Knowledge (Ma et al., Findings 2024)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/2024.findings-acl.612.pdf