Debiasing with Sufficient Projection: A General Theoretical Framework for Vector Representations

Enze Shi, Lei Ding, Linglong Kong, Bei Jiang

Abstract
Pre-trained vector representations in natural language processing often inadvertently encode undesirable social biases. Identifying and removing unwanted biased information from vector representation is an evolving and significant challenge. Our study uniquely addresses this issue from the perspective of statistical independence, proposing a framework for reducing bias by transforming vector representations to an unbiased subspace using sufficient projection. The key to our framework lies in its generality: it adeptly mitigates bias across both debiasing and fairness tasks, and across various vector representation types, including word embeddings and output representations of transformer models. Importantly, we establish the connection between debiasing and fairness, offering theoretical guarantees and elucidating our algorithm’s efficacy. Through extensive evaluation of intrinsic and extrinsic metrics, our method achieves superior performance in bias reduction while maintaining high task performance, and offers superior computational efficiency.
Anthology ID:
2024.naacl-long.332
Volume:
Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)
Month:
June
Year:
2024
Address:
Mexico City, Mexico
Editors:
Kevin Duh, Helena Gomez, Steven Bethard
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
5960–5975
Language:
URL:
https://aclanthology.org/2024.naacl-long.332
DOI:
10.18653/v1/2024.naacl-long.332
Bibkey:
Cite (ACL):
Enze Shi, Lei Ding, Linglong Kong, and Bei Jiang. 2024. Debiasing with Sufficient Projection: A General Theoretical Framework for Vector Representations. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 5960–5975, Mexico City, Mexico. Association for Computational Linguistics.
Cite (Informal):
Debiasing with Sufficient Projection: A General Theoretical Framework for Vector Representations (Shi et al., NAACL 2024)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-4/2024.naacl-long.332.pdf
PDF Search