IBADR: an Iterative Bias-Aware Dataset Refinement Framework for Debiasing NLU models

Xiaoyue Wang, Xin Liu, Lijie Wang, Yaoxiang Wang, Jinsong Su, Hua Wu


Abstract
As commonly-used methods for debiasing natural language understanding (NLU) models, dataset refinement approaches heavily rely on manual data analysis, and thus maybe unable to cover all the potential biased features. In this paper, we propose IBADR, an Iterative Bias-Aware Dataset Refinement framework, which debiases NLU models without predefining biased features. We maintain an iteratively expanded sample pool. Specifically, at each iteration, we first train a shallow model to quantify the bias degree of samples in the pool. Then, we pair each sample with a bias indicator representing its bias degree, and use these extended samples to train a sample generator. In this way, this generator can effectively learn the correspondence relationship between bias indicators and samples. Furthermore, we employ the generator to produce pseudo samples with fewer biased features by feeding specific bias indicators. Finally, we incorporate the generated pseudo samples into the pool. Experimental results and in-depth analyses on two NLU tasks show that IBADR not only significantly outperforms existing dataset refinement approaches, achieving SOTA, but also is compatible with model-centric methods.
Anthology ID:
2023.emnlp-main.569
Volume:
Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
Month:
December
Year:
2023
Address:
Singapore
Editors:
Houda Bouamor, Juan Pino, Kalika Bali
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
9176–9186
Language:
URL:
https://aclanthology.org/2023.emnlp-main.569
DOI:
10.18653/v1/2023.emnlp-main.569
Bibkey:
Cite (ACL):
Xiaoyue Wang, Xin Liu, Lijie Wang, Yaoxiang Wang, Jinsong Su, and Hua Wu. 2023. IBADR: an Iterative Bias-Aware Dataset Refinement Framework for Debiasing NLU models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9176–9186, Singapore. Association for Computational Linguistics.
Cite (Informal):
IBADR: an Iterative Bias-Aware Dataset Refinement Framework for Debiasing NLU models (Wang et al., EMNLP 2023)
Copy Citation:
PDF:
https://preview.aclanthology.org/emnlp22-frontmatter/2023.emnlp-main.569.pdf
Video:
 https://preview.aclanthology.org/emnlp22-frontmatter/2023.emnlp-main.569.mp4