Abstract
In this paper we present a thorough investigation of automatic bias recognition on BASIL, a dataset of political news which has been annotated with different kinds of biases. We begin by unveiling several inconsistencies in prior work using this dataset, showing that most approaches focus only on certain task formulations while ignoring others, and also failing to report important evaluation details. We provide a comprehensive categorization of these approaches, as well as a more uniform and clear set of evaluation metrics. We argue about the importance of the missing formulations and also propose the novel task of simultaneously detecting different kinds of biases in news. In our work, we tackle bias on six different BASIL classification tasks in a unified manner. Eventually, we introduce a simple yet effective approach based on data augmentation and preprocessing which is generic and works very well across models and task formulations, allowing us to obtain state-of-the-art results. We also perform ablation studies on some tasks to quantify the strength of data augmentation and preprocessing, and find that they correlate positively on all bias tasks.- Anthology ID:
- 2023.fever-1.7
- Volume:
- Proceedings of the Sixth Fact Extraction and VERification Workshop (FEVER)
- Month:
- May
- Year:
- 2023
- Address:
- Dubrovnik, Croatia
- Editors:
- Mubashara Akhtar, Rami Aly, Christos Christodoulopoulos, Oana Cocarascu, Zhijiang Guo, Arpit Mittal, Michael Schlichtkrull, James Thorne, Andreas Vlachos
- Venue:
- FEVER
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 66–77
- Language:
- URL:
- https://aclanthology.org/2023.fever-1.7
- DOI:
- 10.18653/v1/2023.fever-1.7
- Cite (ACL):
- Iffat Maab, Edison Marrese-Taylor, and Yutaka Matsuo. 2023. An Effective Approach for Informational and Lexical Bias Detection. In Proceedings of the Sixth Fact Extraction and VERification Workshop (FEVER), pages 66–77, Dubrovnik, Croatia. Association for Computational Linguistics.
- Cite (Informal):
- An Effective Approach for Informational and Lexical Bias Detection (Maab et al., FEVER 2023)
- PDF:
- https://preview.aclanthology.org/dois-2013-emnlp/2023.fever-1.7.pdf