Adaptive Ranking-based Sample Selection for Weakly Supervised Class-imbalanced Text Classification

Linxin Song, Jieyu Zhang, Tianxiang Yang, Masayuki Goto

Abstract
To obtain a large amount of training labels inexpensively, researchers have recently adopted the weak supervision (WS) paradigm, which leverages labeling rules to synthesize training labels rather than using individual annotations to achieve competitive results for natural language processing (NLP) tasks. However, data imbalance is often overlooked in applying the WS paradigm, despite being a common issue in a variety of NLP tasks. To address this challenge, we propose Adaptive Ranking-based Sample Selection (ARS2), a model-agnostic framework to alleviate the data imbalance issue in the WS paradigm. Specifically, it calculates a probabilistic margin score based on the output of the current model to measure and rank the cleanliness of each data point. Then, the ranked data are sampled based on both class-wise and rule-aware ranking. In particular, the two sample strategies corresponds to our motivations: (1) to train the model with balanced data batches to reduce the data imbalance issue and (2) to exploit the expertise of each labeling rule for collecting clean samples. Experiments on four text classification datasets with four different imbalance ratios show that ARS2 outperformed the state-of-the-art imbalanced learning and WS methods, leading to a 2%-57.8% improvement on their F1-score.
Anthology ID:
2022.findings-emnlp.119
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2022
Month:
December
Year:
2022
Address:
Abu Dhabi, United Arab Emirates
Editors:
Yoav Goldberg, Zornitsa Kozareva, Yue Zhang
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1641–1655
Language:
URL:
https://aclanthology.org/2022.findings-emnlp.119
DOI:
10.18653/v1/2022.findings-emnlp.119
Bibkey:
Cite (ACL):
Linxin Song, Jieyu Zhang, Tianxiang Yang, and Masayuki Goto. 2022. Adaptive Ranking-based Sample Selection for Weakly Supervised Class-imbalanced Text Classification. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 1641–1655, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Cite (Informal):
Adaptive Ranking-based Sample Selection for Weakly Supervised Class-imbalanced Text Classification (Song et al., Findings 2022)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-4/2022.findings-emnlp.119.pdf
PDF Search