Abstract
BERT-based models have had strong performance on leaderboards, yet have been demonstrably worse in real-world settings requiring generalization. Limited quantities of training data is considered a key impediment to achieving generalizability in machine learning. In this paper, we examine the impact of training data quality, not quantity, on a model’s generalizability. We consider two characteristics of training data: the portion of human-adversarial (h-adversarial), i.e. sample pairs with seemingly minor differences but different ground-truth labels, and human-affable (h-affable) training samples, i.e. sample pairs with minor differences but the same ground-truth label. We find that for a fixed size of training samples, as a rule of thumb, having 10-30% h-adversarial instances improves the precision, and therefore F1, by up to 20 points in the tasks of text classification and relation extraction. Increasing h-adversarials beyond this range can result in performance plateaus or even degradation. In contrast, h-affables may not contribute to a model’s generalizability and may even degrade generalization performance.- Anthology ID:
- 2023.findings-emnlp.512
- Volume:
- Findings of the Association for Computational Linguistics: EMNLP 2023
- Month:
- December
- Year:
- 2023
- Address:
- Singapore
- Editors:
- Houda Bouamor, Juan Pino, Kalika Bali
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 7637–7649
- Language:
- URL:
- https://aclanthology.org/2023.findings-emnlp.512
- DOI:
- 10.18653/v1/2023.findings-emnlp.512
- Cite (ACL):
- Aparna Elangovan, Estrid He, Yuan Li, and Karin Verspoor. 2023. Effects of Human Adversarial and Affable Samples on BERT Generalization. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7637–7649, Singapore. Association for Computational Linguistics.
- Cite (Informal):
- Effects of Human Adversarial and Affable Samples on BERT Generalization (Elangovan et al., Findings 2023)
- PDF:
- https://preview.aclanthology.org/emnlp-22-attachments/2023.findings-emnlp.512.pdf