How Does Data Corruption Affect Natural Language Understanding Models? A Study on GLUE datasets

Aarne Talman, Marianna Apidianaki, Stergios Chatzikyriakidis, Jörg Tiedemann

Abstract
A central question in natural language understanding (NLU) research is whether high performance demonstrates the models’ strong reasoning capabilities. We present an extensive series of controlled experiments where pre-trained language models are exposed to data that have undergone specific corruption transformations. These involve removing instances of specific word classes and often lead to non-sensical sentences. Our results show that performance remains high on most GLUE tasks when the models are fine-tuned or tested on corrupted data, suggesting that they leverage other cues for prediction even in non-sensical contexts. Our proposed data transformations can be used to assess the extent to which a specific dataset constitutes a proper testbed for evaluating models’ language understanding capabilities.
Anthology ID:
2022.starsem-1.20
Volume:
Proceedings of the 11th Joint Conference on Lexical and Computational Semantics
Month:
July
Year:
2022
Address:
Seattle, Washington
Editors:
Vivi Nastase, Ellie Pavlick, Mohammad Taher Pilehvar, Jose Camacho-Collados, Alessandro Raganato
Venue:
*SEM
SIG:
SIGLEX
Publisher:
Association for Computational Linguistics
Note:
Pages:
226–233
Language:
URL:
https://aclanthology.org/2022.starsem-1.20
DOI:
10.18653/v1/2022.starsem-1.20
Bibkey:
Cite (ACL):
Aarne Talman, Marianna Apidianaki, Stergios Chatzikyriakidis, and Jörg Tiedemann. 2022. How Does Data Corruption Affect Natural Language Understanding Models? A Study on GLUE datasets. In Proceedings of the 11th Joint Conference on Lexical and Computational Semantics, pages 226–233, Seattle, Washington. Association for Computational Linguistics.
Cite (Informal):
How Does Data Corruption Affect Natural Language Understanding Models? A Study on GLUE datasets (Talman et al., *SEM 2022)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingest-acl-2023-videos/2022.starsem-1.20.pdf
Code
 helsinki-nlp/nlu-dataset-diagnostics
Data
CoLA, GLUE, MRPC, MultiNLI, QNLI, SST, SST-2
PDF Search Code