How Does Data Corruption Affect Natural Language Understanding Models? A Study on GLUE datasets
Aarne Talman, Marianna Apidianaki, Stergios Chatzikyriakidis, Jörg Tiedemann
Abstract
A central question in natural language understanding (NLU) research is whether high performance demonstrates the models’ strong reasoning capabilities. We present an extensive series of controlled experiments where pre-trained language models are exposed to data that have undergone specific corruption transformations. These involve removing instances of specific word classes and often lead to non-sensical sentences. Our results show that performance remains high on most GLUE tasks when the models are fine-tuned or tested on corrupted data, suggesting that they leverage other cues for prediction even in non-sensical contexts. Our proposed data transformations can be used to assess the extent to which a specific dataset constitutes a proper testbed for evaluating models’ language understanding capabilities.- Anthology ID:
- 2022.starsem-1.20
- Volume:
- Proceedings of the 11th Joint Conference on Lexical and Computational Semantics
- Month:
- July
- Year:
- 2022
- Address:
- Seattle, Washington
- Venue:
- *SEM
- SIG:
- SIGSEM
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 226–233
- Language:
- URL:
- https://aclanthology.org/2022.starsem-1.20
- DOI:
- 10.18653/v1/2022.starsem-1.20
- Cite (ACL):
- Aarne Talman, Marianna Apidianaki, Stergios Chatzikyriakidis, and Jörg Tiedemann. 2022. How Does Data Corruption Affect Natural Language Understanding Models? A Study on GLUE datasets. In Proceedings of the 11th Joint Conference on Lexical and Computational Semantics, pages 226–233, Seattle, Washington. Association for Computational Linguistics.
- Cite (Informal):
- How Does Data Corruption Affect Natural Language Understanding Models? A Study on GLUE datasets (Talman et al., *SEM 2022)
- PDF:
- https://preview.aclanthology.org/ingestion-script-update/2022.starsem-1.20.pdf
- Code
- helsinki-nlp/nlu-dataset-diagnostics
- Data
- CoLA, GLUE, MRPC, MultiNLI, QNLI, SST, SuperGLUE