Linguistically-Informed Transformations (LIT): A Method for Automatically Generating Contrast Sets

Chuanrong Li, Lin Shengshuo, Zeyu Liu, Xinyi Wu, Xuhui Zhou, Shane Steinert-Threlkeld

Abstract
Although large-scale pretrained language models, such as BERT and RoBERTa, have achieved superhuman performance on in-distribution test sets, their performance suffers on out-of-distribution test sets (e.g., on contrast sets). Building contrast sets often requires human-expert annotation, which is expensive and hard to create on a large scale. In this work, we propose a Linguistically-Informed Transformation (LIT) method to automatically generate contrast sets, which enables practitioners to explore linguistic phenomena of interests as well as compose different phenomena. Experimenting with our method on SNLI and MNLI shows that current pretrained language models, although being claimed to contain sufficient linguistic knowledge, struggle on our automatically generated contrast sets. Furthermore, we improve models’ performance on the contrast sets by applying LIT to augment the training data, without affecting performance on the original data.
Anthology ID:
2020.blackboxnlp-1.12
Volume:
Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP
Month:
November
Year:
2020
Address:
Online
Venue:
BlackboxNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
126–135
Language:
URL:
https://aclanthology.org/2020.blackboxnlp-1.12
DOI:
10.18653/v1/2020.blackboxnlp-1.12
Bibkey:
Cite (ACL):
Chuanrong Li, Lin Shengshuo, Zeyu Liu, Xinyi Wu, Xuhui Zhou, and Shane Steinert-Threlkeld. 2020. Linguistically-Informed Transformations (LIT): A Method for Automatically Generating Contrast Sets. In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 126–135, Online. Association for Computational Linguistics.
Cite (Informal):
Linguistically-Informed Transformations (LIT): A Method for Automatically Generating Contrast Sets (Li et al., BlackboxNLP 2020)
Copy Citation:
PDF:
https://preview.aclanthology.org/auto-file-uploads/2020.blackboxnlp-1.12.pdf
Optional supplementary material:
 2020.blackboxnlp-1.12.OptionalSupplementaryMaterial.pdf
Data
MultiNLI
PDF Search Optional supplementary material