Testing the Generalization Power of Neural Network Models across NLI Benchmarks

Aarne Talman, Stergios Chatzikyriakidis

Abstract
Neural network models have been very successful in natural language inference, with the best models reaching 90% accuracy in some benchmarks. However, the success of these models turns out to be largely benchmark specific. We show that models trained on a natural language inference dataset drawn from one benchmark fail to perform well in others, even if the notion of inference assumed in these benchmarks is the same or similar. We train six high performing neural network models on different datasets and show that each one of these has problems of generalizing when we replace the original test set with a test set taken from another corpus designed for the same task. In light of these results, we argue that most of the current neural network models are not able to generalize well in the task of natural language inference. We find that using large pre-trained language models helps with transfer learning when the datasets are similar enough. Our results also highlight that the current NLI datasets do not cover the different nuances of inference extensively enough.
Anthology ID:
W19-4810
Volume:
Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP
Month:
August
Year:
2019
Address:
Florence, Italy
Editors:
Tal Linzen, Grzegorz Chrupała, Yonatan Belinkov, Dieuwke Hupkes
Venue:
BlackboxNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
85–94
Language:
URL:
https://aclanthology.org/W19-4810
DOI:
10.18653/v1/W19-4810
Bibkey:
Cite (ACL):
Aarne Talman and Stergios Chatzikyriakidis. 2019. Testing the Generalization Power of Neural Network Models across NLI Benchmarks. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 85–94, Florence, Italy. Association for Computational Linguistics.
Cite (Informal):
Testing the Generalization Power of Neural Network Models across NLI Benchmarks (Talman & Chatzikyriakidis, BlackboxNLP 2019)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-2/W19-4810.pdf
Data
Flickr30k, MultiNLI, SICK
PDF Search