Abstract
State-of-the-art deep-learning-based approaches to Natural Language Processing (NLP) are credited with various capabilities that involve reasoning with natural language texts. %However, reasoning in this setting is often ill-defined and shallow. In this paper we carry out a large-scale empirical study investigating the detection of formally valid inferences in controlled fragments of natural language for which the satisfiability problem becomes increasingly complex. We find that, while transformer-based language models perform surprisingly well in these scenarios, a deeper analysis reveals that they appear to overfit to superficial patterns in the data rather than acquiring the logical principles governing the reasoning in these fragments.- Anthology ID:
- 2022.emnlp-main.768
- Volume:
- Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
- Month:
- December
- Year:
- 2022
- Address:
- Abu Dhabi, United Arab Emirates
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 11184–11199
- Language:
- URL:
- https://aclanthology.org/2022.emnlp-main.768
- DOI:
- 10.18653/v1/2022.emnlp-main.768
- Cite (ACL):
- Viktor Schlegel, Kamen Pavlov, and Ian Pratt-Hartmann. 2022. Can Transformers Reason in Fragments of Natural Language?. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11184–11199, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
- Cite (Informal):
- Can Transformers Reason in Fragments of Natural Language? (Schlegel et al., EMNLP 2022)
- PDF:
- https://preview.aclanthology.org/remove-xml-comments/2022.emnlp-main.768.pdf