Abstract
In this work, we present our empirical attempt to identify the proper strategy of using Transformer Language Models to identify sentences consistent with commonsense. We tackle the first two tasks from the ComVE competition. The starting point for our work is the BERT assumption according to which a large number of NLP tasks can be solved with pre-trained Transformers with no substantial task-specific changes of the architecture. However, our experiments show that the encoding strategy can have a great impact on the quality of the fine-tuning. The combination between cross-encoding and multi-input models worked better than one cross-encoder and allowed us to achieve comparable results with the state-of-the-art without the use of any external data.- Anthology ID:
- 2020.deelio-1.10
- Volume:
- Proceedings of Deep Learning Inside Out (DeeLIO): The First Workshop on Knowledge Extraction and Integration for Deep Learning Architectures
- Month:
- November
- Year:
- 2020
- Address:
- Online
- Venue:
- DeeLIO
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 80–88
- Language:
- URL:
- https://aclanthology.org/2020.deelio-1.10
- DOI:
- 10.18653/v1/2020.deelio-1.10
- Cite (ACL):
- Sonia Cibu and Anca Marginean. 2020. Commonsense Statements Identification and Explanation with Transformer-based Encoders. In Proceedings of Deep Learning Inside Out (DeeLIO): The First Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pages 80–88, Online. Association for Computational Linguistics.
- Cite (Informal):
- Commonsense Statements Identification and Explanation with Transformer-based Encoders (Cibu & Marginean, DeeLIO 2020)
- PDF:
- https://preview.aclanthology.org/auto-file-uploads/2020.deelio-1.10.pdf