Enhancing Natural Language Representation with Large-Scale Out-of-Domain Commonsense

Wanyun Cui, Xingran Chen


Abstract
We study how to enhance text representation via textual commonsense. We point out that commonsense has the nature of domain discrepancy. Namely, commonsense has different data formats and is domain-independent from the downstream task. This nature brings challenges to introducing commonsense in general text understanding tasks. A typical method of introducing textual knowledge is continuing pre-training over the commonsense corpus. However, it will cause catastrophic forgetting to the downstream task due to the domain discrepancy. In addition, previous methods of directly using textual descriptions as extra input information cannot apply to large-scale commonsense.In this paper, we propose to use large-scale out-of-domain commonsense to enhance text representation. In order to effectively incorporate the commonsense, we proposed OK-Transformer (Out-of-domain Knowledge enhanced Transformer). OK-Transformer effectively integrates commonsense descriptions and enhances them to the target text representation. In addition, OK-Transformer can adapt to the Transformer-based language models (e.g. BERT, RoBERTa) for free, without pre-training on large-scale unsupervised corpora. We have verified the effectiveness of OK-Transformer in multiple applications such as commonsense reasoning, general text classification, and low-resource commonsense settings.
Anthology ID:
2022.findings-acl.138
Volume:
Findings of the Association for Computational Linguistics: ACL 2022
Month:
May
Year:
2022
Address:
Dublin, Ireland
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1746–1756
Language:
URL:
https://aclanthology.org/2022.findings-acl.138
DOI:
10.18653/v1/2022.findings-acl.138
Bibkey:
Cite (ACL):
Wanyun Cui and Xingran Chen. 2022. Enhancing Natural Language Representation with Large-Scale Out-of-Domain Commonsense. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1746–1756, Dublin, Ireland. Association for Computational Linguistics.
Cite (Informal):
Enhancing Natural Language Representation with Large-Scale Out-of-Domain Commonsense (Cui & Chen, Findings 2022)
Copy Citation:
PDF:
https://preview.aclanthology.org/emnlp-22-attachments/2022.findings-acl.138.pdf
Code
 chenxran/ok-transformer
Data
CommonsenseQAGLUEQNLIWSCWinoGrande