CoCoLM: Complex Commonsense Enhanced Language Model with Discourse Relations

Changlong Yu, Hongming Zhang, Yangqiu Song, Wilfred Ng

Abstract
Large-scale pre-trained language models have demonstrated strong knowledge representation ability. However, recent studies suggest that even though these giant models contain rich simple commonsense knowledge (e.g., bird can fly and fish can swim.), they often struggle with complex commonsense knowledge that involves multiple eventualities (verb-centric phrases, e.g., identifying the relationship between “Jim yells at Bob” and “Bob is upset”). To address this issue, in this paper, we propose to help pre-trained language models better incorporate complex commonsense knowledge. Unlike direct fine-tuning approaches, we do not focus on a specific task and instead propose a general language model named CoCoLM. Through the careful training over a large-scale eventuality knowledge graph ASER, we successfully teach pre-trained language models (i.e., BERT and RoBERTa) rich multi-hop commonsense knowledge among eventualities. Experiments on multiple commonsense tasks that require the correct understanding of eventualities demonstrate the effectiveness of CoCoLM.
Anthology ID:
2022.findings-acl.93
Volume:
Findings of the Association for Computational Linguistics: ACL 2022
Month:
May
Year:
2022
Address:
Dublin, Ireland
Editors:
Smaranda Muresan, Preslav Nakov, Aline Villavicencio
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1175–1187
Language:
URL:
https://aclanthology.org/2022.findings-acl.93
DOI:
10.18653/v1/2022.findings-acl.93
Bibkey:
Cite (ACL):
Changlong Yu, Hongming Zhang, Yangqiu Song, and Wilfred Ng. 2022. CoCoLM: Complex Commonsense Enhanced Language Model with Discourse Relations. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1175–1187, Dublin, Ireland. Association for Computational Linguistics.
Cite (Informal):
CoCoLM: Complex Commonsense Enhanced Language Model with Discourse Relations (Yu et al., Findings 2022)
Copy Citation:
PDF:
https://preview.aclanthology.org/naacl-24-ws-corrections/2022.findings-acl.93.pdf
Code
 hkust-knowcomp/co2lm
Data
COPA, ConceptNet, LAMA, ROCStories
PDF Search Code