Abstract
Previous studies have shown the efficacy of knowledge augmentation methods in pretrained language models. However, these methods behave differently across domains and downstream tasks. In this work, we investigate the augmentation of pretrained language models with knowledge graph data in the cause-effect relation classification and commonsense causal reasoning tasks. After automatically verbalizing triples in ATOMIC2020, a wide coverage commonsense reasoning knowledge graph, we continually pretrain BERT and evaluate the resulting model on cause-effect pair classification and answering commonsense causal reasoning questions. Our results show that a continually pretrained language model augmented with commonsense reasoning knowledge outperforms our baselines on two commonsense causal reasoning benchmarks, COPA and BCOPA-CE, and a Temporal and Causal Reasoning (TCR) dataset, without additional improvement in model architecture or using quality-enhanced data for fine-tuning.- Anthology ID:
- 2022.csrr-1.6
- Volume:
- Proceedings of the First Workshop on Commonsense Representation and Reasoning (CSRR 2022)
- Month:
- May
- Year:
- 2022
- Address:
- Dublin, Ireland
- Venue:
- CSRR
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 43–48
- Language:
- URL:
- https://aclanthology.org/2022.csrr-1.6
- DOI:
- 10.18653/v1/2022.csrr-1.6
- Cite (ACL):
- Pedram Hosseini, David A. Broniatowski, and Mona Diab. 2022. Knowledge-Augmented Language Models for Cause-Effect Relation Classification. In Proceedings of the First Workshop on Commonsense Representation and Reasoning (CSRR 2022), pages 43–48, Dublin, Ireland. Association for Computational Linguistics.
- Cite (Informal):
- Knowledge-Augmented Language Models for Cause-Effect Relation Classification (Hosseini et al., CSRR 2022)
- PDF:
- https://preview.aclanthology.org/ingestion-script-update/2022.csrr-1.6.pdf
- Code
- phosseini/causal-reasoning
- Data
- ATOMIC, BCOPA-CE, COPA, GLUCOSE, TCR