Abstract
Causal reasoning is a critical component of human cognition and is required across a range of question-answering (QA) tasks (such as abductive reasoning, commonsense QA, and procedural reasoning). Research on causal QA has been underdefined, task-specific, and limited in complexity. Recent advances in foundation language models (such as BERT, ERNIE, and T5) have shown the efficacy of pre-trained models across diverse QA tasks. However, there is limited research exploring the causal reasoning capabilities of those language models and no standard evaluation benchmark. To unify causal QA research, we propose CALM-Bench, a multi-task benchmark for evaluating causality-aware language models (CALM). We present a standardized definition of causal QA tasks and show empirically that causal reasoning can be generalized and transferred across different QA tasks. Additionally, we share a strong multi-task baseline model which outperforms single-task fine-tuned models on the CALM-Bench tasks.- Anthology ID:
- 2023.findings-eacl.23
- Volume:
- Findings of the Association for Computational Linguistics: EACL 2023
- Month:
- May
- Year:
- 2023
- Address:
- Dubrovnik, Croatia
- Editors:
- Andreas Vlachos, Isabelle Augenstein
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 296–311
- Language:
- URL:
- https://aclanthology.org/2023.findings-eacl.23
- DOI:
- 10.18653/v1/2023.findings-eacl.23
- Cite (ACL):
- Dhairya Dalal, Paul Buitelaar, and Mihael Arcan. 2023. CALM-Bench: A Multi-task Benchmark for Evaluating Causality-Aware Language Models. In Findings of the Association for Computational Linguistics: EACL 2023, pages 296–311, Dubrovnik, Croatia. Association for Computational Linguistics.
- Cite (Informal):
- CALM-Bench: A Multi-task Benchmark for Evaluating Causality-Aware Language Models (Dalal et al., Findings 2023)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-5/2023.findings-eacl.23.pdf