Abstract
Demonstration-based learning has shown impressive performance in exploiting pretrained language models under few-shot learning settings. It is interesting to see that demonstrations, even those composed of random tokens, can still improve performance. In this paper, we build a Structural Causal Model (SCM) to understand demonstration-based learning from causal perspectives and interpret random demonstrations as interventions on the demonstration variable within the causal model. We investigate the causal effects and find that the concurrence of specific words in the demonstration will induce bias, while randomly sampled tokens in the demonstration do not. Based on this finding, we further propose simple ways to construct random demonstrations, which even outperform hand-crafted, meaningful demonstrations on public sequence labeling benchmarks.- Anthology ID:
- 2023.acl-short.125
- Volume:
- Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
- Month:
- July
- Year:
- 2023
- Address:
- Toronto, Canada
- Editors:
- Anna Rogers, Jordan Boyd-Graber, Naoaki Okazaki
- Venue:
- ACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 1465–1475
- Language:
- URL:
- https://aclanthology.org/2023.acl-short.125
- DOI:
- 10.18653/v1/2023.acl-short.125
- Cite (ACL):
- Ruiyi Zhang and Tong Yu. 2023. Understanding Demonstration-based Learning from a Causal Perspective. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1465–1475, Toronto, Canada. Association for Computational Linguistics.
- Cite (Informal):
- Understanding Demonstration-based Learning from a Causal Perspective (Zhang & Yu, ACL 2023)
- PDF:
- https://preview.aclanthology.org/ingest-acl-2023-videos/2023.acl-short.125.pdf