Abstract
Language tasks involving character-level manipulations (e.g., spelling corrections, arithmetic operations, word games) are challenging for models operating on subword units. To address this, we develop a causal intervention framework to learn robust and interpretable character representations inside subword-based language models. Our method treats each character as a typed variable in a causal model and learns such causal structures by adapting the interchange intervention training method of Geiger et al. (2021). We additionally introduce a suite of character-level tasks that systematically vary in their dependence on meaning and sequence-level context. While character-level models still perform best on purely form-based tasks like string reversal, our method outperforms character-level models on more complex tasks that blend form, meaning, and context, such as spelling correction in context and word search games. Compared with standard subword-based models, our approach also significantly improves robustness on unseen token sequences and leads to human-interpretable internal representations of characters.- Anthology ID:
- 2023.findings-acl.770
- Volume:
- Findings of the Association for Computational Linguistics: ACL 2023
- Month:
- July
- Year:
- 2023
- Address:
- Toronto, Canada
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 12163–12180
- Language:
- URL:
- https://aclanthology.org/2023.findings-acl.770
- DOI:
- 10.18653/v1/2023.findings-acl.770
- Cite (ACL):
- Jing Huang, Zhengxuan Wu, Kyle Mahowald, and Christopher Potts. 2023. Inducing Character-level Structure in Subword-based Language Models with Type-level Interchange Intervention Training. In Findings of the Association for Computational Linguistics: ACL 2023, pages 12163–12180, Toronto, Canada. Association for Computational Linguistics.
- Cite (Informal):
- Inducing Character-level Structure in Subword-based Language Models with Type-level Interchange Intervention Training (Huang et al., Findings 2023)
- PDF:
- https://preview.aclanthology.org/remove-xml-comments/2023.findings-acl.770.pdf