Abstract
Mechanisms for encoding positional information are central for transformer-based language models. In this paper, we analyze the position embeddings of existing language models, finding strong evidence of translation invariance, both for the embeddings themselves and for their effect on self-attention. The degree of translation invariance increases during training and correlates positively with model performance. Our findings lead us to propose translation-invariant self-attention (TISA), which accounts for the relative position between tokens in an interpretable fashion without needing conventional position embeddings. Our proposal has several theoretical advantages over existing position-representation approaches. Proof-of-concept experiments show that it improves on regular ALBERT on GLUE tasks, while only adding orders of magnitude less positional parameters.- Anthology ID:
- 2021.acl-short.18
- Volume:
- Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)
- Month:
- August
- Year:
- 2021
- Address:
- Online
- Editors:
- Chengqing Zong, Fei Xia, Wenjie Li, Roberto Navigli
- Venues:
- ACL | IJCNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 130–140
- Language:
- URL:
- https://aclanthology.org/2021.acl-short.18
- DOI:
- 10.18653/v1/2021.acl-short.18
- Cite (ACL):
- Ulme Wennberg and Gustav Eje Henter. 2021. The Case for Translation-Invariant Self-Attention in Transformer-Based Language Models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 130–140, Online. Association for Computational Linguistics.
- Cite (Informal):
- The Case for Translation-Invariant Self-Attention in Transformer-Based Language Models (Wennberg & Henter, ACL-IJCNLP 2021)
- PDF:
- https://preview.aclanthology.org/aacl-23-doi-ingestion/2021.acl-short.18.pdf
- Code
- ulmewennberg/tisa
- Data
- GLUE, QNLI