Abstract
In sequence modeling, certain tokens are usually less ambiguous than others, and representations of these tokens require fewer refinements for disambiguation. However, given the nature of attention-based models like Transformer and UT (universal transformer), all tokens are equally processed towards depth. Inspired by the equilibrium phenomenon, we present a lazy transition, a mechanism to adjust the significance of iterative refinements for each token representation. Our lazy transition is deployed on top of UT to build LT (lazy transformer), where all tokens are processed unequally towards depth. Eventually, LT is encouraged to oscillate around a relaxed equilibrium. Our experiments show that LT outperforms baseline models on several tasks of machine translation, pre-training, Learning to Execute, and LAMBADA.- Anthology ID:
- 2022.acl-long.208
- Volume:
- Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Month:
- May
- Year:
- 2022
- Address:
- Dublin, Ireland
- Venue:
- ACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 2904–2924
- Language:
- URL:
- https://aclanthology.org/2022.acl-long.208
- DOI:
- 10.18653/v1/2022.acl-long.208
- Cite (ACL):
- Xi Ai and Bin Fang. 2022. Leveraging Relaxed Equilibrium by Lazy Transition for Sequence Modeling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2904–2924, Dublin, Ireland. Association for Computational Linguistics.
- Cite (Informal):
- Leveraging Relaxed Equilibrium by Lazy Transition for Sequence Modeling (Ai & Fang, ACL 2022)
- PDF:
- https://preview.aclanthology.org/nodalida-main-page/2022.acl-long.208.pdf
- Data
- LAMBADA, MultiNLI