Leap-of-Thought: Accelerating Transformers via Dynamic Token Routing
Yeachan Kim, Junho Kim, Jun-Hyung Park, Mingyu Lee, SangKeun Lee
Abstract
Computational inefficiency in transformers has been a long-standing challenge, hindering the deployment in resource-constrained or real-time applications. One promising approach to mitigate this limitation is to progressively remove less significant tokens, given that the sequence length strongly contributes to the inefficiency. However, this approach entails a potential risk of losing crucial information due to the irrevocable nature of token removal. In this paper, we introduce Leap-of-Thought (LoT), a novel token reduction approach that dynamically routes tokens within layers. Unlike previous work that irrevocably discards tokens, LoT enables tokens to ‘leap’ across layers. This ensures that all tokens remain accessible in subsequent layers while reducing the number of tokens processed within layers. We achieve this by pairing the transformer with dynamic token routers, which learn to selectively process tokens essential for the task. Evaluation results clearly show that LoT achieves a substantial improvement in computational efficiency. Specifically, LoT attains up to 25x faster inference time without a significant loss in accuracy- Anthology ID:
- 2023.emnlp-main.976
- Volume:
- Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
- Month:
- December
- Year:
- 2023
- Address:
- Singapore
- Editors:
- Houda Bouamor, Juan Pino, Kalika Bali
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 15757–15769
- Language:
- URL:
- https://aclanthology.org/2023.emnlp-main.976
- DOI:
- 10.18653/v1/2023.emnlp-main.976
- Cite (ACL):
- Yeachan Kim, Junho Kim, Jun-Hyung Park, Mingyu Lee, and SangKeun Lee. 2023. Leap-of-Thought: Accelerating Transformers via Dynamic Token Routing. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 15757–15769, Singapore. Association for Computational Linguistics.
- Cite (Informal):
- Leap-of-Thought: Accelerating Transformers via Dynamic Token Routing (Kim et al., EMNLP 2023)
- PDF:
- https://preview.aclanthology.org/proper-vol2-ingestion/2023.emnlp-main.976.pdf