SHARCS: Efficient Transformers Through Routing with Dynamic Width Sub-networks
Mohammadreza Salehi, Sachin Mehta, Aditya Kusupati, Ali Farhadi, Hannaneh Hajishirzi
Abstract
We introduce SHARCS for adaptive inference that takes into account the hardness of input samples. SHARCS can train a router on any transformer network, enabling the model to direct different samples to sub-networks with varying widths. Our experiments demonstrate that: (1) SHARCS outperforms or complements existing per-sample adaptive inference methods across various classification tasks in terms of accuracy vs. FLOPs; (2) SHARCS generalizes across different architectures and can be even applied to compressed and efficient transformer encoders to further improve their efficiency; (3) SHARCS can provide a 2 times inference speed up at an insignificant drop in accuracy.- Anthology ID:
- 2023.findings-emnlp.706
- Volume:
- Findings of the Association for Computational Linguistics: EMNLP 2023
- Month:
- December
- Year:
- 2023
- Address:
- Singapore
- Editors:
- Houda Bouamor, Juan Pino, Kalika Bali
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 10519–10532
- Language:
- URL:
- https://aclanthology.org/2023.findings-emnlp.706
- DOI:
- 10.18653/v1/2023.findings-emnlp.706
- Cite (ACL):
- Mohammadreza Salehi, Sachin Mehta, Aditya Kusupati, Ali Farhadi, and Hannaneh Hajishirzi. 2023. SHARCS: Efficient Transformers Through Routing with Dynamic Width Sub-networks. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10519–10532, Singapore. Association for Computational Linguistics.
- Cite (Informal):
- SHARCS: Efficient Transformers Through Routing with Dynamic Width Sub-networks (Salehi et al., Findings 2023)
- PDF:
- https://preview.aclanthology.org/add_acl24_videos/2023.findings-emnlp.706.pdf