Easy and Efficient Transformer: Scalable Inference Solution For Large NLP Model
Gongzheng Li, Yadong Xi, Jingzhen Ding, Duan Wang, Ziyang Luo, Rongsheng Zhang, Bai Liu, Changjie Fan, Xiaoxi Mao, Zeng Zhao
Abstract
Recently, large-scale transformer-based models have been proven to be effective over various tasks across many domains. Nevertheless, applying them in industrial production requires tedious and heavy works to reduce inference costs. To fill such a gap, we introduce a scalable inference solution: Easy and Efficient Transformer (EET), including a series of transformer inference optimization at the algorithm and implementation levels. First, we design highly optimized kernels for long inputs and large hidden sizes. Second, we propose a flexible CUDA memory manager to reduce the memory footprint when deploying a large model. Compared with the state-of-the-art transformer inference library (Faster Transformer v4.0), EET can achieve an average of 1.40-4.20x speedup on the transformer decoder layer with an A100 GPU.- Anthology ID:
- 2022.naacl-industry.8
- Volume:
- Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Track
- Month:
- July
- Year:
- 2022
- Address:
- Hybrid: Seattle, Washington + Online
- Editors:
- Anastassia Loukina, Rashmi Gangadharaiah, Bonan Min
- Venue:
- NAACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 62–68
- Language:
- URL:
- https://aclanthology.org/2022.naacl-industry.8
- DOI:
- 10.18653/v1/2022.naacl-industry.8
- Cite (ACL):
- Gongzheng Li, Yadong Xi, Jingzhen Ding, Duan Wang, Ziyang Luo, Rongsheng Zhang, Bai Liu, Changjie Fan, Xiaoxi Mao, and Zeng Zhao. 2022. Easy and Efficient Transformer: Scalable Inference Solution For Large NLP Model. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Track, pages 62–68, Hybrid: Seattle, Washington + Online. Association for Computational Linguistics.
- Cite (Informal):
- Easy and Efficient Transformer: Scalable Inference Solution For Large NLP Model (Li et al., NAACL 2022)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-2/2022.naacl-industry.8.pdf