Abstract
Transformer and its variants have achieved great success in natural language processing. Since Transformer models are huge in size, serving these models is a challenge for real industrial applications. In this paper, we propose , a highly efficient inference library for models in the Transformer family. includes a series of GPU optimization techniques to both streamline the computation of Transformer layers and reduce memory footprint. supports models trained using PyTorch and Tensorflow. Experimental results on standard machine translation benchmarks show that achieves up to 14x speedup compared with TensorFlow and 1.4x speedup compared with , a concurrent CUDA implementation. The code will be released publicly after the review.- Anthology ID:
- 2021.naacl-industry.15
- Volume:
- Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Papers
- Month:
- June
- Year:
- 2021
- Address:
- Online
- Editors:
- Young-bum Kim, Yunyao Li, Owen Rambow
- Venue:
- NAACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 113–120
- Language:
- URL:
- https://aclanthology.org/2021.naacl-industry.15
- DOI:
- 10.18653/v1/2021.naacl-industry.15
- Cite (ACL):
- Xiaohui Wang, Ying Xiong, Yang Wei, Mingxuan Wang, and Lei Li. 2021. LightSeq: A High Performance Inference Library for Transformers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Papers, pages 113–120, Online. Association for Computational Linguistics.
- Cite (Informal):
- LightSeq: A High Performance Inference Library for Transformers (Wang et al., NAACL 2021)
- PDF:
- https://preview.aclanthology.org/proper-vol2-ingestion/2021.naacl-industry.15.pdf
- Code
- bytedance/lightseq