Abstract
Graphics Processing Units (GPUs) are commonly used to train and evaluate neural networks efficiently. While previous work in deep learning has focused on accelerating operations on dense matrices/tensors on GPUs, efforts have concentrated on operations involving sparse data structures. Operations using sparse structures are common in natural language models at the input and output layers, because these models operate on sequences over discrete alphabets. We present two new GPU algorithms: one at the input layer, for multiplying a matrix by a few-hot vector (generalizing the more common operation of multiplication by a one-hot vector) and one at the output layer, for a fused softmax and top-N selection (commonly used in beam search). Our methods achieve speedups over state-of-the-art parallel GPU baselines of up to 7x and 50x, respectively. We also illustrate how our methods scale on different GPU architectures.- Anthology ID:
- P19-1626
- Volume:
- Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
- Month:
- July
- Year:
- 2019
- Address:
- Florence, Italy
- Editors:
- Anna Korhonen, David Traum, Lluís Màrquez
- Venue:
- ACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 6215–6224
- Language:
- URL:
- https://aclanthology.org/P19-1626
- DOI:
- 10.18653/v1/P19-1626
- Cite (ACL):
- Arturo Argueta and David Chiang. 2019. Accelerating Sparse Matrix Operations in Neural Networks on Graphics Processing Units. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6215–6224, Florence, Italy. Association for Computational Linguistics.
- Cite (Informal):
- Accelerating Sparse Matrix Operations in Neural Networks on Graphics Processing Units (Argueta & Chiang, ACL 2019)
- PDF:
- https://preview.aclanthology.org/ingest-acl-2023-videos/P19-1626.pdf