Abstract
We propose an efficient batching strategy for variable-length decoding on GPU architectures. During decoding, when candidates terminate or are pruned according to heuristics, our streaming approach periodically “refills” the batch before proceeding with a selected subset of candidates. We apply our method to variable-width beam search on a state-of-the-art machine translation model. Our method decreases runtime by up to 71% compared to a fixed-width beam search baseline and 17% compared to a variable-width baseline, while matching baselines’ BLEU. Finally, experiments show that our method can speed up decoding in other domains, such as semantic and syntactic parsing.- Anthology ID:
- 2020.emnlp-main.366
- Volume:
- Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
- Month:
- November
- Year:
- 2020
- Address:
- Online
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 4526–4535
- Language:
- URL:
- https://aclanthology.org/2020.emnlp-main.366
- DOI:
- 10.18653/v1/2020.emnlp-main.366
- Cite (ACL):
- Kevin Yang, Violet Yao, John DeNero, and Dan Klein. 2020. A Streaming Approach For Efficient Batched Beam Search. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4526–4535, Online. Association for Computational Linguistics.
- Cite (Informal):
- A Streaming Approach For Efficient Batched Beam Search (Yang et al., EMNLP 2020)
- PDF:
- https://preview.aclanthology.org/paclic-22-ingestion/2020.emnlp-main.366.pdf
- Code
- yangkevin2/emnlp2020-stream-beam-mt
- Data
- Penn Treebank