@inproceedings{yang-etal-2020-streaming,
    title = "A Streaming Approach For Efficient Batched Beam Search",
    author = "Yang, Kevin  and
      Yao, Violet  and
      DeNero, John  and
      Klein, Dan",
    editor = "Webber, Bonnie  and
      Cohn, Trevor  and
      He, Yulan  and
      Liu, Yang",
    booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)",
    month = nov,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/jlcl-multiple-ingestion/2020.emnlp-main.366/",
    doi = "10.18653/v1/2020.emnlp-main.366",
    pages = "4526--4535",
    abstract = "We propose an efficient batching strategy for variable-length decoding on GPU architectures. During decoding, when candidates terminate or are pruned according to heuristics, our streaming approach periodically {\textquotedblleft}refills{\textquotedblright} the batch before proceeding with a selected subset of candidates. We apply our method to variable-width beam search on a state-of-the-art machine translation model. Our method decreases runtime by up to 71{\%} compared to a fixed-width beam search baseline and 17{\%} compared to a variable-width baseline, while matching baselines' BLEU. Finally, experiments show that our method can speed up decoding in other domains, such as semantic and syntactic parsing."
}