@inproceedings{hsu-etal-2025-lets,
    title = "Let{'}s Fuse Step by Step: A Generative Fusion Decoding Algorithm with {LLM}s for Robust and Instruction-Aware {ASR} and {OCR}",
    author = "Hsu, Chan-Jan  and
      Chen, Yi-Chang  and
      Liao, Feng-Ting  and
      Ho, Pei-Chen  and
      Wang, Yu-Hsiang  and
      Hsu, Po-Chun  and
      Shiu, Da-shan",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/display_plenaries/2025.findings-acl.1281/",
    pages = "24959--24973",
    ISBN = "979-8-89176-256-5",
    abstract = "We introduce ``Generative Fusion Decoding'' (GFD), a novel shallow fusion framework, utilized to integrate large language models(LLMs) into cross-modal text recognition systems inlculding automatic speech recognition (ASR) and optical character recognition (OCR). We derive the formulas necessary to enable GFD to operate across mismatched token spaces of different models by calculating likelihood at the byte level, thereby enabling seamless fusion and synchronous progression during the decoding process. GFD is plug-and-play bydesign, making it readily compatible with various auto-regressive models without the need for any re-training. GFD proves effective for general ASR and OCR tasks through intermediate and frequent interactions with LLMs, surpassing cascaded methods in English and Mandarin benchmarks. In addition, GFD transfers in-context learning abilities of LLMs and allows for adaptive ASR in instruction-aware andlong-context settings, yielding significant WER reductions of up to 17.7{\%}."
}