@inproceedings{xiao-etal-2025-whisper,
    title = "Whisper-{UT}: A Unified Translation Framework for Speech and Text",
    author = "Xiao, Cihan  and
      Wiesner, Matthew  and
      Chakraborty, Debashish  and
      Kriz, Reno  and
      Cunningham, Keith  and
      Murray, Kenton  and
      Duh, Kevin  and
      Tavarez-Arce, Luis  and
      McNamee, Paul  and
      Khudanpur, Sanjeev",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-main.148/",
    pages = "3000--3016",
    ISBN = "979-8-89176-332-6",
    abstract = "Encoder-decoder models have achieved remarkable success in speech and text tasks, yet efficiently adapting these models to diverse uni/multi-modal scenarios remains an open challenge. In this paper, we propose Whisper-UT, a unified and efficient framework that leverages lightweight adapters to enable seamless adaptation across tasks, including a multi-modal machine translation (MMT) task that explicitly conditions translation on both speech and source language text inputs. By incorporating ASR hypotheses or ground-truth transcripts as prompts, this approach not only enables the system to process both modalities simultaneously but also enhances speech translation (ST) performance through a 2-stage decoding strategy. We demonstrate our methods using the Whisper model, though in principle they are general and could be applied to similar multitask models. We highlight the effectiveness of cross-modal and cross-task fine-tuning, which improves performance without requiring 3-way parallel data. Our results underscore the flexibility, efficiency, and general applicability of the proposed framework for multi-modal translation."
}