@inproceedings{shan-etal-2025-enhancing,
    title = "Enhancing Speech Large Language Models with Prompt-Aware Mixture of Audio Encoders",
    author = "Shan, Weiqiao  and
      Li, Yuang  and
      Zhang, Yuhao  and
      Luo, Yingfeng  and
      Xu, Chen  and
      Zhao, Xiaofeng  and
      Meng, Long  and
      Lu, Yunfei  and
      Zhang, Min  and
      Yang, Hao  and
      Xiao, Tong  and
      Zhu, JingBo",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-main.974/",
    pages = "19316--19331",
    ISBN = "979-8-89176-332-6",
    abstract = "Connecting audio encoders with large language models (LLMs) allows the LLM to perform various audio understanding tasks, such as automatic speech recognition (ASR) and audio captioning (AC). Most research focuses on training an adapter layer to generate a unified audio feature for the LLM. However, different tasks may require distinct features that emphasize either semantic or acoustic aspects, making task-specific audio features more desirable. In this paper, we propose Prompt-aware Mixture (PaM) to enhance the Speech LLM that uses multiple audio encoders. Our approach involves using different experts to extract different features based on the prompt that indicates different tasks. Experiments demonstrate that with PaM, only one Speech LLM surpasses the best performances achieved by all single-encoder Speech LLMs on ASR, speaker number verification, and AC tasks. PaM also outperforms other feature fusion baselines, such as concatenation and averaging."
}