@inproceedings{chen-etal-2025-wavrag,
    title = "{W}av{RAG}: Audio-Integrated Retrieval Augmented Generation for Spoken Dialogue Models",
    author = "Chen, Yifu  and
      Ji, Shengpeng  and
      Wang, Haoxiao  and
      Wang, Ziqing  and
      Chen, Siyu  and
      He, Jinzheng  and
      Xu, Jin  and
      Zhao, Zhou",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.613/",
    pages = "12505--12523",
    ISBN = "979-8-89176-251-0",
    abstract = "Retrieval Augmented Generation (RAG) has gained widespread adoption owing to its capacity to empower large language models (LLMs) to integrate external knowledge. However, existing RAG frameworks are primarily designed for text-based LLMs and rely on Automatic Speech Recognition to process speech input, which discards crucial audio information, risks transcription errors, and increases computational overhead. Therefore, we introduce WavRAG, the first retrieval augmented generation framework with native, end-to-end audio support. WavRAG offers two key features: 1) Bypassing ASR, WavRAG directly processes raw audio for both embedding and retrieval. 2) WavRAG integrates audio and text into a unified knowledge representation. Specifically, we propose the WavRetriever to facilitate the retrieval from a text-audio hybrid knowledge base, and further enhance the in-context capabilities of spoken dialogue models through the integration of chain-of-thought reasoning. In comparison to state-of-the-art ASR-Text RAG pipelines, WavRAG achieves comparable retrieval performance while delivering a 10x acceleration. Furthermore, WavRAG{'}s unique text-audio hybrid retrieval capability extends the boundaries of RAG to the audio modality."
}