@inproceedings{lei-etal-2026-moe,
    title = "{M}o{E} Adapter for Large Audio Language Models: Sparsity, Disentanglement, and Gradient-Conflict-Free",
    author = "Lei, Yishu  and
      He, Shuwei  and
      Jing, Hu  and
      Zhang, Dan  and
      Luo, Xianlong  and
      Zhu, Danxiang  and
      Feng, Shikun  and
      Liu, Rui  and
      HE, Jingzhou  and
      Sun, Yu  and
      Wu, Hua  and
      Wang, Haifeng",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.840/",
    pages = "17039--17050",
    ISBN = "979-8-89176-395-1",
    abstract = "Extending the input modality of Large Language Models (LLMs) to the audio domain is essential for achieving comprehensive multimodal perception. However, it is well-known that acoustic information is intrinsically heterogeneous, entangling attributes such as speech, music, and environmental context. Existing research is limited to a dense, parameter-shared adapter to model these diverse patterns, which induces gradient conflict during optimization, as parameter updates required for distinct attributes contradict each other. To address this limitation, we introduce the MoE-Adapter, a sparse Mixture-of-Experts (MoE) architecture designed to decouple acoustic information. Specifically, it employs a dynamic gating mechanism that routes audio tokens to specialized experts capturing complementary feature subspaces while retaining shared experts for global context, thereby mitigating gradient conflicts and enabling fine-grained feature learning. Comprehensive experiments show that the MoE-Adapter achieves superior performance on both audio semantic and paralinguistic tasks, consistently outperforming dense linear baselines with comparable computational costs. To facilitate future research, our code are publicly available at https://github.com/Alittleegg/Eureka-Audio."
}