@inproceedings{liu-etal-2026-modularmoe,
    title = "{M}odular{M}o{E}: Fast {LLM} Customization with Parameter-Sharing Mixture-of-Experts for Low-Resource Settings",
    author = "Liu, Jiaxing  and
      Qi, Qi  and
      Sun, Haifeng  and
      Li, Dunjun  and
      Zhuang, Zirui  and
      He, Bo  and
      Yang, Xiang  and
      Liu, Cong  and
      Liao, Jianxin  and
      Wang, Jingyu",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.174/",
    pages = "3562--3575",
    ISBN = "979-8-89176-395-1",
    abstract = "The massive size of Large Language Models (LLMs) imposes substantial computational and storage burdens, particularly on devices with limited hardware resources. Compared to foundation models, smaller and more specialized models are often more suitable for practical deployment. Existing customization approaches, such as the conventional ``prune-then-finetune'' paradigm or task-agnostic deployment strategies, either incur excessive computational costs or lead to suboptimal task performance. The recently popular Mixture-of-Experts (MoE) architecture exhibits a strong ability to mitigate inter-task interference, offering a new perspective on model deployment. In this paper, we introduce ModularMoE, a training framework that converts pre-trained LLMs into parameter-sharing MoE models for lightweight deployment. Exploiting the emergent modularity within LLMs, we split the feed-forward layers into multiple disjoint modules. Each expert is then constructed as a combination of such modules, enabling knowledge sharing across experts and thereby improving parameter efficiency within MoEs. Extensive experiments across multiple downstream tasks demonstrate that ModularMoE outperforms other state-of-the-art baselines at the same sparsity level, achieving an average performance improvement of 4.10{\%} to 28.75{\%} while delivering up to 2.71{\texttimes} inference speedup."
}