@inproceedings{wu-etal-2026-union,
    title = "Union-of-Experts: Neurons in Mixture-of-Experts are Secretly Routers",
    author = "Wu, Songhao  and
      Lv, Ang  and
      Xie, Ruobing  and
      Sun, Samm  and
      Wang, Di  and
      Yan, Rui  and
      Lin, Yankai",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.acl-long.1675/",
    pages = "36193--36206",
    ISBN = "979-8-89176-390-6",
    abstract = "Mixture-of-Experts (MoE) models rely on an external router to assign tokens to experts. This design inherently separates the routing decision from each expert{'}s internal capabilities, leading to suboptimal performance. In this work, we address this limitation with Union-of-Experts (UoE), an MoE variant that performs ``expert-autonomous routing''. The core mechanism of UoE is to pre-designate a minute fraction of neurons within each expert as ``routing neurons''. Experts autonomously select relevant tokens by comparing the activation intensity of these neurons, aligning routing decisions with each expert{'}s functional profile. To prevent the waste of activations from unselected experts' routing neurons, we aggregate all routing neuron outputs and sum them into the final layer output. This aggregation acts as a novel virtual shared expert whose parameters are distributed across the individual experts, and improves overall parameter efficiency. We pre-train UoE models with up to 3B parameters, demonstrating that they outperform traditional MoEs with matched efficiency. Furthermore, our analysis of the routing neurons provides valuable insights into expert-autonomous selection and the broader routing mechanisms of MoE models."
}