@inproceedings{ke-etal-2026-outcome,
    title = "From Outcome to Process: Optimizing {M}o{E} Load Balancing with {MCTS}",
    author = "Ke, Wenjun  and
      Xu, Hengyuan  and
      Shang, Ziyu  and
      He, Yao  and
      Wang, Jiahao  and
      Xu, Zijie  and
      Wang, Peng  and
      Lou, Yuhang  and
      Liu, Jiajun",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.1440/",
    pages = "28831--28848",
    ISBN = "979-8-89176-395-1",
    abstract = "Mixture of Experts (MoE) dynamically routes inputs to specialized expert networks, enabling large language models to scale capacity with low inference overhead. To further improve MoE{'}s parameter efficiency in resource-constrained scenarios, LoRA{--}MoE integrates LoRA for lightweight adaptation while preserving MoE{'}s specialization. Despite these benefits, the effectiveness of LoRA{--}MoE still hinges on balanced expert utilization, where certain experts dominate activations while most remain underutilized. Existing balancing strategies focus on constraining the final distribution of expert usage, but overlook the routing decisions made at each layer. As a result, imbalances gradually accumulate across the routing hierarchy. To address this challenge, we propose LayerMoE, a novel three-stage framework that leverages process-level rewards to guide balanced expert routing. Specifically, to overcome the limitation of focusing only on final losses and ignoring intermediate routing, we introduce Monte Carlo Tree Search (MCTS)-based sampling that decomposes outcome-level supervision into layer-wise reward signals, guiding expert choices throughout the routing process. For efficiency, we organize Transformer layers into groups, which constrain the search space of MCTS and keep exploration overhead tractable while retaining the hierarchical structure. Extensive experiments on representative datasets (e.g., ARC, RACE, OBQA) show that applying LayerMoE consistently improves the performance of state-of-the-art LoRA-MoE baselines, yielding an average accuracy gain of 1.39{\%}. Notably, the maximum improvement reaches 2.50{\%}."
}