@inproceedings{zhu-etal-2024-llama,
    title = "{LL}a{MA}-{M}o{E}: Building Mixture-of-Experts from {LL}a{MA} with Continual Pre-Training",
    author = "Zhu, Tong  and
      Qu, Xiaoye  and
      Dong, Daize  and
      Ruan, Jiacheng  and
      Tong, Jingqi  and
      He, Conghui  and
      Cheng, Yu",
    editor = "Al-Onaizan, Yaser  and
      Bansal, Mohit  and
      Chen, Yun-Nung",
    booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/landing_page/2024.emnlp-main.890/",
    doi = "10.18653/v1/2024.emnlp-main.890",
    pages = "15913--15923",
    abstract = "Mixture-of-Experts (MoE) has gained increasing popularity as a promising framework for scaling up large language models (LLMs). However, training MoE from scratch in a large-scale setting still suffers from data-hungry and instability problems. Motivated by this limit, we investigate building MoE models from existing dense large language models. Specifically, based on the well-known LLaMA-2 7B model, we obtain an MoE model by: (1) Expert Construction, which partitions the parameters of original Feed-Forward Networks (FFNs) into multiple experts; (2) Continual pre-training, which further trains the transformed MoE model and additional gate networks. In this paper, we comprehensively explore different methods for expert construction and various data sampling strategies for continual pre-training. After these stages, our LLaMA-MoE models could maintain language abilities and route the input tokens to specific experts with part of the parameters activated. Empirically, by training 200B tokens, LLaMA-MoE-3.5B models significantly outperform dense models that contain similar activation parameters."
}