@inproceedings{zhang-etal-2025-bts,
    title = "{BTS}: Harmonizing Specialized Experts into a Generalist {LLM}",
    author = "Zhang, Qizhen  and
      Bhargava, Prajjwal  and
      Bi, Chloe  and
      Cai, Chris X.  and
      Foerster, Jakob Nicolaus  and
      Fu, Jeremy  and
      Koura, Punit Singh  and
      Silva, Ruan  and
      Shen, Sheng  and
      Dinan, Emily  and
      Gururangan, Suchin  and
      Lewis, Mike",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-main.347/",
    pages = "6827--6845",
    ISBN = "979-8-89176-332-6",
    abstract = "We present Branch-Train-Stitch (BTS), an efficient and flexible training algorithm for combining independently trained large language model (LLM) experts into a single, capable generalist model. Following Li et al., we start with a single seed language model which is branched into domain-specific (e.g., coding or math) experts with continual pretraining. BTS combines experts into a generalist model using lightweight stitch layers, which are inserted between frozen experts and the seed LLM, and trained on a small datamix of the expert domains. Stitch layers enable the seed LLM to integrate representations from any number of experts during the forward pass, allowing it to generalize to new domains, despite remaining frozen. Because BTS does not alter the constituent LLMs, BTS provides a modular and flexible approach: experts can be easily removed and new experts can be added with only a small amount of training. Compared to alternative model merging approaches, BTS yields the best generalist performance on a variety of downstream tasks, retaining the specialized capabilities of each of the experts."
}