@inproceedings{aycock-etal-2026-limits,
    title = "On the Limits of Model Merging for Multilinguality in Pre-Training",
    author = "Aycock, Seth  and
      Vitiugin, Fedor  and
      Umnov, Aleksandr  and
      Monz, Christof  and
      Sima{'}an, Khalil",
    editor = "Huang, Kaiyu  and
      Mo, Fengran  and
      Chen, Pinzhen  and
      Jiang, Meng",
    booktitle = "Proceedings of the 1st Workshop on Multilinguality in the Era of Large Language Models ({M}e{LLM} 2026)",
    month = jul,
    year = "2026",
    address = "San Diego, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl-workshops/2026.mellm-1.15/",
    pages = "159--169",
    ISBN = "979-8-89176-430-9",
    abstract = "Endowing models with consistent multilingual performance can be achieved by {\_}mixing{\_} pre-training data, or post-training approaches such as language-specific model {\_}merging{\_}. In this work, we test whether merging can be applied to monolingually pre-trained models. We conduct a controlled study on the efficacy of mixed, merged, and monolingual pre-training setups. We find that while monolingual pre-training results in strong in-language performance, merging any combination of monolingual models leads to performance collapse due to interference. Our analysis suggests representational similarity is a prerequisite for model merging. We therefore conclude that the flexibility of merging in fine-tuning does not extend trivially to language-specific pre-training."
}