@inproceedings{janeiro-etal-2025-mixture,
    title = "Mixture of Languages: Improved Multilingual Encoders Through Language Grouping",
    author = "Janeiro, Jo{\~a}o Maria  and
      Alastruey, Belen  and
      Massa, Francisco  and
      Elbayad, Maha  and
      Piwowarski, Benjamin  and
      Gallinari, Patrick  and
      Barrault, Loic",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-main.1509/",
    pages = "29695--29710",
    ISBN = "979-8-89176-332-6",
    abstract = "We propose Mixture of Languages (MoL), a new strategy to pretrain largely multilingual encoders. Recent work in this field has relied on training transformer encoders on a large amount of multilingual data, with all parameters shared across all languages, without studying how to optimally balance language transfer and interference to achieve better performance. To address this, MoL proposes to group languages based on their similarity, and add parallel, sparsely activated layers that process each group independently. This architecture allows MoL to boost language transfer while minimizing interference, without increasing the active parameter count. We show that MoL largely outperforms a dense counterpart trained with the same configuration, as well as MoE models and public multilingual encoders such as XLM-R or mBERT on downstream tasks."
}