@inproceedings{shao-etal-2026-role,
    title = "The Role of Mixed-Language Documents for Multilingual Large Language Model Pretraining",
    author = "Shao, Jiandong  and
      Tang, Raphael  and
      Zhang, Crystina  and
      Sevegnani, Karin  and
      Stenetorp, Pontus  and
      Yang, Jianfei  and
      Lu, Yao",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.acl-long.1706/",
    pages = "36807--36818",
    ISBN = "979-8-89176-390-6",
    abstract = "Multilingual large language models achieve impressive cross-lingual performance despite largely monolingual pretraining. While bilingual data in pretraining corpora is widely believed to enable these abilities, details of its contributions remain unclear. We investigate this question by pretraining models from scratch under controlled conditions, comparing the standard web corpus with a monolingual-only version that removes all multilingual documents. Despite constituting only 2{\%} of the corpus, removing bilingual data causes translation performance to drop 56{\%} in BLEU, while behaviour on cross-lingual QA and general reasoning tasks remains stable, with training curves largely overlapping the baseline. To understand this asymmetry, we categorize bilingual data into parallel (14{\%}), code-switching (72{\%}), and miscellaneous documents (14{\%}) based on the semantic relevance of content in different languages. We then conduct granular ablations by reintroducing parallel or code-switching data into the monolingual-only corpus. Our experiments reveal that parallel data almost fully restores translation performance (91{\%} of the unfiltered baseline), whereas code-switching contributes minimally. Other cross-lingual tasks remain largely unaffected by either type. These findings reveal that translation critically depends on systematic token-level alignments from parallel data, whereas cross-lingual understanding and reasoning appear to be achievable even without bilingual data."
}