@inproceedings{ji-etal-2026-data,
    title = "Data-Centric Continual Pre-training for 500+ Languages: A New Bilingual Translation Corpus and Multilingual Models",
    author = "Ji, Shaoxiong  and
      Li, Zihao  and
      Paavola, Jaakko  and
      Luo, Hengyu  and
      Tiedemann, J{\textbackslash}{''}org",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.937/",
    pages = "18776--18807",
    ISBN = "979-8-89176-395-1",
    abstract = "This paper investigates a critical design decision in the practice of massively multilingual continual pre-training {---} the inclusion of parallel data. Specifically, we study the impact of bilingual translation data for massively multilingual language adaptation of the Llama 3 family of models to 500 languages. To this end, we construct a bilingual translation corpus named OUR{\_}DATA, containing data from more than 2,500 language pairs. Subsequently, we develop the OUR{\_}MODEL Llama 3 suite of four massively multilingual models {---} continually pre-trained from the Llama 3 family of base models extensively on diverse data mixes up to 671B tokens {---} and explore the effect of continual pre-training with or without bilingual translation data. Comprehensive evaluation across 7 tasks and 12 benchmarks demonstrates that bilingual data tends to enhance language transfer and performance, particularly for low-resource languages. We open-source the OUR{\_}DATA corpus, OUR{\_}MODEL Llama 3 suite artefacts, code, and model generations."
}