@inproceedings{seto-etal-2025-assessing,
    title = "Assessing the Role of Data Quality in Training Bilingual Language Models",
    author = "Seto, Skyler  and
      Ter Hoeve, Maartje  and
      de Seyssel, Maureen  and
      Grangier, David",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2025",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/name-variant-enfa-fane/2025.findings-emnlp.1236/",
    doi = "10.18653/v1/2025.findings-emnlp.1236",
    pages = "22694--22720",
    ISBN = "979-8-89176-335-7",
    abstract = "Bilingual and multilingual language models offer a promising path toward scaling NLP systems across diverse languages and users. However, their performance often varies wildly between languages as prior works show that adding more languages can degrade performance for some languages (such as English), while improving others (typically more data constrained languages). In this work, we investigate causes of these inconsistencies by comparing bilingual and monolingual language models. Our analysis reveals that unequal data quality, not just data quantity, is a major driver of performance degradation in bilingual settings. We propose a simple yet effective data filtering strategy to select higher-quality bilingual training data with only high quality English data. Applied to French, German, and Chinese, our approach improves monolingual performance by 2{--}4{\%} and reduces bilingual model performance gaps to 1{\%}. These results highlight the overlooked importance of data quality in multilingual pretraining and offer a practical recipe for balancing performance."
}