@inproceedings{ali-etal-2025-judging,
    title = "Judging Quality Across Languages: A Multilingual Approach to Pretraining Data Filtering with Language Models",
    author = {Ali, Mehdi  and
      Brack, Manuel  and
      L{\"u}bbering, Max  and
      Wendt, Elias  and
      Khan, Abbas Goher  and
      Rutmann, Richard  and
      Jude, Alex  and
      Kraus, Maurice  and
      Weber, Alexander Arno  and
      Stollenwerk, Felix  and
      Kacz{\'e}r, David  and
      Mai, Florian  and
      Flek, Lucie  and
      Sifa, Rafet  and
      Flores-Herr, Nicolas  and
      Koehler, Joachim  and
      Schramowski, Patrick  and
      Fromm, Michael  and
      Kersting, Kristian},
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-main.449/",
    pages = "8870--8909",
    ISBN = "979-8-89176-332-6",
    abstract = "High-quality multilingual training data is essential for effectively pretraining large language models (LLMs). Yet, the availability of suitable open-source multilingual datasets remains limited. Existing state-of-the-art datasets mostly rely on heuristic filtering methods, restricting both their cross-lingual transferability and scalability. Here, we introduce JQL, a systematic approach that efficiently curates diverse and high-quality multilingual data at scale while significantly reducing computational demands. JQL distills LLMs' annotation capabilities into lightweight annotators based on pretrained multilingual embeddings. These models exhibit robust multilingual and cross-lingual performance, even for languages and scripts unseen during training. Evaluated empirically across 35 languages, the resulting annotation pipeline substantially outperforms current heuristic filtering methods like Fineweb2. JQL notably enhances downstream model training quality and increases data retention rates. Our research provides practical insights and valuable resources for multilingual data curation, raising the standards of multilingual dataset development."
}