@article{oepen-etal-2026-large,
    title = "Very Large-Scale Multilingual Resources for {LLM}s and {MT}. Mono- and Bi-lingual Data, Multilingual Evaluation, and Pre-Trained Models",
    author = "Oepen, Stephan  and
      Arefyev, Nikolay  and
      Aulamo, Mikko  and
      Ba{\~n}{\'o}n, Marta  and
      Buljan, Maja  and
      Burchell, Laurie V.  and
      Charpentier, Lucas Georges Gabriel  and
      Chen, Pinzhen  and
      Fedorova, Mariia  and
      de Gibert, Ona  and
      Haddow, Barry  and
      Haji{\v{c}}, Jan  and
      Helcl, Jindrich  and
      Kutuzov, Andrey  and
      Laippala, Veronika  and
      Li, Zihao  and
      Malik, Bhavitvya  and
      Mikhailov, Vladislav  and
      Myntti, Amanda  and
      O'Brien, Dayy{\'a}n  and
      Polakova, Lucie  and
      Ram{\'i}rez-S{\'a}nchez, Gema  and
      Siewert, Janine  and
      Stepachev, Pavel  and
      Tiedemann, Joerg  and
      Vahtola, Teemu  and
      Varis, Dusan  and
      Vitiugin, Fedor  and
      Zaragoza, Jaume",
    editor = "Piperidis, Stelios  and
      Bel, N{\'u}ria  and
      van den Heuvel, Henk  and
      Ide, Nancy  and
      Krek, Simon  and
      Toral, Antonio",
    journal = "International Conference on Language Resources and Evaluation",
    volume = "main",
    month = may,
    year = "2026",
    address = "Palma de Mallorca, Spain",
    publisher = "ELRA Language Resource Association",
    url = "https://preview.aclanthology.org/ingest-lrec/2026.lrec-main.110/",
    pages = "1409--1434",
    abstract = "We present an ongoing initiative to provide open, very large, high-quality, and richly annotated textual datasets for almost 200 languages. At 30 trillion tokens, this is likely the largest generally available multilingual collection of LLM pre-training data. These datasets are derived from web crawls from different sources and accompanied with a complete, open-source pipeline for document selection from web archives, text extraction from HTML, language identification for noisy texts, exact and near-deduplication, annotation with, among others, register labels, text quality estimates, and personally identifiable information; and final selection and filtering. We report on data quality probes through contrastive and analytical statistics, through manual inspection of samples for some 20 languages, and through end-to-end evaluation of various language model architectures trained on this data. For multilingual LLM evaluation, we provide a comprehensive collection of benchmarks for nine European languages, with special emphasis on natively created tasks, mechanisms to mitigate prompt sensitivity, and refined normalization and aggregation of scores. Additionally, we train and evaluate a family of 57 monolingual encoder{--}decoder models, as well as about 30 ``smallish'' monolingual GPT-like reference models. Besides the monolingual data and models, we also present a very large collection of parallel texts automatically mined from this data, together with a novel parallel corpus synthesized via machine translation."
}