Towards a Cleaner Document-Oriented Multilingual Crawled Corpus
Julien Abadji, Pedro Ortiz Suarez, Laurent Romary, Benoît Sagot
Abstract
The need for large corpora raw corpora has dramatically increased in recent years with the introduction of transfer learning and semi-supervised learning methods to Natural Language Processing. And while there have been some recent attempts to manually curate the amount of data necessary to train large language models, the main way to obtain this data is still through automatic web crawling. In this paper we take the existing multilingual web corpus OSCAR and its pipeline Ungoliant that extracts and classifies data from Common Crawl at the line level, and propose a set of improvements and automatic annotations in order to produce a new document-oriented version of OSCAR that could prove more suitable to pre-train large generative language models as well as hopefully other applications in Natural Language Processing and Digital Humanities.- Anthology ID:
- 2022.lrec-1.463
- Volume:
- Proceedings of the Thirteenth Language Resources and Evaluation Conference
- Month:
- June
- Year:
- 2022
- Address:
- Marseille, France
- Editors:
- Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Jan Odijk, Stelios Piperidis
- Venue:
- LREC
- SIG:
- Publisher:
- European Language Resources Association
- Note:
- Pages:
- 4344–4355
- Language:
- URL:
- https://aclanthology.org/2022.lrec-1.463
- DOI:
- Cite (ACL):
- Julien Abadji, Pedro Ortiz Suarez, Laurent Romary, and Benoît Sagot. 2022. Towards a Cleaner Document-Oriented Multilingual Crawled Corpus. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 4344–4355, Marseille, France. European Language Resources Association.
- Cite (Informal):
- Towards a Cleaner Document-Oriented Multilingual Crawled Corpus (Abadji et al., LREC 2022)
- PDF:
- https://preview.aclanthology.org/proper-vol2-ingestion/2022.lrec-1.463.pdf
- Data
- C4, CCNet, OSCAR