@inproceedings{dent-etal-2025-identifying,
    title = "Identifying Rare Languages in {C}ommon {C}rawl Data is a Needles-in-a-Haystack Problem",
    author = "Dent, Rasul  and
      Ortiz Suarez, Pedro  and
      Cl{\'e}rice, Thibault  and
      Sagot, Beno{\^i}t",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2025",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/name-variant-v-g-vinod-vydiswaran/2025.findings-emnlp.77/",
    doi = "10.18653/v1/2025.findings-emnlp.77",
    pages = "1460--1473",
    ISBN = "979-8-89176-335-7",
    abstract = "Automatic language identification is frequently framed as a multi-class classification problem. However, when creating digital corpora for less commonly written languages, it may be more appropriate to consider it a data mining problem. For these varieties, one knows ahead of time that the vast majority of documents are of little interest. By minimizing resources spent on classifying such documents, we can create corpora covering previously overlooked languages faster than existing pipelines. To demonstrate the effectiveness of the targeted mining perspective, we introduce a new pipeline that can filter a single snapshot in two hours. We also provide web corpora for several French-based Creoles."
}