@inproceedings{zhang-etal-2025-autonomous,
    title = "Autonomous Data Selection with Zero-shot Generative Classifiers for Mathematical Texts",
    author = "Zhang, Yifan  and
      Luo, Yifan  and
      Yuan, Yang  and
      Yao, Andrew C",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/landing_page/2025.findings-acl.216/",
    pages = "4168--4189",
    ISBN = "979-8-89176-256-5",
    abstract = "We present Autonomous Data Selection (AutoDS), a method that leverages base language models as zero-shot ``generative classifiers'' to automatically curate high-quality mathematical texts. Unlike prior approaches that require human annotations or training a dedicated data filter, AutoDS relies solely on a model{'}s logits to determine whether a given passage is mathematically informative and educational. By integrating AutoDS into a continual pretraining pipeline, we substantially boost downstream performance on challenging math benchmarks (MATH, GSM8K, and BBH) while using far fewer tokens than previous methods. Empirically, our approach achieves roughly a twofold improvement in pretraining token efficiency over strong baselines, underscoring the potential of self-directed data selection in enhancing mathematical reasoning. We will release our curated dataset to facilitate future research in automated domain-specific data curation."
}