@inproceedings{ying-etal-2025-data,
    title = "Data-Efficient Selection via Grammatical Complexity in Continual Pre-training of Domain-Specific {LLM}s",
    author = "Ying, Yizhou  and
      Zhang, Geng  and
      Danxin, Cui  and
      Du, Chengyu  and
      Yue, Guanglei  and
      Jiang, Sihang  and
      Liang, Jiaqing  and
      Fu, Yifei  and
      Hu, Hailin  and
      Xiao, Yanghua",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-luhme/2025.emnlp-main.1121/",
    doi = "10.18653/v1/2025.emnlp-main.1121",
    pages = "22066--22080",
    ISBN = "979-8-89176-332-6",
    abstract = "Data efficiency is crucial in domain-specific continual pre-training (CPT) of large language models (LLMs), especially under resource constraints. Aiming for ``small data, big impact,'' this work addresses the limitations of existing domain-specific data selection strategies, which often rely on scarce labeled data or computationally expensive LLMs. We introduce CDF Sampling with Grammatical Complexity (CDF-GC), an annotation-independent, efficient and interpretable data selection framework for CPT. Our approach comprehensively evaluates grammatical complexity using lexical diversity and syntactic complexity, and employs a cumulative distribution function (CDF)-based sampling strategy to balance complexity and diversity. To validate the effectiveness of CDF-GC, we conducted experiments on a financial dataset. The results demonstrate that CDF-GC significantly outperforms baselines, achieving 2.0{\%} improvement in financial QA at the same selection ratio and even surpassing full-data training by 1.7{\%} using only 20{\%} of the data."
}