@inproceedings{chang-etal-2024-target,
    title = "Target-Aware Language Modeling via Granular Data Sampling",
    author = "Chang, Ernie  and
      Lin, Pin-Jie  and
      Li, Yang  and
      Zhao, Changsheng  and
      Kim, Daeil  and
      Rabatin, Rastislav  and
      Liu, Zechun  and
      Shi, Yangyang  and
      Chandra, Vikas",
    editor = "Al-Onaizan, Yaser  and
      Bansal, Mohit  and
      Chen, Yun-Nung",
    booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/jlcl-multiple-ingestion/2024.emnlp-main.719/",
    doi = "10.18653/v1/2024.emnlp-main.719",
    pages = "12927--12935",
    abstract = "Language model pretraining generally targets a broad range of use cases and incorporates data from diverse sources. However, there are instances where we desire a model that excels in specific areas without markedly compromising performance in other areas. A cost-effective and straightforward approach is sampling with low-dimensional data features, which allows selecting large-scale pretraining data for domain-specific use cases. In this work, we revisit importance sampling with n-gram features consisting of multi-granular tokens, which strikes a good balance between sentence compression and representation capabilities. We observed the sampled data to have a high correlation with the target downstream task performance *while preserving its effectiveness on other tasks*. This leads to the proposed data sampling paradigm where language models can be pretrained more efficiently on selected documents. On eight benchmarks we demonstrate with {\textasciitilde}1{\%} of the data, pretrained models perform on par with the full RefinedWeb data and outperform randomly selected samples for model sizes ranging from 125M to 1.5B."
}