@inproceedings{du-etal-2025-disentangling,
    title = "Disentangling the Roles of Representation and Selection in Data Pruning",
    author = "Du, Yupei  and
      Song, Yingjin  and
      Wong, Hugh Mee  and
      Ignatev, Daniil  and
      Gatt, Albert  and
      Nguyen, Dong",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.821/",
    pages = "16791--16809",
    ISBN = "979-8-89176-251-0",
    abstract = "Data pruning{---}selecting small but impactful subsets{---}offers a promising way to efficiently scale NLP model training. However, existing methods often involve many different design choices, which have not been systematically studied. This limits future developments. In this work, we decompose data pruning into two key components: data representation and selection algorithm, and systematically analyze their influence on selected instances. Our theoretical and empirical results highlight the crucial role of representations: better representations, e.g., training gradients, generally lead to better selected instances, regardless of the chosen selection algorithm. Furthermore, different selection algorithms excel in different settings, and none consistently outperform the others. Moreover, the selection algorithms do not always align with their intended objectives: for example, algorithms designed for the same objective can select drastically different instances, highlighting the need for careful evaluation."
}