@inproceedings{wang-etal-2025-data-whisperer,
    title = "Data Whisperer: Efficient Data Selection for Task-Specific {LLM} Fine-Tuning via Few-Shot In-Context Learning",
    author = "Wang, Shaobo  and
      Jin, Xiangqi  and
      Wang, Ziming  and
      Wang, Jize  and
      Zhang, Jiajun  and
      Li, Kaixin  and
      Wen, Zichen  and
      Li, Zhong  and
      He, Conghui  and
      Hu, Xuming  and
      Zhang, Linfeng",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.1135/",
    pages = "23287--23305",
    ISBN = "979-8-89176-251-0",
    abstract = "Fine-tuning large language models (LLMs) on task-specific data is essential for their effective deployment. As dataset sizes grow, efficiently selecting optimal subsets for training becomes crucial to balancing performance and computational costs. Traditional data selection methods often require fine-tuning a scoring model on the target dataset, which is time-consuming and resource-intensive, or rely on heuristics that fail to fully leverage the model{'}s predictive capabilities. To address these challenges, we propose Data Whisperer, an efficient, training-free, attention-based method that leverages few-shot in-context learning with the model to be fine-tuned. Comprehensive evaluations were conducted on both raw and synthetic datasets across diverse tasks and models. Notably, Data Whisperer achieves superior performance compared to the full GSM8K dataset on the Llama-3-8B-Instruct model, using just 10{\%} of the data, and outperforms existing methods with a 3.1-point improvement and a 7.4{\texttimes} speedup."
}