@inproceedings{shi-etal-2025-dids,
    title = "{DIDS}: Domain Impact-aware Data Sampling for Large Language Model Training",
    author = "Shi, Weijie  and
      Zhang, Jipeng  and
      Wu, Yaguang  and
      Fang, Jingzhi  and
      Zhang, Shibo  and
      Zhao, Yao  and
      Chen, Hao  and
      Zhang, Ruiyuan  and
      Cui, Yue  and
      Zhu, Jia  and
      Han, Sirui  and
      Xu, Jiajie  and
      Zhou, Xiaofang",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-main.215/",
    pages = "4330--4350",
    ISBN = "979-8-89176-332-6",
    abstract = "Large language models (LLMs) are commonly trained on multi-domain datasets, where domain sampling strategies significantly impact model performance due to varying domain importance across downstream tasks. Existing approaches for optimizing domain-level sampling strategies struggle with maintaining intra-domain consistency and accurately measuring domain impact. In this paper, we present Domain Impact-aware Data Sampling (DIDS). To ensure intra-domain consistency, a gradient clustering algorithm is proposed to group training data based on their learning effects, where a proxy language model and dimensionality reduction are employed to reduce computational overhead. To accurately measure domain impact, we develop a Fisher Information Matrix (FIM) guided metric that quantifies how domain-specific parameter updates affect the model{'}s output distributions on downstream tasks, with theoretical guarantees. Furthermore, to determine optimal sampling ratios, DIDS combines both the FIM-guided domain impact assessment and loss learning trajectories that indicate domain-specific potential, while accounting for diminishing marginal returns. Extensive experiments demonstrate that DIDS achieves 3.4{\%} higher average performance while maintaining comparable training efficiency. The code is available at https://github.com/shiweijiezero/DIDS."
}