@inproceedings{li-etal-2025-support,
    title = "On Support Samples of Next Word Prediction",
    author = "Li, Yuqian  and
      Du, Yupei  and
      Liu, Yufang  and
      Feng, Feifei  and
      Feng, Mou Xiao  and
      Wu, Yuanbin",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.507/",
    pages = "10277--10289",
    ISBN = "979-8-89176-251-0",
    abstract = "Language models excel in various tasks by making complex decisions, yet understanding the rationale behind these decisions remains a challenge. This paper investigates \textit{data-centric interpretability} in language models, focusing on the next-word prediction task. Using representer theorem, we identify two types of \textit{support samples}{---}those that either promote or deter specific predictions. Our findings reveal that being a support sample is an intrinsic property, predictable even before training begins. Additionally, while non-support samples are less influential in direct predictions, they play a critical role in preventing overfitting and shaping generalization and representation learning. Notably, the importance of non-support samples increases in deeper layers, suggesting their significant role in intermediate representation formation.These insights shed light on the interplay between data and model decisions, offering a new dimension to understanding language model behavior and interpretability."
}