@inproceedings{harada-etal-2025-massive,
    title = "Massive Supervised Fine-tuning Experiments Reveal How Data, Layer, and Training Factors Shape {LLM} Alignment Quality",
    author = "Harada, Yuto  and
      Yamauchi, Yusuke  and
      Oda, Yusuke  and
      Oseki, Yohei  and
      Miyao, Yusuke  and
      Takagi, Yu",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-main.1138/",
    pages = "22371--22392",
    ISBN = "979-8-89176-332-6",
    abstract = "Supervised fine-tuning (SFT) is a critical step in aligning large language models (LLMs) with human instructions and values, yet many aspects of SFT remain poorly understood. We trained a wide range of base models on a variety of datasets including code generation, mathematical reasoning, and general-domain tasks, resulting in 1,000+ SFT models under controlled conditions. We then identified the dataset properties that matter most and examined the layer-wise modifications introduced by SFT.Our findings reveal that some training{--}task synergies persist across all models while others vary substantially, emphasizing the importance of model-specific strategies. Moreover, we demonstrate that perplexity consistently predicts SFT effectiveness, often surpassing superficial similarity between the training data and the benchmark, and that mid-layer weight changes correlate most strongly with performance gains. We release these 1,000+ SFT models and benchmark results to accelerate further research. All resources are available at \url{https://github.com/llm-jp/massive-sft}."
}