@inproceedings{li-etal-2026-select,
    title = "Select Before Use: On the Importance of Reference Model Selection in Preference Alignment",
    author = "Li, Muyang  and
      Wu, Runze  and
      Zhao, Xiangyu  and
      Han, Bo  and
      Dong, Daoyi  and
      Liu, Tongliang",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.acl-long.780/",
    pages = "17151--17171",
    ISBN = "979-8-89176-390-6",
    abstract = "The post-training stage of Large Language Models (LLMs) typically involves Supervised Fine-Tuning (SFT) followed by preference alignment to ensure LLM to generate safe, helpful, and instruction-aligned content. The SFT model critically serves as both the initialization and reference model for subsequent preference alignment. However, an essential yet often neglected question is the optimal selection of the SFT checkpoint for this role. We show that checkpoint selection substantially affects final performance, and that the common practice of choosing the minimum validation-loss checkpoint often fails, due to a fundamental conflict between SFT{'}s focus on imitation and alignment{'}s goal of response discriminability. To this end, we propose RewardRank, a simple, effective, training-free metrics for estimating initial implicit alignment between reference model and preference objective. Empirical evidence suggests that, using our selected model as reference can gain up to 67.6{\%} relative increase on length-controlled win rate on the popular Zephyr recipe comparing to baselines."
}