@inproceedings{hua-etal-2025-intuitive,
    title = "Intuitive Fine-Tuning: Towards Simplifying Alignment into a Single Process",
    author = "Hua, Ermo  and
      Qi, Biqing  and
      Zhang, Kaiyan  and
      Tian, Kai  and
      Lv, Xingtai  and
      Ding, Ning  and
      Zhou, Bowen",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/acl25-workshop-ingestion/2025.acl-long.6/",
    pages = "121--136",
    ISBN = "979-8-89176-251-0",
    abstract = "Supervised Fine-Tuning (SFT) and Preference Optimization (PO) are key processes for aligning Language Models (LMs) with human preferences post pre-training. While SFT excels in efficiency and PO in effectiveness, they are often combined sequentially without integrating their optimization objectives. This approach ignores the opportunities to bridge their paradigm gap and take the strengths from both. In this paper, we interpret SFT and PO with two sub-processes {---} *Preference Estimation* and *Transition Optimization* {---} defined at token level within the Markov Decision Process (MDP). This modeling shows that SFT is only a special case of PO with inferior estimation and optimization. PO estimates the model{'}s preference by its entire generation, while SFT only scores model{'}s subsequent predicted tokens based on prior tokens from ground truth answer. These priors deviates from model{'}s distribution, hindering the preference estimation and transition optimization. Building on this view, we introduce **Intuitive Fine-Tuning (IFT)** to integrate SFT and PO into a single process. Through a temporal residual connection, IFT brings better estimation and optimization by capturing LMs' intuitive sense of its entire answers. But it solely relies on a single policy and the same volume of non-preference-labeled data as SFT. Our experiments show that IFT performs comparably or even superiorly to SFT and some typical PO methods across several tasks, particularly those requires generation, reasoning, and fact-following abilities. An explainable Frozen Lake game further validates the effectiveness of IFT for getting competitive policy."
}