@inproceedings{mao-etal-2024-dont,
    title = "Don`t Forget Your Reward Values: Language Model Alignment via Value-based Calibration",
    author = "Mao, Xin  and
      Li, Feng-Lin  and
      Xu, Huimin  and
      Zhang, Wei  and
      Chen, Wang  and
      Luu, Anh Tuan",
    editor = "Al-Onaizan, Yaser  and
      Bansal, Mohit  and
      Chen, Yun-Nung",
    booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/add-emnlp-2024-awards/2024.emnlp-main.976/",
    doi = "10.18653/v1/2024.emnlp-main.976",
    pages = "17622--17642",
    abstract = "While Reinforcement Learning from Human Feedback (RLHF) significantly enhances the generation quality of Large Language Models (LLMs), recent studies have raised concerns regarding the complexity and instability associated with the Proximal Policy Optimization (PPO) algorithm, proposing a series of order-based alignment methods as viable alternatives. This paper delves into existing order-based methods, unifying them into one framework and examining their inefficiencies in utilizing reward values. Building upon these findings, we propose a new Value-based Calibration (VCB) method to better align LLMs with human preferences. Experimental results demonstrate that VCB surpasses existing alignment methods on AI assistant and summarization datasets, providing impressive generalizability, robustness, and diversity in different settings."
}