@inproceedings{chen-etal-2025-diffpo,
    title = "{D}iff{PO}: Diffusion-styled Preference Optimization for Inference Time Alignment of Large Language Models",
    author = "Chen, Ruizhe  and
      Chai, Wenhao  and
      Yang, Zhifei  and
      Zhang, Xiaotian  and
      Wang, Ziyang  and
      Quek, Tony  and
      Zhou, Joey Tianyi  and
      Poria, Soujanya  and
      Liu, Zuozhu",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.926/",
    pages = "18910--18925",
    ISBN = "979-8-89176-251-0",
    abstract = "Inference-time alignment provides an efficient alternative for aligning LLMs with humans. However, these approaches still face challenges, such as limited scalability due to policy-specific value functions and latency during the inference phase. In this paper, we propose a novel approach, Diffusion-styled Preference Optimization (DiffPO), which provides an efficient and policy-agnostic solution for aligning LLMs with humans. By directly performing alignment at sentence level, DiffPO avoids the time latency associated with token-level generation. Designed as a plug-and-play module, DiffPO can be seamlessly integrated with various base models to enhance their alignment. Extensive experiments on AlpacaEval 2, MT-bench, and HH-RLHF demonstrate that DiffPO achieves superior alignment performance across various settings, achieving a favorable trade-off between alignment quality and inference-time latency. Furthermore, DiffPO demonstrates model-agnostic scalability, significantly improving the performance of large models such as Llama-3-70B."
}