@inproceedings{qiu-etal-2025-treebon,
    title = "{T}ree{B}o{N}: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Sampling",
    author = "Qiu, Jiahao  and
      Lu, Yifu  and
      Zeng, Yifan  and
      Guo, Jiacheng  and
      Geng, Jiayi  and
      Zhu, Chenhao  and
      Juan, Xinzhe  and
      Yang, Ling  and
      Wang, Huazheng  and
      Huang, Kaixuan  and
      Wu, Yue  and
      Wang, Mengdi",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2025",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/author-page-yu-wang-polytechnic/2025.findings-emnlp.1140/",
    doi = "10.18653/v1/2025.findings-emnlp.1140",
    pages = "20894--20917",
    ISBN = "979-8-89176-335-7",
    abstract = "Inference-time alignment enhances the performance of large language models without requiring additional training or fine-tuning but presents challenges due to balancing computational efficiency with high-quality output. Best-of-N (BoN) sampling, as a simple yet powerful approach, generates multiple responses and selects the best one, achieving improved performance but with a high computational cost. We propose TreeBoN, a novel framework that integrates a speculative tree-search strategy into Best-of-N (BoN) Sampling. TreeBoN maintains a set of parent nodes, iteratively branching and pruning low-quality responses, thereby reducing computational overhead while maintaining high output quality. Our approach also leverages token-level rewards from Direct Preference Optimization (DPO) to guide tree expansion and prune low-quality paths. We evaluate TreeBoN using AlpacaFarm, UltraFeedback, GSM8K, HH-RLHF, and TutorEval datasets, demonstrating consistent improvements. Specifically, TreeBoN achieves a 65{\%} win rate at maximum lengths of 192 and 384 tokens, outperforming standard BoN with the same computational cost. Furthermore, TreeBoN achieves around a 60{\%} win rate across longer responses, showcasing its scalability and alignment efficacy."
}