@inproceedings{hu-etal-2025-openrlhf,
    title = "{O}pen{RLHF}: A Ray-based Easy-to-use, Scalable and High-performance {RLHF} Framework",
    author = "Hu, Jian  and
      Wu, Xibin  and
      Shen, Wei  and
      Liu, Jason Klein  and
      Wang, Weixun  and
      Jiang, Songlin  and
      Wang, Haoran  and
      Chen, Hao  and
      Chen, Bin  and
      Fang, Wenkai  and
      Xianyu  and
      Cao, Yu  and
      Xu, Haotian  and
      Liu, Yiming",
    editor = {Habernal, Ivan  and
      Schulam, Peter  and
      Tiedemann, J{\"o}rg},
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-demos.48/",
    pages = "656--666",
    ISBN = "979-8-89176-334-0",
    abstract = "Large Language Models (LLMs) fine-tuned via Reinforcement Learning from Human Feedback (RLHF) and Reinforcement Learning with Verifiable Rewards (RLVR) significantly improve the alignment of human-AI values and further raise the upper bound of AI capabilities, particularly in reasoning-intensive, long-context Chain-of-Thought (long-CoT) tasks. However, existing RLHF (or RLVR) frameworks commonly face challenges such as inference bottlenecks and complexity barriers, restricting their accessibility for newcomers. To bridge this gap, we introduce \textbf{OpenRLHF}, a user-friendly, scalable, and easy-to-learn open-source RLHF framework built upon Ray, vLLM, DeepSpeed, and HuggingFace Transformers, featuring a simplified design, clear code structure, and comprehensive documentation to facilitate entry for researchers and practitioners. Experimental results show that OpenRLHF achieves superior training efficiency with speedups ranging from 1.22{\texttimes} to 1.68{\texttimes} across different model sizes compared to state-of-the-art frameworks, while requiring significantly fewer lines of code for implementation. OpenRLHF is publicly available at \url{https://github.com/OpenRLHF/OpenRLHF}, and has already been adopted by leading institutions to accelerate RLHF research and learning."
}