@inproceedings{guo-etal-2025-general,
    title = "From General Reward to Targeted Reward: Improving Open-ended Long-context Generation Models",
    author = "Guo, Zhihan  and
      Wu, Jiele  and
      Cui, Wenqian  and
      Zhang, Yifei  and
      Hu, Minda  and
      Wang, Yufei  and
      King, Irwin",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/cawl-year/2025.emnlp-main.260/",
    doi = "10.18653/v1/2025.emnlp-main.260",
    pages = "5151--5166",
    ISBN = "979-8-89176-332-6",
    abstract = "Current research on long-form context in Large Language Models (LLMs) primarily focuses on the understanding of long-contexts, the \textbf{Open-ended Long Text Generation} (Open-LTG) remains insufficiently explored. Training a long text generation model requires curation of gold-standard reference data, which is typically nonexistent for informative Open-LTG tasks. However, previous methods only utilize general assessments as reward signals, which limits accuracy. To bridge this gap, we introduce \textbf{ProxyReward}, an innovative reinforcement learning (RL) based framework, which includes a data synthesis method and a novel reward signal. Firstly, \textbf{ProxyReward Dataset} synthesis is accomplished through simple prompts that enables the model to create automatically, obviating extensive labeled data or significant manual effort. Secondly, \textbf{ProxyReward Signal} offers a targeted evaluation of information comprehensiveness and accuracy for specific questions. The experimental results indicate that our method ProxyReward \textbf{surpasses even GPT-4-Turbo}. It can significantly enhance performance by 20{\%} on the Open-LTG task when training widely used open-source models, while also surpassing the LLM-as-a-Judge approach. Our work presents effective methods to enhance the ability of LLMs to address complex open-ended questions posed by humans."
}