@inproceedings{qiu-etal-2025-reward,
    title = "Reward Generalization in {RLHF}: A Topological Perspective",
    author = "Qiu, Tianyi Alex  and
      Zeng, Fanzhi  and
      Ji, Jiaming  and
      Yan, Dong  and
      Wang, Kaile  and
      Zhou, Jiayi  and
      Han, Yang  and
      Dai, Josef  and
      Pan, Xuehai  and
      Yang, Yaodong",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/landing_page/2025.findings-acl.820/",
    pages = "15884--15930",
    ISBN = "979-8-89176-256-5",
    abstract = "Existing alignment methods share a common topology of information flow, where reward information is collected from humans, modeled with preference learning, and used to tune language models. However, this shared topology has not been systematically characterized, nor have its alternatives been thoroughly explored, leaving the problems of low data efficiency and unreliable generalization unaddressed. As a solution, we introduce a theory of **reward generalization** in reinforcement learning from human feedback (RLHF), focusing on the **topology of information flow** at both macro and micro levels. At the macro level, we portray the RLHF information flow as an autoencoding process over behavior distributions, formalizing the RLHF objective of distributional consistency between human preference and model behavior. At the micro level, we present *induced Bayesian networks* to model the impact of dataset topologies on reward generalization. Combining analysis on both levels, we propose **reward modeling from tree-structured preference information**. It is shown to reduce reward uncertainty by up to $\Theta(\log n/\log\log n)$ times compared to baselines, where $n$ is the dataset size. Validation on three NLP tasks shows that it achieves an average win rate of 65{\%} against baselines, thus improving reward generalization *for free* via topology design, while *reducing* the amount of data requiring annotation."
}