@inproceedings{liang-etal-2026-rlhs,
    title = "{RLHS}: Mitigating Misalignment in {RLHF} with Hindsight Simulation",
    author = "Liang, Kaiqu  and
      Hu, Haimin  and
      Liu, Ryan  and
      Griffiths, Thomas L.  and
      Fisac, Jaime Fern{\textbackslash}'andez",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.556/",
    pages = "11457--11483",
    ISBN = "979-8-89176-395-1",
    abstract = "While Reinforcement Learning from Human Feedback (RLHF) has shown promise in aligning generative AI, we present empirical evidence that it can also cause severe, systematic misalignment. We hypothesize that this stems from evaluator feedback depending on downstream outcome predictions (foresight) that can be influenced by the AI{'}s output, inducing Goodhart{'}s law dynamics. We present a theoretical analysis showing that conditioning evaluator feedback on downstream observations (hindsight) inhibits this effect by decoupling the alignment signal from potentially compromised predictions{---}crucially, the result holds even if the observed outcomes are sampled from the AI{'}s own world model. Building on this insight, we introduce Reinforcement Learning from Hindsight Simulation (RLHS), which presents plausible simulated outcomes to evaluators before eliciting feedback. We validate RLHS across three consultancy settings{---}marketplace interactions, restaurant recommendations, and online course advising{---}using both online (PPO) and offline (DPO) fine-tuning methods, and show that it substantially improves alignment over RLHF in experiments and human evaluations. We perform post-hoc benchmark evaluations on TruthfulQA, HaluEval, and TrustLLM, finding that even after single-task fine-tuning, RLHF misalignment persists, while RLHS consistently outperforms baselines and demonstrates strong out-of-domain generalization."
}