@inproceedings{chai-etal-2026-activation,
    title = "Activation Reward Models for Few-Shot Model Alignment",
    author = "Chai, Tianning  and
      Mitra, Chancharik  and
      Huang, Brandon  and
      Gare, Gautam Rajendrakumar  and
      Lin, Zhiqiu  and
      Arbelle, Assaf  and
      Karlinsky, Leonid  and
      Feris, Rogerio  and
      Darrell, Trevor  and
      Ramanan, Deva  and
      Herzig, Roei",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.1709/",
    pages = "34201--34217",
    ISBN = "979-8-89176-395-1",
    abstract = "Aligning Large Language Models (LLMs) and Large Multimodal Models (LMMs) to human preferences is crucial for improving their real-world behavior. A common approach is to use reward models that enable reinforcement-learning post-training. However, traditional reward modeling requires finetuning on large preference datasets, limiting adaptability to new preferences. We introduce Activation Reward Models (Activation RMs){---}the first mechanistic interpretability approach that steers LLM activations to align with few-shot preference data without finetuning. Our method combines activation denoising and output token likelihood scoring, achieving state-of-the-art performance on standard reward modeling benchmarks, surpassing zero-shot, few-shot, and voting-based baselines. We further demonstrate that Activation RMs mitigate reward hacking behaviors and remain robust to noisy exemplars and spurious reward signals. To evaluate this, we propose PreferenceHack, a novel few-shot benchmark testing reward models on reward hacking in a paired preference format, where Activation RMs achieve state-of-the-art performance, surpassing GPT-4o."
}