@inproceedings{liao-etal-2026-intrinsic,
    title = "Intrinsic Mutual Information as a Modulator for Preference Optimization",
    author = "Liao, Peng  and
      Zheng, Peijia  and
      Li, Lingbo  and
      Liang, Shangsong  and
      Chen, Lin",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.185/",
    pages = "3790--3808",
    ISBN = "979-8-89176-395-1",
    abstract = "Offline preference optimization methods, such as Direct Preference Optimization (DPO), offer significant advantages in aligning Large Language Models (LLMs) with human values. However, achieving optimal performance with these methods typically involves additional hyperparameter tuning, resulting in substantial time overhead. Although prior work has proposed a range of improvements, these methods remain limited in effectiveness and have not fully eliminated reliance on hyperparameter tuning. In this work, we introduce RMiPO, a lightweight and efficient framework for offline preference optimization. RMiPO leverages intrinsic **R**esponse-level **M**utual **i**nformation for **P**reference **O**ptimization with hyperparameter modulation, dynamically decoupling preference contributions at negligible additional computational cost. Extensive experimental results demonstrate that RMiPO achieves consistently superior performance over existing methods while reducing training overhead by more than 15{\%}. Our code is available at https://github.com/liavonpenn/rmipo."
}