@inproceedings{liang-etal-2026-adaptive,
    title = "Adaptive Helpfulness{--}Harmlessness Alignment with Preference Vectors",
    author = "Liang, Ren-Wei  and
      Hsu, Chin Ting  and
      Yu, Chan-Hung  and
      Agrawal, Saransh  and
      Huang, Shih-Cheng  and
      Lin, Chieh-Yen  and
      Chen, Shang-Tse  and
      Huang, Kuan-Hao  and
      Sun, Shao-Hua",
    editor = "Demberg, Vera  and
      Inui, Kentaro  and
      Marquez, Llu{\'i}s",
    booktitle = "Proceedings of the 19th Conference of the {E}uropean Chapter of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = mar,
    year = "2026",
    address = "Rabat, Morocco",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-eacl/2026.eacl-long.77/",
    pages = "1646--1668",
    ISBN = "979-8-89176-380-7",
    abstract = "Ensuring that large language models (LLMs) are both helpful and harmless is a critical challenge, as overly strict constraints can lead to excessive refusals, while permissive models risk generating harmful content. Existing approaches, such as reinforcement learning from human feedback (RLHF) and direct preference optimization (DPO), attempt to balance these trade-offs but suffer from performance conflicts, limited controllability, and poor extendability. To address these issues, we propose Preference Vector, a novel framework inspired by task arithmetic. Instead of optimizing multiple preferences within a single objective, we train separate models on individual preferences, extract behavior shifts as preference vectors, and dynamically merge them at test time. This modular approach enables fine-grained, user-controllable preference adjustments and facilitates seamless integration of new preferences without retraining. Experiments show that our proposed Preference Vector framework improves helpfulness without excessive conservatism, allows smooth control over preference trade-offs, and supports scalable multi-preference alignment."
}