@inproceedings{wang-etal-2024-conditional,
    title = "Conditional Language Policy: A General Framework For Steerable Multi-Objective Finetuning",
    author = "Wang, Kaiwen  and
      Kidambi, Rahul  and
      Sullivan, Ryan  and
      Agarwal, Alekh  and
      Dann, Christoph  and
      Michi, Andrea  and
      Gelmi, Marco  and
      Li, Yunxuan  and
      Gupta, Raghav  and
      Dubey, Kumar Avinava  and
      Rame, Alexandre  and
      Ferret, Johan  and
      Cideron, Geoffrey  and
      Hou, Le  and
      Yu, Hongkun  and
      Ahmed, Amr  and
      Mehta, Aranyak  and
      Hussenot, Leonard  and
      Bachem, Olivier  and
      Leurent, Edouard",
    editor = "Al-Onaizan, Yaser  and
      Bansal, Mohit  and
      Chen, Yun-Nung",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2024",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/jlcl-multiple-ingestion/2024.findings-emnlp.118/",
    doi = "10.18653/v1/2024.findings-emnlp.118",
    pages = "2153--2186",
    abstract = "Reward-based finetuning is crucial for aligning language policies with intended behaviors (*e.g.*, creativity and safety). A key challenge is to develop steerable language models that trade-off multiple (conflicting) objectives in a flexible and efficient manner. This paper presents Conditional Language Policy (CLP), a general framework for finetuning language models on multiple objectives. Building on techniques from multi-task training and parameter-efficient finetuning, CLP learn steerable models that effectively trade-off conflicting objectives at *inference time*. Notably, this does not require training or maintaining multiple models to achieve different trade-offs between the objectives. Through extensive experiments and ablations on two summarization datasets, we show that CLP learns steerable language models that outperform and Pareto-dominate the existing approaches for multi-objective"
}