@inproceedings{meng-etal-2024-attribute,
    title = "Attribute Controlled Fine-tuning for Large Language Models: A Case Study on Detoxification",
    author = "Meng, Tao  and
      Mehrabi, Ninareh  and
      Goyal, Palash  and
      Ramakrishna, Anil  and
      Galstyan, Aram  and
      Zemel, Richard  and
      Chang, Kai-Wei  and
      Gupta, Rahul  and
      Peris, Charith",
    editor = "Al-Onaizan, Yaser  and
      Bansal, Mohit  and
      Chen, Yun-Nung",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2024",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest_wac_2008/2024.findings-emnlp.779/",
    doi = "10.18653/v1/2024.findings-emnlp.779",
    pages = "13329--13341",
    abstract = "We propose a constraint learning schema forfine-tuning Large Language Models (LLMs)with attribute control. Given a training corpusand control criteria formulated as a sequence-level constraint on model outputs, our methodfine-tunes the LLM on the training corpus whileenhancing constraint satisfaction with minimalimpact on its utility and generation quality.Specifically, our approach regularizes the LLMtraining by penalizing the KL divergence be-tween the desired output distribution, which sat-isfies the constraints, and the LLM`s posterior.This regularization term can be approximatedby an auxiliary model trained to decomposethe sequence-level constraints into token-levelguidance, allowing the term to be measuredby a closed-form formulation. To further im-prove efficiency, we design a parallel schemefor concurrently updating both the LLM andthe auxiliary model. We evaluate the empiricalperformance of our approach by controlling thetoxicity when training an LLM. We show thatour approach leads to an LLM that producesfewer inappropriate responses while achievingcompetitive performance on benchmarks and atoxicity detection task"
}