@inproceedings{yang-etal-2025-dpo,
    title = "How Does {DPO} Reduce Toxicity? A Mechanistic Neuron-Level Analysis",
    author = "Yang, Yushi  and
      Sondej, Filip  and
      Mayne, Harry  and
      Lee, Andrew  and
      Mahdi, Adam",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-main.1501/",
    pages = "29512--29531",
    ISBN = "979-8-89176-332-6",
    abstract = "Safety fine-tuning algorithms reduce harmful outputs in language models, yet their mechanisms remain under-explored. Direct Preference Optimization (DPO) is a popular choice of algorithm, but prior explanations{---}attributing its effects solely to dampened toxic neurons in the MLP layers{---}are incomplete. In this study, we analyse four language models (Llama-3.1-8B, Gemma-2-2B, Mistral-7B, GPT-2-Medium) and show that toxic neurons only account for 2.5{\%} to 24{\%} of DPO{'}s effects across models. Instead, DPO induces distributed activation shifts across all MLP neurons to create a net toxicity reduction. We attribute this reduction to four neuron groups{---}two aligned with reducing toxicity and two promoting anti-toxicity{---}whose combined effects replicate DPO across models. To further validate this understanding, we develop an activation editing method that mimics DPO through distributed shifts along a toxicity representation. This method outperforms DPO in reducing toxicity while preserving perplexity, without requiring any weight updates. Our work provides a mechanistic understanding of DPO and introduces an efficient, tuning-free alternative for safety fine-tuning."
}