@inproceedings{zhao-etal-2025-mpo,
    title = "{MPO}: Multilingual Safety Alignment via Reward Gap Optimization",
    author = "Zhao, Weixiang  and
      Hu, Yulin  and
      Deng, Yang  and
      Wu, Tongtong  and
      Zhang, Wenxuan  and
      Guo, Jiahe  and
      Zhang, An  and
      Zhao, Yanyan  and
      Qin, Bing  and
      Chua, Tat-Seng  and
      Liu, Ting",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.1149/",
    pages = "23564--23587",
    ISBN = "979-8-89176-251-0",
    abstract = "Large language models (LLMs) have become increasingly central to AI applications worldwide, necessitating robust multilingual safety alignment to ensure secure deployment across diverse linguistic contexts. Existing preference learning methods for safety alignment, such as RLHF and DPO, are primarily monolingual and struggle with noisy multilingual data. To address these limitations, we introduce \textbf{Mul}tilingual reward ga\textbf{P} \textbf{O}ptimization (\textbf{MPO}), a novel approach that leverages the well-aligned safety capabilities of the dominant language (\textit{e.g.}, English) to improve safety alignment across multiple languages. MPO directly minimizes the reward gap difference between the dominant language and target languages, effectively transferring safety capabilities while preserving the original strengths of the dominant language. Extensive experiments on three LLMs, LLaMA-3.1, Gemma-2 and Qwen2.5, validate MPO{'}s efficacy in multilingual safety alignment without degrading general multilingual utility."
}