@inproceedings{hong-etal-2025-cross,
    title = "Cross-lingual Transfer of Reward Models in Multilingual Alignment",
    author = "Hong, Jiwoo  and
      Lee, Noah  and
      Mart{\'i}nez-Casta{\~n}o, Rodrigo  and
      Rodr{\'i}guez, C{\'e}sar  and
      Thorne, James",
    editor = "Chiruzzo, Luis  and
      Ritter, Alan  and
      Wang, Lu",
    booktitle = "Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers)",
    month = apr,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2025.naacl-short.8/",
    pages = "82--94",
    ISBN = "979-8-89176-190-2",
    abstract = "Reinforcement learning with human feedback (RLHF) is shown to largely benefit from precise reward models (RMs). However, recent studies in reward modeling schemes are skewed towards English, limiting the applicability of RLHF in multilingual alignments. In this work, we investigate the cross-lingual transfer of RMs trained in diverse languages, primarily from English. Our experimental results demonstrate the strong cross-lingual transfer of English RMs, exceeding target language RMs by 3{\textasciitilde}4{\%} average increase in Multilingual RewardBench. Furthermore, we analyze the cross-lingual transfer of RMs through the representation shifts. Finally, we perform multilingual alignment to exemplify how cross-lingual transfer in RM propagates to enhanced multilingual instruction-following capability."
}