@inproceedings{ye-etal-2025-cpo,
    title = "{CPO}: Addressing Reward Ambiguity in Role-playing Dialogue via Comparative Policy Optimization",
    author = "Ye, Jing  and
      Wang, Rui  and
      Wu, Yuchuan  and
      Ma, Victor  and
      Fang, Feiteng  and
      Huang, Fei  and
      Li, Yongbin",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2025",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/author-page-yu-wang-polytechnic/2025.findings-emnlp.18/",
    doi = "10.18653/v1/2025.findings-emnlp.18",
    pages = "297--323",
    ISBN = "979-8-89176-335-7",
    abstract = "Reinforcement Learning Fine-Tuning (RLFT) has achieved notable success in tasks with objectively verifiable answers (e.g., code generation, mathematical reasoning), yet struggles with open-ended subjective tasks like role-playing dialogue. Traditional reward modeling approaches, which rely on independent sample-wise scoring, face dual challenges: subjective evaluation criteria and unstable reward signals. Motivated by the insight that human evaluation inherently combines explicit criteria with implicit comparative judgments, we propose \textbf{Comparative Policy Optimization (CPO)}. CPO redefines the reward evaluation paradigm by shifting from sample-wise scoring to comparative group-wise scoring. Building on the same principle, we introduce the \textbf{CharacterArena} evaluation framework, which comprises two stages: (1) \textit{Contextualized Multi-turn Role-playing Simulation}, and (2) \textit{Trajectory-level Comparative Evaluation}. By operationalizing subjective scoring via objective trajectory comparisons, CharacterArena minimizes contextual bias and enables more robust and fair performance evaluation. Empirical results on CharacterEval, CharacterBench, and CharacterArena confirm that CPO effectively mitigates reward ambiguity and leads to substantial improvements in dialogue quality."
}