@inproceedings{anschel-etal-2025-group,
    title = "Group-Aware Reinforcement Learning for Output Diversity in Large Language Models",
    author = "Anschel, Oron  and
      Shoshan, Alon  and
      Botach, Adam  and
      Hakimi, Shunit Haviv  and
      Gendler, Asaf  and
      Ben Baruch, Emanuel  and
      Bhonker, Nadav  and
      Kviatkovsky, Igor  and
      Aggarwal, Manoj  and
      Medioni, Gerard",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/author-page-yu-wang-polytechnic/2025.emnlp-main.1649/",
    doi = "10.18653/v1/2025.emnlp-main.1649",
    pages = "32382--32403",
    ISBN = "979-8-89176-332-6",
    abstract = "Large Language Models (LLMs) often suffer from mode collapse, repeatedly generating the same few completions even when many valid answers exist, limiting their diversity across a wide range of tasks. We introduce \textbf{Group-Aware Policy Optimization (GAPO)}, a simple extension of the recent and popular Group Relative Policy Optimization (GRPO) that computes rewards over the group as a whole. GAPO enables learning from the group-level properties such as diversity and coverage.We demonstrate GAPO using a frequency-aware reward function that encourages uniform sampling over valid LLM completions, and show that GAPO-trained models produce valid and more diverse model responses.Beyond this setup, GAPO generalizes to open-ended prompts and improves response diversity without compromising accuracy on standard LLM benchmarks (GSM8K, MATH, HumanEval, MMLU-Pro). Our code will be made publicly available."
}