@inproceedings{wu-etal-2026-imbalanced,
    title = "Imbalanced Gradients in {RL} Post-Training of Multi-Task {LLM}s",
    author = "Wu, Runzhe  and
      Samanta, Ankur  and
      Jain, Ayush  and
      Fujimoto, Scott  and
      Kwon, Jeongyeol  and
      Kretzu, Ben  and
      Yu, Youliang  and
      Hassani, Kaveh  and
      Vidolov, Boris  and
      Efroni, Yonathan",
    editor = "Demberg, Vera  and
      Inui, Kentaro  and
      Marquez, Llu{\'i}s",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {EACL} 2026",
    month = mar,
    year = "2026",
    address = "Rabat, Morocco",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-eacl/2026.findings-eacl.164/",
    pages = "3137--3150",
    ISBN = "979-8-89176-386-9",
    abstract = "Multi-task post-training of large language models (LLMs) is typically performed by mixing datasets from different tasks and optimizing them jointly. This approach implicitly assumes that all tasks contribute gradients of similar magnitudes; when this assumption fails, optimization becomes biased toward large-gradient tasks. In this paper, however, we show that this assumption fails in RL post-training: certain tasks produce significantly larger gradients, thus biasing updates toward those tasks. Such gradient imbalance would be justified only if larger gradients implied larger learning gains on the tasks (i.e., larger performance improvements){---}but we find this is not true. Large-gradient tasks can achieve similar or even much lower learning gains than small-gradient ones. Further analyses reveal that these gradient imbalances cannot be explained by typical training statistics such as training rewards or advantages, suggesting that they arise from the *inherent* differences between tasks. This cautions against naive dataset mixing and calls for future work on principled gradient-level corrections for LLMs."
}