@inproceedings{liu-etal-2026-heterogeneous,
    title = "Heterogeneous Adaptive Policy Optimization: Tailoring Optimization to Every Token{'}s Nature",
    author = "Liu, Zheng  and
      Liu, Mengjie  and
      Wen, Siwei  and
      Cai, Mengzhang  and
      Cui, Bin  and
      He, Conghui  and
      Zhang, Wentao",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/revision-previews/2026.acl-long.272/",
    doi = "10.18653/v1/2026.acl-long.272",
    pages = "6023--6045",
    ISBN = "979-8-89176-390-6",
    abstract = "Using entropy as a measure of heterogeneity to guide optimization has emerged as a crucial research direction in Reinforcement Learning for LLMs. However, existing methods typically treat it as a discrete filter or post-hoc regulator rather than a core optimization driver. To fully leverage the potential of entropy and achieve fine-grained regulation, we introduce \textbf{H}eterogeneous \textbf{A}daptive \textbf{P}olicy \textbf{O}ptimization (HAPO), a token-aware algorithm that continuously adapts optimization dynamics based on token-level entropy throughout the entire training process. Our algorithm includes four key components: (1) \textbf{Adaptive Temperature Sampling} that adjusts sampling temperature in real time, promoting exploration at high-entropy tokens. (2) \textbf{Token-Level Group Average Advantage Estimation} that estimates advantages at token level, accounting for sequence-length effects while preserving non-biased treatment.(3) \textbf{Differential Advantage Redistribution} that leverages entropy and importance ratios to adjust advantages for tokens with clear signals. (4) \textbf{Asymmetric Adaptive Clipping} that dynamically adjusts clipping boundaries based on token-level entropy. Through systematic investigation of entropy, we embed token-level treatment into every stage. Extensive experiments on mathematical reasoning, code, and logic tasks across multiple models demonstrate HAPO{'}s consistent superiority over DAPO."
}