@inproceedings{li-etal-2026-soar,
    title = "{SOAR}: Supervision from Observation for Agentic Reinforcement Learning",
    author = "Li, Meng  and
      Li, Lei  and
      Wang, Xiting  and
      Yuan, Yi  and
      Wei, Zheng  and
      Brucebian  and
      Li, Zang",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.acl-long.1624/",
    pages = "35175--35197",
    ISBN = "979-8-89176-390-6",
    abstract = "Agentic reinforcement learning enables large language models to solve long-horizon tasks by interacting with the environment and internalizing tool-use behavior into their reasoning. Prior work assigns supervision primarily based on outcome rewards or external reward models, but largely ignores environment observations, a critical source of learning. Consequently, agents may identify successful actions without understanding how the environment responds, producing suboptimal policies. To address this, we propose SOAR (Supervision from Observation for Agentic Reinforcement Learning), which assigns positive advantages to observation tokens proportional to the negative entropy of preceding actions. This encourages the agent to learn from outcomes of confident actions, grounding policy updates in environment dynamics and improving anticipation of tool-call consequences. Empirical results across three domains and 14 benchmarks show that SOAR improves performance, yielding gains of up to 7.0{\%} on general reasoning tasks and 16.9{\%} on deep research tasks, while reducing erroneous and inefficient tool usage."
}