@inproceedings{liu-etal-2025-contextual,
    title = "Contextual Experience Replay for Self-Improvement of Language Agents",
    author = "Liu, Yitao  and
      Si, Chenglei  and
      Narasimhan, Karthik R  and
      Yao, Shunyu",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.694/",
    pages = "14179--14198",
    ISBN = "979-8-89176-251-0",
    abstract = "Large language model (LLM) agents have been applied to sequential decision-making tasks such as web navigation, but without any environment-specific experiences, they often fail in these complex tasks. Moreover, current LLM agents are not designed to continually learn from past experiences during inference time, which could be crucial for them to gain these environment-specific experiences. To address this, we propose Contextual Experience Replay (CER), a training-free framework to enable efficient self-improvement for language agents in their context window. Specifically, CER accumulates and synthesizes past experiences into a dynamic memory buffer. These experiences encompass environment dynamics and common decision-making patterns, allowing the agents to retrieve and augment themselves with relevant knowledge in new tasks, enhancing their adaptability in complex environments. We evaluate CER on the challenging WebArena and VisualWebArena benchmarks. On VisualWebArena, CER surpasses the tree search method with much fewer token costs and achieves the state-of-the-art performance of 31.9{\%}. On WebArena, CER also gets a competitive average success rate of 36.7{\%}, relatively improving the success rate of the GPT-4o agent baseline by 51.0{\%}. We also conduct a comprehensive analysis on it to prove its efficiency, validity and understand it better."
}