@inproceedings{zhao-etal-2026-safety,
    title = "On Safety Risks in Experience-Driven Self-Evolving Agents",
    author = "Zhao, Weixiang  and
      Zhang, Yichen  and
      Wang, Yingshuo  and
      Deng, Yang  and
      Zhao, Yanyan  and
      Zhi, Xuda  and
      Huang, Yongbo  and
      He, Hao  and
      Che, Wanxiang  and
      Qin, Bing  and
      Liu, Ting",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.2091/",
    pages = "42145--42169",
    ISBN = "979-8-89176-395-1",
    abstract = "Experience-driven self-evolution has emerged as a promising paradigm for improving the autonomy of large language model agents, yet its reliance on self-curated experience introduces underexplored safety risks. In this study, we investigate how experience accumulation and utilization in self-evolving agents affect safety performance across web-based and embodied environments. Notably, experience gathered solely from benign tasks can still compromise safety in high-risk scenarios. Further analysis attributes this degradation to the execution-oriented nature of accumulated experience, which reinforces agents' tendency to act rather than refuse. In more realistic settings where agents encounter both benign and harmful tasks, refusal-related experience mitigates safety decline but induces over-refusal, revealing a fundamental safety{--}utility trade-off. Overall, our findings expose inherent limitations of current self-evolving agents and call for more principled strategies to ensure safe and reliable adaptation."
}