@inproceedings{bian-etal-2026-realmem,
    title = "{R}eal{M}em: Benchmarking {LLM}s in Real-World Memory-Driven Interaction",
    author = "Bian, Haonan  and
      Yao, Zhiyuan  and
      Hu, Sen  and
      Xu, Zishan  and
      Zhang, Shaolei  and
      Guo, Yifu  and
      Yang, Ziliang  and
      Han, Xueran  and
      Wang, Huacan  and
      Chen, Ronghao",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.703/",
    pages = "14349--14365",
    ISBN = "979-8-89176-395-1",
    abstract = "As Large Language Models (LLMs) evolve from static dialogue interfaces to autonomous general agents, effective memory is paramount to ensuring long-term consistency. However, existing benchmarks primarily focus on casual conversation or task-oriented dialogue, failing to capture ``long-term project-oriented'' interactions where agents must track evolving goals. To bridge this gap, we introduce RealMem, the first benchmark grounded in realistic project scenarios. RealMem comprises over 2,000 cross-session dialogues across eleven scenarios, utilizing natural user queries for evaluation. We propose a synthesis pipeline that integrates Project Foundation Construction, Multi-Agent Dialogue Generation, and Memory and Schedule Management to simulate the dynamic evolution of memory. Experiments reveal that current memory systems face significant challenges in managing the long-term project states and dynamic context dependencies inherent in real-world projects. Our code and datasets are available at https://anonymous.4open.science/r/realmem-A1E4."
}