@inproceedings{rakotonirina-etal-2025-tools,
    title = "From Tools to Teammates: Evaluating {LLM}s in Multi-Session Coding Interactions",
    author = {Rakotonirina, Nathana{\"e}l Carraz  and
      Hamdy, Mohammed  and
      Campos, Jon Ander  and
      Weber, Lucas  and
      Testoni, Alberto  and
      Fadaee, Marzieh  and
      Pezzelle, Sandro  and
      Del Tredici, Marco},
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/acl25-workshop-ingestion/2025.acl-long.964/",
    pages = "19609--19642",
    ISBN = "979-8-89176-251-0",
    abstract = "Large Language Models (LLMs) are increasingly used in working environments for a wide range of tasks, excelling at solving individual problems in isolation. However, are they also able to effectively collaborate over long-term interactions? To investigate this, we introduce MemoryCode, a synthetic multi-session dataset designed to test LLMs' ability to track and execute simple coding instructions amid irrelevant information, simulating a realistic setting. While all the models we tested handle isolated instructions well, even the performance of state-of-the-art models like GPT-4o deteriorates when instructions are spread across sessions. Our analysis suggests this is due to their failure to retrieve and integrate information over long interaction chains. Our results highlight a fundamental limitation of current LLMs, restricting their ability to collaborate effectively in long interactions."
}