@inproceedings{cheng-etal-2025-codemenv,
    title = "{CODEMENV}: Benchmarking Large Language Models on Code Migration",
    author = "Cheng, Keyuan  and
      Shen, Xudong  and
      Yang, Yihao  and
      TengyueWang, TengyueWang  and
      Cao, Yang  and
      Ali, Muhammad Asif  and
      Wang, Hanbin  and
      Hu, Lijie  and
      Wang, Di",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.findings-acl.140/",
    pages = "2719--2744",
    ISBN = "979-8-89176-256-5",
    abstract = "Large language models (LLMs) have demonstrated remarkable proficiency in handling a wide range of tasks within the software engineering domain, but their ability to perform code migration{---}adapting code to different environments{---}remains underexplored. In this work, we propose a novel benchmark, : \textbf{Code} \textbf{M}igration Across \textbf{Env}ironment, designed to evaluate LLMs' performance in handling code migration tasks. The benchmark comprises 922 data points across 19 Python and Java packages, offering three tasks to systematically evaluate code migration: identifying version-incompatible functions, determining function changes, and adapting code to target environments. Experimental evaluation of across seven LLMs revealed an average pass@1 rate of 26.50{\%}, with GPT-4o performing best at 43.84{\%}. We highlight our key findings as follows: (i) LLMs are more familiar with newer function versions, making them better at migrating legacy code, and (ii) a logical inconsistency where LLMs sometimes identify irrelevant function changes for the target migration environment."
}