@inproceedings{chen-etal-2025-benchmarking-large,
    title = "Benchmarking Large Language Models Under Data Contamination: A Survey from Static to Dynamic Evaluation",
    author = "Chen, Simin  and
      Chen, Yiming  and
      Li, Zexin  and
      Jiang, Yifan  and
      Wan, Zhongwei  and
      He, Yixin  and
      Ran, Dezhi  and
      Gu, Tianle  and
      Li, Haizhou  and
      Xie, Tao  and
      Ray, Baishakhi",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/automate-pypi-publishing/2025.emnlp-main.511/",
    doi = "10.18653/v1/2025.emnlp-main.511",
    pages = "10080--10098",
    ISBN = "979-8-89176-332-6",
    abstract = "In the era of evaluating large language models (LLMs), data contamination has become an increasingly prominent concern. To address this risk, LLM benchmarking has evolved from a \textit{static} to a \textit{dynamic} paradigm. In this work, we conduct an in-depth analysis of existing \textit{static} and \textit{dynamic} benchmarks for evaluating LLMs. We first examine methods that enhance \textit{static} benchmarks and identify their inherent limitations. We then highlight a critical gap{---}the lack of standardized criteria for evaluating \textit{dynamic} benchmarks. Based on this observation, we propose a series of optimal design principles for \textit{dynamic} benchmarking and analyze the limitations of existing \textit{dynamic} benchmarks.This survey provides a concise yet comprehensive overview of recent advancements in data contamination research, offering valuable insights and a clear guide for future research efforts. We maintain a GitHub repository to continuously collect both static and dynamic benchmarking methods for LLMs. The repository can be found at this link."
}