@inproceedings{zhang-etal-2026-llmeval,
    title = "{LLME}val-Fair: A Large-Scale Longitudinal Study on Robust and Fair Evaluation of Large Language Models",
    author = "Zhang, Ming  and
      Shen, Yujiong  and
      Deng, Jingyi  and
      Wang, Yuhui  and
      Sha, Huayu  and
      Tan, Kexin  and
      Peng, Qiyuan  and
      Zhang, Yue  and
      Wang, Junzhe  and
      Liu, Shichun  and
      Huang, Yueyuan  and
      Tong, Jingqi  and
      Jiang, Changhao  and
      Wu, Yilong  and
      Zhang, Zhihao  and
      Wu, Mingqi  and
      Chai, Mingxu  and
      Xi, Zhiheng  and
      Dou, Shihan  and
      Gui, Tao  and
      Zhang, Qi  and
      Huang, Xuanjing",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.acl-long.972/",
    pages = "21246--21273",
    ISBN = "979-8-89176-390-6",
    abstract = "Existing evaluation of Large Language Models (LLMs) on static benchmarks is vulnerable to data contamination and leaderboard overfitting, critical issues that obscure true model capabilities. To address this, we introduce LLMEval-Fair, a framework for dynamic evaluation of LLMs. LLMEval-Fair is built on a proprietary bank of 220k graduate-level questions, from which it dynamically samples unseen test sets for each evaluation run. Its automated pipeline ensures integrity via contamination-resistant data curation, a novel anti-cheating architecture, and a calibrated LLM-as-a-judge process achieving 90{\%} agreement with human experts, complemented by a relative ranking system for fair comparison. An 30-month longitudinal study of nearly 60 leading models reveals a performance ceiling on knowledge memorization and exposes data contamination vulnerabilities undetectable by static benchmarks. The framework demonstrates exceptional robustness in ranking stability and consistency, providing strong empirical validation for the dynamic evaluation paradigm. LLMEval-Fair offers a robust and credible methodology for assessing the true capabilities of LLMs beyond leaderboard scores, promoting the development of more trustworthy evaluation standards."
}