@inproceedings{zhang-etal-2026-test,
    title = "Test of Time: Rethinking Temporal Signal of Benchmark Contamination",
    author = "Zhang, Terry Jingchen  and
      Dev, Gopal  and
      Wang, Ning  and
      Obreiter, Max  and
      Jiang, Wenyuan  and
      Pandey, Punya Syon  and
      Samway, Keenan  and
      Huang, Yinya  and
      Sch{\textbackslash}{''}olkopf, Bernhard  and
      Sachan, Mrinmaya  and
      Jin, Zhijing",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.acl-long.1693/",
    pages = "36538--36555",
    ISBN = "979-8-89176-390-6",
    abstract = "Post-cutoff performance decay has been widely interpreted as a temporal signal for benchmark contamination.We critically examine this belief and demonstrate that this temporal signal is highly sensitive to how benchmark questions are constructed.Specifically, we show that LLM-generated questions can produce remarkably different temporal patterns compared to fill-in-the-blank questions directly retrieved from the very same materials.We validated this finding on previous benchmarks that reported clear post-cutoff performance decay such as LiveCodeBench and further showed simple LLM transformation could effectively remove this temporal pattern when evaluated on the same models.We also provide a mechanistic understanding of our observation using influence function analysis.Overall, this work offers a new perspective on the sensitivity of temporal contamination signal and highlights the need for more robust contamination detection methods for reliable AI evaluation."
}