@inproceedings{zhang-etal-2026-test,
    title = "Test of Time: Rethinking Temporal Signal of Benchmark Contamination",
    author = {Zhang, Terry Jingchen  and
      Dev, Gopal  and
      Wang, Ning  and
      Obreiter, Max  and
      Pandey, Punya Syon  and
      Samway, Keenan  and
      Jiang, Wenyuan  and
      Huang, Yinya  and
      Sch{\"o}lkopf, Bernhard  and
      Sachan, Mrinmaya  and
      Jin, Zhijing},
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/bulk-corrections-2026-07-02/2026.acl-long.1693/",
    doi = "10.18653/v1/2026.acl-long.1693",
    pages = "36538--36555",
    ISBN = "979-8-89176-390-6",
    abstract = "Post-cutoff performance decay of LLMs has been widely interpreted as a temporal signal for benchmark contamination, where public information released before the training cutoff may have been included into training corpora and inflated model performance by memorization. We critically examine this view and demonstrate that this temporal signal is highly sensitive to how benchmark questions are constructed, even if the underlying source material remains invariant. Specifically, we show that LLM-transformed questions can produce remarkably different temporal patterns compared to fill-in-the-blank (cloze) questions directly retrieved from the very same documents. We validate this effect on prior benchmarks that report clear post-cutoff decay (LiveCodeBench), and show that a simple LLM-driven transformation of the same problems can effectively remove the temporal pattern. We further provide a mechanistic understanding of this phenomenon using influence function analysis. Overall, our results suggest that post-cutoff performance decay is a sensitive contamination signal, motivating more robust contamination probes for reliable LLM evaluation."
}