@inproceedings{wei-etal-2025-equibench,
    title = "{E}qui{B}ench: Benchmarking Large Language Models' Reasoning about Program Semantics via Equivalence Checking",
    author = "Wei, Anjiang  and
      Cao, Jiannan  and
      Li, Ran  and
      Chen, Hongyu  and
      Zhang, Yuhui  and
      Wang, Ziheng  and
      Liu, Yuan  and
      Teixeira, Thiago S. F. X.  and
      Yang, Diyi  and
      Wang, Ke  and
      Aiken, Alex",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-main.1718/",
    pages = "33856--33869",
    ISBN = "979-8-89176-332-6",
    abstract = "As large language models (LLMs) become integral to code-related tasks, a central question emerges: Do LLMs truly understand program semantics? We introduce EquiBench, a new benchmark for evaluating LLMs through equivalence checking, i.e., determining whether two programs produce identical outputs for all possible inputs. Unlike prior code generation benchmarks, this task directly tests a model{'}s ability to reason about program semantics. EquiBench consists of 2400 program pairs across four languages and six categories. These pairs are generated through program analysis, compiler scheduling, and superoptimization, ensuring high-confidence labels, nontrivial difficulty, and full automation. We evaluate 19 state-of-the-art LLMs and find that in the most challenging categories, the best accuracies are 63.8{\%} and 76.2{\%}, only modestly above the 50{\%} random baseline. Further analysis reveals that models often rely on syntactic similarity rather than exhibiting robust reasoning about program semantics, highlighting current limitations. Our code and dataset are publicly available at https://github.com/Anjiang-Wei/equibench"
}