@inproceedings{lim-etal-2026-contracteval,
    title = "{C}ontract{E}val: A Benchmark for Evaluating Contract-Satisfying Assertions in Code Generation",
    author = "Lim, Soohan  and
      Hahn, Joonghyuk  and
      Park, Hyunwoo  and
      Ko, Sang-Ki  and
      Han, Yo-Sub",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.2112/",
    pages = "42549--42566",
    ISBN = "979-8-89176-395-1",
    abstract = "Current code generation evaluation measures functional correctness on well-formed inputs that satisfy all input preconditions. This paradigm has a critical limitation: task descriptions often leave these preconditions implicit, while evaluation filters out inputs that violate them. As a result, generated code may achieve high pass@k scores while failing to enforce the preconditions that the task actually requires. To address this gap, we introduce **ContractEval**, a benchmark for evaluating whether generated code enforces such preconditions{---}commonly referred to as contracts. Built on HumanEval+ and MBPP+, ContractEval consists of 364 tasks, each with three components: (i) descriptions reconstructed to explicitly state the contracts, (ii) test cases synthesized through a neuro-symbolic pipeline that pairs an LLM with an SMT solver to evaluate whether generated code satisfies these contracts, and (iii) reference code combined with contracts. Using ContractEval to evaluate five representative open-source code LLMs, we reveal a stark disparity between functional correctness and contract satisfaction. Under standard prompting, these models achieve pass@1 of 75-82{\%} with 0{\%} contract satisfaction. Even when contracts are explicitly stated in the prompt, the satisfaction rate reaches only 23-41{\%}. This indicates that current LLMs struggle to satisfy contracts in their generated code, establishing contract satisfaction as a crucial and previously overlooked axis of code generation quality. Our code is available at https://github.com/suhanmen/ContractEval."
}