Yun Peng
2026
CoRE: A Fine-Grained Code Reasoning Benchmark Beyond Output Prediction
Jun Gao | Yun Peng | Qian Qiao | Changhai Zhou | Yuhua Zhou | Shiyang Zhang | Shichao Weng | Zhenchang Xing | Xiaoxue Ren
Findings of the Association for Computational Linguistics: ACL 2026
Jun Gao | Yun Peng | Qian Qiao | Changhai Zhou | Yuhua Zhou | Shiyang Zhang | Shichao Weng | Zhenchang Xing | Xiaoxue Ren
Findings of the Association for Computational Linguistics: ACL 2026
Despite strong performance on code generation tasks, it remains unclear whether large language models (LLMs) genuinely reason about code execution. Existing code reasoning benchmarks primarily evaluate final output correctness under a single canonical implementation, leaving two critical aspects underexplored: (1) whether LLMs predictions are consistent to functionally equivalent implementations, and (2) whether LLMs can accurately reason about intermediate execution states. We introduce CoRE, a Code Reasoning benchmark that evaluates code reasoning through implementation invariance and process transparency. Extensive evaluations on eight frontier LLMs reveal two fundamental limitations. First, models exhibit a substantial robustness gap, with performance varying significantly across equivalent implementations. Second, we observe superficial execution, where models arrive at correct final outputs without correctly reasoning about intermediate execution states. Together, these findings demonstrate that output-only evaluations are insufficient for assessing code reasoning and position CoRE as a necessary benchmark for evaluating robust and faithful code reasoning.
MulVul: Retrieval-augmented Multi-Agent Code Vulnerability Detection via Cross-Model Prompt Evolution
Zihan Wu | Jie Xu | Yun Peng | Chun Yong Chong | Xiaohua Jia
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Zihan Wu | Jie Xu | Yun Peng | Chun Yong Chong | Xiaohua Jia
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Large Language Models (LLMs) struggle to automate real-world vulnerability detection due to two key limitations: the heterogeneity of vulnerability patterns undermines the effectiveness of a single unified model, and manual prompt engineering for massive weakness categories is unscalable.To address these challenges, we propose MulVul, a retrieval-augmented multi-agent framework designed for precise and broad-coverage vulnerability detection. MulVul adopts a coarse-to-fine strategy: a Router agent first predicts the top- coarse categories and then forwards the input to specialized Detector agents, which identify the exact vulnerability types. Both agents use evidence retrieved from vulnerability knowledge bases to mitigate hallucinations. Crucially, to automate the generation of specialized prompts, we design Cross-Model Prompt Evolution, a prompt optimization mechanism where a generator LLM iteratively refines candidate prompts while a distinct executor LLM validates their effectiveness. This decoupling mitigates the self-correction bias inherent in single-model optimization. Evaluated on 130 CWE types, MulVul achieves 34.79% Macro-F1, outperforming the best baseline by 41.5%. Ablation studies validate cross-model prompt evolution, which boosts performance by 51.6% over manual prompts by effectively handling diverse vulnerability patterns.