Qingyuan Liang
2026
TRACE: Evaluating Execution Efficiency of LLM-Based Code Translation
Zhihao Gong | Zeyu Sun | Dong Huang | Qingyuan Liang | Jie M. Zhang | Dan Hao
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Zhihao Gong | Zeyu Sun | Dong Huang | Qingyuan Liang | Jie M. Zhang | Dan Hao
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
While Large Language Models (LLMs) have substantially improved the functional correctness of code translation, the critical dimension of execution efficiency remains overlooked. We present Trace, the first benchmark to explicitly assess efficiency in LLM-translated code. Trace includes 1,000 efficiency-critical tasks across C++, Java, and Python, each augmented with stress tests that reveal efficiency disparities often overlooked by small-scale tests. Using Trace, we conduct an extensive evaluation of 28 representative LLMs and highlight several key insights: 1) Correctness and efficiency are often misaligned: the correctness leader Claude-Sonnet-4-Think achieves only moderate time efficiency, outperformed by smaller open-source LLMs such as Qwen2.5-Coder-14B-Instruct. 2) Inefficiency is both prevalent and patterned: 23.5% of correct translations suffer from notable inefficiency, mainly arising from algorithm implementation discrepancy (11.9%), language construct mismatch (66.4%), and resource management inefficiency (21.7%).3) Inference-time prompt strategies bring only modest improvements, indicating that simple prompting alone is insufficient to improve translation efficiency. Together, our results establish execution efficiency as an essential dimension of code translation and position Trace as a principled foundation for efficiency-oriented evaluation. Our code and data are available at: https://github.com/Albert-Gong/TRACE.
2025
Grammar-Based Code Representation: Is It a Worthy Pursuit for LLMs?
Qingyuan Liang | Zhao Zhang | Zeyu Sun | Zheng Lin | Qi Luo | Yueyi Xiao | Yizhou Chen | Yuqun Zhang | Haotian Zhang | Lu Zhang | Bin Chen | Yingfei Xiong
Findings of the Association for Computational Linguistics: ACL 2025
Qingyuan Liang | Zhao Zhang | Zeyu Sun | Zheng Lin | Qi Luo | Yueyi Xiao | Yizhou Chen | Yuqun Zhang | Haotian Zhang | Lu Zhang | Bin Chen | Yingfei Xiong
Findings of the Association for Computational Linguistics: ACL 2025
Grammar serves as a cornerstone in programming languages and software engineering, providing frameworks to define the syntactic space and program structure. Existing research demonstrates the effectiveness of grammar-based code representations in small-scale models, showing their ability to reduce syntax errors and enhance performance. However, as language models scale to the billion level or beyond, syntax-level errors become rare, making it unclear whether grammar information still provides performance benefits. To explore this, we develop a series of billion-scale GrammarCoder models, incorporating grammar rules in the code generation process. Experiments on HumanEval (+) and MBPP (+) demonstrate a notable improvement in code generation accuracy. Further analysis shows that grammar-based representations enhance LLMs’ ability to discern subtle code differences, reducing semantic errors caused by minor variations. These findings suggest that grammar-based code representations remain valuable even in billion-scale models, not only by maintaining syntax correctness but also by improving semantic differentiation.