@inproceedings{zhang-etal-2025-physreason,
    title = "{P}hys{R}eason: A Comprehensive Benchmark towards Physics-Based Reasoning",
    author = "Zhang, Xinyu  and
      Dong, Yuxuan  and
      Wu, Yanrui  and
      Huang, Jiaxing  and
      Jia, Chengyou  and
      Fernando, Basura  and
      Shou, Mike Zheng  and
      Zhang, Lingling  and
      Liu, Jun",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.811/",
    pages = "16593--16615",
    ISBN = "979-8-89176-251-0",
    abstract = "Large language models demonstrate remarkable capabilities across various domains, especially mathematics and logic reasoning. However, current evaluations overlook physics-based reasoning - a complex task requiring physics theorems and constraints. We present PhysReason, a 1,200-problem benchmark comprising knowledge-based (25{\%}) and reasoning-based (75{\%}) problems, where the latter are divided into three difficulty levels (easy, medium, hard). Notably, problems require an average of 8.1 solution steps, with hard requiring 15.6, reflecting the complexity of physics-based reasoning. We propose the Physics Solution Auto Scoring Framework, incorporating efficient answer-level and comprehensive step-level evaluations. Top-performing models like Deepseek-R1, Gemini-2.0-Flash-Thinking, and o3-mini-high achieve less than 60{\%} on answer-level evaluation, with performance dropping from knowledge questions (75.11{\%}) to hard problems (31.95{\%}). Through step-level evaluation, we identified four key bottlenecks: Physics Theorem Application, Physics Process Understanding, Calculation, and Physics Condition Analysis. These findings position PhysReason as a novel and comprehensive benchmark for evaluating physics-based reasoning capabilities in large language models."
}