@inproceedings{chernyshev-etal-2025-u,
    title = "{U}-{MATH}: A University-Level Benchmark for Evaluating Mathematical Skills in Large Language Models",
    author = "Chernyshev, Konstantin  and
      Polshkov, Vitaliy  and
      Stepanov, Vlad  and
      Myasnikov, Alex  and
      Artemova, Ekaterina  and
      Miasnikov, Alexei  and
      Tilga, Sergei",
    editor = "Dhole, Kaustubh  and
      Clinciu, Miruna",
    booktitle = "Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM{\texttwosuperior})",
    month = jul,
    year = "2025",
    address = "Vienna, Austria and virtual meeting",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/corrections-2025-08/2025.gem-1.77/",
    pages = "974--1001",
    ISBN = "979-8-89176-261-9",
    abstract = "Current evaluations of mathematical skills in Large Language Models are constrained by benchmarks lacking scope, particularly for multi-modal problems {---} frequently relying on school-level, niche Olympiad-style, simple quiz-format, or relatively small datasets.To address this, we introduce **U-MATH**, a novel benchmark comprising **1,100** unpublished open-ended university-level problems sourced from current US curricula, with **20{\%}** incorporating visual elements. Given the free-form nature of U-MATH problems, we employ LLM judges for solution evaluation and release $\boldsymbol{\mu}$**-MATH**, a meta-evaluation benchmark composed of **1,084** U-MATH-derived tasks enabling precise assessment of these judges.Benchmarking leading LLMs reveals marked limitations in multi-modal reasoning, with maximum accuracy reaching 93.1{\%} on textual tasks but only 58.5{\%} on visual ones. Furthermore, solution judgment proves challenging, requiring the most advanced models to achieve meaningfully high performance, even still peaking at an imperfect F1-score of 90.1{\%}."
}