@inproceedings{nalbandyan-etal-2025-score,
    title = "{SCORE}: Systematic {CO}nsistency and Robustness Evaluation for Large Language Models",
    author = "Nalbandyan, Grigor  and
      Shahbazyan, Rima  and
      Bakhturina, Evelina",
    editor = "Chen, Weizhu  and
      Yang, Yi  and
      Kachuee, Mohammad  and
      Fu, Xue-Yong",
    booktitle = "Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 3: Industry Track)",
    month = apr,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/Ingest-2025-COMPUTEL/2025.naacl-industry.39/",
    pages = "470--484",
    ISBN = "979-8-89176-194-0",
    abstract = "Typical evaluations of Large Language Models (LLMs) report a single metric per dataset, often representing the model`s best-case performance under carefully selected settings. Unfortunately, this approach overlooks model robustness and reliability in real-world applications. For instance, simple paraphrasing of prompts on the MMLU-Pro dataset causes accuracy fluctuations of up to 10{\%}, while reordering answer choices in the AGIEval dataset results in accuracy differences of up to 6.1{\%}. While some studies discuss issues with LLM robustness, there is no unified or centralized framework for evaluating the robustness of language models. To address this gap and consolidate existing research on model robustness, we present SCORE (\textbf{S}ystematic \textbf{CO}nsistency and \textbf{R}obustness \textbf{E}valuation), a comprehensive framework for non-adversarial evaluation of LLMs. The SCORE framework evaluates models by repeatedly testing them on the same benchmarks in various setups to give a realistic estimate of their accuracy and consistency. We will make the code publicly available to facilitate further development and research."
}