@inproceedings{etzine-etal-2025-revitalizing,
    title = "Revitalizing Saturated Benchmarks: A Weighted Metric Approach for Differentiating Large Language Model Performance",
    author = "Etzine, Bryan  and
      Hashemi, Masoud  and
      Madhusudhan, Nishanth  and
      Davasam, Sagar  and
      Sharma, Roshnee  and
      Madhusudhan, Sathwik Tejaswi  and
      Yadav, Vikas",
    editor = "Cao, Trista  and
      Das, Anubrata  and
      Kumarage, Tharindu  and
      Wan, Yixin  and
      Krishna, Satyapriya  and
      Mehrabi, Ninareh  and
      Dhamala, Jwala  and
      Ramakrishna, Anil  and
      Galystan, Aram  and
      Kumar, Anoop  and
      Gupta, Rahul  and
      Chang, Kai-Wei",
    booktitle = "Proceedings of the 5th Workshop on Trustworthy NLP (TrustNLP 2025)",
    month = may,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2025.trustnlp-main.33/",
    pages = "511--523",
    ISBN = "979-8-89176-233-6",
    abstract = "Existing benchmarks are becoming saturated and less effective in evaluating model performance due to factors such as data contamination and the advancing capabilities of the Large Language Models (LLMs). This paper introduces EMDM (Enhanced Model Differentiation Metric), a novel weighted metric designed to revitalize existing benchmarks. EMDM implements a weighting schema for samples based on their complexity and requisite knowledge, utilizing the performance of a baseline LLM in two experimental setups: ``Unguided'', where the model has no prior exposure to test samples, and ``Guided'', where the model has prior knowledge about the desired answer. This schema is leveraged in an optimization objective to assign weights to test samples, distinguishing instances of varying complexity. EMDM accounts for both answer correctness and the depth and accuracy of reasoning, offering a more nuanced evaluation of model performance. By weighting test examples based on their required reasoning and knowledge, EMDM achieves a distinguishing range of evaluation scores of 46{\%} among various LLMs, compared to just 17{\%} with traditional exact match (EM) metrics, thereby highlighting the saturation of current evaluation methods."
}