@inproceedings{madhyastha-2025-task,
    title = "Task-Aware Evaluation and Error-Overlap Analysis for Large Language Models",
    author = "Madhyastha, Pranava",
    editor = {Sinha, Aman  and
      V{\'a}zquez, Ra{\'u}l  and
      Mickus, Timothee  and
      Agarwal, Rohit  and
      Buhnila, Ioana  and
      Schmidtov{\'a}, Patr{\'i}cia  and
      Gamba, Federica  and
      Prasad, Dilip K.  and
      Tiedemann, J{\"o}rg},
    booktitle = "Proceedings of the 1st Workshop on Confabulation, Hallucinations and Overgeneration in Multilingual and Practical Settings (CHOMPS 2025)",
    month = dec,
    year = "2025",
    address = "Mumbai, India",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-ijcnlp-aacl/2025.chomps-main.1/",
    pages = "1--10",
    ISBN = "979-8-89176-308-1",
    abstract = "Public leaderboards for large language models often rely on aggregate scores that conceal critical information about model behavior. In this paper, we present a methodology for task-aware evaluation that combines (i) correctness metrics aligned with task semantics compliance checks for instruction-following and numeric equivalence for mathematics with (ii) pairwise error-overlap analysis to identify complementary model pairs. We apply this methodology to 17 outputs of recent state of the art and frontier LLMs across multiple-choice QA, instruction-following, and mathematical reasoning tasks. We observe that task-aware metrics can reorder model rankings relative to generic lexical metrics, and that error-overlap patterns vary substantially across model pairs and scenarios. We finally conclude by discussing implications for model selection, routing strategies, and LLM-as-judge calibration, and release our analysis pipeline to support further investigation."
}