@inproceedings{zhang-etal-2025-remembering,
    title = "From Remembering to Metacognition: Do Existing Benchmarks Accurately Evaluate {LLM}s?",
    author = "Zhang, Geng  and
      Ying, Yizhou  and
      Jiang, Sihang  and
      Liang, Jiaqing  and
      Yue, Guanglei  and
      Fu, Yifei  and
      Hu, Hailin  and
      Xiao, Yanghua",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2025",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/name-variant-enfa-fane/2025.findings-emnlp.724/",
    doi = "10.18653/v1/2025.findings-emnlp.724",
    pages = "13440--13457",
    ISBN = "979-8-89176-335-7",
    abstract = "Despite the rapid development of large language models (LLMs), existing benchmark datasets often focus on low-level cognitive tasks, such as factual recall and basic comprehension, while providing limited coverage of higher-level reasoning skills, including analysis, evaluation, and creation. In this work, we systematically assess the cognitive depth of popular LLM benchmarks using Bloom{'}s Taxonomy to evaluate both the cognitive and knowledge dimensions.Our analysis reveals a pronounced imbalance: most datasets concentrate on ``Remembering'' and ``Understanding'', with metacognitive and creative reasoning largely underrepresented. We also find that incorporating higher-level cognitive instructions into the current instruction fine-tuning process improves model performance. These findings highlight the importance of future benchmarks incorporating metacognitive evaluations to more accurately assess and enhance model performance."
}