@inproceedings{li-etal-2026-awarenessbench,
    title = "{A}wareness{B}ench: Assessing Cognitive Capabilities of Language Models",
    author = "Li, Xiaojian  and
      Xu, Rongwu  and
      Zhang, Tianyun  and
      Wang, Yue  and
      Chen, Shuo  and
      Lyu, Qiner  and
      Zhang, Briana  and
      Yang, Peiran  and
      Chen, Kyle Xue  and
      Shi, Haoyuan  and
      Wang, Yu  and
      Xu, Wei",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.acl-long.124/",
    pages = "2682--2741",
    ISBN = "979-8-89176-390-6",
    abstract = "As language models (LMs) exhibit increasingly consciousness-like behaviors, evaluating their cognitive abilities becomes essential. We introduce $\texttt{AwarenessBench}$, the first comprehensive benchmark for assessing the cognitive abilities of LMs in four dimensions: metacognition, self-awareness, social awareness, and situational awareness, covering 15 cognitive functions and 14,381 samples. Evaluating 18 state-of-the-art LMs, we find that all consistently surpass random baselines, with more advanced models performing better. We further compare LMs with human performance across three demographic groups, where the best-performing model surpasses human averages overall, but most still fall markedly short in metacognition and self-awareness. Finally, we show that awareness is a distinct capability: progress in language modeling or reasoning does not necessarily translate into improved cognition."
}