@inproceedings{zhang-etal-2026-sentient,
    title = "Sentient Agent as a Judge: Evaluating Higher-Order Social Cognition in Large Language Models",
    author = "Zhang, Bang  and
      Ma, Ruotian  and
      Jiang, Qingxuan  and
      Wang, Peisong  and
      Chen, Jiaqi  and
      Xie, Zheng  and
      Chen, Xingyu  and
      Wang, Yue  and
      Ye, Fanghua  and
      Li, Jian  and
      Yang, Yifan  and
      Tu, Zhaopeng  and
      Li, Xiaolong",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl-workshops/2026.findings-acl.1905/",
    pages = "38196--38224",
    ISBN = "979-8-89176-395-1",
    abstract = "Assessing how well a large language model (LLM) understands human, rather than merely text, remains an open challenge.To bridge the gap, we introduce Sentient Agent as a Judge(SAGE), an automated evaluation framework that measures an LLM{'}s higher-order social cognition.SAGE instantiates a ``Sentient Agent'' {--} an LLM-powered agent that simulates human-like emotional changes and inner thoughts to provide a more realistic evaluation of the tested model in multi-turn conversations.At every turn, the agent reasons about (i) how its emotion changes, (ii) how it feels, and (iii) how it should reply, yielding a numerical emotion trajectory and interpretable inner thoughts.Experiments on 100 supportive-dialogue scenarios show that the final Sentient emotion score correlates strongly with Barrett-Lennard Relationship Inventory (BLRI) ratings and utterance-level empathy metrics, validating psychological fidelity. Human evaluation further demonstrates 85.3{\%} consistency between the agent{'}s emotional reasoning and human judgments. We also build a public Sentient Leaderboard covering 18 commercial and open-source models that uncovers substantial gaps (up to 4$\times$) between frontier systems (GPT-4o-Latest, Gemini2.5-Pro) and earlier baselines, gaps not reflected in conventional leaderboards (e.g. Arena). SAGE thus provides a principled, scalable, and interpretable tool for tracking progress toward genuinely empathetic and socially adept language agents."
}