@inproceedings{spaulding-etal-2025-role,
    title = "On the Role of Semantic Proto-roles in Semantic Analysis: What do {LLM}s know about agency?",
    author = "Spaulding, Elizabeth  and
      Ahmed, Shafiuddin Rehan  and
      Martin, James",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/corrections-2025-08/2025.findings-acl.623/",
    doi = "10.18653/v1/2025.findings-acl.623",
    pages = "12027--12048",
    ISBN = "979-8-89176-256-5",
    abstract = "Large language models (LLMs) are increasingly used in decision-making contexts, yet their ability to reason over event structure{---}an important component in the situational awareness needed to make complex decisions{---}is not well understood. By operationalizing proto-role theory, which characterizes agents via properties such as *instigation* and *volition* and patients via properties such as *change of state*, we examine the ability of LLMs to answer questions that require complex, multi-step event reasoning. Specifically, we investigate the extent to which LLMs capture semantic roles such as ``agent'' and ``patient'' through zero-shot prompts, and whether incorporating semantic proto-role labeling (SPRL) context improves semantic role labeling (SRL) performance in a zero-shot setting. We find that, while SPRL context sometimes degrades SRL accuracy in high-performing models (e.g., GPT-4o), it also uncovers an internal consistency between SPRL and SRL predictions that mirrors linguistic theory, and provides evidence that LLMs implicitly encode consistent multi-dimensional event role knowledge. Furthermore, our experiments support prior work showing that LLMs underperform human annotators in complex semantic analysis."
}