@inproceedings{jiang-etal-2026-investigating,
    title = "Investigating Human and {LLM}s' Decisions in Unverifiable Environments: A Case Study with {G}it{H}ub Activity Overview",
    author = "Jiang, Zheng  and
      Wang, Wei  and
      Zhang, Gaowei  and
      Feng, Yang  and
      Wang, Yi",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.1374/",
    pages = "27591--27618",
    ISBN = "979-8-89176-395-1",
    abstract = "The behaviors of Large Language Models (LLMs) as artificial social actors are largely underexplored, particularly in unverifiable scenarios where conventional benchmarking has little to help improve their abilities. Thus, examining their behaviors in such scenarios can help understand and improve LLMs' capabilities of simulating real-world social actors in many tasks such as LLM-empowered social agents. We draw a typical unverifiable scenario{--}a simplified pull request scenario on GitHub focusing on decision-making based on Activity Overview signal{--}to investigate how human and LLMs behave. We introduce a systematic method to collect, compare, and reason about human and LLMs' decisions. Our results reveal that there are both similarities and differences between human and LLMs' decisions, and proprietary LLMs generally behave more like human than open-source LLMs do. We further find that human and LLMs may rely on different information and reasoning mechanisms in decision-making. Our study thus urges more future work on human and LLMs decision-making in unverifiable environments."
}