@inproceedings{ivey-etal-2026-real,
    title = "Real or Robotic? Assessing Whether {LLM}s Accurately Simulate Qualities of Human Responses in Human-{LLM} Dialogue",
    author = "Ivey, Jonathan  and
      Kumar, Shivani  and
      Liu, Jiayu  and
      Shen, Hua  and
      Rakshit, Sushrita  and
      Raju, Rohan  and
      Zhang, Haotian  and
      Ananthasubramaniam, Aparna  and
      Kim, Junghwan  and
      Yi, Bowen  and
      Wright, Dustin  and
      Israeli, Abraham  and
      M{\{}{\textbackslash}o{\}}ller, Anders Giovanni  and
      Zhang, Lechen  and
      Jurgens, David",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.2060/",
    pages = "41395--41432",
    ISBN = "979-8-89176-395-1",
    abstract = "Building datasets for dialogue tasks is expensive and time-consuming, requiring recruitment, training, and data collection from study participants. In response, much recent work has sought to use large language models (LLMs) to simulate both human-human and human-LLM interactions, as they have been shown to generate convincingly human-like text in many settings. However, how well do LLM-based simulations reflect real human dialogue? In this work, we answer this question by generating a large-scale dataset of 100,000 paired LLM-LLM and human-LLM dialogues from the WildChat dataset and quantifying how well the LLM simulations align with their human counterparts. Overall, we find relatively low alignment between simulations and human interactions, with systematic differences in multiple textual properties, including style and conversational dynamics. Further, we find that models perform similarly in simulating English, Chinese, and Russian dialogues. Our results also suggest that LLMs only simulate a narrow range of the overall distribution of human dialogue, as they perform better on the subset of humans who write similarly to the LLM{'}s own style."
}