@inproceedings{kim-etal-2026-lovers,
    title = "Are they lovers or friends? Evaluating {LLM}s' Social Reasoning in {E}nglish and {K}orean Dialogues",
    author = "Kim, Eunsu  and
      Park, Junyeong  and
      Oh, Juhyun  and
      Park, Kiwoong  and
      Song, Seyoung  and
      Do{\textbackslash}u{\{}g{\}}ru{\textbackslash}{''}oz, A. Seza  and
      Oh, Alice  and
      Kim, Najoung",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.acl-long.1074/",
    pages = "23431--23451",
    ISBN = "979-8-89176-390-6",
    abstract = "As LLMs are increasingly deployed in real-world interactions, their social reasoning in interpersonal communication becomes critical. To explore their capabilities, we introduce SCRIPTS, a 1.1k-dialogue dataset in English and Korean, sourced from movie scripts and propose a social reasoning task based on SCRIPTS that evaluates the capacity of LLMs to infer the social relationships (e.g., friends, lovers) between speakers in each dialogue. Evaluating nine models on our task, current LLMs achieve around 75{--}80{\%} on the English dataset and 58{--}69{\%} in Korean, and models predict an Unlikely relationship in 10{--}25{\%} of responses in both languages.Furthermore, we find that thinking models and chain-of-thought prompting provide minimal benefits for social reasoning and occasionally amplify social biases.In sum, there are significant limitations in current LLMs' social reasoning capabilities, especially for Korean, highlighting the need for efforts to develop socially-aware LLMs across languages."
}