@inproceedings{li-etal-2025-third,
    title = "Can Third Parties Read Our Emotions?",
    author = "Li, Jiayi  and
      Zhou, Yingfan  and
      Narayanan Venkit, Pranav  and
      Islam, Halima Binte  and
      Arya, Sneha  and
      Wilson, Shomir  and
      Rajtmajer, Sarah",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.1042/",
    pages = "21478--21499",
    ISBN = "979-8-89176-251-0",
    abstract = "Natural Language Processing tasks that aim to infer an author{'}s private states, e.g., emotions and opinions, from their written text, typically rely on datasets annotated by third-party annotators. However, the assumption that third-party annotators can accurately capture authors' private states remains largely unexamined. In this study, we present human subjects experiments on emotion recognition tasks that directly compare third-party annotations with first-party (author-provided) emotion labels. Our findings reveal significant limitations in third-party annotations{---}whether provided by human annotators or large language models (LLMs){---}in faithfully representing authors' private states. However, LLMs outperform human annotators nearly across the board. We further explore methods to improve third-party annotation quality. We find that demographic similarity between first-party authors and third-party human annotators enhances annotation performance. While incorporating first-party demographic information into prompts leads to a marginal but statistically significant improvement in LLMs' performance. We introduce a framework for evaluating the limitations of third-party annotations and call for refined annotation practices to accurately represent and model authors' private states."
}