@inproceedings{liu-etal-2025-understand,
    title = "Understand User Opinions of Large Language Models via {LLM}-Powered In-the-Moment User Experience Interviews",
    author = "Liu, Mengqiao  and
      Wang, Tevin  and
      Cohen, Cassandra A.  and
      Li, Sarah  and
      Xiong, Chenyan",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/transition-to-people-yaml/2025.findings-acl.714/",
    doi = "10.18653/v1/2025.findings-acl.714",
    pages = "13872--13893",
    ISBN = "979-8-89176-256-5",
    abstract = "Which large language model (LLM) is better? Every evaluation tells a story, but what do users really think about current LLMs? This paper presents CLUE, an LLM-powered interviewer that conducts in-the-moment user experience interviews, right after users interact with LLMs, and automatically gathers insights about user opinions from massive interview logs. We conduct a study with thousands of users to understand user opinions on mainstream LLMs, recruiting users to first chat with a target LLM and then be interviewed by CLUE. Our experiments demonstrate that CLUE captures interesting user opinions, e.g., the bipolar views on the displayed reasoning process of DeepSeek-R1 and demands for information freshness and multi-modality. Our code and data are at https://github.com/cxcscmu/LLM-Interviewer."
}