@inproceedings{li-etal-2025-large-language,
    title = "Large Language Models Struggle to Describe the Haystack without Human Help: A Social Science-Inspired Evaluation of Topic Models",
    author = "Li, Zongxia  and
      Calvo-Bartolom{\'e}, Lorena  and
      Hoyle, Alexander Miserlis  and
      Xu, Paiheng  and
      Stephens, Daniel Kofi  and
      Fung, Juan Francisco  and
      Dima, Alden  and
      Boyd-Graber, Jordan Lee",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.375/",
    pages = "7583--7604",
    ISBN = "979-8-89176-251-0",
    abstract = "A common use of NLP is to facilitate the understanding of large document collections, with models based on Large Language Models (LLMs) replacing probabilistic topic models. Yet the effectiveness of LLM-based approaches in real-world applications remains under explored. This study measures the knowledge users acquire with topic models{---}including traditional, unsupervised and supervised LLM- based approaches{---}on two datasets. While LLM-based methods generate more human- readable topics and show higher average win probabilities than traditional models for data exploration, they produce overly generic topics for domain-specific datasets that do not easily allow users to learn much about the documents. Adding human supervision to LLM-based topic models improves data exploration by addressing hallucination and genericity but requires more human efforts. In contrast, traditional models like Latent Dirichlet Allocation (LDA) remain effective for exploration but are less user-friendly. This paper provides best practices{---}there is no one right model, the choice of models is situation-specific{---}and suggests potential improvements for scalable LLM- based topic models."
}