@inproceedings{hutson-etal-2025-guessinggame,
    title = "{G}uessing{G}ame: Measuring the Informativeness of Open-Ended Questions in Large Language Models",
    author = "Hutson, Dylan  and
      Vennemeyer, Daniel  and
      Deshmukh, Aneesh  and
      Zhan, Justin  and
      Jiang, Tianyu",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/name-variant-enfa-fane/2025.emnlp-main.876/",
    doi = "10.18653/v1/2025.emnlp-main.876",
    pages = "17344--17360",
    ISBN = "979-8-89176-332-6",
    abstract = "We introduce GuessingGame, a protocol for evaluating large language models (LLMs) as strategic question-askers in open-ended, open-domain settings. A Guesser LLM identifies a hidden object by posing free-form questions to an Oracle{---}without predefined choices or candidate lists. To measure question quality, we propose two information gain (IG) metrics: a Bayesian method that tracks belief updates over semantic concepts using LLM-scored relevance, and an entropy-based method that filters candidates via ConceptNet. Both metrics are model-agnostic and support post hoc analysis. Across 858 games with multiple models and prompting strategies, higher IG strongly predicts efficiency: a one-standard-deviation IG increase reduces expected game length by 43{\%}. Prompting constraints guided by IG{---}such as enforcing question diversity{---}enable weaker models to match GPT-4o. These results show that question-asking in LLMs is both measurable and improvable, and crucial for interactive reasoning."
}