@inproceedings{mundada-etal-2026-evaluating,
    title = "Evaluating Language Model Pluralism through In-the-wild Crowd Discussions",
    author = "Mundada, Gagan  and
      Surana, Rohan  and
      Swaminathan, Nandhini  and
      Majumder, Bodhisattwa Prasad  and
      Wu, Junda  and
      McAuley, Julian  and
      Xie, Zhouhang",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.acl-long.1957/",
    pages = "42273--42296",
    ISBN = "979-8-89176-390-6",
    abstract = "When answering subjective questions, an ideal LLM should surface diverse plausible perspectives rather than favoring a single viewpoint, a characteristic known as pluralism. Recent studies show that modern LLMs optimized through preference alignment systematically favor certain positions on subjective queries, making pluralism evaluation increasingly important. However, existing evaluation methods focus dominantly on multiple-choice and question-answering tasks, leaving open-ended generation largely unaddressed.We propose PLURALEVAL, an evaluation framework that assesses LLM pluralism in open-ended generation by comparing outputs against free-form crowd responses. Our approach decomposes ground-truth responses into atomic, non-overlapping claims, then evaluates whether LLMs adequately cover this diverse claim space. We then introduce WildSCOPE, a multi-domain dataset of natural crowd responses, and demonstrate that PLURALEVAL captures novel insights, such as the collapse of pluralism through sycophancy, where LLM systematically degrades in overton pluralism when a user{'}s belief is revealed. Finally, we discuss the value and actionable insights for preserving and encouraging pluralism from LLM deployers' side."
}