@inproceedings{kantharuban-etal-2025-stereotype,
    title = "Stereotype or Personalization? User Identity Biases Chatbot Recommendations",
    author = "Kantharuban, Anjali  and
      Milbauer, Jeremiah  and
      Sap, Maarten  and
      Strubell, Emma  and
      Neubig, Graham",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/landing_page/2025.findings-acl.1254/",
    pages = "24418--24436",
    ISBN = "979-8-89176-256-5",
    abstract = "While personalized recommendations are often desired by users, it can be difficult in practice to distinguish cases of bias from cases of personalization: we find that models generate racially stereotypical recommendations regardless of whether the user revealed their identity intentionally through explicit indications or unintentionally through implicit cues. We demonstrate that when people use large language models (LLMs) to generate recommendations, the LLMs produce responses that reflect both what the user wants and who the user is. We argue that chatbots ought to transparently indicate when recommendations are influenced by a user{'}s revealed identity characteristics, but observe that they currently fail to do so. Our experiments show that even though a user{'}s revealed identity significantly influences model recommendations ($p < 0.001$), model responses obfuscate this fact in response to user queries. This bias and lack of transparency occurs consistently across multiple popular consumer LLMs and for four American racial groups."
}