@inproceedings{rupprecht-etal-2026-prompt,
    title = "Prompt Perturbations Reveal Human-Like Biases in Large Language Model Survey Responses",
    author = "Rupprecht, Jens  and
      Ahnert, Georg  and
      Strohmaier, Markus",
    editor = "Card, Dallas  and
      Field, Anjalie  and
      Keith, Katherine  and
      Mendelsohn, Julia",
    booktitle = "Proceedings of the Seventh Workshop on Natural Language Processing and Computational Social Science",
    month = jul,
    year = "2026",
    address = "San Diego",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl-workshops/2026.nlpcss-1.1/",
    pages = "1--21",
    ISBN = "979-8-89176-426-2",
    abstract = "Large Language Models (LLMs) are increasingly used as proxies for human subjects in social science surveys, but their reliability and susceptibility to known human-like response biases, such as $\textit{central tendency, opinion floating}$ and $\textit{primacy bias}$ are poorly understood. This work investigates the response robustness of LLMs in normative survey contexts{---}we test 18 LLMs on questions taken from the World Values Survey (WVS), applying a comprehensive set of ten perturbations to both question phrasing and answer option structure, resulting in over 334,800 simulated survey interviews. In doing so, we not only reveal LLMs' vulnerabilities to perturbations but also show that almost all tested models exhibit a consistent $\textit{recency bias}$, disproportionately favoring the last-presented answer option. While larger models are generally more robust, all models remain sensitive to semantic variations like paraphrasing and to combined perturbations. This underscores the critical importance of prompt design and robustness testing when using LLMs to generate synthetic survey data."
}