@inproceedings{wachter-etal-2025-llms,
    title = "Are {LLM}s (Really) Ideological? An {IRT}-based Analysis and Alignment Tool for Perceived Socio-Economic Bias in {LLM}s",
    author = "Wachter, Jasmin  and
      Radloff, Michael  and
      Smolej, Maja  and
      Kinder-Kurlanda, Katharina",
    editor = "Dhole, Kaustubh  and
      Clinciu, Miruna",
    booktitle = "Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM{\texttwosuperior})",
    month = jul,
    year = "2025",
    address = "Vienna, Austria and virtual meeting",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/corrections-2025-08/2025.gem-1.9/",
    pages = "99--120",
    ISBN = "979-8-89176-261-9",
    abstract = "We introduce an Item Response Theory (IRT)-based framework to detect and quantify ideological bias in large language models (LLMs) without relying on subjective human judgments. Unlike prior work, our two-stage approach distinguishes between response avoidance and expressed bias by modeling `Prefer Not to Answer' (PNA) behaviors and calibrating ideological leanings based on open-ended responses. We fine-tune two LLM families to represent liberal and conservative baselines, and validate our approach using a 105-item ideological test inventory. Our results show that off-the-shelve LLMs frequently avoid engagement with ideological prompts, calling into question previous claims of partisan bias. This framework provides a statistically grounded and scalable tool for LLM alignment and fairness assessment. The general methodolody can also be applied to other forms of bias and languages."
}