@inproceedings{alipour-etal-2025-robustness,
    title = "Robustness and Confounders in the Demographic Alignment of {LLM}s with Human Perceptions of Offensiveness",
    author = "Alipour, Shayan  and
      Sen, Indira  and
      Samory, Mattia  and
      Mitra, Tanu",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/corrections-2025-08/2025.findings-acl.1136/",
    doi = "10.18653/v1/2025.findings-acl.1136",
    pages = "22025--22047",
    ISBN = "979-8-89176-256-5",
    abstract = "Despite a growing literature finding that large language models (LLMs) exhibit demographic biases, reports with whom they align best are hard to generalize or even contradictory. In this work, we examine the alignment of LLMs with human annotations in five offensive language datasets, comprising approximately 220K annotations. While demographic traits, particularly race, influence alignment, these effects vary across datasets and are often entangled with other factors. Confounders introduced in the annotation process{---}such as document difficulty, annotator sensitivity, and within-group agreement{---}account for more variation in alignment patterns than demographic traits. Alignment increases with annotator sensitivity and group agreement, and decreases with document difficulty. Our results underscore the importance of multi-dataset analyses and confounder-aware methodologies in developing robust measures of demographic bias."
}