@inproceedings{chen-etal-2025-causally,
    title = "Causally Testing Gender Bias in {LLM}s: A Case Study on Occupational Bias",
    author = "Chen, Yuen  and
      Raghuram, Vethavikashini Chithrra  and
      Mattern, Justus  and
      Mihalcea, Rada  and
      Jin, Zhijing",
    editor = "Chiruzzo, Luis  and
      Ritter, Alan  and
      Wang, Lu",
    booktitle = "Findings of the Association for Computational Linguistics: NAACL 2025",
    month = apr,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/moar-dois/2025.findings-naacl.281/",
    doi = "10.18653/v1/2025.findings-naacl.281",
    pages = "4984--5004",
    ISBN = "979-8-89176-195-7",
    abstract = "Generated texts from large language models (LLMs) have been shown to exhibit a variety of harmful, human-like biases against various demographics. These findings motivate research efforts aiming to understand and measure such effects. This paper introduces a causal formulation for bias measurement in generative language models. Based on this theoretical foundation, we outline a list of desiderata for designing robust bias benchmarks. We then propose a benchmark called OccuGender, with a bias-measuring procedure to investigate occupational gender bias. We test several state-of-the-art open-source LLMs on OccuGender, including Llama, Mistral, and their instruction-tuned versions. The results show that these models exhibit substantial occupational gender bias. Lastly, we discuss prompting strategies for bias mitigation and an extension of our causal formulation to illustrate the generalizability of our framework."
}