@inproceedings{eskandari-miandoab-etal-2026-breaking,
    title = "Breaking the Benchmark: Revealing {LLM} Bias via Minimal Contextual Augmentation",
    author = "Eskandari Miandoab, Kaveh  and
      Kamruzzaman, Mahammed  and
      Gharooni, Arshia  and
      Kim, Gene Louis  and
      Sarathy, Vasanth  and
      Mehrabi, Ninareh",
    editor = "Piperidis, Stelios  and
      Bel, N{\'u}ria  and
      van den Heuvel, Henk  and
      Ide, Nancy  and
      Krek, Simon  and
      Toral, Antonio",
    booktitle = "Proceedings of the Fifteenth Language Resources and Evaluation Conference",
    month = may,
    year = "2026",
    address = "Palma de Mallorca, Spain",
    publisher = "ELRA Language Resource Association",
    url = "https://preview.aclanthology.org/cawl-year/2026.lrec-1.322/",
    doi = "10.63317/5a6nbh2tnoeb",
    pages = "4070--4092",
    abstract = "Large Language Models have been shown to demonstrate stereotypical biases in their representations and behavior due to the discriminative nature of the data that they have been trained on. Despite significant progress in the development of methods and models that refrain from using stereotypical information in their decision-making, recent work has shown that approaches used for bias alignment are brittle. In this work, we introduce a novel and general augmentation framework that involves three plug-and-play steps and is applicable to a number of fairness evaluation benchmarks. Through application of augmentation to a fairness evaluation dataset (Bias Benchmark for Question Answering (BBQ)), we find that Large Language Models (LLMs), including state-of-the-art open and closed weight models, are susceptible to perturbations to their inputs, showcasing a higher likelihood to behave stereotypically. Furthermore, we find that such models are more likely to have biased behavior in cases where the target demographic belongs to a community less studied by the literature, underlining the need to expand the fairness and safety research to include more diverse communities."
}