@inproceedings{bouchard-2026-bring,
    title = "Bring Your Own Prompts: Use-Case-Specific Bias and Fairness Evaluation for {LLM}s",
    author = "Bouchard, Dylan",
    editor = "Chakravarthi, Bharathi Raja  and
      B, Bharathi  and
      Buitelaar, Paul  and
      Thenmozhi, Durairaj  and
      Garc{\'i}a Cumbreras, Miguel {\'A}ngel  and
      Jim{\'e}nez Zafra, Salud Mar{\'i}a",
    booktitle = "Proceedings of the Sixth Workshop on Language Technology for Equality, Diversity, Inclusion",
    month = jul,
    year = "2026",
    address = "Virtual (Online)",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl-workshops/2026.ltedi-1.2/",
    pages = "10--26",
    ISBN = "979-8-89176-424-8",
    abstract = "Bias and fairness risks in Large Language Models (LLMs) vary substantially across deployment contexts, yet existing approaches lack systematic guidance for selecting appropriate evaluation metrics. We present a decision framework that maps LLM use cases, characterized by a model and population of prompts, to relevant bias and fairness metrics based on task type, whether prompts contain protected attribute mentions, and stakeholder priorities. Our framework addresses toxicity, stereotyping, counterfactual unfairness, and allocational harms, and introduces novel metrics based on stereotype classifiers and counterfactual adaptations of text similarity measures. We release an open-source Python library, langfair, for practical adoption. Extensive experiments on use cases across five LLMs and five prompt populations demonstrate that fairness risks cannot be reliably assessed from benchmark performance alone: results on one prompt dataset likely overstate or understate risks for another, underscoring that fairness evaluation must be grounded in the specific deployment context."
}