@inproceedings{lan-etal-2025-f2bench,
    title = "{F}{\texttwosuperior}{B}ench: An Open-ended Fairness Evaluation Benchmark for {LLM}s with Factuality Considerations",
    author = "Lan, Tian  and
      Li, Jiang  and
      Wang, Yemin  and
      Liu, Xu  and
      Su, Xiangdong  and
      Gao, Guanglai",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-main.105/",
    pages = "2031--2046",
    ISBN = "979-8-89176-332-6",
    abstract = "With the growing adoption of large language models (LLMs) in NLP tasks, concerns about their fairness have intensified. Yet, most existing fairness benchmarks rely on closed-ended evaluation formats, which diverge from real-world open-ended interactions. These formats are prone to position bias and introduce a ``minimum score'' effect, where models can earn partial credit simply by guessing. Moreover, such benchmarks often overlook factuality considerations rooted in historical, social, physiological, and cultural contexts, and rarely account for intersectional biases. To address these limitations, we propose F{\texttwosuperior}Bench: an open-ended fairness evaluation benchmark for LLMs that explicitly incorporates factuality considerations. F{\texttwosuperior}Bench comprises 2,568 instances across 10 demographic groups and two open-ended tasks. By integrating text generation, multi-turn reasoning, and factual grounding, F{\texttwosuperior}Bench aims to more accurately reflect the complexities of real-world model usage. We conduct a comprehensive evaluation of several LLMs across different series and parameter sizes. Our results reveal that all models exhibit varying degrees of fairness issues. We further compare open-ended and closed-ended evaluations, analyze model-specific disparities, and provide actionable recommendations for future model development. Our code and dataset are publicly available at https://github.com/VelikayaScarlet/F2Bench."
}