@inproceedings{chiu-etal-2025-culturalbench,
    title = "{C}ultural{B}ench: A Robust, Diverse and Challenging Benchmark for Measuring {LM}s' Cultural Knowledge Through Human-{AI} Red-Teaming",
    author = "Chiu, Yu Ying  and
      Jiang, Liwei  and
      Lin, Bill Yuchen  and
      Park, Chan Young  and
      Li, Shuyue Stella  and
      Ravi, Sahithya  and
      Bhatia, Mehar  and
      Antoniak, Maria  and
      Tsvetkov, Yulia  and
      Shwartz, Vered  and
      Choi, Yejin",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.1247/",
    pages = "25663--25701",
    ISBN = "979-8-89176-251-0",
    abstract = "Robust, diverse, and challenging cultural knowledge benchmarks are essential for measuring our progress towards making LMs that are helpful across diverse cultures. We introduce CulturalBench: a set of 1,696 human-written and human-verified questions to assess LMs' cultural knowledge, covering 45 global regions including underrepresented ones like Bangladesh, Zimbabwe, and Peru. Questions are each verified by five independent annotators and span 17 diverse topics ranging from food preferences to greeting etiquette. We construct CulturalBench using methods inspired by Human-AI Red-Teaming. Compared to human performance (92.4{\%} accuracy), the hard version of CulturalBench is challenging even for the best-performing frontier LMs, ranging from 28.7{\%} to 61.5{\%} in accuracy. We find that LMs often struggle with tricky questions that have multiple correct answers (e.g., What utensils do the Chinese usually use?), revealing a tendency to overfit to a single answer. Our results indicate that GPT-4o substantially outperform other models across cultures, besting local providers (e.g., Mistral on European culture and DeepSeek on Chinese culture). Across the board, models under-perform on questions related to North Africa, South America and Middle East."
}