@inproceedings{liu-etal-2026-gaoyao,
    title = "The {G}ao{Y}ao Benchmark: A Comprehensive Framework for Evaluating Multilingual and Multicultural Abilities of Large Language Models",
    author = "Liu, Yilun  and
      Zhao, Chunguang  and
      Piao, Mengyao  and
      Miao, Lingqi  and
      Tao, Shimin  and
      HE, Minggui  and
      Liu, Chenxin  and
      Li, Zhang  and
      Mahongxia  and
      Guo, Jiaxin  and
      Liu, Chen  and
      Deng, Liqun  and
      Wei, Jiansheng  and
      Meng, Xiaojun  and
      Du, Fanyi  and
      Wei, Daimeng  and
      Xiao, Yanghua",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.acl-long.977/",
    pages = "21364--21384",
    ISBN = "979-8-89176-390-6",
    abstract = "Evaluating the multilingual and multicultural capabilities of Large Language Models (LLMs) is essential for their global utility. However, current benchmarks face three critical limitations: (1) fragmented evaluation dimensions that often neglect deep cultural nuances; (2) insufficient language coverage in subjective tasks relying on low-quality machine translation; and (3) shallow analysis that lacks diagnostic depth beyond simple rankings. To address these, we introduce GaoYao, a comprehensive benchmark with 182.3k samples, 26 languages and 51 nations/areas. First, GaoYao proposes a unified framework categorizing evaluation tasks into three cultural layers (General Multilingual, Cross-cultural, Monocultural) and nine cognitive sub-layers. Second, we achieve native-quality expansion by leveraging experts to rigorously localize subjective benchmarks into 19 languages and synthesizing cross-cultural test sets for 34 cultures, surpassing prior coverage by up to 111{\%}. Third, we conduct an in-depth diagnostic analysis on 20+ flagship and compact LLMs. Our findings reveal significant geographical performance disparities and distinct gaps between tasks, offering a reliable map for future work. We release the benchmark."
}