@inproceedings{tan-etal-2025-chinese,
    title = "{C}hinese {S}afety{QA}: A Safety Short-form Factuality Benchmark for Large Language Models",
    author = "Tan, Yingshui  and
      Zheng, Boren  and
      Zheng, Baihui  and
      Cao, Kerui  and
      Jing, Huiyun  and
      Wei, Jincheng  and
      Liu, Jiaheng  and
      He, Yancheng  and
      Su, Wenbo  and
      Zhu, Xiaoyong  and
      Zheng, Bo  and
      Zhang, Kaifu",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.732/",
    pages = "15053--15076",
    ISBN = "979-8-89176-251-0",
    abstract = "With the rapid advancement of Large Language Models (LLMs), significant safety concerns have emerged. Fundamentally, the safety of large language models is closely linked to the accuracy, comprehensiveness, and clarity of their understanding of safety knowledge, particularly in domains such as law, policy and ethics. This factuality ability is crucial in determining whether these models can be deployed and applied safely and compliantly within specific regions. To address these challenges and better evaluate the factuality ability of LLMs to answer short question, we introduce the Chinese SafetyQA benchmark. Chinese SafetyQA has several properties (i.e., Chinese, Diverse, High-quality, Static, Easy-to-evaluate, safety-related, harmless). Based on Chinese SafetyQA, we perform a comprehensive evaluation on the factuality abilities of existing LLMs and analyze how these capabilities relate to LLM abilities, e.g., RAG ability and robustness against attacks."
}