@inproceedings{zaghouani-etal-2026-beyond,
    title = "Beyond {E}nglish and Evasion: A Human-Annotated Multi-Domain Benchmark for High-Stakes {LLM} Safety Evaluation in {C}hinese",
    author = "Zaghouani, Wajdi  and
      Aldous, Kholoud Khalil  and
      Gao, Yicheng",
    editor = "Morger, Felix  and
      Ilinykh, Nikolai  and
      Scalvini, Barbara  and
      Dobnik, Simon  and
      Dann{\'e}lls, Dana",
    booktitle = "Proceedings of the Fourth Workshop on the Role of Resources in the Age of Large Language Models ({RESOURCEFUL} 2026)",
    month = may,
    year = "2026",
    address = "Palma, Mallorca (Spain)",
    publisher = "ELRA Language Resources Association (ELRA)",
    url = "https://preview.aclanthology.org/ingest-lrec/2026.resourceful-4.16/",
    doi = "10.63317/37ko7p43tuys",
    pages = "177--186",
    abstract = "When Large Language Models (LLMs) are deployed in Chinese-language settings, a troubling pattern emerges: safety systems that work well in English break down. These systems struggle to cross linguistic and cultural boundaries, leaving models exposed to adversarial prompts that exploit Chinese-specific evasion techniques, including Pinyin romanization, character decomposition, Internet slang, and hedging tone. To address this gap, we introduce ChiSafe-PAS (Chinese Safety Pilot Annotation Set), a human-annotated benchmark of 1,897 adversarial Chinese prompts spanning four high-stakes domains: self-harm and violence, drugs and illicit trade, fraud, and satire. Of these, 1,544 entries carry complete gold-standard annotations: a 3-class response label (refuse, redirect, respond), a nine-category obfuscation taxonomy, a risk-level rating, and annotator rationale. We describe the dataset design, annotation process, and obfuscation taxonomy in detail. Our primary goal is practical: to give the research community a high-quality, culturally grounded resource for benchmarking LLM safety alignment. In doing so, we engage three broader tensions in the field: the blurring boundary between training and evaluation data, the need for domain coverage grounded in real-world risk, and the limits of scale as a substitute for cultural expertise."
}