@inproceedings{ji-etal-2025-pku,
    title = "{PKU}-{S}afe{RLHF}: Towards Multi-Level Safety Alignment for {LLM}s with Human Preference",
    author = "Ji, Jiaming  and
      Hong, Donghai  and
      Zhang, Borong  and
      Chen, Boyuan  and
      Dai, Josef  and
      Zheng, Boren  and
      Qiu, Tianyi Alex  and
      Zhou, Jiayi  and
      Wang, Kaile  and
      Li, Boxun  and
      Han, Sirui  and
      Guo, Yike  and
      Yang, Yaodong",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.1544/",
    pages = "31983--32016",
    ISBN = "979-8-89176-251-0",
    abstract = "In this work, we introduce the PKU-SafeRLHF dataset, designed to promote research on safety alignment in large language models (LLMs). As a sibling project to SafeRLHF and BeaverTails, we separate annotations of helpfulness and harmlessness for question-answering pairs, providing distinct perspectives on these coupled attributes. Overall, we provide 44.6k refined prompts and 265k question-answer pairs with safety meta-labels for 19 harm categories and three severity levels ranging from minor to severe, with answers generated by Llama-family models. Based on this, we collected 166.8k preference data, including dual-preference (helpfulness and harmlessness decoupled) and single-preference data (trade-off the helpfulness and harmlessness from scratch), respectively. Using the large-scale annotation data, we further train severity-sensitive moderation for the risk control of LLMs and safety-centric RLHF algorithms for the safety alignment of LLMs. We believe this dataset will be a valuable resource for the community, aiding in the safe deployment of LLMs."
}