@inproceedings{liu-etal-2024-alignbench,
    title = "{A}lign{B}ench: Benchmarking {C}hinese Alignment of Large Language Models",
    author = "Liu, Xiao  and
      Lei, Xuanyu  and
      Wang, Shengyuan  and
      Huang, Yue  and
      Feng, Andrew  and
      Wen, Bosi  and
      Cheng, Jiale  and
      Ke, Pei  and
      Xu, Yifan  and
      Tam, Weng Lam  and
      Zhang, Xiaohan  and
      Sun, Lichao  and
      Gu, Xiaotao  and
      Wang, Hongning  and
      Zhang, Jing  and
      Huang, Minlie  and
      Dong, Yuxiao  and
      Tang, Jie",
    editor = "Ku, Lun-Wei  and
      Martins, Andre  and
      Srikumar, Vivek",
    booktitle = "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = aug,
    year = "2024",
    address = "Bangkok, Thailand",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2024.acl-long.624/",
    doi = "10.18653/v1/2024.acl-long.624",
    pages = "11621--11640",
    abstract = "Alignment has become a critical step for instruction-tuned Large Language Models (LLMs) to become helpful assistants. However, effective evaluation of alignment for emerging Chinese LLMs is still significantly lacking, calling for real-scenario grounded, open-ended, challenging and automatic evaluations tailored for alignment. To fill in this gap, we introduce AlignBench, a comprehensive multi-dimensional benchmark for evaluating LLMs' alignment in Chinese. We tailor a human-in-the-loop data curation pipeline, containing 8 main categories, 683 real-scenario rooted queries and corresponding human verified references.To ensure references' correctness, each knowledge-intensive query is accompanied with evidences collected from reliable webpages (including the url and quotation) by our annotators.For automatic evaluation, our benchmark employs a rule-calibrated multi-dimensional LLM-as-Judge (CITATION) with Chain-of-Thought to generate explanations and final ratings as evaluations, ensuring high reliability and interpretability.All evaluation codes and data are publicly available at \url{https://github.com/THUDM/AlignBench}"
}