@inproceedings{yao-etal-2025-value,
    title = "Value Compass Benchmarks: A Comprehensive, Generative and Self-Evolving Platform for {LLM}s' Value Evaluation",
    author = "Yao, Jing  and
      Yi, Xiaoyuan  and
      Duan, Shitong  and
      Wang, Jindong  and
      Bai, Yuzhuo  and
      Huang, Muhua  and
      Ou, Yang  and
      Li, Scarlett  and
      Zhang, Peng  and
      Lu, Tun  and
      Dou, Zhicheng  and
      Sun, Maosong  and
      Evans, James  and
      Xie, Xing",
    editor = "Mishra, Pushkar  and
      Muresan, Smaranda  and
      Yu, Tao",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-demo.64/",
    pages = "666--678",
    ISBN = "979-8-89176-253-4",
    abstract = "As large language models (LLMs) are gradually integrated into human daily life, assessing their underlying values becomes essential for understanding their risks and alignment with specific preferences. Despite growing efforts, current value evaluation methods face two key challenges. C1. Evaluation Validity: Static benchmarks fail to reflect intended values or yield informative results due to data contamination or a ceiling effect. C2. Result Interpretation: They typically reduce the pluralistic and often incommensurable values to one-dimensional scores, which hinders users from gaining meaningful insights and guidance. To address these challenges, we present Value Compass Benchmarks, the first dynamic, online and interactive platform specially devised for comprehensive value diagnosis of LLMs. It (1) grounds evaluations in multiple basic value systems from social science; (2) develops a generative evolving evaluation paradigm that automatically creates real-world test items co-evolving with ever-advancing LLMs; (3) offers multi-faceted result interpretation, including (i) fine-grained scores and case studies across 27 value dimensions for 33 leading LLMs, (ii) customized comparisons, and (iii) visualized analysis of LLMs' alignment with cultural values. We hope Value Compass Benchmarks serves as a navigator for further enhancing LLMs' safety and alignment, benefiting their responsible and adaptive development."
}