@inproceedings{feng-etal-2025-sample,
    title = "Sample-Efficient Human Evaluation of Large Language Models via Maximum Discrepancy Competition",
    author = "Feng, Kehua  and
      Ding, Keyan  and
      Hongzhi, Tan  and
      Ma, Kede  and
      Wang, Zhihua  and
      Guo, Shuangquan  and
      Yuzhou, Cheng  and
      Sun, Ge  and
      Zheng, Guozhou  and
      Zhang, Qiang  and
      Chen, Huajun",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.535/",
    pages = "10913--10947",
    ISBN = "979-8-89176-251-0",
    abstract = "The past years have witnessed a proliferation of large language models (LLMs). Yet, reliable evaluation of LLMs is challenging due to the inaccuracy of standard metrics in human perception of text quality and the inefficiency in sampling informative test examples for human evaluation. This paper presents a sample-efficient human evaluation method for LLMs based on the principle of MAximum Discrepancy (MAD) competition. MAD automatically selects a small set of informative input instructions, each of which maximizes the discrepancy of two LLMs' reponses, which are subsequently subject to three-alternative forced choice by human subjects. The pairwise comparison results of multiple LLMs are then aggregated into a global ranking using the Elo rating system. We compare eight representative LLMs in terms of four skills: knowledge understanding, mathematical reasoning, writing, and coding. Experimental results show that the proposed method reliably achieves the ``golden'' ranking of LLMs with a minimum set of input instructions, which in turn reveal their relative strengths and weaknesses, and offers valuable insights for further LLM advancement."
}