@inproceedings{zhang-etal-2025-reasonerrank,
    title = "{R}easoner{R}ank: Redefining Language Model Evaluation with Ground-Truth-Free Ranking Frameworks",
    author = "Zhang, Jiamu  and
      Yuan, Jiayi  and
      Wen, Andrew  and
      Le, Hoang Anh Duy  and
      Chuang, Yu-Neng  and
      Choi, Soo-Hyun  and
      Chen, Rui  and
      Hu, Xia",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/landing_page/2025.findings-acl.700/",
    pages = "13623--13639",
    ISBN = "979-8-89176-256-5",
    abstract = "Large Language Models (LLMs) are increasingly adopted across real-world applications, yet traditional evaluations rely on expensive, domain-specific ground-truth labels that are often unavailable or infeasible. We introduce a ground-truth-free evaluation framework focused on reasoning consistency and instruction following, shifting the emphasis from correctness{---}which is elusive without labels{---}to transparent, coherent, evidence-based reasoning. Each model response must include a direct answer, a structured multi-step explanation, and supporting evidence, all assessed via semantic similarity and output adherence checks. We further propose TopK-ReRank, which refines rankings by constructing a consensus answer from the most reliable models, reducing ambiguity across diverse reasoning styles. Experiments show that our framework outperforms existing label-free methods, including majority voting, triplet ranking, and peer-review approaches, providing a more interpretable and efficient alternative for evaluating LLMs in the absence of ground-truth labels."
}